This PR introduces a `NativeAssetsManifest.json` next to the
`AssetManifest.bin` and `FontManifest.json`. This removes the need for
embedding the native assets mapping inside the kernel file and enables
decoupling native assets building and bundling from the kernel
compilation in flutter tools. This means `flutter run` no longer does a
dry run of `hook/build.dart` hooks.
(It also means all isolate groups will have the same native assets.
However, since Flutter does not support `Isolate.spawnUri` from kernel
files, this is not a regression.)
Implementation details:
* g3 is still using kernel embedding.
https://github.com/flutter/flutter/pull/142016 introduced an argument to
embed a `native_assets.yaml` inside `flutter attach` and `flutter run`
(the outer flutter process), but it is not used in `flutter assemble`
(the inner process when doing `flutter run`). So, those arguments need
to still be respected. However, all other logic related to embedding a
yaml encoding in the kernel file has been removed.
* All dry-run logic has been removed. 🎉
* The `KernelSnapshot` target no longer depends on the
`InstallCodeAssets` target. Instead, the various OS-specific
"BundleAsset" targets now depend on the `InstallCodeAssets` target. The
`InstallCodeAssets` invokes the build hooks and produces the
`NativeAssetsManifest.json`. The various "BundleAsset" commands
synchronize the `NativeAssetsManifest.json` to the app bundle.
* `InstallCodeAssets` produces a `native_assets.json`, which is renamed
to `NativeAssetsManifest.json` in the various "Bundle" targets. This
means that all unit tests of the "Bundle" targets now need to create
this file. (Similar to how `app.dill` is expected to exist because
`KernelSnapshot` is a dependency of the "Bundle" targets.)
* Because dynamic libraries need to be code signed (at least on iOS and
MacOS), the bundling of the dylibs is _not_ migrated to reuse
`_updateDevFS` (which is used for ordinary assets). Only the 2nd and 3rd
invocation of `flutter assemble` from `xcodebuild` has access to the
code signing identity.
Relevant tests:
* test/integration.shard/isolated/native_assets_test.dart - runs
`flutter run` with native assets including hot restart and hot reload.
TODO:
* Undo engine-roll in this PR after engine has rolled in.
Issue:
* https://github.com/flutter/flutter/issues/154425
Related PRs:
* https://dart-review.googlesource.com/c/sdk/+/388161
* https://github.com/flutter/engine/pull/56727
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Currently the `NativeAsset` target in flutter tools is responsible for
two things:
* performing the dart build (in the app as well as all transitive pub
dependencies)
* taking output (shared libraries) from this build and copying them
around
This intermingling of responsibilities leads to more complex code and
potentially unnecessary work: If the source code changed (e.g. `.c`
files change) we have to run the dart build again. But doing so may
result in the same shared libraries (e.g. adding comments to the `.c`
code). Currently we're going to copy the shared libraries despite them
having not changed, which then may cause upstream things to be dirtied
(if it's based on timestamp of files) and re-built.
Instead this PR splits this `NativeAsset` into the two orthogonal pieces
* `DartBuild` target that is responsible for the dart build
* `InstallCodeAssets` that is responsible for copying shared libraries
to the right place and producing a `native_assets.yaml`.
This decoupling is also preparation for a future where a dart build can
produce other kinds of assets (e.g. data assets) and is used in the web
build as well. (The web build would use `DartBuild` but not
`InstalCodeAssets`).
Work towards https://github.com/flutter/flutter/issues/56591.
I explicitly want an LGTM from @andrewkolos @jmagman @jonahwilliams before merging.
---
After this PR, `<Plugin>.isDevDependency` is resolved based on the following logic, IFF:
- The plugin comes from a package _A_ listed in the app's package's `dev_dependencies: ...`
- The package _A_ is not a normal dependency of any transitive non-dev dependency of the app
See [`compute_dev_dependencies_test.dart`](51676093a3/packages/flutter_tools/test/general.shard/compute_dev_dependencies_test.dart) for probably the best specification of this behavior.
We (still) do not write the property to disk (i.e. it never makes it to `.flutter-plugins-dependencies`), so there is no impact to build artifacts at this time; that would come in a follow-up PR (and then follow-up follow-up PRs for the various build systems in both Gradle and Xcode to actually use that value to omit dependencies).
Some tests had to be updated; for the most part it was updating the default `ProcessManager` because a call to `dart pub deps --json` is now made in code that computes what plugins are available, but there should be no change in behavior.
_/cc @jonasfj @sigurdm for FYI only (we talked on an internal thread about this; see https://github.com/dart-lang/sdk/issues/56968)._
_/cc @camsim99 @cbracken @johnmccutchan for visibility on the change._
Work towards https://github.com/flutter/flutter/issues/157819. **No behavior changes as a result of this PR**.
Based on a proof of concept by @jonahwilliams (https://github.com/flutter/flutter/pull/157818).
The existence of this flag (which for the time being, defaults to `true`) implies the following:
1. The (legacy, deprecated) `.flutter-plugins` file is not generated:
https://docs.flutter.dev/release/breaking-changes/flutter-plugins-configuration
2. The (legacy, deprecated) `package:flutter_gen` is not synthetically generated:
https://github.com/flutter/website/pull/11343
(awaiting website approvers, but owners approve this change)
This change creates `useImplicitPubspecResolution` and plumbs it through as a required variable, parsing it from a `FlutterCommand.globalResults` where able. In tests, I've defaulted the value to `true` 100% of the time - except for places where the value itself is acted on directly, in which case there are true and false test-cases (e.g. localization and i10n based classes and functions).
I'm not extremely happy this needed to change 50+ files, but is sort of a result of how inter-connected many of the elements of the tools are. I believe keeping this as an explicit (flagged) argument will be our best way to ensure the default behavior changes consistently and that tests are running as expected.
There have been various requests for this for a while. See https://github.com/flutter/flutter/issues/96283 for an example. This has become more important with dart2wasm builds in the mix, as the profiling versions of the dart2wasm builds are a lot less debuggable than the debug versions. Most of this is already wired up so it just requires taking out a few explicit checks for it and making sure that we compile with the right optimization levels when compiling debug.
Fix JS compilation to use the command 'compile js' instead of using
snapshot names to invoke dart2js
The Dart SDK is switching all tools to use AOT snapshots instead of JIT
snapshots and using snapshot names directly causes things to break.
Dart2js updated its CLI to support generating a 'dump-info' json file by passing a "--stage" option. The "dump-info-all" stage performs a full compilation (from the provided dill) and then also generates the dump info file.
Tested via the `flutter-dev` CLI locally. This results in the same output but with the addition of an extra `main.dart.js.info.json` file.
There are three categories of binaries produced as part of the framework artifacts:
* Those that use APIs that require entitlements and must be code-signed; e.g. gen_snapshot
* Those that do not use APIs that require entitlements and must be code-signed; e.g. Flutter.framework dylib.
* Those that do not need to be code-signed; e.g. Flutter.dSYM symbols.
We are adding the third category in https://github.com/flutter/engine/pull/54977. The Cocoon code signing aspect of this was handled in https://github.com/flutter/cocoon/pull/3890.
This ensures these files don't get copied into the build output should they appear in the artifact cache.
Issue: https://github.com/flutter/flutter/issues/154571
As of Xcode 16, App Store validation now requires that apps uploaded to the App store bundle dSYM debug information bundles for each Framework they embed.
dSYM bundles are packaged in the FlutterMacOS.xcframework shipped in the `darwin-x64-release` tools archive as of engine patches:
* https://github.com/flutter/engine/pull/54696
This copies the FlutterMacOS.framework.dSYM bundle from the tools cache to the build outputs produced by `flutter build macOS`.
Fixes: https://github.com/flutter/flutter/issues/153879
Turns out just supporting the right value for `kDebugMode` was a lot simpler than I thought. Debug builds used to never go through the build system code path when using `flutter run`, but now that we have wasm this can occur with the run command.
This should address https://github.com/flutter/flutter/issues/148850
For the necessary background knowledge, see the flutter.dev content on [Resolution-aware image assets](https://docs.flutter.dev/ui/assets/assets-and-images#resolution-aware) and [Conditional bundling of assets based on app flavor](https://docs.flutter.dev/ui/assets/assets-and-images#conditional-bundling-of-assets-based-on-app-flavor) if you don't have a basic understanding of these features.
Fixes https://github.com/flutter/flutter/issues/151813 by using unique temporary directories, per asset file, for transformations. Currently, only a single directory is used and the name of the temporary files was based only on the basename of files. This means that `assets/image.png` and `assets/2x/image.png` would share an output path (`<temp dir path>/image.png`), causing a race. If this quick and rough explanation is a bit confusing, the original issueâ#151813âprovides a full repro and correct identification of the exact cause of the failure that can occur in the asset transformation process.
This will make
* `flutter run` have source maps enabled by default
* `flutter build` have source maps disabled by default
which mirrors what happens already today with the js compilers.
For local development this works quite well - even better than with
dart2js (see dart2js issues in [0]).
We do have some follow-up items for source maps in dart2wasm compiler,
see [1]
[0]
[flutter/flutter/issues/151641](https://github.com/flutter/flutter/issues/151641)
[1]
[dart-lang/sdk/issues/56232](https://github.com/dart-lang/sdk/issues/56232)
If the user specifies the `--no-web-resources-cdn` or `--local-web-sdk`, we should use the local version of CanvasKit. `flutter.js` now has a flag that can be specified in the build configuration that tells it to load locally instead.
Also, added a link to the relevant docs in the web template warnings.
This addresses https://github.com/flutter/flutter/issues/148713
Also fixes https://github.com/flutter/flutter/issues/145559
- When `--web-renderer` is omitted, keep the value `null` until it later materializes to either `canvaskit` or `skwasm`.
- No more hardcoded defaults anywhere. We use `WebRendererMode.defaultForJs/defaultForWasm` instead.
- When in `--wasm` mode, the JS fallback is now `canvaskit` instead of `auto`.
- Add test for defaulting to `skwasm` when `--wasm` is enabled.
Fixes https://github.com/flutter/flutter/issues/149826
Dart2wasm only needs a platform file, which contains the compiled
`dart:*` libraries. There's no need to specify a seperate `--dart-sdk`
option (anymore).
(See also https://dart-review.googlesource.com/c/sdk/+/366821)
Fixes https://github.com/flutter/flutter/issues/148354
Fixes https://github.com/flutter/flutter/issues/147142
Closes https://github.com/flutter/flutter/pull/147144
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
Fix dependency tracking in the build.
* https://github.com/flutter/flutter/issues/147643
## Testing
It's not clear to me where to test the caching behavior of specific targets.
* test/general.shard/build_system/targets/common_test.dart
* This doesn't test the caching behavior of any targets
* test/general.shard/build_system/build_system_test.dart
* test/general.shard/cache_test.dart
* Both of these don't test specific `Target`s, these have `TestTarget`s.
* test/integration.shard/isolated/native_assets_test.dart
* This could work, but it's an integration test that already takes long to run.
Uses kernel concatenation to concatenate the `program.dill` and `native_assets.dill`.
This will enable compiling `native_assets.dill` after `program.dill` so that tree-shaking information can be used.
Issue:
* https://github.com/flutter/flutter/issues/146270
## Implementation choices
* We can either run the frontend_server twice as a process invocation (current PR), or keep the process running after the dart compilation and then shut it down after native assets compilation to kernel. Keeping the process running would require more coordination between different `Target`s.
* We can either chose to only do separate kernel file compilation in release (AOT) builds, or do it both for JIT and AOT (current PR). Doing it differently in JIT/AOT means more conditionals in lib/src/build_system/targets/, doing it single-shot in JIT might be more performant though.
Update: Both of these are mitigated by not running the kernel compiler process if there are no native assets at all.
## Implementation notes
This only updates `flutter assemble`.
The kernel compilation calls in `flutter run` do not require kernel concatenation. The native-assets-kernel-mapping in `flutter run` contains results from `--dry-run` invocations of `hook/build.dart` (and in the future `hook/link.dart`). These `--dry-run` invocations do not get access to tree-shaking information, so the `link` hook to be introduced later can be run before Dart compilation.
## Tests
* Updated the `Target`s tests to reflect the new implementation.
## Related CLs
* frontend_server support: https://dart-review.googlesource.com/c/sdk/+/363567
* Same PR for Dart standalone: https://dart-review.googlesource.com/c/sdk/+/362384
Here's another PR with a couple of typos fixed. As you can see there was a typo in _fileReferenceI**n**dentifiers_, in class _ParsedProjectInfo._ Maybe we should do some check on that since I'm not sure if that property is used somewhere outside Flutter?
From https://github.com/flutter/flutter/issues/143348#issuecomment-2016047148:
> before we ship, we should add a printTrace to the tool about each asset transformer we're invoking and the path/arguments we called it with
I think this is a good idea since asset transformers can be arbitrary Dart programsâmeaning that a lot can go wrong when running them. For example, they can hang indefinitely or perform some sort of I/O that later results in a tool crash. Knowing that asset transformation was involved when debugging a crash (or a slow/stuck `flutter build`) could be useful, so I think adding a `printTrace` or two is a good idea (or at least not a bad one).
Changing the web renderer doesn't directly modify the environment's dart defines, and so doesn't do a full build invalidation. We need to include the web renderer in the build key for the compiler configuration. This information is used directly by the web targets to modify the dart defines that are passed into the compiler, so we need to rebuild if this information changes.
This makes several changes to flutter web app bootstrapping.
* The build now produces a `flutter_bootstrap.js` file.
* By default, this file does the basic streamlined startup of a flutter app with the service worker settings and no user configuration.
* The user can also put a `flutter_bootstrap.js` file in the `web` subdirectory in the project directory which can have whatever custom bootstrapping logic they'd like to write instead. This file is also templated, and can use any of the tokens that can be used with the `index.html` (with the exception of `{{flutter_bootstrap_js}}`, see below).
* Introduced a few new templating tokens for `index.html`:
* `{{flutter_js}}` => inlines the entirety of `flutter.js`
* `{{flutter_service_worker_version}}` => replaced directly by the service worker version. This can be used instead of the script that sets the `serviceWorkerVersion` local variable that we used to have by default.
* `{{flutter_bootstrap_js}}` => inlines the entirety of `flutter_bootstrap.js` (this token obviously doesn't apply to `flutter_bootstrap.js` itself).
* Changed `IndexHtml` to be called `WebTemplate` instead, since it is used for more than just the index.html now.
* We now emit warnings at build time for certain deprecated flows:
* Warn on the old service worker version pattern (i.e.`(const|var) serviceWorkerVersion = null`) and recommends using `{{flutter_service_worker_version}}` token instead
* Warn on use of `FlutterLoader.loadEntrypoint` and recommend using `FlutterLoader.load` instead
* Warn on manual loading of `flutter_service_worker.js`.
* The default `index.html` on `flutter create` now uses an async script tag with `flutter_bootstrap.js`.
Reverts: flutter/flutter#144752
Initiated by: andrewkolos
Reason for reverting: compilation issue has turned the tree red
Original PR Author: andrewkolos
Reviewed By: {christopherfujino}
This change reverts the following previous change:
In service of https://github.com/flutter/flutter/issues/143348
When invoking a package to transform an asset, we set `FLUTTER_BUILD_MODE` to the CLI name of the build mode being used. Inspired by https://github.com/flutter/flutter/issues/101077#issuecomment-1890379501:
> Do transformers know whether they get executed in debug or release mode? I kinda imagine that being useful. Ex: There's a transformer that optimizes the file size of images. Depending on the amount and size of the images, that could take a significant amount of time. Therefore, I might want to only execute it in release builds.
Note for the reviewer: the interesting part of this change can be found in the commit [set environment variable to build mode when running asset transformer…](579912d470). The rest of the change is updating call sites with a new argument.
In service of https://github.com/flutter/flutter/issues/143348
When invoking a package to transform an asset, we set `FLUTTER_BUILD_MODE` to the CLI name of the build mode being used. Inspired by https://github.com/flutter/flutter/issues/101077#issuecomment-1890379501:
> Do transformers know whether they get executed in debug or release mode? I kinda imagine that being useful. Ex: There's a transformer that optimizes the file size of images. Depending on the amount and size of the images, that could take a significant amount of time. Therefore, I might want to only execute it in release builds.
Note for the reviewer: the interesting part of this change can be found in the commit [set environment variable to build mode when running asset transformerâ¦](579912d470). The rest of the change is updating call sites with a new argument.
This fixes https://github.com/flutter/flutter/issues/143703
We need to make sure that when source maps are enabled for the dart2js target, it advertises the sourcemap file as one of its outputs so that the web release bundle can copy it over.
The flutter engine & framework can opt out of this optimization for
individual classes / class hierarchies via:
* `@pragma(flutter:keep-to-string)`
* `@pragma(flutter:keep-to-string-in-subtypes)`
Or by using the convenience constant `@keepToString` from `dart:ui`.
=> This aligns the build process more with g3 (which already does this)
Closes https://github.com/flutter/flutter/issues/52759
Partial implementation of https://github.com/flutter/flutter/issues/143348
This enables asset transformation during hot reload (except for web, because that has its own implementation of `DevFS` ð). Asset transformers will be reapplied after changing any asset and performing a hot reload during `flutter run`.
So far `flutter build web --wasm` was always stripping wasm symbols
except if `--no-strip-wasm` is passed.
=> Ensure that in profile mode we also keep the symbols