This PR rolls in a number of breaking changes from dart-lang/native:
* `BuildMode` is no longer part of the protocol, so Flutter no longer
passes it in.
* This means all code dealing with the name conflict between
`native_assets_cli.BuildMode` and `flutter_tools.BuildMode` has been
cleaned up.
* Also, the logs no longer mention the build mode.
* The tests still exercise both modes, because linking only happens in
release mode.
* `OS` is no longer part of the main protocol, but of the "code"
"protocol extension".
* The code now aligns more with `OS?` being nullable in a bunch of
places, since it is nullable if there's no code assets.
* The OS-specific config is nested in an object per OS.
* `CCompilerConfig`s fields are non-nullable now.
* So instead of passing an object with nullable fields around, a null
instead of the object is returned in various places.
* `FileSystem` is now passed in to the native assets builder.
This PR contains no feature changes.
This PR will need to be followed up by restricting what environment
variables are passed in (similar to
https://github.com/dart-lang/native/pull/1764), I will do this in a
follow up PR.
Tests:
* All existing features should be covered by existing tests.
dart_style 3.0.1 comes with some minor style fixes:
https://github.com/dart-lang/dart_style/blob/main/CHANGELOG.md#301
This PR applies this fixes in bulk across the repository. (Otherwise,
the next person touching these files would have been the one updating
them to the new format by running the formatter).
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.
**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.
---------
Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
This PR introduces a `NativeAssetsManifest.json` next to the
`AssetManifest.bin` and `FontManifest.json`. This removes the need for
embedding the native assets mapping inside the kernel file and enables
decoupling native assets building and bundling from the kernel
compilation in flutter tools. This means `flutter run` no longer does a
dry run of `hook/build.dart` hooks.
(It also means all isolate groups will have the same native assets.
However, since Flutter does not support `Isolate.spawnUri` from kernel
files, this is not a regression.)
Implementation details:
* g3 is still using kernel embedding.
https://github.com/flutter/flutter/pull/142016 introduced an argument to
embed a `native_assets.yaml` inside `flutter attach` and `flutter run`
(the outer flutter process), but it is not used in `flutter assemble`
(the inner process when doing `flutter run`). So, those arguments need
to still be respected. However, all other logic related to embedding a
yaml encoding in the kernel file has been removed.
* All dry-run logic has been removed. 🎉
* The `KernelSnapshot` target no longer depends on the
`InstallCodeAssets` target. Instead, the various OS-specific
"BundleAsset" targets now depend on the `InstallCodeAssets` target. The
`InstallCodeAssets` invokes the build hooks and produces the
`NativeAssetsManifest.json`. The various "BundleAsset" commands
synchronize the `NativeAssetsManifest.json` to the app bundle.
* `InstallCodeAssets` produces a `native_assets.json`, which is renamed
to `NativeAssetsManifest.json` in the various "Bundle" targets. This
means that all unit tests of the "Bundle" targets now need to create
this file. (Similar to how `app.dill` is expected to exist because
`KernelSnapshot` is a dependency of the "Bundle" targets.)
* Because dynamic libraries need to be code signed (at least on iOS and
MacOS), the bundling of the dylibs is _not_ migrated to reuse
`_updateDevFS` (which is used for ordinary assets). Only the 2nd and 3rd
invocation of `flutter assemble` from `xcodebuild` has access to the
code signing identity.
Relevant tests:
* test/integration.shard/isolated/native_assets_test.dart - runs
`flutter run` with native assets including hot restart and hot reload.
TODO:
* Undo engine-roll in this PR after engine has rolled in.
Issue:
* https://github.com/flutter/flutter/issues/154425
Related PRs:
* https://dart-review.googlesource.com/c/sdk/+/388161
* https://github.com/flutter/engine/pull/56727
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
The dart-lang/native repository contains a `Target` class that is almost
not needed anymore. The remaining uses are mainly due to kernel asset
mapping (which we may be able to remove in the future).
This PR removes usages of that `Target` (in favor of `Architecture`)
class in most places in flutter tools.
This makes the code also cleaner because we no longer have an implicit
assumption that
a `List<Target>` all belong to the same operating system.
Currently the `NativeAsset` target in flutter tools is responsible for
two things:
* performing the dart build (in the app as well as all transitive pub
dependencies)
* taking output (shared libraries) from this build and copying them
around
This intermingling of responsibilities leads to more complex code and
potentially unnecessary work: If the source code changed (e.g. `.c`
files change) we have to run the dart build again. But doing so may
result in the same shared libraries (e.g. adding comments to the `.c`
code). Currently we're going to copy the shared libraries despite them
having not changed, which then may cause upstream things to be dirtied
(if it's based on timestamp of files) and re-built.
Instead this PR splits this `NativeAsset` into the two orthogonal pieces
* `DartBuild` target that is responsible for the dart build
* `InstallCodeAssets` that is responsible for copying shared libraries
to the right place and producing a `native_assets.yaml`.
This decoupling is also preparation for a future where a dart build can
produce other kinds of assets (e.g. data assets) and is used in the web
build as well. (The web build would use `DartBuild` but not
`InstalCodeAssets`).
After the dart build is done, the flutter tool has to bundle the
produced shared libraries, which it does that by copying them around.
Though the code assumed that all code assets are shared libraries to be
bundled, whereas in fact one can have code assets without any actual
code (ones that are installed on the target system already or artificial
code assets whose symbols get resolved from executable / process).
=> Using non-bundled code assets currently results in null pointer
exceptions and/or cast errors.
=> We update the copy code to only operate on code assets that have a
shared library to bundle.
We also update the copy routines by removing copy&past'ed - but slightly
different - printing code into the shared caller function.
In release builds linking of native assets is enabled. The build step is
only a temprary step, it's output is given to the link step which then
returns all final assets (effectively a map-reduce system). Assets that
aren't sent to a specific linker could be conceptually viewed as sent to
a linker that will emit it's input as-is.
=> The code currently took output of build & link step and therefore
accumulated assets that aren't explicitly sent to a linker twice.
=> This led to performing work twice for those (e.g. copying them twice)
This PR changes this such that if linking mode is enabled, we only rely
on the output of the link phase.
That in return means many tests that mock the native asset builds need
to be updated to mock the output of the link phase.
Closes https://github.com/flutter/flutter/issues/158012.
This is (effectively) a user-facing NOP, which is exchanging an
on-by-default command-line argument (`--implicit-pubspec-resolution`)
for an off-by-default global feature flag
(`explicit-package-dependencies`). It matches the mental model better,
is less painstaking to maintain and feed throughout, and will be easier
to globally flip on/off in a future PR.
---------
Co-authored-by: Andrew Kolos <andrewrkolos@gmail.com>
Almost all of the code is just adopting to changes to the APIs of
`package:native_assets_builder`, `package:native_assets_cli` and
`package:native_toolchain_c`
There's only two semantic changes
* Removes a test that checks for a verification error if a build hook
produces a static library if the preferred linking mode is dynamic:
=> The test is written in a very hacky way. By monkey patching the build
config.json that flutter build actually made. This monkey patching
relies on package:cli_config which is now no longer used.
=> The actual code that checks for this mismatch lives in
dart-lang/native repository and is tested there. So there's really no
need to duplicate that.
* The `package:native_assets_builder` no longer knows about code assets.
This is something a user of that package (e.g. flutter tools) adds. Now
the dry-run functionality will invoke build hooks who produce code
assets without an architecture.
=> The `package:native_assets_builder` used to expand such a code asset
to N different code assets (one for each supported architecture)
=> This logic was now moved to flutter tools. => In the near future
we're going to this dry-run complexity, which will then also get rid of
this uglyness (of expanding to all archs of an OS).
Work towards https://github.com/flutter/flutter/issues/157819. **No behavior changes as a result of this PR**.
Based on a proof of concept by @jonahwilliams (https://github.com/flutter/flutter/pull/157818).
The existence of this flag (which for the time being, defaults to `true`) implies the following:
1. The (legacy, deprecated) `.flutter-plugins` file is not generated:
https://docs.flutter.dev/release/breaking-changes/flutter-plugins-configuration
2. The (legacy, deprecated) `package:flutter_gen` is not synthetically generated:
https://github.com/flutter/website/pull/11343
(awaiting website approvers, but owners approve this change)
This change creates `useImplicitPubspecResolution` and plumbs it through as a required variable, parsing it from a `FlutterCommand.globalResults` where able. In tests, I've defaulted the value to `true` 100% of the time - except for places where the value itself is acted on directly, in which case there are true and false test-cases (e.g. localization and i10n based classes and functions).
I'm not extremely happy this needed to change 50+ files, but is sort of a result of how inter-connected many of the elements of the tools are. I believe keeping this as an explicit (flagged) argument will be our best way to ensure the default behavior changes consistently and that tests are running as expected.
The "link-dry-run" functionality was never used in flutter (even before
the recent refactoring).
I think we can remove this "link-dry-run" concept everywhere.
PR to remove this in dart-lang/native:
https://github.com/dart-lang/native/pull/1613
Changes to original CL: The code that issues an error on unsupported
operating system in the dry-run case was missing a case for iOS and
Android
Original CL description
tl;dr Removes 50% (>1650 locs) of native asset related code in
`packages/flutter_tools`
Before this PR the invocation of dart build/link/dry-run was implemented
per OS. This lead to very large code duplication of almost identical,
but slightly different code. It also led to similarly duplicated test
code.
Almost the entire dart build/link/dry-run implementation is identical
across OSes. There's small variations:
- configuration of the build (e.g. android/macos/ios version, ios sdk,
...)
- determining target locations & copying the final shared libraries
This PR unifies the implementation by reducing the code to basically two
main functions:
* `runFlutterSpecificDartBuild` which is responsible for
- obtain flutter configuration
- perform dart build (& link)
- determine target location & install binaries
* `runFlutterSpecificDartDryRunOnPlatforms` which is responsible for a
similar (but not same):
- obtain flutter configuration
- perform dart dry run
- determine target location
these two functions will call out to helpers for the OS specific
functionality:
* `_assetTargetLocationsForOS` for determining the location of the code
assets
* `_copyNativeCodeAssetsForOS` for copying the code assets (and possibly
overriting the install name, etc)
=> Since we get rid of the code duplication across OSes and have only a
single code path for the build/link/dry-run, we can also remove the
duplicated tests that were pretty much identical across OSes.
We also harden the building code by adding asserts, e.g.
* the dry fun functionality should never be used by `flutter test`
* the `build/native_assets/<os>/native_assets.yaml` should only be used
by `flutter test` and the dry-run of `flutter run`
=> We change the tests to also comply with these invariants (so the
tests are not testing things that cannot happen in reality)
We also rename `{,Flutter}NativeAssetsBuildRunner` to disambiguate it
from the `package:native_asset_builder`'s `NativeAssetsBuildRunner`.
Reverts: flutter/flutter#155430
Initiated by: eyebrowsoffire
Reason for reverting: Postsubmit failures closing the tree. See the following examples:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20native_assets_ios/5738/overviewhttps://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_mokey%20native_assets_android/583/overviewhttps://ci.chromium.org/ui/p/flutter/builders/prod/Linux_pixel_7pro%20native_assets_android/4075/overviewhttps://ci.chromium.org/u
Original PR Author: mkustermann
Reviewed By: {bkonyi, dcharkes}
This change reverts the following previous change:
tl;dr Removes 50% (>1650 locs) of native asset related code in `packages/flutter_tools`
Before this PR the invocation of dart build/link/dry-run was implemented per OS. This lead to very large code duplication of almost identical, but sligthly different code. It also led to similarly duplicated test code.
Almost the entire dart build/link/dry-run implementation is identical across OSes. There's small variations:
- configuration of the build (e.g. android/macos/ios version, ios sdk, ...)
- determining target locations & copying the final shared libraries
This PR unifies the implementation by reducing the code to basically two main functions:
* `runFlutterSpecificDartBuild` which is responsible for
- obtain flutter configuration
- perform dart build (& link)
- determine target location & install binaries
* `runFlutterSpecificDartDryRunOnPlatforms` which is responsible for a similar (but not same):
- obtain flutter configuration
- perform dart dry run
- determine target location
these two functions will call out to helpers for the OS specific functionality:
* `_assetTargetLocationsForOS` for determining the location of the code assets
* `_copyNativeCodeAssetsForOS` for copying the code assets (and possibly overriting the install name, etc)
=> Since we get rid of the code duplication across OSes and have only a single code path for the build/link/dry-run, we can also remove the duplicated tests that were pretty much identical across OSes.
We also harden the building code by adding asserts, e.g.
* the dry fun functionality should never be used by `flutter test`
* the `build/native_assets/<os>/native_assets.yaml` should only be used by `flutter test` and the dry-run of `flutter run`
=> We change the tests to also comply with these invariants (so the tests are not testing things that cannot happen in reality)
We also rename `{,Flutter}NativeAssetsBuildRunner` to disambiguate it from the `package:native_asset_builder`'s `NativeAssetsBuildRunner`.
We also reorganize the main code to make it readable from top-down and make members private where they can be.
tl;dr Removes 50% (>1650 locs) of native asset related code in
`packages/flutter_tools`
Before this PR the invocation of dart build/link/dry-run was implemented
per OS. This lead to very large code duplication of almost identical,
but slightly different code. It also led to similarly duplicated test
code.
Almost the entire dart build/link/dry-run implementation is identical
across OSes. There's small variations:
- configuration of the build (e.g. android/macos/ios version, ios sdk, ...)
- determining target locations & copying the final shared libraries
This PR unifies the implementation by reducing the code to basically two
main functions:
* `runFlutterSpecificDartBuild` which is responsible for
- obtain flutter configuration
- perform dart build (& link)
- determine target location & install binaries
* `runFlutterSpecificDartDryRunOnPlatforms` which is responsible for a
similar (but not same):
- obtain flutter configuration
- perform dart dry run
- determine target location
these two functions will call out to helpers for the OS specific
functionality:
* `_assetTargetLocationsForOS` for determining the location of the code
assets
* `_copyNativeCodeAssetsForOS` for copying the code assets (and possibly
overriting the install name, etc)
=> Since we get rid of the code duplication across OSes and have only a
single code path for the build/link/dry-run, we can also remove the
duplicated tests that were pretty much identical across OSes.
We also harden the building code by adding asserts, e.g.
* the dry fun functionality should never be used by `flutter test`
* the `build/native_assets/<os>/native_assets.yaml` should only be used
by `flutter test` and the dry-run of `flutter run`
=> We change the tests to also comply with these invariants (so the
tests are not testing things that cannot happen in reality)
We also rename `{,Flutter}NativeAssetsBuildRunner` to disambiguate it
from the `package:native_asset_builder`'s `NativeAssetsBuildRunner`.
Rolls native deps to the latest version, and cleans up deprecated field from template.
Tests:
* All the unit and integration tests for native assets. The template and dependencies are exercised in the integration test.
Since `package:native_assets_builder` already checks for having no static libraries as output, the custom check in flutter_tools is removed. The tests stubbing out the native assets builder exercising the custom check are also removed. (The integration tests now check for the error message from the native assets builder.)
Native libraries that are contributed by native asset builders can depend on each other. For macOS and iOS, native libraries are repackaged into Frameworks, which renders install names that have been written into dependent libraries invalid.
With this change, a mapping between old and new install names is maintained, and install names in dependent libraries are rewritten as a final step.
Related to https://github.com/dart-lang/native/issues/190
Stop running link hooks in debug mode.
Rationale: link hooks only get access to tree-shaking info in release builds, so they can't do anything meaningful in debug builds. Debug builds should be fast as development cycle, so running less is better.
More details:
* https://github.com/dart-lang/native/issues/1252
Also: rolls packages to latest versions.
## Implementation details
The decision whether linking is enabled is made as follows:
* For normal builds `build_info.dart::BuildMode` is used to determine whether Dart is compiled in JIT or AOT mode.
* Testers always run in JIT, so no linking.
* Native asset dry runs only run for JIT builds (e.g only when hot reload and hot restart are enabled).
## Testing
The integration test is updated to output an asset for linking if `BuildConfig.linkingEnabled` is true, and to output an asset for bundling directly if linking is not enabled.
This PR adds support invoking `link.dart` hooks.
Link hooks can add new assets. Link hooks can transform assets sent to link hook from build hooks.
This PR does not yet add support for getting tree-shake information in the link hooks. This is pending on defining the `resources.json` format (https://github.com/dart-lang/sdk/issues/55494).
Issue:
* https://github.com/flutter/flutter/issues/146263
## Implementation considerations
The build hooks could be run before Dart compilation and the link hooks after Dart compilation. (This is how it's done in Dart standalone.) However, due to the way the `Target`s are set up, this would require two targets and serializing and deserializing the `BuildResult` in between these. This would lead to more code but no benefits. Currently there is nothing that mandates running build hooks before Dart compilation.
## Testing
* The unit tests verify that the native_assets_builder `link` and `linkDryRun` would be invoked with help of the existing fake.
* The native assets integration test now also invokes an FFI call of a package that adds the asset during the link hook instead of the build hook.
* In order to keep coverage of the `flutter create --template=package_ffi`, `flutter create` is still run and the extra dependency is added and an extra ffi call is added. (Open to alternative suggestions.)
*Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.*
*List which issues are fixed by this PR. You must list at least one issue. An issue is not required if the PR fixes something trivial like a typo.*
*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
Reland of https://github.com/flutter/flutter/pull/142709.
The revert of the revert is in the first commit, the fix in the commit on top.
The move of the fakes for packages/flutter_tools/test/general.shard/resident_runner_test.dart was erroneous before, as it was trying to use setters instead of a private field. This PR changes the private `_devFS` field in the fake to be a public `fakeDevFS` in line with other fakes.
## Original PR description
Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.
This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.
In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:
* bin/fuchsia_tester.dart
* lib/executable.dart
The build_system/targets dir is already excluded in other build systems.
So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.
Context:
* https://github.com/flutter/flutter/issues/142041
Reverts flutter/flutter#142709
Initiated by: vashworth
Reason for reverting: `Mac tool_tests_general` started failing on this commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_tests_general/15552/overview
Original PR Author: dcharkes
Reviewed By: {christopherfujino, chingjun, reidbaker}
This change reverts the following previous change:
Original Description:
Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.
This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.
In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:
* bin/fuchsia_tester.dart
* lib/executable.dart
The build_system/targets dir is already excluded in other build systems.
So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.
Context:
* https://github.com/flutter/flutter/issues/142041
Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.
This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.
In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:
* bin/fuchsia_tester.dart
* lib/executable.dart
The build_system/targets dir is already excluded in other build systems.
So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.
Context:
* https://github.com/flutter/flutter/issues/142041
