Roll packages to the ones published today.
Note that `native_assets_cli` was split up into `hooks` and
`code_assets`, and `native_assets_builder` was renamed to
`hooks_runner`.
This PR is the dual of
https://dart-review.googlesource.com/c/sdk/+/420700 in the Dart SDK.
This PR adds support for user-defines for hooks in the root package
`pubspec.yaml`.
```yaml
hooks:
user_defines:
hook_user_defines: # package name
magic_value: 1000
```
For more design considerations see:
* https://github.com/dart-lang/native/issues/39
### Testing
This PR is covered by
`test/integration.shard/isolated/native_assets_test.dart`.
In this PR, we add a new test project
`dev/integration_tests/hook_user_defines/` and add a dependency on that
project in the mentioned test.
The integration test already covers `flutter run`, `flutter test`, and
`flutter build`.
This PR rolls in a number of breaking changes from dart-lang/native:
* `BuildMode` is no longer part of the protocol, so Flutter no longer
passes it in.
* This means all code dealing with the name conflict between
`native_assets_cli.BuildMode` and `flutter_tools.BuildMode` has been
cleaned up.
* Also, the logs no longer mention the build mode.
* The tests still exercise both modes, because linking only happens in
release mode.
* `OS` is no longer part of the main protocol, but of the "code"
"protocol extension".
* The code now aligns more with `OS?` being nullable in a bunch of
places, since it is nullable if there's no code assets.
* The OS-specific config is nested in an object per OS.
* `CCompilerConfig`s fields are non-nullable now.
* So instead of passing an object with nullable fields around, a null
instead of the object is returned in various places.
* `FileSystem` is now passed in to the native assets builder.
This PR contains no feature changes.
This PR will need to be followed up by restricting what environment
variables are passed in (similar to
https://github.com/dart-lang/native/pull/1764), I will do this in a
follow up PR.
Tests:
* All existing features should be covered by existing tests.
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.
**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.
---------
Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
Currently the `NativeAsset` target in flutter tools is responsible for
two things:
* performing the dart build (in the app as well as all transitive pub
dependencies)
* taking output (shared libraries) from this build and copying them
around
This intermingling of responsibilities leads to more complex code and
potentially unnecessary work: If the source code changed (e.g. `.c`
files change) we have to run the dart build again. But doing so may
result in the same shared libraries (e.g. adding comments to the `.c`
code). Currently we're going to copy the shared libraries despite them
having not changed, which then may cause upstream things to be dirtied
(if it's based on timestamp of files) and re-built.
Instead this PR splits this `NativeAsset` into the two orthogonal pieces
* `DartBuild` target that is responsible for the dart build
* `InstallCodeAssets` that is responsible for copying shared libraries
to the right place and producing a `native_assets.yaml`.
This decoupling is also preparation for a future where a dart build can
produce other kinds of assets (e.g. data assets) and is used in the web
build as well. (The web build would use `DartBuild` but not
`InstalCodeAssets`).
Almost all of the code is just adopting to changes to the APIs of
`package:native_assets_builder`, `package:native_assets_cli` and
`package:native_toolchain_c`
There's only two semantic changes
* Removes a test that checks for a verification error if a build hook
produces a static library if the preferred linking mode is dynamic:
=> The test is written in a very hacky way. By monkey patching the build
config.json that flutter build actually made. This monkey patching
relies on package:cli_config which is now no longer used.
=> The actual code that checks for this mismatch lives in
dart-lang/native repository and is tested there. So there's really no
need to duplicate that.
* The `package:native_assets_builder` no longer knows about code assets.
This is something a user of that package (e.g. flutter tools) adds. Now
the dry-run functionality will invoke build hooks who produce code
assets without an architecture.
=> The `package:native_assets_builder` used to expand such a code asset
to N different code assets (one for each supported architecture)
=> This logic was now moved to flutter tools. => In the near future
we're going to this dry-run complexity, which will then also get rid of
this uglyness (of expanding to all archs of an OS).
Changes to original CL: The code that issues an error on unsupported
operating system in the dry-run case was missing a case for iOS and
Android
Original CL description
tl;dr Removes 50% (>1650 locs) of native asset related code in
`packages/flutter_tools`
Before this PR the invocation of dart build/link/dry-run was implemented
per OS. This lead to very large code duplication of almost identical,
but slightly different code. It also led to similarly duplicated test
code.
Almost the entire dart build/link/dry-run implementation is identical
across OSes. There's small variations:
- configuration of the build (e.g. android/macos/ios version, ios sdk,
...)
- determining target locations & copying the final shared libraries
This PR unifies the implementation by reducing the code to basically two
main functions:
* `runFlutterSpecificDartBuild` which is responsible for
- obtain flutter configuration
- perform dart build (& link)
- determine target location & install binaries
* `runFlutterSpecificDartDryRunOnPlatforms` which is responsible for a
similar (but not same):
- obtain flutter configuration
- perform dart dry run
- determine target location
these two functions will call out to helpers for the OS specific
functionality:
* `_assetTargetLocationsForOS` for determining the location of the code
assets
* `_copyNativeCodeAssetsForOS` for copying the code assets (and possibly
overriting the install name, etc)
=> Since we get rid of the code duplication across OSes and have only a
single code path for the build/link/dry-run, we can also remove the
duplicated tests that were pretty much identical across OSes.
We also harden the building code by adding asserts, e.g.
* the dry fun functionality should never be used by `flutter test`
* the `build/native_assets/<os>/native_assets.yaml` should only be used
by `flutter test` and the dry-run of `flutter run`
=> We change the tests to also comply with these invariants (so the
tests are not testing things that cannot happen in reality)
We also rename `{,Flutter}NativeAssetsBuildRunner` to disambiguate it
from the `package:native_asset_builder`'s `NativeAssetsBuildRunner`.
Reverts: flutter/flutter#155430
Initiated by: eyebrowsoffire
Reason for reverting: Postsubmit failures closing the tree. See the following examples:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20native_assets_ios/5738/overviewhttps://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_mokey%20native_assets_android/583/overviewhttps://ci.chromium.org/ui/p/flutter/builders/prod/Linux_pixel_7pro%20native_assets_android/4075/overviewhttps://ci.chromium.org/u
Original PR Author: mkustermann
Reviewed By: {bkonyi, dcharkes}
This change reverts the following previous change:
tl;dr Removes 50% (>1650 locs) of native asset related code in `packages/flutter_tools`
Before this PR the invocation of dart build/link/dry-run was implemented per OS. This lead to very large code duplication of almost identical, but sligthly different code. It also led to similarly duplicated test code.
Almost the entire dart build/link/dry-run implementation is identical across OSes. There's small variations:
- configuration of the build (e.g. android/macos/ios version, ios sdk, ...)
- determining target locations & copying the final shared libraries
This PR unifies the implementation by reducing the code to basically two main functions:
* `runFlutterSpecificDartBuild` which is responsible for
- obtain flutter configuration
- perform dart build (& link)
- determine target location & install binaries
* `runFlutterSpecificDartDryRunOnPlatforms` which is responsible for a similar (but not same):
- obtain flutter configuration
- perform dart dry run
- determine target location
these two functions will call out to helpers for the OS specific functionality:
* `_assetTargetLocationsForOS` for determining the location of the code assets
* `_copyNativeCodeAssetsForOS` for copying the code assets (and possibly overriting the install name, etc)
=> Since we get rid of the code duplication across OSes and have only a single code path for the build/link/dry-run, we can also remove the duplicated tests that were pretty much identical across OSes.
We also harden the building code by adding asserts, e.g.
* the dry fun functionality should never be used by `flutter test`
* the `build/native_assets/<os>/native_assets.yaml` should only be used by `flutter test` and the dry-run of `flutter run`
=> We change the tests to also comply with these invariants (so the tests are not testing things that cannot happen in reality)
We also rename `{,Flutter}NativeAssetsBuildRunner` to disambiguate it from the `package:native_asset_builder`'s `NativeAssetsBuildRunner`.
We also reorganize the main code to make it readable from top-down and make members private where they can be.
tl;dr Removes 50% (>1650 locs) of native asset related code in
`packages/flutter_tools`
Before this PR the invocation of dart build/link/dry-run was implemented
per OS. This lead to very large code duplication of almost identical,
but slightly different code. It also led to similarly duplicated test
code.
Almost the entire dart build/link/dry-run implementation is identical
across OSes. There's small variations:
- configuration of the build (e.g. android/macos/ios version, ios sdk, ...)
- determining target locations & copying the final shared libraries
This PR unifies the implementation by reducing the code to basically two
main functions:
* `runFlutterSpecificDartBuild` which is responsible for
- obtain flutter configuration
- perform dart build (& link)
- determine target location & install binaries
* `runFlutterSpecificDartDryRunOnPlatforms` which is responsible for a
similar (but not same):
- obtain flutter configuration
- perform dart dry run
- determine target location
these two functions will call out to helpers for the OS specific
functionality:
* `_assetTargetLocationsForOS` for determining the location of the code
assets
* `_copyNativeCodeAssetsForOS` for copying the code assets (and possibly
overriting the install name, etc)
=> Since we get rid of the code duplication across OSes and have only a
single code path for the build/link/dry-run, we can also remove the
duplicated tests that were pretty much identical across OSes.
We also harden the building code by adding asserts, e.g.
* the dry fun functionality should never be used by `flutter test`
* the `build/native_assets/<os>/native_assets.yaml` should only be used
by `flutter test` and the dry-run of `flutter run`
=> We change the tests to also comply with these invariants (so the
tests are not testing things that cannot happen in reality)
We also rename `{,Flutter}NativeAssetsBuildRunner` to disambiguate it
from the `package:native_asset_builder`'s `NativeAssetsBuildRunner`.
Rolls native deps to the latest version, and cleans up deprecated field from template.
Tests:
* All the unit and integration tests for native assets. The template and dependencies are exercised in the integration test.
Since `package:native_assets_builder` already checks for having no static libraries as output, the custom check in flutter_tools is removed. The tests stubbing out the native assets builder exercising the custom check are also removed. (The integration tests now check for the error message from the native assets builder.)
Stop running link hooks in debug mode.
Rationale: link hooks only get access to tree-shaking info in release builds, so they can't do anything meaningful in debug builds. Debug builds should be fast as development cycle, so running less is better.
More details:
* https://github.com/dart-lang/native/issues/1252
Also: rolls packages to latest versions.
## Implementation details
The decision whether linking is enabled is made as follows:
* For normal builds `build_info.dart::BuildMode` is used to determine whether Dart is compiled in JIT or AOT mode.
* Testers always run in JIT, so no linking.
* Native asset dry runs only run for JIT builds (e.g only when hot reload and hot restart are enabled).
## Testing
The integration test is updated to output an asset for linking if `BuildConfig.linkingEnabled` is true, and to output an asset for bundling directly if linking is not enabled.
This PR adds support invoking `link.dart` hooks.
Link hooks can add new assets. Link hooks can transform assets sent to link hook from build hooks.
This PR does not yet add support for getting tree-shake information in the link hooks. This is pending on defining the `resources.json` format (https://github.com/dart-lang/sdk/issues/55494).
Issue:
* https://github.com/flutter/flutter/issues/146263
## Implementation considerations
The build hooks could be run before Dart compilation and the link hooks after Dart compilation. (This is how it's done in Dart standalone.) However, due to the way the `Target`s are set up, this would require two targets and serializing and deserializing the `BuildResult` in between these. This would lead to more code but no benefits. Currently there is nothing that mandates running build hooks before Dart compilation.
## Testing
* The unit tests verify that the native_assets_builder `link` and `linkDryRun` would be invoked with help of the existing fake.
* The native assets integration test now also invokes an FFI call of a package that adds the asset during the link hook instead of the build hook.
* In order to keep coverage of the `flutter create --template=package_ffi`, `flutter create` is still run and the extra dependency is added and an extra ffi call is added. (Open to alternative suggestions.)
Reland of https://github.com/flutter/flutter/pull/142709.
The revert of the revert is in the first commit, the fix in the commit on top.
The move of the fakes for packages/flutter_tools/test/general.shard/resident_runner_test.dart was erroneous before, as it was trying to use setters instead of a private field. This PR changes the private `_devFS` field in the fake to be a public `fakeDevFS` in line with other fakes.
## Original PR description
Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.
This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.
In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:
* bin/fuchsia_tester.dart
* lib/executable.dart
The build_system/targets dir is already excluded in other build systems.
So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.
Context:
* https://github.com/flutter/flutter/issues/142041
Reverts flutter/flutter#142709
Initiated by: vashworth
Reason for reverting: `Mac tool_tests_general` started failing on this commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_tests_general/15552/overview
Original PR Author: dcharkes
Reviewed By: {christopherfujino, chingjun, reidbaker}
This change reverts the following previous change:
Original Description:
Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.
This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.
In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:
* bin/fuchsia_tester.dart
* lib/executable.dart
The build_system/targets dir is already excluded in other build systems.
So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.
Context:
* https://github.com/flutter/flutter/issues/142041
Native assets in other build systems are not built with `package:native_assets_builder` invoking `build.dart` scripts. Instead all packages have their own blaze rules. Therefore we'd like to not depend on `package:native_assets_builder` from flutter tools in g3 at all.
This PR aims to move the imports of `native_assets_builder` and `native_assets_cli` into the `isolated/` directory and into the files with a `main` function that are not used in with other build systems.
In order to be able to remove all imports in files used by other build systems, two new interfaces are added `HotRunnerNativeAssetsBuilder` and `TestCompilerNativeAssetsBuilder`. New parameters are then piped all the way through from the entry points:
* bin/fuchsia_tester.dart
* lib/executable.dart
The build_system/targets dir is already excluded in other build systems.
So, after this PR only the two above files and build_system/targets import from `isolated/native_assets/` and only `isolated/native_assets/` import `package:native_assets_cli` and `package:native_assets_builder`.
Context:
* https://github.com/flutter/flutter/issues/142041
