My newly added DDC macrobenchmarks tests have been timing out on the
test infra. It turns out that `headless` Chrome is significantly slower
than headful Chrome (reasons unknown, but perhaps related to hardware
acceleration). Using `--headless=new` instead of `flutter run`'s
headless flag (which uses the old headless mode) makes my local runs
significantly (10x+) faster.
*This time for sure*
Bypasses this error in the test infra `[2025-04-30 20:32:46.191597]
[STDOUT] [CHROME STDERR]:
[CHROME]:[3497:3497:0430/203246.172288:ERROR:ozone_platform_x11.cc(244)]
Missing X server or $DISPLAY`.
Resolves this error on the bots (I hope!):
```
[2025-04-29 15:56:48.817223] [STDOUT] [CHROME STDERR]: [CHROME]:[41258:41258:0429/155648.790553:FATAL:zygote_host_impl_linux.cc(126)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
```
Notable changes:
* Allows macrobenchmarks to run via `flutter run`.
* Splits macrobenchmarking between "orchestration logic" and "app
serving" (served on separate ports on the same machine to keep the
scheme consistent with the `flutter build` path).
* Adds an intercepted entrypoint for web benchmarks. We can't pass flags
to the app since it's not supported, so I've hard-coded the
orchestration server's port.
* Adding logic to connect to an existing Chrome debugger instance (vs
spawning one) for benchmarks.
We're adding macrobenchmark support to DDC (AMD modules and our new
hot-reload-capable module system) to be aware of any current/future
performance regressions. I'm not terribly aware of the metrics
collection pipeline; please let me know if we need to do any more work
to see these tests run/report numbers to the proper dashboards.
As per https://github.com/flutter/flutter/issues/162620, we are going to
run the web benchmarks at `-O2` for a trial period to evaluate the
performance difference, which will give us some data on whether we can
consider changing to `-O2` by default.
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.
**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.
---------
Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
Fix the `--ab` option in the benchmark harness:
- Make `--local-engine-host` optional. The web engine doesn't need it, so it doesn't build it. But the tool crashes by failing to find it.
- Disable icon tree shaking because `--ab` runs against local engine build, whose Dart kernel version is out of sync with the framework, which crashes the const finder.
Update: Accidentally use `--O4` instead of `-O4` in `dev/devicelab/lib/tasks/web_benchmarks.dart` update.
Original description:
* Make `flutter build web` have one option that determins the
optimization level: `-O<level>` / `--optimization-level=<level>` =>
Defaulting to -O4 => Will apply to both dart2js and dart2wasm
* Deprecate `--dart2js-optimization=O<level>`
* Disentagle concept of optimization from concept of static symbols =>
Add a `--strip-wasm` / `--no-strip-wasm` flag that determins whether
static symbols are kept in the resulting wasm file.
* Remove copy&past'ed code in the tests for wasm build tests
* Cleanup some artifacts code, now that we no longer use `wasm-opt`
inside flutter tools
Reverts flutter/flutter#143517
Initiated by: dnfield
Reason for reverting: broke CI, see https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20web_benchmarks_skwasm/3446/overview
Original PR Author: mkustermann
Reviewed By: {eyebrowsoffire}
This change reverts the following previous change:
Original Description:
* Make `flutter build web` have one option that determins the optimization level: `-O<level>` / `--optimization-level=<level>` => Defaulting to -O4 => Will apply to both dart2js and dart2wasm
* Deprecate `--dart2js-optimization=O<level>`
* Disentagle concept of optimization from concept of static symbols => Add a `--strip-wasm` / `--no-strip-wasm` flag that determins whether static symbols are kept in the resulting wasm file.
* Remove copy&past'ed code in the tests for wasm build tests
* Cleanup some artifacts code, now that we no longer use `wasm-opt` inside flutter tools
* Make `flutter build web` have one option that determins the
optimization level: `-O<level>` / `--optimization-level=<level>` =>
Defaulting to -O4 => Will apply to both dart2js and dart2wasm
* Deprecate `--dart2js-optimization=O<level>`
* Disentagle concept of optimization from concept of static symbols =>
Add a `--strip-wasm` / `--no-strip-wasm` flag that determins whether
static symbols are kept in the resulting wasm file.
* Remove copy&past'ed code in the tests for wasm build tests
* Cleanup some artifacts code, now that we no longer use `wasm-opt`
inside flutter tools
This is an attempt at a reland of https://github.com/flutter/flutter/pull/141396
The main changes here that are different than the original PR is fixes to wire up the `flutter test` command properly with the web renderer.
Dual Web Compile has had some issues where `flutter test` is not respecting the `--web-renderer` flag for some reason. I haven't gotten entirely to the bottom of the issue, but for now we need to rever these changes while I investigate. This reverts the following PRs:
https://github.com/flutter/flutter/pull/143128https://github.com/flutter/flutter/pull/141396
While doing this revert, I had a few merge conflicts with https://github.com/flutter/flutter/pull/142760, and I tried to resolve the merge conflicts within the spirit of that PR's change, but @chingjun I might need your input on whether the imports I have modified are okay with regards to the change you were making.
This implements dual compile via the newly available flutter.js bootstrapping APIs for intelligent build fallback.
* Users can now use the `FlutterLoader.load` API from flutter.js
* Flutter tool injects build info into the `index.html` of the user so that the bootstrapper knows which build variants are available to bootstrap
* The semantics of the `--wasm` flag for `flutter build web` have changed:
- Instead of producing a separate `build/web_wasm` directory, the output goes to the `build/web` directory like a normal web build
- Produces a dual build that contains two build variants: dart2wasm+skwasm and dart2js+CanvasKit. The dart2wasm+skwasm will only work on Chrome in a cross-origin isolated context, all other environments will fall back to dart2js+CanvasKit.
- `--wasm` and `--web-renderer` are now mutually exclusive. Since there are multiple build variants with `--wasm`, the web renderer cannot be expressed via a single command-line flag. For now, we are hard coding what build variants are produced with the `--wasm` flag, but I plan on making this more customizable in the future.
* Build targets now can optionally provide a "build key" which can uniquely identify any specific parameterization of that build target. This way, the build target can invalidate itself by changing its build key. This works a bit better than just stuffing everything into the environment defines because (a) it doesn't invalidate the entire build, just the targets which are affected and (b) settings for multiple build variants don't translate well to the flat map of environment defines.
Because the cost of type checks dominate our dart2wasm benchmarks, we've
decided to pass `--omit-type-checks` for now.
This was previously reverted because the skwasm benchmarks were broken
in general for a separate reason, and my getting rid of `bringup: true`
broke the tree. I ended up fixing the benchmarks and getting rid of
`bringup: true` in a separate commit, so this just adds the flag only.
We've decided to use the `--omit-type-checks` flag for our dart2wasm benchmarks. Right now, many of the benchmark results are dominated by type checks and most of what we are actually trying to measure get drowned out in the noise.
This enables benchmarks for the Skwasm renderer, compiled with
dart2wasm.
Platform views aren't supported in Skwasm yet, so we are skipping those
benchmarks for now.
By default, the browser fuzzes the timer APIs such that they have a granularity of approximately 100 microseconds (this is due to Spectre mitigation techniques). However, many of the thing we are trying to measure actually have a much finer granularity than 100 microseconds. As a result, many of our benchmarks are extremely noisy and don't provide accurate data.
By serving the initial script files with the `Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp` HTTP headers, the browser runs the benchmarks in a `crossOriginIsolated` context, which restores the fine granularity of APIs such as `performance.now()` to microsecond precision.
Also, we were considering anything an outlier that was more than one standard deviation away from the mean. In a normal distribution, that means we are only capturing 68% of the data and the rest are considered outliers. This is not ideal. Doing two standard deviations away captures 95% of the data, and the outliers are in the remaining 5%, which seems much more reasonable.
This adds avoid_dynamic_calls to the list of lints, and fixes all instances where it was violated.
Importantly, this lint is NOT turned on for flutter/packages/test, because those changes are happening in another PR: #84478
