My newly added DDC macrobenchmarks tests have been timing out on the
test infra. It turns out that `headless` Chrome is significantly slower
than headful Chrome (reasons unknown, but perhaps related to hardware
acceleration). Using `--headless=new` instead of `flutter run`'s
headless flag (which uses the old headless mode) makes my local runs
significantly (10x+) faster.
*This time for sure*
Bypasses this error in the test infra `[2025-04-30 20:32:46.191597]
[STDOUT] [CHROME STDERR]:
[CHROME]:[3497:3497:0430/203246.172288:ERROR:ozone_platform_x11.cc(244)]
Missing X server or $DISPLAY`.
Resolves this error on the bots (I hope!):
```
[2025-04-29 15:56:48.817223] [STDOUT] [CHROME STDERR]: [CHROME]:[41258:41258:0429/155648.790553:FATAL:zygote_host_impl_linux.cc(126)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
```
Notable changes:
* Allows macrobenchmarks to run via `flutter run`.
* Splits macrobenchmarking between "orchestration logic" and "app
serving" (served on separate ports on the same machine to keep the
scheme consistent with the `flutter build` path).
* Adds an intercepted entrypoint for web benchmarks. We can't pass flags
to the app since it's not supported, so I've hard-coded the
orchestration server's port.
* Adding logic to connect to an existing Chrome debugger instance (vs
spawning one) for benchmarks.
The keyboard hot restart test listens to `flutter run` logs and then
does an action. However, the test prints the `flutter run` log after the
action. This results in confusing logs.
For example:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8716606099574306145/+/u/run_keyboard_hot_restart_ios/stdout
```
═══════════╡ ••• Launch app and wait for keyboard to be visible ••• ╞═══════════
...
stdout: Installing and launching... 37.2s
stdout: Syncing files to device swarming’s iPhone...
════╡ ••• Update the app to no longer force the keyboard to be visible ••• ╞════
════════════════════════╡ ••• Hot restart the app ••• ╞═════════════════════════
════════════╡ ••• Wait until the keyboard is no longer visible ••• ╞════════════
stdout: flutter: Keyboard is open
```
Here, the test waits until the keyboard is visible before moving onto
the test's next step. However, the log that triggers this action
(`flutter: Keyboard is open`) is printed after the action was taken
(`Update the app to no longer force the keyboard to be visible` and
`Wait until the keyboard is no longer visible`). This is confusing.
Follow-up to: https://github.com/flutter/flutter/pull/167013
Part of: https://github.com/flutter/flutter/issues/10713
https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.staging/Mac_arm64_ios%20imitation_game_flutter
The swiftui benchmark has been failing in bring up because the project
wasnt getting correctly codesigned. Also, fixes the fact that I flipped
the naming of imitation_game_flutter and imitation_game_swiftui 😶🌫️
Also removes extraneous targets from the project
https://ci.chromium.org/ui/p/flutter/builders/try.shadow/Mac_arm64_ios%20imitation_game_flutter/14/overview
Tested with `led`, proof that it's working post this PR ^
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
We're adding macrobenchmark support to DDC (AMD modules and our new
hot-reload-capable module system) to be aware of any current/future
performance regressions. I'm not terribly aware of the metrics
collection pipeline; please let me know if we need to do any more work
to see these tests run/report numbers to the proper dashboards.
We currently benchmark path tessellation on Android and iOS, but only
for filled paths. We will now run those same benchmarks also with
stroking in a new set of benchmarks.
This reverts commit
8dccbc33df.
Find/replace didn't catch the use of `jvmTarget = "1.8"` in
`dev/integration_tests/android_engine_test/android/app/build.gradle` and
`android_engine_test` needed to have `"com.android.application"`
updated.
Should resolve issues related to the following messages being output on
stderr:
```
warning: [options] source value 8 is obsolete and will be removed in a future release"
warning: [options] target value 8 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
```
Fixes https://github.com/flutter/flutter/issues/165973
The cost of bootstapping the initial PSOs can regress cold startup time
for customer money. As an experiment, attempt to defer PSO construction
to skia like.
---------
Co-authored-by: Aaron Clarke <aaclarke@google.com>
Co-authored-by: gaaclarke <30870216+gaaclarke@users.noreply.github.com>
<!-- start_original_pr_link -->
Reverts: flutter/flutter#165261
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: jonahwilliams
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: bork the tree
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: jonahwilliams
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {gaaclarke}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
The cost of bootstapping the initial PSOs can regress cold startup time
for customer money. As an experiment, attempt to defer PSO construction
to skia like.
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
The cost of bootstapping the initial PSOs can regress cold startup time
for customer money. As an experiment, attempt to defer PSO construction
to skia like.
---------
Co-authored-by: Aaron Clarke <aaclarke@google.com>
Co-authored-by: gaaclarke <30870216+gaaclarke@users.noreply.github.com>
This benchmark is a benchmark of the integration test binding which IMO
is not that important given that we continue to rely on flutter_driver
for actual performance metrics.
Fixes https://github.com/flutter/flutter/issues/162615
Can test 2 different ways.
On a mac (or linux machine) with adb on the path (or android sdk set in
ANDROID_HOME) and an emulator running (or physical device attached) that
is api 30 or higher.
```
cd dev/devicelab
dart bin/test_runner.dart test -t android_display_cutout
```
OR
```
dev/integration_tests/display_cutout_rotation
flutter drive integration_test/display_cutout_test.dart
```
Proof the test ran successfully
```
[2025-02-12 08:08:22.069817] [STDOUT] Removing Synthetic notch...
[2025-02-12 08:08:22.071147] [STDOUT] Executing "/b/s/w/ir/cache/android/sdk/platform-tools/adb -s emulator-5554 shell cmd overlay disable com.android.internal.display.cutout.emulation.tall" in "/b/s/w/ir/x/w/rc/tmpk3k3yhhp/flutter sdk/dev/integration_tests/display_cutout_rotation/" with environment {BOT: true, LANG: en_US.UTF-8}
[2025-02-12 08:08:22.862219] [STDOUT] Checking for reboot
[android_defines_test] Process terminated with exit code 0.
Task result:
{
"success": true,
"data": null,
"detailFiles": [],
"benchmarkScoreKeys": [],
"reason": "success"
}
```
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8723125792202374961/+/u/run_android_defines_test/stdout
All checks passed
https://github.com/flutter/flutter/pull/162641/checks?check_run_id=36991537539
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
Someone could always resurrect it via `git`-magic (or GitHub), but let's
not maintain (including Gradle lock files, Dart analysis, etc) unused
code that hasn't run in 12+ months.
/cc @johnmccutchan
As per https://github.com/flutter/flutter/issues/162620, we are going to
run the web benchmarks at `-O2` for a trial period to evaluate the
performance difference, which will give us some data on whether we can
consider changing to `-O2` by default.
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.
**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.
---------
Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
When running `dart format` over these lines the `// ignore` ended up on
a line where it wasn't properly ignoring the lint. This adjusts the
placement of `// ignore`s so they will continue to ignore the right
thing even after the code is auto formatted.
I am hoping that if we do this now the large PR that formats the entire
repo will go in smoother without manual intervention.
Fix the `--ab` option in the benchmark harness:
- Make `--local-engine-host` optional. The web engine doesn't need it, so it doesn't build it. But the tool crashes by failing to find it.
- Disable icon tree shaking because `--ab` runs against local engine build, whose Dart kernel version is out of sync with the framework, which crashes the const finder.
**Three things**
Re-lands #154374
New: fix `platform_channels_benchmarks` to print the "done" key. Updated notes for the microbenchmark parser. There are no other users of `microbenchmarks.readJsonResults`.
Re-Re-land: Uninstall microbenchmarks before running them.
Flakes in https://github.com/flutter/flutter/issues/153828 stem from adb saying the app isn't installed, but then failing to install wtih -r. Several other tests uninstall the app before trying to run it. Previous fix called uninstall between tests, but iOS takes 12 to 13 seconds to perform uninstall / install, which timed out the test. Just uninstall the one time since we only care about any lingering apps with different keys.
Potential solution https://github.com/flutter/flutter/issues/153828
Re-land Make things go fast
Instead of installing 21 different compilations of the same app to get results; compile and run them together. Locally on Mac+iOS, this should takes ~3 minutes instead of ~15 minutes.
Reverts: flutter/flutter#154374
Initiated by: jtmcdole
Reason for reverting: A different benchmark was using the microbenchmark parser and timing out.
Original PR Author: jtmcdole
Reviewed By: {zanderso}
This change reverts the following previous change:
Two things:
**Re-land**: Uninstall microbenchmarks before running them.
Flakes in #153828 stem from adb saying the app isn't installed, but then failing to install wtih `-r`. Several other tests uninstall the app before trying to run it. Previous fix called uninstall between tests, but iOS takes 12 to 13 seconds to perform uninstall / install, which timed out the test. Just uninstall the one time since we only care about any lingering apps with different keys.
Potential solution #153828
**Make things go fast**
Instead of installing 21 different compilations of the same app to get results; compile and run them together. Locally on Mac+iOS, this should takes ~3 minutes instead of ~15 minutes.
Two things:
**Re-land**: Uninstall microbenchmarks before running them.
Flakes in #153828 stem from adb saying the app isn't installed, but then failing to install wtih `-r`. Several other tests uninstall the app before trying to run it. Previous fix called uninstall between tests, but iOS takes 12 to 13 seconds to perform uninstall / install, which timed out the test. Just uninstall the one time since we only care about any lingering apps with different keys.
Potential solution #153828
**Make things go fast**
Instead of installing 21 different compilations of the same app to get results; compile and run them together. Locally on Mac+iOS, this should takes ~3 minutes instead of ~15 minutes.
Reverts: flutter/flutter#154183
Initiated by: zanderso
Reason for reverting: microbenchmarks timing out on CI.
Original PR Author: jtmcdole
Reviewed By: {andrewkolos}
This change reverts the following previous change:
Flakes in #153828 stem from adb saying the app isn't installed, but then failing to install wtih `-r`. Several other tests uninstall the app before trying to run it.
Potential corrects #153828
Flakes in #153828 stem from adb saying the app isn't installed, but then
failing to install wtih `-r`. Several other tests uninstall the app
before trying to run it.
Potential corrects #153828
A recent change to the flutter_drone recipe is setting FLUTTER_TEST_OUTPUTS_DIR in the environment (see https://flutter.googlesource.com/recipes/+/b111cd3ef2297e66905cc48c2cfedce0cf1ba46f)
If FLUTTER_TEST_OUTPUTS_DIR is set, then perf_tests will use an output path based on that variable. perf_tests_test needs to use the same logic when looking for the expected output file.
