flutter/dev/benchmarks/macrobenchmarks
Gray Mackall 2948917a47
[Reland] Force automatic ndk download when building for Android (#160260)
Relands https://github.com/flutter/flutter/pull/159756.

The original caused some stderr because
1. the configurable `ndkVersion` value that AGP provides used to be null
by default.
2. Now it is not, instead it has a default value
3. `integration_test` doesn't actually use the NDK, but default for
`integration_test` was higher that `flutter.ndkVersion`
4. We have code that checks if a plugin uses a higher ndk version and
tells you to consider upgrading
https://github.com/flutter/flutter/blob/master/packages/flutter_tools/gradle/src/main/groovy/flutter.groovy#L921

This reverts the revert, and then aligns `integration_test`'s NDK
version (which again, it doesn't actually use, so I'm not too concerned
about the downgrade) with `flutter.ndkVersion`.

It also starts using the `flutter.min/compile/ndkVersion` values in
`integration_test`, which was unrelated to the original pr. Can remove
if you think it should land separately.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [ ] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

---------

Co-authored-by: Gray Mackall <mackall@google.com>
2024-12-13 20:27:23 +00:00
..
android [Reland] Force automatic ndk download when building for Android (#160260) 2024-12-13 20:27:23 +00:00
assets
ios [flutter_tools] opt iOS/macOS apps out of Metal API validation via migrator, update templates in repo. (#159228) 2024-11-21 01:36:55 +00:00
lib [web] Enable platform view benchmarks in Skwasm (#160186) 2024-12-12 20:32:27 +00:00
macos [flutter_tools] opt iOS/macOS apps out of Metal API validation via migrator, update templates in repo. (#159228) 2024-11-21 01:36:55 +00:00
test Add benchmark for scrolling very long pictures (#133434) 2023-08-31 19:03:14 +00:00
test_driver Remove android stack_size_test (#153695) 2024-08-19 12:25:35 -07:00
test_memory
web Add benchmarks for single-threaded Skwasm. (#158027) 2024-11-04 16:38:10 +00:00
.gitignore [web] Use plain platform views in benchmarks (#126080) 2023-05-05 14:17:34 +00:00
pubspec.yaml Bump Dart SDK to 3.7 (#159812) 2024-12-05 21:39:53 +00:00
README.md [web] Remove the benchmarks of the HTML renderer (#158520) 2024-11-15 16:24:20 +00:00

Macrobenchmarks

Performance benchmarks use either flutter drive or the web benchmark harness.

Mobile benchmarks

Cull opacity benchmark

To run the cull opacity benchmark on a device:

flutter drive --profile -t test_driver/run_app.dart --driver test_driver/cull_opacity_perf_test.dart

Results should be in the file build/cull_opacity_perf.timeline_summary.json.

More detailed logs should be in build/cull_opacity_perf.timeline.json.

Cubic bezier benchmark

To run the cubic-bezier benchmark on a device:

flutter drive --profile -t test_driver/run_app.dart --driver test_driver/cubic_bezier_perf_test.dart

Results should be in the file build/cubic_bezier_perf.timeline_summary.json.

More detailed logs should be in build/cubic_bezier_perf.timeline.json.

Backdrop filter benchmark

To run the backdrop filter benchmark on a device: To run a mobile benchmark on a device:

flutter drive --profile -t test_driver/run_app.dart --driver test_driver/[test_name]_test.dart

Results should be in the file build/[test_name].timeline_summary.json.

More detailed logs should be in build/[test_name].timeline.json.

The key [test_name] can be:

  • animated_placeholder_perf
  • backdrop_filter_perf
  • color_filter_and_fade_perf
  • cubic_bezier_perf
  • cull_opacity_perf
  • fading_child_animation_perf
  • imagefiltered_transform_animation_perf
  • multi_widget_construction_perf
  • picture_cache_perf
  • post_backdrop_filter_perf
  • simple_animation_perf
  • textfield_perf
  • fullscreen_textfield_perf

E2E benchmarks

(On-going work)

E2E-based tests are driven independent of the host machine. The following tests are E2E:

  • cull_opacity_perf.dart
  • multi_widget_construction_perf

These tests should be run by:

flutter drive --profile -t test/[test_name]_e2e.dart --driver test_driver/e2e_test.dart

Web benchmarks

Web benchmarks are compiled from the same entry point in lib/web_benchmarks.dart.

How to write a web benchmark

Create a new file for your benchmark under lib/src/web. See bench_draw_rect.dart as an example.

Choose one of the two benchmark types:

  • A "raw benchmark" records performance metrics from direct interactions with dart:ui with no framework. This kind of benchmark is good for benchmarking low-level engine primitives, such as layer, picture, and semantics performance.
  • A "widget benchmark" records performance metrics using a widget. This kind of benchmark is good for measuring the performance of widgets, often together with engine work that widget-under-test incurs.
  • A "widget build benchmark" records the cost of building a widget from nothing. This is different from the "widget benchmark" because typically the latter only performs incremental UI updates, such as an animation. In contrast, this benchmark pumps an empty frame to clear all previously built widgets and rebuilds them from scratch.

For a raw benchmark extend RawRecorder (tip: you can start by copying bench_draw_rect.dart).

For a widget benchmark extend WidgetRecorder (tip: you can start by copying bench_simple_lazy_text_scroll.dart).

For a widget build benchmark extend WidgetBuildRecorder (tip: you can start by copying bench_build_material_checkbox.dart).

Pick a unique benchmark name and class name and add it to the benchmarks list in lib/web_benchmarks.dart.

How to run a web benchmark

Web benchmarks can be run using flutter run in debug, profile, and release modes, using either the CanvasKit or the Skwasm rendering backend. Note, however, that running in debug mode will result in worse numbers. Profile mode is useful for profiling in Chrome DevTools because the numbers are close to release mode and the profile contains unobfuscated names.

Example:

cd dev/benchmarks/macrobenchmarks

# Runs in profile mode using the CanvasKit renderer
flutter run --web-renderer=canvaskit --profile -d web-server lib/web_benchmarks.dart

You can also run all benchmarks exactly as the devicelab runs them:

cd dev/devicelab

# Runs using the CanvasKit renderer
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t bin/tasks/web_benchmarks_canvaskit.dart

Frame policy test

File test/frame_policy.dart and its driving script test_driver/frame_policy_test.dart are used for testing fullyLive and benchmarkLive policies in terms of its effect on WidgetTester.handlePointerEventRecord.