flutter/dev/benchmarks/macrobenchmarks
Jenn Magder 404655aae1
Update gradle memory properties in example and test projects (#163798)
Match the example and integration test apps to the the properties in the
`flutter create` template, updated in
https://github.com/flutter/flutter/pull/156201.

We discussed perhaps making this an auto-migration for existing Flutter
apps (if they've never edited the template from the old values), but it
probably makes sense for devs to manage this themselves depending on how
beefy the machine they are running on is.

Filed https://github.com/flutter/flutter/issues/163801 to show a nicer
message when the user hits this so they can manage
`java.lang.OutOfMemoryError: Java heap space` errors themselves.

Speculatively hoping this helps with the `Java heap space` build test
flakes
https://github.com/flutter/flutter/issues/163121#issuecomment-2665783740



## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
2025-02-21 21:08:20 +00:00
..
android Update gradle memory properties in example and test projects (#163798) 2025-02-21 21:08:20 +00:00
assets [web] Add wrapbox scroll benchmark (#70966) 2020-12-02 08:30:42 -08:00
ios [flutter_tools] opt iOS/macOS apps out of Metal API validation via migrator, update templates in repo. (#159228) 2024-11-21 01:36:55 +00:00
lib Introduce caching mechanism during compile semantics tree monorepo and formatted version (#161195) 2025-02-06 03:08:23 +00:00
macos [flutter_tools] opt iOS/macOS apps out of Metal API validation via migrator, update templates in repo. (#159228) 2024-11-21 01:36:55 +00:00
test remove frame policy benchmark. (#163245) 2025-02-13 20:26:04 +00:00
test_driver remove frame policy benchmark. (#163245) 2025-02-13 20:26:04 +00:00
test_memory Clean up the bindings APIs. (#89451) 2022-02-03 14:55:15 -08:00
web Add benchmarks for single-threaded Skwasm. (#158027) 2024-11-04 16:38:10 +00:00
.gitignore [web] Use plain platform views in benchmarks (#126080) 2023-05-05 14:17:34 +00:00
pubspec.yaml Roll pub packages (#163083) 2025-02-11 23:47:59 +00:00
README.md [web] Remove HTML from bots, test suites and e2e tests (#161537) 2025-01-22 18:50:46 +00:00

Macrobenchmarks

Performance benchmarks use either flutter drive or the web benchmark harness.

Mobile benchmarks

Cull opacity benchmark

To run the cull opacity benchmark on a device:

flutter drive --profile -t test_driver/run_app.dart --driver test_driver/cull_opacity_perf_test.dart

Results should be in the file build/cull_opacity_perf.timeline_summary.json.

More detailed logs should be in build/cull_opacity_perf.timeline.json.

Cubic bezier benchmark

To run the cubic-bezier benchmark on a device:

flutter drive --profile -t test_driver/run_app.dart --driver test_driver/cubic_bezier_perf_test.dart

Results should be in the file build/cubic_bezier_perf.timeline_summary.json.

More detailed logs should be in build/cubic_bezier_perf.timeline.json.

Backdrop filter benchmark

To run the backdrop filter benchmark on a device: To run a mobile benchmark on a device:

flutter drive --profile -t test_driver/run_app.dart --driver test_driver/[test_name]_test.dart

Results should be in the file build/[test_name].timeline_summary.json.

More detailed logs should be in build/[test_name].timeline.json.

The key [test_name] can be:

  • animated_placeholder_perf
  • backdrop_filter_perf
  • color_filter_and_fade_perf
  • cubic_bezier_perf
  • cull_opacity_perf
  • fading_child_animation_perf
  • imagefiltered_transform_animation_perf
  • multi_widget_construction_perf
  • picture_cache_perf
  • post_backdrop_filter_perf
  • simple_animation_perf
  • textfield_perf
  • fullscreen_textfield_perf

E2E benchmarks

(On-going work)

E2E-based tests are driven independent of the host machine. The following tests are E2E:

  • cull_opacity_perf.dart
  • multi_widget_construction_perf

These tests should be run by:

flutter drive --profile -t test/[test_name]_e2e.dart --driver test_driver/e2e_test.dart

Web benchmarks

Web benchmarks are compiled from the same entry point in lib/web_benchmarks.dart.

How to write a web benchmark

Create a new file for your benchmark under lib/src/web. See bench_draw_rect.dart as an example.

Choose one of the two benchmark types:

  • A "raw benchmark" records performance metrics from direct interactions with dart:ui with no framework. This kind of benchmark is good for benchmarking low-level engine primitives, such as layer, picture, and semantics performance.
  • A "widget benchmark" records performance metrics using a widget. This kind of benchmark is good for measuring the performance of widgets, often together with engine work that widget-under-test incurs.
  • A "widget build benchmark" records the cost of building a widget from nothing. This is different from the "widget benchmark" because typically the latter only performs incremental UI updates, such as an animation. In contrast, this benchmark pumps an empty frame to clear all previously built widgets and rebuilds them from scratch.

For a raw benchmark extend RawRecorder (tip: you can start by copying bench_draw_rect.dart).

For a widget benchmark extend WidgetRecorder (tip: you can start by copying bench_simple_lazy_text_scroll.dart).

For a widget build benchmark extend WidgetBuildRecorder (tip: you can start by copying bench_build_material_checkbox.dart).

Pick a unique benchmark name and class name and add it to the benchmarks list in lib/web_benchmarks.dart.

How to run a web benchmark

Web benchmarks can be run using flutter run in debug, profile, and release modes, using either the CanvasKit or the Skwasm rendering backend. Note, however, that running in debug mode will result in worse numbers. Profile mode is useful for profiling in Chrome DevTools because the numbers are close to release mode and the profile contains unobfuscated names.

Example:

cd dev/benchmarks/macrobenchmarks

# Runs in profile mode
flutter run --profile -d web-server lib/web_benchmarks.dart

You can also run all benchmarks exactly as the devicelab runs them:

cd dev/devicelab

# Runs using the CanvasKit renderer
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t bin/tasks/web_benchmarks_canvaskit.dart

Frame policy test

File test/frame_policy.dart and its driving script test_driver/frame_policy_test.dart are used for testing fullyLive and benchmarkLive policies in terms of its effect on WidgetTester.handlePointerEventRecord.