flutter/dev/integration_tests/android_engine_test
Matan Lurey e517ae3457
Refactor android_engine_test, make it easier to debug/deflake locally. (#161534)
The goal here is to have a great standalone `android_engine_test` suite
that [replaces
`scenario_app/android`](https://github.com/flutter/flutter/pull/160992).

No test is _functionally_ changed in this PR, but overview of changes:
- Finished renaming the suite `android_engine_tests` instead of
`flutter_driver_android`
- Added instructions and an environment variable for local generation of
golden-files (`UPDATE_GOLDENS=1`)
- Added explanations of the individual tests, where they live, and how
to run them locally
- Added a hybrid-composition (HC, not TLHC, which is already tested)
test
- Renamed "other_smiley" to "surface_texture_smiley" (and renamed the
original to "surface_producer_smiley")
- Removed unnecessary ".android" suffix (we will not run this on
anything but Android)
- Added a `tool/deflake.dart` to run a test suite 10x (or custom) times
locally to try and determine flakiness

After this PR, I'll add flags to let you control variants and name the
screenshots accordingly, i.e.:
- API v34 or v35
- OpenGLES or Vulkan (will require an `AndroidManifest.xml` edit during
the test instrumentation)
2025-01-15 01:05:31 +00:00
..
android Add a virtual-display (VD) platform view test, and refactor tests a bit. (#161349) 2025-01-10 03:23:40 +00:00
lib Refactor android_engine_test, make it easier to debug/deflake locally. (#161534) 2025-01-15 01:05:31 +00:00
test_driver Refactor android_engine_test, make it easier to debug/deflake locally. (#161534) 2025-01-15 01:05:31 +00:00
tool Refactor android_engine_test, make it easier to debug/deflake locally. (#161534) 2025-01-15 01:05:31 +00:00
.metadata Rename native_driver to android_{driver_extensions|engine_test} (#161263) 2025-01-08 20:28:38 +00:00
pubspec.yaml Update package revisions to latest (#161525) 2025-01-14 18:22:42 +00:00
README.md Refactor android_engine_test, make it easier to debug/deflake locally. (#161534) 2025-01-15 01:05:31 +00:00

android_engine_test

This directory contains a sample app and tests that demonstrate how to use the (experimental) native Flutter Driver API to drive Flutter apps that run on Android devices or emulators, interact with and capture screenshots of the app, and compare the screenshots against golden images.

Caution

This test suite is a very end-to-end suite that is testing a combination of the graphics backend, the Android embedder, the Flutter framework, and Flutter tools, and only useful when the documentation and naming stays up to date and is clearly actionable.

Please take extra care when updating the test suite to also update the REAMDE.

How it runs on CI (LUCI)

See dev/bots/suite_runners/run_android_engine_tests.dart, but tl;dr:

# TIP: If golden-files do not exist locally, this command will fail locally.
SHARD=android_engine_tests bin/cache/dart-sdk/bin/dart dev/bots/test.dart

Running the apps and tests

Each lib/{prefix}_main.dart file is a standalone Flutter app that you can run on an Android device or emulator.

flutter_rendered_blue_rectangle

This app displays a full screen blue rectangle. It mostly serves as a test that Flutter can run at all on the target device, and that the Flutter (native) driver can take a screenshot and compare it to a golden image. If this app or test fails, it's likely none of the other apps or tests will work either.

# Run the app
$ flutter run lib/flutter_rendered_blue_rectangle_main.dart

# Run the test
$ flutter drive lib/flutter_rendered_blue_rectangle_main.dart

external_texture/surface_producer_smiley_face

This app displays a full screen rectangular deformed smiley face with a yellow background. It tests the SurfaceProducer API end-to-end, including historic regression cases around backgrounding the app, trimming memory, and resuming the app.

# Run the app
$ flutter run lib/external_texture/surface_producer_smiley_face_main.dart

# Run the test
$ flutter drive lib/external_texture/surface_producer_smiley_face_main.dart

external_texture/surface_texture_smiley_face

This app displays a full screen rectangular deformed smiley face with a yellow background. It tests the SurfaceTexture API end-to-end.

# Run the app
$ flutter run lib/external_texture/surface_texture_smiley_face_main.dart

# Run the test
$ flutter drive lib/external_texture/surface_texture_smiley_face_main.dart

platform_view/hybrid_composition_platform_view

This app displays a blue orange gradient, the app is backgrounded, and then resumed. It tests the Hybrid Composition implementation.

# Run the app
$ flutter run lib/platform_view/hybrid_composition_platform_view_main.dart

# Run the test
$ flutter drive lib/platform_view/hybrid_composition_platform_view_main.dart

platform_view/texture_layer_hybrid_composition_platform_view

This app displays a blue orange gradient, the app is backgrounded, and then resumed. It tests the Texture Layer Hybrid Composition implementation.

# Run the app
$ flutter run lib/platform_view/texture_layer_hybrid_composition_platform_view_main.dart

# Run the test
$ flutter drive lib/platform_view/texture_layer_hybrid_composition_platform_view_main.dart

platform_view/virtual_display_platform_view

This app displays a blue orange gradient, the app is backgrounded, and then resumed. It tests the Virtual Display implementation.

# Run the app
$ flutter run lib/platform_view/virtual_display_platform_view_main.dart

# Run the test
$ flutter drive lib/platform_view/virtual_display_platform_view_main.dart

platform_view_tap_color_change

This app displays a blue rectangle, using platform views, which upon being tapped (natively, not by Flutter), changes from blue to red.

# Run the app
$ flutter run lib/platform_view_tap_color_change_main.dart

# Run the test
$ flutter drive lib/platform_view_tap_color_change_main_test.dart

Deflaking

Use tool/deflake.dart <path/to/lib/main.dart> to, in 1-command:

  • Build an APK.
  • Establish a baseline set of golden-files locally.
  • Run N tests (by default, 10) in the same state, asserting the same output.

For example:

dart tool/deflake.dart lib/flutter_rendered_blue_rectangle_main.dart

For more options, see dart tool/deflake.dart --help.