From 1df8b0f8942df06ae9b474f377c9fd18bb501a2c Mon Sep 17 00:00:00 2001 From: Kate Lovett Date: Tue, 21 May 2024 14:31:54 -0500 Subject: [PATCH] Updated Flutter FirebaseLab Tests (markdown) --- Flutter-FirebaseLab-Tests.md | 96 +----------------------------------- 1 file changed, 1 insertion(+), 95 deletions(-) diff --git a/Flutter-FirebaseLab-Tests.md b/Flutter-FirebaseLab-Tests.md index eb3d8e7..92df667 100644 --- a/Flutter-FirebaseLab-Tests.md +++ b/Flutter-FirebaseLab-Tests.md @@ -1,95 +1 @@ -## FirebaseLab Tests - -Flutter FirebaseLab tests are used to build flutter applications and to run them using different versions of emulators and physical devices. - -These tests consist of two parts: - -* [Firebaselab recipe](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/firebaselab/firebaselab.py) -* [.ci.yaml](https://github.com/flutter/flutter/blob/master/.ci.yaml#L413) configuration file - -The recipe supports three properties: **physical\_devices** to specify actual hardware connected to firebase infra to run tests, **virtual\_devices** to specify the virtual devices(avd) to use and **task\_name** for selecting the integration test to build. - -physical\_device and virtual\_device are strings that take the **MODEL\_ID** format defined by firebase. Use `gcloud firebase test android models list` (assuming you have [gcloud](https://cloud.google.com/sdk/gcloud) installed) for a list of possible model id’s. - -Task name is the subdirectory of [dev/integration\_tests](https://github.com/flutter/flutter/tree/master/dev/integration_tests) (e.g. android\_views, channels, etc ) that contains the integration test to build. - -The following is an example of the properties format: - -```json -physical_devices: >- - [ - "--device", "model=oriole,version=33", - "--device", "model=griffin,version=24" - ], -virtual_device: >- - [ - "--device", "model=Nexus5,version=21", - "--device", "model=Nexus6P,version=27" - ] -``` - -The recipe executes the following workflow: - - - -1. Reads **physical\_devices** property and if not empty it builds an app bundle for the integration test referenced by **task\_name**. -2. Reads **virtual\_devices** property and if not empty it builds an apk for the integration test. Apks are built for virtual\_devices to prevent them from picking the wrong binary and using runtime translation. -3. Uses the `gcloud firebase` command to upload the binary and delegates the execution of the test to firebase lab. -4. The gcloud command blocks until the execution is complete -5. The recipe reads the logcat and the test succeeds if no `E/flutter` is found in the logcat file. -6. If the test fails it retries for maximum of 3 times -7. The recipe also supports [infra\_failure\_codes = (1, 15, 20)](https://firebase.google.com/docs/test-lab/ios/command-line#script-exit-codes) to prevent firebaselab infrastructure failures from closing the tree. - - -## Prerequisites (googlers) - -This is only required for manually running the steps - -* [Install gcloud CLI](https://cloud.google.com/sdk/docs/install) -* Request access on demand write access to `flutter-infra-staging` -* Build the application and run the gcloud command. You can check the [firebaselab recipe](https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/firebaselab/firebaselab.py) for the details. - -## Adding a Firebaselab Test - -* Step 1: Select the integration test to use from [dev/integration\_tests](https://github.com/flutter/flutter/tree/master/dev/integration_tests) -* Step 2: Select the physical and virtual devices to run the test on. You can use `gcloud firebase test android models list` and `gcloud firebase test ios models list` to find the available devices. -* Step 3: Write a .ci.yaml target configuration in the [flutter/flutter .ci.yaml](https://github.com/flutter/flutter/blob/master/.ci.yaml) file providing the task\_name, virtual\_devices, physical\_devices and recipe properties. -* Step 4: Create a PR with the new target. The presubmit checks will run basic validations on yaml format. -* Step 5: Wait for the change to propagate. -* Step 6: Fix any potential issues and remove `bringup: true` to validate the changes end to end in presubmit. - -The following is an example of a full target: **Linux firebase_oriol33_abstract_method_smoke_test**. By convention the name should follow the format “<host os> firebase\_<model id>\_<taskname>” - -In the example below `recipe: firebaselab/firebaselab` refers to the recipe to use to run the test and should probably always be the same, `dependencies` refers to the android sdk to use for the test, in general all tests should use the same android sdk unless specifically testing something that changes across android sdk versions, `task_name` refers to the integration test to use see the definition above for where to find the code that is run, `physical_devices` `virtual_devices` are defined above. - -```yaml - - name: Linux firebase_oriol33_abstract_method_smoke_test - # This is required for new tests to allow the - # configuration to propagate. - bringup: true - recipe: firebaselab/firebaselab - # The unit for timeout is minutes. 1 hour is enough - # for most use cases unless the test is using a device - # with low capacity and the queue is expected to be - # longer than 30 minutes. - timeout: 60 - Properties: - # These top level dependencies are shared between firebaselab - # tests. For the current values you can copy paste the - # dependencies from another firebaselab target. Changing these - # values are only necessary when upgrading to a new android sdk - # version. - dependencies: >- - [ - {"dependency": "android\_sdk", "version": "version:33v6"} - ] - # Use for metrics collection and to be able to filter tasks** - # in swarming. - tags: > - ["firebaselab"] - task_name: abstract_method_smoke_test - physical_devices: >- - ["--device", "model=oriole,version=33"] - virtual_devices: >- - [] -``` \ No newline at end of file +This page has migrated to https://github.com/flutter/flutter/blob/master/docs/infra/Flutter-FirebaseLab-Tests.md \ No newline at end of file