These devicelab tests are incredibly flaky. Currently they will build & install the same application 15 times. This causes the temp storage to fill up on android, and has a good chance of flaking on iOS due to install issues.
The change from 3 to 15 increased total test time for 2 -> 12 minutes on iOS, or roughly 5 more test equivalents for every single iOS test. Reduce the iteration count back to 5
Uninstall the app after each run so temp storage does not fill up.
These devicelab tests are incredibly flaky. Currently they will build & install the same application 15 times. This causes the temp storage to fill up on android, and has a good chance of flaking on iOS due to install issues.
The change from 3 to 15 increased total test time for 2 -> 12 minutes on iOS, or roughly 5 more test equivalents for every single iOS test. Reduce the iteration count back to 5
Uninstall the app after each run so temp storage does not fill up.
Remove devicelab specific code for shutting down gradle daemon, add --android-gradle-daemon option to build/run/drive`. Avoids need for un-tested devicelab specific handler. There are also some feature requests for this, so 2 birds one stone.
Example:
flutter build apk --no-android-gradle-daemon will pass --no-daemon on to gradle
Remove devicelab specific code for shutting down gradle daemon, add --android-gradle-daemon option to build/run/drive`. Avoids need for un-tested devicelab specific handler. There are also some feature requests for this, so 2 birds one stone.
Example:
flutter build apk --no-android-gradle-daemon will pass --no-daemon on to gradle
- run_without_leak_tests: have been disabled for months
- build_benchmark: the tooling work here is mostly done and we're not tracking further improvements, free up more devicelab capacity
- system_debug_ios: does not work post iOS13
- mac_enable_twc: not adding more mac tests to devicelab
- hello_world_start_up: disabled
Any tests that we think will be valuable in the future can be resurrected from the git history.
--use-application-binary allows running with an already built APK. This can be useful for speeding up CI test cases, or in our case eventually supporting some sort of build server. Demonstrate that this works by updating the old gallery test to use it. Fixes#56604
Co-authored-by: Jenn Magder <magder@google.com>
A change which sped up hot restart locally caused many of the devicelab measures to regress. I think this is because we do not measure when the isolate is actually "ready", so starting a reload or restart prematurely can cause time spent doing initialization to be registered as part of the reload operation.
A fix for this would be to have the framework include some sort of "initialization complete" event ... but it is not clear what the correct trigger would be. Perhaps after the first frame is successfully registered?
(9a3a0dc caused the benchmark regression - possibly since we spend less time syncing files now so we start the restart earlier)
Currently these tests are taking CI time without providing any value, because there is no verification that network connectivity is avoided. This is better covered by a unit test that verifies that --offline is passed to pub.
