Add `--ab-local-engine-only` flag to change A/B benchmark aggregate behavior to instead run the number of tasks specified with the local engine only, not the default engine. This is useful when you are trying to get average and noise results for a particular engine branch, without needing the default engine every time you change the engine.
Fixes https://github.com/flutter/flutter/issues/143770
- fix https://github.com/flutter/flutter/issues/53707 by having the test not expect a timeout but instead actually look for the retry message
- simplify the `--task` option to only accept task names rather than also accepting paths
- remove some obsolete options that referred to the manifest which no longer seems to exist
Fixes#134154
This PR also changes the default value of the `--exit` flag from `true` to `false`. Effectively, this is not a change in behavior since `--exit` didn't previously work.
Some places where we are missing a forward of the `--local-engine-host` values in the benchmark code preventing local running of the benchmarks (does not affect CI usages which don't override local engine).
* Testing whether emulator is possible.
* Adding changes to see if emulator can be used from recipe.
* adding emulator support.
* Add the emulator flag for testing.
* Using string for boolean since it cannot be parsed in properties
* Checking to see if these changes are being used.
* Updated bool back to string
* Remove trailing whitespace from file.
* Implement device selection for devicelab/run.dart
* Add test to --device-id option for devicelab/run
* Update dev/devicelab/bin/run.dart by jonahwilliam
* Rename deviceOperatingSystem enum mock -> fake
Co-authored-by: Jonah Williams <jonahwilliams@google.com>
* print intermediate A/B results when not silent
* print raw A/B results when in loud mode
* add tests; handle missing metrics more gracefully
* use less fancy section header on Windows
* Update project.pbxproj files to say Flutter rather than Chromium
Also, the templates now have an empty organization so that we don't cause people to give their apps a Flutter copyright.
* Update the copyright notice checker to require a standard notice on all files
* Update copyrights on Dart files. (This was a mechanical commit.)
* Fix weird license headers on Dart files that deviate from our conventions; relicense Shrine.
Some were already marked "The Flutter Authors", not clear why. Their
dates have been normalized. Some were missing the blank line after the
license. Some were randomly different in trivial ways for no apparent
reason (e.g. missing the trailing period).
* Clean up the copyrights in non-Dart files. (Manual edits.)
Also, make sure templates don't have copyrights.
* Fix some more ORGANIZATIONNAMEs
* Update dependencies with forced upgrade.
Now that https://github.com/flutter/flutter/issues/15261 is fixed, do the update with upgrade.
* Update test package use following upgrade
* Replace addOption(allowMultiple) with addMultiOption
