- run_without_leak_tests: have been disabled for months
- build_benchmark: the tooling work here is mostly done and we're not tracking further improvements, free up more devicelab capacity
- system_debug_ios: does not work post iOS13
- mac_enable_twc: not adding more mac tests to devicelab
- hello_world_start_up: disabled
Any tests that we think will be valuable in the future can be resurrected from the git history.
--use-application-binary allows running with an already built APK. This can be useful for speeding up CI test cases, or in our case eventually supporting some sort of build server. Demonstrate that this works by updating the old gallery test to use it. Fixes#56604
Co-authored-by: Jenn Magder <magder@google.com>
A change which sped up hot restart locally caused many of the devicelab measures to regress. I think this is because we do not measure when the isolate is actually "ready", so starting a reload or restart prematurely can cause time spent doing initialization to be registered as part of the reload operation.
A fix for this would be to have the framework include some sort of "initialization complete" event ... but it is not clear what the correct trigger would be. Perhaps after the first frame is successfully registered?
(9a3a0dc caused the benchmark regression - possibly since we spend less time syncing files now so we start the restart earlier)
Currently these tests are taking CI time without providing any value, because there is no verification that network connectivity is avoided. This is better covered by a unit test that verifies that --offline is passed to pub.
* Implement device selection for devicelab/run.dart
* Add test to --device-id option for devicelab/run
* Update dev/devicelab/bin/run.dart by jonahwilliam
* Rename deviceOperatingSystem enum mock -> fake
Co-authored-by: Jonah Williams <jonahwilliams@google.com>
This rolls the engine from:
965fbbed1776545ee681cb57f3fb1d0c2000bbcb to
b5f5e6332cb4987e9e38ffaa267733ec0a8705ba. A Dart SDK roll in this range
changed the format of the compileExpression RPC VMService endpoint,
necessitating a tool patch.
This adds some information about the chosen device to the output log so that we can tell what type of device/OS Level/API Level the test is running on when doing a postmortem on a failed test.
Work towards #16723
This is only safe to land after #58131 lands in google3. Only build NOTICES in asset manfiest, and load either LICENSE or NOTICES from pubspec dependencies.
When `const LocalPlatform` is used immediately and not passed as a
dependency it does not have utility over the static properties from
`dart:io`. Remove these uses to reduce the noise around how the
instantiable `Platform` class is used.
* print intermediate A/B results when not silent
* print raw A/B results when in loud mode
* add tests; handle missing metrics more gracefully
* use less fancy section header on Windows
Move Flutter Gallery to dev/integration_tests/ as it is an older copy used only for testing. The current version of the Flutter Gallery now lives in https://github.com/flutter/gallery.
dev_finder name is deprecated. device-finder is preferred.
Functionality remains identical, this is an aesthetic change.
This change was generated via:
$ find . -not -path '*/\.*' -type f -exec sed -i 's/(^import.*)!dev_finder/device-finder/g' {} \;
* Update project.pbxproj files to say Flutter rather than Chromium
Also, the templates now have an empty organization so that we don't cause people to give their apps a Flutter copyright.
* Update the copyright notice checker to require a standard notice on all files
* Update copyrights on Dart files. (This was a mechanical commit.)
* Fix weird license headers on Dart files that deviate from our conventions; relicense Shrine.
Some were already marked "The Flutter Authors", not clear why. Their
dates have been normalized. Some were missing the blank line after the
license. Some were randomly different in trivial ways for no apparent
reason (e.g. missing the trailing period).
* Clean up the copyrights in non-Dart files. (Manual edits.)
Also, make sure templates don't have copyrights.
* Fix some more ORGANIZATIONNAMEs
Removes multiple re-entrant calls of bundle and aot and replaces them with a single call to assemble. This restores full caching and will allow follow-up performance improvements when building multiple ABIs
This reverts commit baea9bf7cc.
Additionally, we let the test run on mac8 with iphonexs because the test won't run on Xcode 10.1 (mac3-7). Hence we force it to run on mac8 which currently has Xcode 10.2.
This reverts commit af9f424d5d.
Reverts flutter/flutter#39439
Reason: this passed the local device lab test on my Macbook but it failed in the actual device lab. Will investigate, fix, and reland.
TBR: @dnfield @goderbauer @tvolkert @Hixie
For https://github.com/flutter/flutter/issues/33899
Test added:
- simple_animation_perf_ios
Test modified:
- backdrop_filter_perf_ios__timeline_summary
We'll add the CPU/GPU measurement to more iOS tests
once it's proven to be non-flaky.
This relands https://github.com/flutter/flutter/pull/35297
The followings have been done to fix the broken tests:
1. Add `didSendFirstFrameRasterizedEvent` extension and its tests
2. Wait for `didSendFirstFrameRasterizedEvent` instead of
`didSendFirstFrameEvent` during start up tests
3. Mark missed (probably newly added) start up tests as flaky
`flutter build aar`
This new build command works just like `flutter build apk` or `flutter build appbundle`, but for plugin and module projects.
This PR also refactors how plugins are included in app or module projects. By building the plugins as AARs, the Android Gradle plugin is able to use Jetifier to translate support libraries into AndroidX libraries for all the plugin's native code. Thus, reducing the error rate when using AndroidX in apps.
This change also allows to build modules as AARs, so developers can take these artifacts and distribute them along with the native host app without the need of the Flutter tool. This is a requirement for add to app.
`flutter build aar` generates POM artifacts (XML files) which contain metadata about the native dependencies used by the plugin. This allows Gradle to resolve dependencies at the app level. The result of this new build command is a single build/outputs/repo, the local repository that contains all the generated AARs and POM files.
In a Flutter app project, this local repo is used by the Flutter Gradle plugin to resolve the plugin dependencies. In add to app case, the developer needs to configure the local repo and the dependency manually in `build.gradle`:
repositories {
maven {
url "<path-to-flutter-module>build/host/outputs/repo"
}
}
dependencies {
implementation("<package-name>:flutter_<build-mode>:1.0@aar") {
transitive = true
}
}
`flutter build aar`
This new build command works just like `flutter build apk` or `flutter build appbundle`, but for plugin and module projects.
This PR also refactors how plugins are included in app or module projects. By building the plugins as AARs, the Android Gradle plugin is able to use Jetifier to translate support libraries into AndroidX libraries for all the plugin's native code. Thus, reducing the error rate when using AndroidX in apps.
This change also allows to build modules as AARs, so developers can take these artifacts and distribute them along with the native host app without the need of the Flutter tool. This is a requirement for add to app.
`flutter build aar` generates POM artifacts (XML files) which contain metadata about the native dependencies used by the plugin. This allows Gradle to resolve dependencies at the app level. The result of this new build command is a single build/outputs/repo, the local repository that contains all the generated AARs and POM files.
In a Flutter app project, this local repo is used by the Flutter Gradle plugin to resolve the plugin dependencies. In add to app case, the developer needs to configure the local repo and the dependency manually in `build.gradle`:
repositories {
maven {
url "<path-to-flutter-module>build/host/outputs/repo"
}
}
dependencies {
implementation("<package-name>:flutter_<build-mode>:1.0@aar") {
transitive = true
}
}
This is the first step in getting esoteric devices wired
up to device lab to ensure that we can at least start Flutter
apps in profile mode on such devices.
https://github.com/flutter/flutter/issues/35838
This updates the flutter tool cache to download binary files for ideviceinstaller, ios-deploy, libimobiledevice, and dynamically linked dependencies from Flutter's GCP bucket.
This updates the flutter tool to cache binary files for ideviceinstaller, ios-deploy, libimobiledevice, and dynamically linked dependencies from Flutter's GCP bucket.
* Introduce image_list performance benchmark that runs on jit(debug) build.
We want to track performance of debug build because ongoing work on reducing the cost of spanwing new isolate will impact jit(debug) more than it will aot(release) configurations.
* Newlines and spaces
* Add test case for Flutter Issue #27677 as a benchmark.
See https://github.com/flutter/flutter/issues/27677
I got the following results running the test on a Moto E2 which
will help us determine how much we can gain by analyzing the
operations and eliminating unnecessary repaints based on
dirty rectangles/regions.
no blurs - avg 216.0 fps over 3 samples
blur the group - avg 22.9 fps over 3 samples
blur each txt - avg 3.4 fps over 3 samples
* Added the new benchmark to be tracked on the dashboard.
* Gradle generates ELF shared libraries instead of AOT snapshots.
* `flutter build apk/appbundle` supports multiple `--target-platform` and defaults to `android-arm` and `android-arm64`.
* `flutter build apk` now has a flag called `--split-per-abi`.
Updates documentation and non-public API to use American spellings for
consistency with the rest of the codebase.
No changes to behaviour... other than how it's spelt.
* Clean up some flutter_tools tests
* Remove arbitrary retry that happens even for fundamental errors, and generally clean up _DevFSHttpWriter.
* Update dependencies (requires fixes; see next commit)
* Fixes for new dependencies.
* Added support for authentication codes for the VM service.
Previously, a valid web socket connection would use the following URI:
`ws://127.0.0.1/ws`
Now, by default, the VM service requires a connection to be made with a
URI similar to the following:
`ws://127.0.0.1:8181/Ug_U0QVsqFs=/ws`
where `Ug_U0QVsqFs` is an authentication code generated and shared by
the
service.
This behavior can be disabled with the `--disable-service-auth-codes`
flag.
Previously, I used the Android emulator for testing and everything
seemed to work fine with stdout (if I remember correctly). But our
devicelab uses real Android devices and the warnings are routed to
stderr. Hence change stdout to stderr in the test.
* add trailing commas on list/map/parameters
* add trailing commas on Invocation with nb of arg>1
* add commas for widget containing widgets
* add trailing commas if instantiation contains trailing comma
* revert bad change
This patch adds a default shader warm up process which moves shader compilation from the animation time to the startup time. This also provides an extension for `runApp` so developers can customize the warm up process.
This should reduce our worst_frame_rasterizer_time_millis from ~100ms to ~20-30ms for both flutter_gallery and complex_layout benchmarks. Besides, this should also have a significant improvement on 90th and 99th percentile time (50%-100% speedup in some cases, but I haven't tested them thoroughly; I'll let our device lab collect the data afterwards).
The tradeoff the is the startup time (time to first frame). Our `flutter run --profile --trace-startup` seems to be a little noisy and I see about 100ms-200ms increase in that measurement for complex_layout and flutter_gallery. Note that this only happens on the first run after install or data wipe. Later the Skia persistent cache will remove the overhead.
This also adds a cubic_bezier benchmark to test the custom shader warm up process.
This should fix https://github.com/flutter/flutter/issues/813 (either by `defaultShaderWarmUp`, or a `customShaderWarmUp`).
* Fall back to ANDROID_SDK_ROOT if ANDROID_HOME is not set
And update descriptions to use the non-deprecated ANDROID_SDK_ROOT.
Fixes#15114.
* Remove trailing whitespace
* Update dev/devicelab/lib/framework/adb.dart
Co-Authored-By: DanTup <danny@tuppeny.com>
* Reformat long line
* Revert "Add dashing config file for generating docset from flutter docs (#24374)"
This reverts commit ec8ca8606c.
* Revert "Update driver script to execute test through test_core (#24168)"
This reverts commit 6c62cf337f.
This renames the "module" template to the "application" template, and makes "application" the default. The existing "app" template is now deprecated.
flutter create also now recognizes the type of project in an existing directory, and is able to recreate it without having the template type explicitly specified (although you can still do that). It does this now by first looking in the .metadata file for the new project_type field, and if it doesn't find that, then it looks at the directory structure. Also, the .metadata file is now overwritten even on an existing directory so that 1) the project_type can be added to legacy projects, and 2) the version of Flutter that updated the project last is updated.
I also cleaned up a bunch of things in create_test.dart, added many more tests, and added an example test to the test/ directory in the generated output of the application template.
Fixes#22530Fixes#22344
* Track number of package dependencies in Flutter
Relands #20774.
* Use evalFlutter instead of startProcess.
That way we don't need `flutter` on the PATH.
* Fix TODO syntax.
* Clarify messages for some timeouts, to aid debugging.
* Increase some other timeouts that were a needlessly short, to reduce sources of flakes.
* Remove some more timeouts that were mostly redundant, to remove complexity.
* Minor style cleanup.
* Remove some dangerous traps (specifically, hide the explicit start/end times in TimedEvent since they shouldn't matter).
All temporary directory start with `flutter_` and have their random component separated from the name by a period, as in `flutter_test_bundle.YFYQMY`.
I've tried to find some of the places where we didn't cleanly delete temporary directories, too. This greatly reduces, though it does not entirely eliminate, the directories we leave behind when running tests, especially `flutter_tools` tests.
While I was at it I standardized on `tempDir` as the variable name for temporary directories, since it was the most common, removing occurrences of `temp` and `tmp`, among others.
Also I factored out some common code that used to catch exceptions that happen on Windows, and made more places use that pattern.
* Refactor analysis benchmark and collect more data (#20169)
This does some code cleanup to avoid duplication, improves the logs,
and records min and max values as well as the mean.
* Fix typo that led to infinite loop
* Revert "increase size of user account drawer headers to 48 by 48 (#20266)"
This reverts commit 4a7b4a4dde.
* Revert "EditableText Cursor can be set to not blink for testing (#20004)"
This reverts commit d041b319e8.
* Revert "Refactor analysis benchmark and collect more data (#20169)"
This reverts commit 5ea0a13598.
* Revert "Revert "Roll engine to fed2ea458ed49088d33eddabc546ba56d600c717 (includes dart roll) (#19044)" (#19276)"
This reverts commit cf932490b7 as it also
includes fix for type error that broke tests.
* Add type cast for dart2 type checks.
* Move up to latest goldens
* Make inDirectory() type-parameterized.
* Add typecasting to transitions_perf_test.dart and microbenchmarks.
* Add boolean flag initialization in save_catalog_screenshots.dart
* Add type conversion to gallery transition test
* Roll engine to rolled dart
Add connectionTimeout property to class that extends HttpOverrides.
Upgrade packages.
Add more type fixes.
Update goldens hash.
Add more type fixes.
Upgrade packages.
Fix tests.
Remove strong-mode from analysis options yaml file.
Increase dev/devicelab timeout from 2 to 10 seconds.
Added getVM to waitForViews. Fix type errors in compileExpression.
* Pick up engine revision with fix for microbenchmarks regression.
Increase default timeout from 20/30 to 60s.
This should make it more obvious from the error what's gone wrong if we see a repeat of #19096 (previously the error was something like "Bad state: future already completed").
fuchsia_tester.dart still assumes Dart 1. Previously, it ran tests directly
from source, flutter_platform.dart automatically runs a kernel compile when
operating in Dart 2 mode, but this assumes a functional Dart SDK is available
in the artifacts directly, and fuchsia_tester.dart mocks out the artifacts
directory with an empty temp dir.
Remaining work is:
1. Get the frontend server building as a dependency on Fuchsia.
2. Patch fuchsia_tester.dart to use a valid Dart SDK and frontend server.
This also reverts migration to Dart 2 typedef syntax.
This reverts commit 6c56bb2. (#18362)
This reverts commit 3daebd0. (#18316)
Now that Dart 1 is turned off, reapplying my change to turn on the prefer_generic_function_type_aliases analysis option, and fix all the typedefs to Dart 2 preferred syntax.
Also eliminated the unused analysis_options_repo.yaml file and turned on public_member_api_docs in analysys_options.yaml.
No logic changes, just changing the typedef syntax for all typedefs, and updating analysis options.
* It's time to #deleteDart1 (#18293)
Eliminates support for Dart 1 in flutter_tools, and drops our Dart 1
benchmarks. All commands now run in Dart 1 mode only.
Eliminates --preview-dart-2 / --no-preview-dart-2 support.
* Fix indentation, remove no longer necessary .toList()
* Only push udpated kernel if >0 invalidated srcs
Eliminates support for Dart 1 in flutter_tools, and drops our Dart 1
benchmarks. All commands now run in Dart 1 mode only.
Eliminates --preview-dart-2 / --no-preview-dart-2 support.
* With dart being default, move preview_dart_2 tests to dart1.
* Swap out defaults for preview-dart-2 option in tests. Make dart1 explicit
* Move suffix around
* Rename files
* Roll engine to b6df7a637498ca9beda1fa9cd7210e3202ea599f.
Changes since last roll:
```
b6df7a637 Roll dart to 290c576264faa096a0b3206c71b2435309d9f904. (#4771)
a6764dbd5 Add sources for Fuchsia target. (#4763)
2d5900615 [fuchsia] Remove unused header file. (#4769)
9717063b7 Revert "Roll dart to c080951d45e79cd25df98036c4be835b284a269c. (#4767)" (#4768)
9a9814312 Roll dart to c080951d45e79cd25df98036c4be835b284a269c. (#4767)
e74e8b35c [async] Update includes of async headers to new path (#4760)
e2c4b2760 Use Dart 2 camel case constants in the engine Dart libraries (#4766)
9c1e48434 Updates for Fuchsia roll. (#4765)
14c940e27 Switch from fxl::Mutex to std::mutex (#4764)
debf82c0b Roll Garnet (#4759)
5bffdefbb Use weak pointers to the accesibility bridge from objects vended to the UIKit accessibility framework. (#4761)
```
* Fix some of the generic types specified to avoid runtime errors in
--preview-dart-2
type 'YamlMap' is not a subtype of type 'Map<String, dynamic>' where
YamlMap is from package:yaml/src/yaml_node.dart
Map is from dart:core
String is from dart:core
package:flutter_devicelab/framework/manifest.dart 17:36 loadTaskManifest
test/manifest_test.dart 14:33 main.<fn>.<fn>
package:test/src/backend/declarer.dart 161:27 Declarer.test.<fn>.<fn>.<fn>
* Add dynamic generic types to keep the analyer happy.
* Address code review comment.
* first round fastlane working on Android
* clean up and start ios
* Partial iOS
* Got a fastfile that works on iOS
* Finalize Android push instructions
* Finalize instructions for iOS
* Extra cleanup and optipng
* Disable readme autogen and add a default readme
This replaces the use of ideviceinfo (whose output was then filtered for
device IDs) with an invocation idevice_id -l, which returns the list
directly, one per line.
This patch migrates iOS device listing from using Xcode instruments to
using the libimobiledevice tools idevice_id and ideviceinfo.
ideviceinfo was previously incompatible with iOS 11 physical devices;
this has now been fixed.
In 58fe8237d2 flutter_tools migrated from
libimobiledevice-based device listing on iOS to using Xcode instruments
to work around the lack of support for iOS 11. Using instruments entails
several downsides, including a significantly higher performance hit, and
leaking hung DTServiceHub processes in certain cases when a simulator is
running, necessitating workarounds in which we watched for, and cleaned
up leaked DTServiceHub processes. This patch returns reverts the move to
instruments now that it's no longer necessary.
This reverts commit 58fe8237d2.
* Change some of the dev/devicelab tests to support testing against local engine.
We can already configure flutter tools to use local engine by setting
FLUTTER_ENGINE environment variable. However when this variable is set
this also requires setting --local-engine to specify which flavor of
engine to use.
This change changes tests in dev/devicelab to pass a sensible default for
--local-engine, e.g. when testing hot reload on Android we pass android_debug
and when testing release AOT build for IOS we pass ios_release.
* Fix analysis issues
* Update utils.dart
This fills out the following matrix:
App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview
It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.
Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
I got tired of drive-by spelling fixes, so I figured I'd just take care of them all at once.
This only corrects errors in the dartdocs, not regular comments, and I skipped any sample code in the dartdocs. It doesn't touch any identifiers in the dartdocs either. No code changes, just comments.
* Restructure hot mode test so it runs interactively.
This allows to add a benchmark for hot reload after actual source code change.
* Add curly braces, refactory copyRecursive
* Add hotReloadInitialDevFSSyncMilliseconds to track how long user have to wait before being able to do first reload.
This stat is significantly different between existing and preview-dart-2 setting (for the latter, this stat is ~3x slower: 13s vs ~4s).
* Remove ws
* Cleanup timer-related code
* Clone hot reload benchmark for --preview-dart-2 option.
* Get rid of linux and win preview_dart_2 (only android would be sufficient for now). Refactor code into lib/tasks
* Revert 2016 to 2017
* Mark new test as flaky
* only tap on widgets reachable by hit testing
* use FractionalOffset
* added tests
* check finder finds correct widget
* undo unintentional changes
* address comments
* style fix
* add Directionality in test
* fix analysis warning
* Revert "Revert "fire service protocol events for frames (#11565)" (#11727)"
This reverts commit f25e2f5213.
* move the postEvent() call into a separate method
The main purpose of this PR is to make it so that when you set the
initial route and it's a hierarchical route (e.g. `/a/b/c`), it
implies multiple pushes, one for each step of the route (so in that
case, `/`, `/a`, `/a/b`, and `/a/b/c`, in that order). If any of those
routes don't exist, it falls back to '/'.
As part of doing that, I:
* Changed the default for MaterialApp.initialRoute to honor the
actual initial route.
* Added a MaterialApp.onUnknownRoute for handling bad routes.
* Added a feature to flutter_driver that allows the host test script
and the device test app to communicate.
* Added a test to make sure `flutter drive --route` works.
(Hopefully that will also prove `flutter run --route` works, though
this isn't testing the `flutter` tool's side of that. My main
concern is over whether the engine side works.)
* Fixed `flutter drive` to output the right target file name.
* Changed how the stocks app represents its data, so that we can
show a page for a stock before we know if it exists.
* Made it possible to show a stock page that doesn't exist. It shows
a progress indicator if we're loading the data, or else shows a
message saying it doesn't exist.
* Changed the pathing structure of routes in stocks to work more
sanely.
* Made search in the stocks app actually work (before it only worked
if we happened to accidentally trigger a rebuild). Added a test.
* Replaced some custom code in the stocks app with a BackButton.
* Added a "color" feature to BackButton to support the stocks use case.
* Spaced out the ErrorWidget text a bit more.
* Added `RouteSettings.copyWith`, which I ended up not using.
* Improved the error messages around routing.
While I was in some files I made a few formatting fixes, fixed some
code health issues, and also removed `flaky: true` from some devicelab
tests that have been stable for a while. Also added some documentation
here and there.
Adds an initial integration test that:
1. captures laid-out widget size
2. focuses a text field to show the keyboard and resize the view
3. unfocuses the text field to dismiss the keyboard and resize the view
4. verify that the final widget size matches the initial size
Eliminates Gallery-specific hardcoding of app package and activity name
in the test.
Fixes breakage introduced with 1dc2d4eb43
due to main activity class now being MainActivity.
This remove a very brittle aspect of flutter drive, whereby it would
assume a known port instead of explicitly finding out what it was.
Fixes#7692 and hopefully fixes the devicelab tests.
- output single-line JSON because iOS 9.x cannot output more than one line
- move the workaround for #7433 into LiveTestWidgetsFlutterBinding so that it works in all cases (e.g. animation_bench.dart on iOS)
