Commit Graph

267 Commits

Author SHA1 Message Date
Todd Volkert
0f625196cc
Add plumbing for hello world startup test in devicelab (#36410)
This is the first step in getting esoteric devices wired
up to device lab to ensure that we can at least start Flutter
apps in profile mode on such devices.

https://github.com/flutter/flutter/issues/35838
2019-07-18 11:13:14 -07:00
xster
03220cacd1
Make sure add-to-app build bundle from outer xcodebuild/gradlew sends analytics (#36122) 2019-07-17 11:17:04 -07:00
xster
2d08fec368
Move buildable module test to a module test (#36102) 2019-07-15 18:31:44 -07:00
Jenn Magder
b5c1b61c73
Add platform_interaction_test_swift to devicelab (#35775) 2019-07-10 14:13:01 -07:00
Jonah Williams
91e3b2d671
add iOS build benchmarks (#35749) 2019-07-08 18:09:00 -07:00
liyuqian
34e18d38ce
ios (iPhone6) and iPhone XS tiles_scroll_perf tests (#35556)
For https://github.com/flutter/flutter/issues/31086.

Also test if our mac8 with iPhone XS is working
(https://github.com/flutter/flutter/issues/34590).
2019-07-08 14:13:33 -07:00
Alexander Aprelev
9f285aa9d9
Introduce image_list performance benchmark that runs on jit(debug) build. (#35424)
* Introduce image_list performance benchmark that runs on jit(debug) build.

We want to track performance of debug build because ongoing work on reducing the cost of spanwing new isolate will impact jit(debug) more than it will aot(release) configurations.

* Newlines and spaces
2019-07-08 08:23:06 -07:00
Jonah Williams
05b4c67550
add APK build time benchmarks (#35481) 2019-07-03 10:54:04 -07:00
Alexander Aprelev
3badcf51a4
Introduce image loading performance test. (#34823)
* Introduce image loading performance test.

This example reports how long it takes to load images over the network(locally spawned server).
2019-06-28 15:14:03 -07:00
Alexandre Ardhuin
919dcf53f3
enable lints prefer_spread_collections and prefer_inlined_adds (#35189) 2019-06-27 21:23:16 +02:00
Josh Burton
577c2fc48c Ensures flutter jar is added to all build types on plugin projects (#34573) 2019-06-25 10:40:55 -07:00
Jim Graham
f83fd9d467
Add test case for Flutter Issue #27677 as a benchmark. (#34870)
* Add test case for Flutter Issue #27677 as a benchmark.

See https://github.com/flutter/flutter/issues/27677

I got the following results running the test on a Moto E2 which
will help us determine how much we can gain by analyzing the
operations and eliminating unnecessary repaints based on
dirty rectangles/regions.

no blurs       - avg 216.0 fps over 3 samples
blur the group - avg  22.9 fps over 3 samples
blur each txt  - avg   3.4 fps over 3 samples

* Added the new benchmark to be tracked on the dashboard.
2019-06-24 20:53:22 -07:00
Emmanuel Garcia
c50fa3d2bd Re-land config lib dependencies for flavors (#34668) 2019-06-18 22:37:42 -07:00
Emmanuel Garcia
9dbc66979a
Revert "Config lib dependencies when using flavors (#34592)" (#34655)
This reverts commit 358b9bd381.
2019-06-18 15:30:59 -07:00
Emmanuel Garcia
358b9bd381
Config lib dependencies when using flavors (#34592) 2019-06-17 15:05:16 -07:00
Devon Carew
5f521eea22
add route information to Flutter.Navigation events (#34508)
* add route information to Flutter.Navigation events

* route can be null; re-work how we send route info to accomidate that
2019-06-15 20:40:34 -07:00
Dan Field
3d58400e76
Revert "Revert "redux of a change to use new engine APIs for Flutter.Frame events (#34365)" (#34514)" (#34530)
This reverts commit 8f75d53735.
2019-06-15 00:01:35 -07:00
Keerti Parthasarathy
8f75d53735
Revert "redux of a change to use new engine APIs for Flutter.Frame events (#34365)" (#34514)
This reverts commit ceb09b8bad.
2019-06-14 13:08:14 -07:00
Devon Carew
ceb09b8bad
redux of a change to use new engine APIs for Flutter.Frame events (#34365) 2019-06-14 11:16:54 -07:00
Emmanuel Garcia
21a532698f
Split gradle_plugin_test.dart (#34282)
Fixes timeout when running gradle_plugin_test
2019-06-11 16:24:03 -07:00
Emmanuel Garcia
8627ff433b
Allow multi-abi shared libraries in APKs and App bundles (#34123)
* Gradle generates ELF shared libraries instead of AOT snapshots.
* `flutter build apk/appbundle` supports multiple `--target-platform` and defaults to `android-arm` and `android-arm64`.
* `flutter build apk` now has a flag called `--split-per-abi`.
2019-06-10 13:16:09 -07:00
Jonah Williams
f8f7b89976
Revert "Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles (#33696)" (#34121) 2019-06-08 18:43:25 -07:00
Emmanuel Garcia
966b15b4a4
Generate ELF shared libraries and allow multi-abi libs in APKs and App bundles (#33696)
* Gradle generates ELF shared libraries instead of AOT snapshots.
* `flutter build apk/appbundle` supports multiple `--target-platform` and defaults to `android-arm` and `android-arm64`.
*  `flutter build apk` now has a flag called `--split-per-abi`.
2019-06-08 18:02:32 -07:00
Jonah Williams
50ea6f32d3
add benchmarks to track web size (#33892) 2019-06-05 12:40:21 -07:00
Kaushik Iska
92bfc99132
Added a benchmark for ImageCache (#33814)
* Add an image cache benchmark for a monochrome image
2019-06-04 12:32:52 -07:00
Jonah Williams
83986acbdb
Add a real-er web restart, doctor, workflow (#33786) 2019-06-03 23:19:42 -07:00
Chris Bracken
98ea501995
Revert "Add real-er restart for web using webkit inspection protocol (#33629)" (#33703)
Revert "fix devicelab manfiest (#33698)"

This reverts commit 5a6a00dc54.
This reverts commit 0d79f0fc79.
2019-05-31 21:53:30 -07:00
Jonah Williams
5a6a00dc54
Add real-er restart for web using webkit inspection protocol (#33629) 2019-05-31 17:37:01 -07:00
Chris Bracken
4d9923201b
Correct typos (#33322)
Corects a bnuch of typeos throuhgout teh Fluter codebsae.

Made use of the `misspell` tool:
https://github.com/client9/misspell
2019-05-24 19:12:45 -07:00
Michael Thomsen
7ae3caf309
Rename flutter packages to flutter pub (#33041) 2019-05-21 16:38:58 +02:00
Emmanuel Garcia
90f38907d7
Support ARM 32 and 64 bits in app bundles 2019-05-20 13:09:20 -07:00
Ben Konyi
484f6e754a
Fixed flutter_attach_test not respecting authentication codes (#31321) 2019-04-19 13:55:34 -07:00
Ben Konyi
1459b1e91f
Fixed failing tests caused by introduction of authentication codes (#31315) 2019-04-19 11:45:53 -07:00
Ben Konyi
3764cb8515
Added support for authentication codes for the VM service. (#30857)
* Added support for authentication codes for the VM service.

Previously, a valid web socket connection would use the following URI:

`ws://127.0.0.1/ws`

Now, by default, the VM service requires a connection to be made with a
URI similar to the following:

`ws://127.0.0.1:8181/Ug_U0QVsqFs=/ws`

where `Ug_U0QVsqFs` is an authentication code generated and shared by
the
service.

This behavior can be disabled with the `--disable-service-auth-codes`
flag.
2019-04-18 21:01:50 -07:00
liyuqian
b0937a1e89
Fix the warning test by checking stderr (#30997)
Previously, I used the Android emulator for testing and everything
seemed to work fine with stdout (if I remember correctly). But our
devicelab uses real Android devices and the warnings are routed to
stderr. Hence change stdout to stderr in the test.
2019-04-15 10:19:36 -07:00
liyuqian
eb30745faf
Print warning if flutter drive is run in debug (#30747)
## Description

Print actionable warnings if `flutter drive` (where most of our performance benchmarks come from) is run in debug mode and it tries to gather benchmarks using `traceAction`.

## Related Issues

https://github.com/flutter/flutter/issues/30625

## Tests

I added the following tests:
* drive_perf_debug_warning devicelab test
2019-04-09 18:55:42 -07:00
Jonah Williams
953a947130
add coverage benchmark (#30222) 2019-03-29 17:00:19 -07:00
Jonah Williams
1bfa2f2311
Intercept errors thrown by synchronous Completers in image resolution. (#30139) 2019-03-29 10:05:18 -07:00
Dan Field
ecfdd7e1ea
Detect and cleanup leaky processes (#29196)
* Detect and cleanup leaky processes
* Add flaky tests for detecting leaked processes
2019-03-22 14:32:36 -07:00
Dan Field
2a644f301e
Run non-perf sensitive tests on Cirrus (#27971)
* Run non-perf sensisitive tests on Cirrus
2019-03-10 07:52:44 -07:00
Jonah Williams
d4a441a0d9
Add integration to all targets (#29020) 2019-03-08 12:25:35 -08:00
Alexandre Ardhuin
387f885481
Add missing trailing commas (#28673)
* add trailing commas on list/map/parameters

* add trailing commas on Invocation with nb of arg>1

* add commas for widget containing widgets

* add trailing commas if instantiation contains trailing comma

* revert bad change
2019-03-01 08:17:55 +01:00
liyuqian
19a6a6f431
Reland "Shader warm up (#27660)" (#28537)
This reverts commit adc8e159a5.

This should be safe to land once https://github.com/flutter/flutter/pull/28530 gets merged

Merge on yellow doc test because the doc test is actually green.
2019-02-27 09:33:08 -08:00
liyuqian
adc8e159a5
Revert "Shader warm up (#27660)" (#28376)
This reverts commit a44f174efc.

Reason: start_up tests become flaky.

See https://github.com/flutter/flutter/issues/28374

TBR: xster

Merge on red to fix the tree
2019-02-22 17:32:37 -08:00
liyuqian
a44f174efc
Shader warm up (#27660)
This patch adds a default shader warm up process which moves shader compilation from the animation time to the startup time. This also provides an extension for `runApp` so developers can customize the warm up process.

This should reduce our worst_frame_rasterizer_time_millis from ~100ms to ~20-30ms for both flutter_gallery and complex_layout benchmarks. Besides, this should also have a significant improvement on 90th and 99th percentile time (50%-100% speedup in some cases, but I haven't tested them thoroughly; I'll let our device lab collect the data afterwards).

The tradeoff the is the startup time (time to first frame). Our `flutter run --profile --trace-startup` seems to be a little noisy and I see about 100ms-200ms increase in that measurement for complex_layout and flutter_gallery. Note that this only happens on the first run after install or data wipe. Later the Skia persistent cache will remove the overhead.

This also adds a cubic_bezier benchmark to test the custom shader warm up process.

This should fix https://github.com/flutter/flutter/issues/813 (either by `defaultShaderWarmUp`, or a `customShaderWarmUp`).
2019-02-22 15:37:02 -08:00
Jonah Williams
67cf21577f
Add basic codegen app to be used for integration testing and benchmarks (#27257) 2019-02-14 23:17:16 -08:00
Greg Spencer
da27f62337
Make a kReleaseMode constant that is public. (#27502)
Before this, we had several places where an isReleaseMode was defined, all with the same definition. This just makes it more broadly visible to allow our users to use it, as well as creating debug and profile versions, and adding a device lab test for it.

Since this is a const value, this makes it possible for a developer to easily mark blocks that can be removed at AOT compile time.
2019-02-06 14:37:17 -08:00
liyuqian
de321b9cc1
Add scroll performance test for flutter_gallery (#27513)
Previously, we had a scroll_perf drive test but we were not exercising
it on our device lab. This helps us tracking issues like:
https://github.com/flutter/flutter/issues/24782
2019-02-04 22:54:11 -08:00
Todd Volkert
db7e4fbbc4 Fix test expectations per change made in #26736 (#26809) 2019-01-19 11:32:48 -08:00
Ian Hickson
31a9626c48
[O] Removing all timeouts (mark II) (#26736)
These are essentially self-inflicted race conditions. Instead of timeouts we're going to try a more verbose logging mechanism that points out when things are taking a long time.
2019-01-19 00:31:05 -08:00
Ian Hickson
78f4878fe1
Cleanup device lab test and remove a timeout. (#24581) 2019-01-13 20:44:59 -08:00
Ian Hickson
8426910a19
Revert "[O] Remove many timeouts. (#23531)" (#25646)
This reverts commit 76f70810e4.
2018-12-20 18:46:36 -08:00
Ian Hickson
76f70810e4
[O] Remove many timeouts. (#23531)
* Remove many timeouts.

These are essentially self-inflicted race conditions. Instead of timeouts we're going to try a more verbose logging mechanism that points out when things are taking a long time.

* Get the attach tests to pass.

* Apply review comments from Todd

* More review comment fixes

* Put back the extended timeouts here now that I know why we have them...
2018-12-20 17:10:40 -08:00
liyuqian
8ad4cbe06e
Add cull opacity perf test to device lab (#25381)
For https://github.com/flutter/flutter/issues/24712

This test verifies that https://github.com/flutter/engine/pull/6923 will speedup the average rasterize time of this test from ~150ms to ~10ms

Please see non-auto-generated files in 37b21d9fb4
2018-12-17 22:01:07 -08:00
Jonah Williams
d01f574a4a
Revert "Use stderr instead of stdout to contain errors in flutter attach test" (#25380) 2018-12-14 10:15:31 -08:00
David Shuckerow
2b819dd257
Use stderr instead of stdout to contain errors in flutter attach test (#25305) 2018-12-13 17:50:41 -04:00
Jonah Williams
f30029bae1
Run flutter tests through mini test engine when run directly (flutter run -t test_file) (#24930) 2018-12-12 12:51:16 -08:00
Ian Hickson
bcac1bd5e2
Handle the case where there are no missing dartdocs too... (#24801) 2018-11-27 14:44:53 -08:00
Ian Hickson
e0f642ecbb
Update dartdocs devicelab test to handle current analyzer output (#24582) 2018-11-27 11:29:32 -08:00
Dan Field
0155ee7157
fix devicelab (#24404) 2018-11-15 17:55:01 -08:00
Dan Field
091b73ab6f
fix variable check in xcode_backend.sh (#24368)
* fix test in xcode_backend.sh

* -n

* fix devicelab test

* uncomment good code, check for \"\"
2018-11-14 18:54:51 -08:00
Dan Field
2729385758
Make host app test actually test host app (#24131)
* Make host app test actually test host app

* mark updated test as flaky
2018-11-09 19:09:56 -08:00
Jonah Williams
d52f315c35
Add new benchmark to track cost of dependencies to a User (#23856) 2018-11-02 13:38:54 -07:00
Devon Carew
e30f9a9447
add explicit const keywords (#23652) 2018-10-29 22:39:40 -07:00
Michael Klimushyn
ce434022b5
Fix "E2E test setting and using isolate names (#23388)" (#23438)
Updates gradle.

This reverts commit d07c09c9a1.
2018-10-24 11:44:45 -07:00
Michael Klimushyn
e70f19523b
Revert "E2E test setting and using isolate names (#23388)" (#23416)
This reverts commit 93573de216.

Fails in the devicelab:

```
run:stderr: A problem occurred evaluating project ':app'.
run:stderr: > Could not resolve all files for configuration 'classpath'.2018-10-23T10:46:46.335864:
run:stderr:    > Could not find aapt2-proto.jar (com.android.tools.build:aapt2-proto:0.3.1).2018-10-23T10:46:46.335960:
run:stderr:      Searched in the following locations:2018-10-23T10:46:46.336048:
```
2018-10-23 11:28:12 -07:00
Michael Klimushyn
93573de216
E2E test setting and using isolate names (#23388)
Adds an integration devicelab test that runs an Android app with two
custom named isolates. Tests that the isolate names are present and that
it's possible to attach to just one of the isolates.

Fixes flutter/flutter#22009
2018-10-23 09:30:00 -07:00
Michael Klimushyn
43c99404c5
Test flutter create --offline (#23278)
Verifies that flutter tools isn't rebuilt. Doesn't truly check that the
created output is runnable.
2018-10-22 16:39:28 -07:00
Jacob Richman
e4c2c978a6
Dispatch a Flutter.Navigation event each time navigation occurs. (#23126)
Dispatch a Flutter.Navigation event each time navigation occurs.
2018-10-17 07:40:08 -07:00
Greg Spencer
0ff9e8a928
Rename 'application' back to 'module', and make 'app' the default again for templates. (#22888)
We decided that redefining the default for templates was premature. We're going to go back to having "module" in experimental land again, and we'll try again when we have the feature set fully baked.

This keeps the writing of the .metadata files, and writing the template type to them, because that was a good improvement, and there are still a bunch of added tests that improve our coverage.
2018-10-10 11:01:40 -07:00
Greg Spencer
abb1758edf
Fix places in devicelab tests where it expects the app template. (#22696)
I missed some create template conversions in the devicelab directory.
2018-10-04 15:30:08 -07:00
Alexandre Ardhuin
d340e2f229
apply lint prefer_void_to_null in dev/ (#22661) 2018-10-04 18:44:23 +02:00
Alexandre Ardhuin
2ea1d81cdc
sort_constructors_first (#22575) 2018-10-04 07:28:07 +02:00
Alexandre Ardhuin
f62afdcf57
add missing type parameter on methods (#22096) 2018-10-01 21:29:08 +02:00
Dan Field
8081336966
Fix red tree (#22432)
* fix race in devicelab concurrent hot reload

* only complete future once!
2018-09-28 09:43:42 -07:00
Dan Field
2c61067387
fix race in devicelab concurrent hot reload (#22403) 2018-09-28 09:01:39 -07:00
Mikkel Nygaard Ravn
a600fe7f13 Support materializing Flutter module host app on iOS (#21276)
* Prototype

* Fix paths to Flutter library resources

* Invoke pod install as necessary for materialized modules

* Add devicelab test for module use on iOS

* Remove debug output

* Rebase, reame materialize editable

* Add devicelab test editable iOS host app

* Removed add2app test section
2018-09-25 15:21:13 -04:00
Greg Spencer
eed8c7ad07
Fix the run_release_test devicelab test after changing output. (#22104) 2018-09-20 16:30:07 -07:00
Alexander Aprelev
2fec45b17d
Amend the test following console output changes. (#22093)
This is the follow-up to https://github.com/flutter/flutter/pull/20958.
2018-09-20 12:12:53 -07:00
Danny Tuppeny
4108240a7c
Update devicelab test for rename of materialize -> make-host-app-editable (#22034)
This fixes the devicelab failure after #22006 renamed this command.
2018-09-19 16:12:01 +01:00
Alexandre Ardhuin
d927c93310
Unnecessary new (#20138)
* enable lint unnecessary_new

* fix tests

* fix tests

* fix tests
2018-09-12 08:29:29 +02:00
Alexandre Ardhuin
774ca2f197
enable lint avoid_void_async (#21652) 2018-09-11 07:14:04 +02:00
Jonah Williams
c6e8a513f0
Add android semantics integration test to device lab (#20971) 2018-09-06 12:31:18 -07:00
Danny Tuppeny
7fc9165e4e
Add a detach command to detach without terminating (#21490)
* Add a detach command to detach without terminating (#21376)

* Add a detach command to detach without terminating

Fixes #21154.

* Bump protocol version for app.detach

* Tweak to detach/quit text

* Change logPrefix to named param

* Fix the text that the devicelab attach test looks for
2018-09-06 15:26:55 +01:00
Chris Bracken
9f9676c4c6
Eliminate Dart 1 support in devicelab tests (#21240)
Flutter commands all default to --preview-dart-2. Eliminate support for
the use of this option and the previewDart2 parameter in dev/devicelab
tests.
2018-08-30 17:17:37 -07:00
amirh
1f31c3b354
Run the embedded Android views integration test on the device lab. (#20841) 2018-08-23 10:26:00 -07:00
Ian Hickson
22f376418f
Track number of package dependencies in Flutter (#20831)
* Track number of package dependencies in Flutter

Relands #20774.

* Use evalFlutter instead of startProcess.

That way we don't need `flutter` on the PATH.
2018-08-21 14:02:33 -07:00
Ian Hickson
a56a5c598d
Add ignore_for_file to the technical debt menu (#20755)
If //ignore is bad, ignoring something for a whole file is presumably worse.
2018-08-20 17:59:11 -07:00
Ian Hickson
809721d0ee
Revert "Track number of package dependencies in Flutter" (#20774)
* Revert "Deprecated CupertinoDialog in favor of a new widget called CupertinoP… (#20649)"

This reverts commit 3a4ae280e3.

* Revert "Feature pesto new recipes (#19415)"

This reverts commit e2f3b3d6f2.

* Revert "Use markdown table (#20721)"

This reverts commit 82d43b952a.

* Revert "Documentation regarding tap gesture callbacks (#20647)"

This reverts commit 3acc278521.

* Revert "Add branch to footer information in flutter docs (#20711)"

This reverts commit 9118d3d715.

* Revert "Performance test cleanup (#20652)"

This reverts commit 1993a67381.

* Revert "Track number of package dependencies in Flutter (#20722)"

This reverts commit 03d6f18f4a.
2018-08-17 18:34:05 -07:00
Ian Hickson
03d6f18f4a
Track number of package dependencies in Flutter (#20722) 2018-08-17 13:19:06 -07:00
Ian Hickson
3dec6a6930
Clean up usage of temporary directories (#20682)
All temporary directory start with `flutter_` and have their random component separated from the name by a period, as in `flutter_test_bundle.YFYQMY`.

I've tried to find some of the places where we didn't cleanly delete temporary directories, too. This greatly reduces, though it does not entirely eliminate, the directories we leave behind when running tests, especially `flutter_tools` tests.

While I was at it I standardized on `tempDir` as the variable name for temporary directories, since it was the most common, removing occurrences of `temp` and `tmp`, among others.

Also I factored out some common code that used to catch exceptions that happen on Windows, and made more places use that pattern.
2018-08-17 13:17:23 -07:00
Mikkel Nygaard Ravn
d4e5e1e11e
Materialize Flutter module, Android (#20520) 2018-08-16 13:21:55 +02:00
Ian Hickson
686d8f8a22 Shim package:test to avoid matcher issues (#20602)
* Upgrade everything except matcher.
* Roll matcher (and test)
* Adjust tests that depend on flutter:test directly to depend on a shim
* Require use of package:test shim and remove other references to package:test
2018-08-14 20:33:58 -07:00
Mikkel Nygaard Ravn
e69b434602
Fix broken Flutter module with plugins (#20496) 2018-08-13 10:35:59 +02:00
Ian Hickson
93b0265199
Fix flutter_gallery_instrumentation_test (#20361)
Somehow I forgot to say "super.tap()" when calling "tap()" on the new
superclass, so it was just recursing infinitely but ended up actually
crashing on the first reuse of the finder.

The error was previously swallowed, I made this print it instead.
2018-08-09 16:57:41 -07:00
Ian Hickson
872e88cec7
Reduce noise on the memory benchmarks (#19630) (#20163)
- Check memory usage in release builds, not profile.
- Use multiple runs and average the results.
2018-08-06 12:46:51 -07:00
Mikkel Nygaard Ravn
d1f446e559
Support flutter run/build of module on Android (#20197) 2018-08-04 13:52:09 +02:00
liyuqian
b6cd38d8df
Add tiles_scroll_perf__timeline_summary to devicelab (#20105) 2018-08-02 16:47:58 -07:00
Ian Hickson
a2ca14d71e
Revert "Reduce noise on the memory benchmarks" (#20158)
* Revert "Add pub cache, artifacts, pkgs to Cirrus cache (#20080)"

This reverts commit 07e93b385c.

* Revert "Reduce noise on the memory benchmarks (#19630)"

This reverts commit 8eb5cb7dc0.
2018-08-02 13:56:44 -07:00
Ian Hickson
8eb5cb7dc0
Reduce noise on the memory benchmarks (#19630)
- Check memory usage in release builds, not profile.
- Use multiple runs and average the results.
2018-08-02 12:22:15 -07:00