Commit Graph

184 Commits

Author SHA1 Message Date
liyuqian
adc8e159a5
Revert "Shader warm up (#27660)" (#28376)
This reverts commit a44f174efc.

Reason: start_up tests become flaky.

See https://github.com/flutter/flutter/issues/28374

TBR: xster

Merge on red to fix the tree
2019-02-22 17:32:37 -08:00
liyuqian
a44f174efc
Shader warm up (#27660)
This patch adds a default shader warm up process which moves shader compilation from the animation time to the startup time. This also provides an extension for `runApp` so developers can customize the warm up process.

This should reduce our worst_frame_rasterizer_time_millis from ~100ms to ~20-30ms for both flutter_gallery and complex_layout benchmarks. Besides, this should also have a significant improvement on 90th and 99th percentile time (50%-100% speedup in some cases, but I haven't tested them thoroughly; I'll let our device lab collect the data afterwards).

The tradeoff the is the startup time (time to first frame). Our `flutter run --profile --trace-startup` seems to be a little noisy and I see about 100ms-200ms increase in that measurement for complex_layout and flutter_gallery. Note that this only happens on the first run after install or data wipe. Later the Skia persistent cache will remove the overhead.

This also adds a cubic_bezier benchmark to test the custom shader warm up process.

This should fix https://github.com/flutter/flutter/issues/813 (either by `defaultShaderWarmUp`, or a `customShaderWarmUp`).
2019-02-22 15:37:02 -08:00
Dan Field
04b7a5b366
Reduce memory benchmark perf tests outer loop to 10. (#28281)
* reduce iterations to avoid test timing out
2019-02-21 14:46:24 -08:00
liyuqian
de321b9cc1
Add scroll performance test for flutter_gallery (#27513)
Previously, we had a scroll_perf drive test but we were not exercising
it on our device lab. This helps us tracking issues like:
https://github.com/flutter/flutter/issues/24782
2019-02-04 22:54:11 -08:00
Jason Simmons
b5ca595d61
Remove icudtl.dat from APK asset size checks (#27505) 2019-02-04 12:40:43 -08:00
liyuqian
8ad4cbe06e
Add cull opacity perf test to device lab (#25381)
For https://github.com/flutter/flutter/issues/24712

This test verifies that https://github.com/flutter/engine/pull/6923 will speedup the average rasterize time of this test from ~150ms to ~10ms

Please see non-auto-generated files in 37b21d9fb4
2018-12-17 22:01:07 -08:00
Michael Goderbauer
b8c29a1490
Add benchmark for LICENSE file size (#23948) 2018-11-05 13:57:40 -08:00
Greg Spencer
abb1758edf
Fix places in devicelab tests where it expects the app template. (#22696)
I missed some create template conversions in the devicelab directory.
2018-10-04 15:30:08 -07:00
Alexandre Ardhuin
f62afdcf57
add missing type parameter on methods (#22096) 2018-10-01 21:29:08 +02:00
liyuqian
872eba2d4c
Track 90th, 99th percentile instead of missed_frame_build_budget_count (#21754)
See https://github.com/flutter/flutter/pull/19121#issuecomment-419520765
2018-09-14 13:24:13 -07:00
Alexandre Ardhuin
d927c93310
Unnecessary new (#20138)
* enable lint unnecessary_new

* fix tests

* fix tests

* fix tests
2018-09-12 08:29:29 +02:00
Chris Bracken
9f9676c4c6
Eliminate Dart 1 support in devicelab tests (#21240)
Flutter commands all default to --preview-dart-2. Eliminate support for
the use of this option and the previewDart2 parameter in dev/devicelab
tests.
2018-08-30 17:17:37 -07:00
Ian Hickson
1993a67381
Performance test cleanup (#20652)
* Fix TODO syntax.
* Clarify messages for some timeouts, to aid debugging.
* Increase some other timeouts that were a needlessly short, to reduce sources of flakes.
* Remove some more timeouts that were mostly redundant, to remove complexity.
* Minor style cleanup.
* Remove some dangerous traps (specifically, hide the explicit start/end times in TimedEvent since they shouldn't matter).
2018-08-17 13:19:28 -07:00
Ian Hickson
3dec6a6930
Clean up usage of temporary directories (#20682)
All temporary directory start with `flutter_` and have their random component separated from the name by a period, as in `flutter_test_bundle.YFYQMY`.

I've tried to find some of the places where we didn't cleanly delete temporary directories, too. This greatly reduces, though it does not entirely eliminate, the directories we leave behind when running tests, especially `flutter_tools` tests.

While I was at it I standardized on `tempDir` as the variable name for temporary directories, since it was the most common, removing occurrences of `temp` and `tmp`, among others.

Also I factored out some common code that used to catch exceptions that happen on Windows, and made more places use that pattern.
2018-08-17 13:17:23 -07:00
Ian Hickson
872e88cec7
Reduce noise on the memory benchmarks (#19630) (#20163)
- Check memory usage in release builds, not profile.
- Use multiple runs and average the results.
2018-08-06 12:46:51 -07:00
Ian Hickson
a2ca14d71e
Revert "Reduce noise on the memory benchmarks" (#20158)
* Revert "Add pub cache, artifacts, pkgs to Cirrus cache (#20080)"

This reverts commit 07e93b385c.

* Revert "Reduce noise on the memory benchmarks (#19630)"

This reverts commit 8eb5cb7dc0.
2018-08-02 13:56:44 -07:00
Ian Hickson
8eb5cb7dc0
Reduce noise on the memory benchmarks (#19630)
- Check memory usage in release builds, not profile.
- Use multiple runs and average the results.
2018-08-02 12:22:15 -07:00
Alexandre Ardhuin
eda03e2586
re-re-enable lint unnecessary_const (#20103) 2018-08-02 12:02:32 +02:00
liyuqian
a4cc34dee6
Add tiles_scroll_perf test (#19833) 2018-07-27 14:59:38 -07:00
Todd Volkert
00aac68e2d
Revert flutter/flutter#19592 (#19861)
It was causing problems rolling Flutter into Fuchsia
2018-07-27 08:44:39 -07:00
Alexandre Ardhuin
27018359d2
re-enable lint unnecessary_const (#19592)
* re-enable lint unnecessary_const

* remove trailling whitespaces

* remove unnecessary const (after merge)
2018-07-23 08:31:48 +02:00
Chris Bracken
c0294e3a02
Eliminate Dart 1 benchmarks (#19430)
Flutter app development has been Dart 2 only for a significant amount of
time, and we're about to remove all support for Dart 1.
2018-07-18 15:56:51 -07:00
Ian Hickson
f630f90d6d
Revert "enable lint unnecessary_const (#19342)" (#19423)
This reverts commit cc1cf13eec.
2018-07-16 13:30:27 -07:00
Alexandre Ardhuin
cc1cf13eec
enable lint unnecessary_const (#19342) 2018-07-16 21:43:48 +02:00
liyuqian
8c1d8a6a6b
Add 90th and 99th percentiles to dashboard (#18930)
This fixes #18727
2018-06-28 16:19:23 -07:00
Ian Hickson
35ad2a786d
Remove race conditions involving finding available ports (#18698)
This is an attempt to reland #18488 with less breakage on macOS.
2018-06-27 16:44:28 -07:00
Jason Simmons
43e6372128
Update the ICU data file asset path in the devicelab tests (#18854) 2018-06-26 16:01:31 -07:00
Michael Goderbauer
f11b790b8c
Track size of App.framework and Flutter.framework for Hello World (#18701) 2018-06-22 09:27:31 -07:00
Michael Goderbauer
9a8c4dfef6
Report compressed/uncompressed size of icudtl, snapshots, and libflutter (#18583)
Reported for the hello_world compile benchmark in devicelab for now.
2018-06-18 17:12:14 -07:00
Michael Goderbauer
ebb174b0bf
Add size benchmark for hello_world (#18522) 2018-06-16 00:08:33 -07:00
Michael Goderbauer
2a19dae28b
Fix aot_snapshot_size_* benchmarks (#18520)
... and make sure they fail if they can't report the size.
2018-06-15 17:17:28 -07:00
Ian Hickson
0fa5ba43a1
Revert "Remove race conditions involving finding available ports (#18488)" (#18521)
This reverts commit 7750872251.
2018-06-15 15:37:20 -07:00
Ian Hickson
7750872251
Remove race conditions involving finding available ports (#18488) 2018-06-15 14:16:18 -07:00
Chris Bracken
2ae48845a8
Revert elimination of Dart 1 (#18460)
fuchsia_tester.dart still assumes Dart 1. Previously, it ran tests directly
from source, flutter_platform.dart automatically runs a kernel compile when
operating in Dart 2 mode, but this assumes a functional Dart SDK is available
in the artifacts directly, and fuchsia_tester.dart mocks out the artifacts
directory with an empty temp dir.

Remaining work is:
1. Get the frontend server building as a dependency on Fuchsia.
2. Patch fuchsia_tester.dart to use a valid Dart SDK and frontend server.

This also reverts migration to Dart 2 typedef syntax.

This reverts commit 6c56bb2. (#18362)
This reverts commit 3daebd0. (#18316)
2018-06-13 12:46:39 -07:00
Chris Bracken
3daebd059c
It's time to #deleteDart1 (again) (#18316)
* It's time to #deleteDart1 (#18293)

Eliminates support for Dart 1 in flutter_tools, and drops our Dart 1
benchmarks. All commands now run in Dart 1 mode only.

Eliminates --preview-dart-2 / --no-preview-dart-2 support.

* Fix indentation, remove no longer necessary .toList()

* Only push udpated kernel if >0 invalidated srcs
2018-06-08 15:49:24 -07:00
Chris Bracken
9495df4bc3
Revert "It's time to #deleteDart1 (#18293)" (#18313)
Several benchmarks started failing during kernel compile:
1. Timeouts:
   * commands_test
   * hot_mode_dev_cycle__benchmark
   * run_machine_concurrent_hot_reload
   * service_extensions_test

2. Kernel compiler crash:
   * hot_mode_dev_cycle_linux__benchmark
   * routing_test

3. Unexpected termination:
   * hot_mode_dev_cycle_win__benchmark

This reverts commit ed63e7082a.
2018-06-08 12:43:11 -07:00
Chris Bracken
ed63e7082a
It's time to #deleteDart1 (#18293)
Eliminates support for Dart 1 in flutter_tools, and drops our Dart 1
benchmarks. All commands now run in Dart 1 mode only.

Eliminates --preview-dart-2 / --no-preview-dart-2 support.
2018-06-08 11:04:57 -07:00
Alexandre Ardhuin
09276bea25
enable lint prefer_equal_for_default_values (#18156) 2018-06-05 08:50:40 +02:00
Sarah Zakarias
a3e71bf11f
Add startup test for Platform View example (#17645) 2018-05-16 13:24:47 +02:00
Alexander Aprelev
ffc7fac288
Pass --no-preview-dart-2 for dart1 tests. (#15800) 2018-03-21 13:40:17 -07:00
Alexander Aprelev
21ad09fe57
With dart2 being default, move preview_dart_2 tests to dart1. (#15667)
* With dart being default, move preview_dart_2 tests to dart1.

* Swap out defaults for preview-dart-2 option in tests. Make dart1 explicit

* Move suffix around

* Rename files
2018-03-19 12:11:18 -07:00
Alexander Aprelev
2f8474f9aa
Roll engine to b6df7a637498ca9beda1fa9cd7210e3202ea599f. (#15444)
* Roll engine to b6df7a637498ca9beda1fa9cd7210e3202ea599f.

Changes since last roll:
```
b6df7a637 Roll dart to 290c576264faa096a0b3206c71b2435309d9f904. (#4771)
a6764dbd5 Add sources for Fuchsia target. (#4763)
2d5900615 [fuchsia] Remove unused header file. (#4769)
9717063b7 Revert "Roll dart to c080951d45e79cd25df98036c4be835b284a269c. (#4767)" (#4768)
9a9814312 Roll dart to c080951d45e79cd25df98036c4be835b284a269c. (#4767)
e74e8b35c [async] Update includes of async headers to new path (#4760)
e2c4b2760 Use Dart 2 camel case constants in the engine Dart libraries (#4766)
9c1e48434 Updates for Fuchsia roll. (#4765)
14c940e27 Switch from fxl::Mutex to std::mutex (#4764)
debf82c0b Roll Garnet (#4759)
5bffdefbb Use weak pointers to the accesibility bridge from objects vended to the UIKit accessibility framework. (#4761)
```
2018-03-12 15:44:25 -07:00
xster
82ddabd32a
Fastlane instructives for tool publishing the Gallery (#14347)
* first round fastlane working on Android

* clean up and start ios

* Partial iOS

* Got a fastfile that works on iOS

* Finalize Android push instructions

* Finalize instructions for iOS

* Extra cleanup and optipng

* Disable readme autogen and add a default readme
2018-01-31 11:21:37 -08:00
Leaf Petersen
b3cfa785c4
Uncontroversial and backwards compatible 2.0 SDK fixes (#13723)
Small code changes as part of moving the framework SDK forward to a 2.0 dev version.
2017-12-21 13:39:03 -08:00
Yegor
1c6decaa5e
update APK path in devicelab tests (#13540)
* update APK path in devicelab tests

* add missing transitive dependency on archive

* use path used by flutter tools
2017-12-13 13:12:10 -08:00
Vyacheslav Egorov
41a5e5d8f6 Change few devicelab tests to support testing against local engine (#13268)
* Change some of the dev/devicelab tests to support testing against local engine.

We can already configure flutter tools to use local engine by setting
FLUTTER_ENGINE environment variable. However when this variable is set
this also requires setting --local-engine to specify which flavor of
engine to use.

This change changes tests in dev/devicelab to pass a sensible default for
--local-engine, e.g. when testing hot reload on Android we pass android_debug
and when testing release AOT build for IOS we pass ios_release.

* Fix analysis issues

* Update utils.dart
2017-11-30 12:15:27 -08:00
Ian Hickson
1bef312fb1
Actually run "flutter create" before compiling the default app... (#13246) 2017-11-29 15:49:50 -08:00
Ian Hickson
216ff81cd2
Fix devicelab (#13239) 2017-11-28 17:41:40 -08:00
Ian Hickson
d104106e78
Refactor compilation devicelab tests (#13218)
This fills out the following matrix:

App: `flutter create` default app, Complex Layout, and Gallery.
Platforms: Linux/Android, Mac/iOS, Windows/Android
Build modes: full release, aot blobs, debug
Dart: normal, 2.0 preview

It also renames all the tests that used "build" to mean compilation
build to use "compile" instead, so that it's much clearer on the
dashboard.

Also, mark some flaky tests that are no longer flaky as not flaky.
Also, remove uses of call() since that's probably going to go away at
some point.
2017-11-28 17:21:01 -08:00
Michael Goderbauer
2a11e896ee Update clean command in devicelab tests (#12640) 2017-10-19 13:20:32 -07:00
Alexander Aprelev
c5750cd7ea Introduce --preview-dart-2 option to run new frontend compiler in flutter tools. (#11741)
This adds --preview-dart-2 flag that enables use of Dart 2.0 Frontend in Flutter tools.
2017-08-31 13:35:55 -07:00
Michael Goderbauer
4b957bf028 Keep provisioning step (#10736) 2017-06-15 18:23:37 -07:00
yjbanov
b9d5327919 perf_tests.dart: const constructors; constructors first 2017-05-26 09:44:33 -07:00
yjbanov
f59a67707d track debug build times; switch from --profile to --release 2017-05-26 09:44:33 -07:00
Chris Bracken
28feba043f Update Gallery plugin integration (#9823)
Enable plugin infra for both iOS and Android post-#9791.
2017-05-04 19:02:00 -07:00
Chris Bracken
eacf793b35 Use default FlutterActivity in non-plugin examples (#9770)
Brings hello_world and flutter_gallery Android code in line with the
corresponding iOS code.
2017-05-03 14:20:17 -07:00
Chris Bracken
7f9d859e68 Pass package/activity in AndroidBackButtonMemoryTest (#9738)
Eliminates Gallery-specific hardcoding of app package and activity name
in the test.

Fixes breakage introduced with 1dc2d4eb43
due to main activity class now being MainActivity.
2017-05-02 15:44:03 -07:00
xster
1a4bf9b66d Fix flutter view project's automated tests (#9712)
Remove checked in .lock, remove checked in cocoapods scripts in xcode project, let the automated test run pod install specifically for this test
2017-05-01 18:23:06 -07:00
Chris Bracken
ffa8483f3b Update hello_world_memory_test package name (#9704)
Broken by 89af5ec8c7.
2017-05-01 14:18:00 -07:00
Mikkel Nygaard Ravn
70ff50f929 Integration test for channel communication (#9621) 2017-04-27 08:44:28 +02:00
Alexandre Ardhuin
189028a958 Separate the 'if' expression from its statement (#9177) 2017-04-04 21:45:50 +02:00
Sarah Zakarias
d274888be6 Rename platform_services to platform_channel (#9018)
* Rename platform_services to platform_channel

* rename in README

* renamed tests

* update modules.xml
2017-03-27 11:17:31 +02:00
Yegor
bc8cc80b9f Xcode 8-compatible provisioning profiles in devicelab tests (#8987) 2017-03-23 17:35:04 -07:00
Sarah Zakarias
18db5a2e68 Add start-up test for flutter_view sample (#8892)
* Add start-up test for flutter_view sample

* add reportMetrics flag to startup test

* rewording in manifest.yaml
2017-03-23 13:20:00 +01:00
Sarah Zakarias
d1f73fd0ad Add integration test for the Platform Services app (#8826) 2017-03-20 22:19:03 +01:00
Ian Hickson
2bbe82ef30 Increase timeout and verbosity to diagnose devicelab failure (#8832) 2017-03-16 15:02:43 -07:00
Chris Bracken
6c97dd2c6c Declare locals as final where not reassigned (dev) (#8574) 2017-03-03 18:06:08 -08:00
Alexandre Ardhuin
e9a775bf38 prefer const constructors (#8316) 2017-02-21 14:54:29 -08:00
Ian Hickson
21434fcf1a Refactor 'flutter drive' to get the observatory port from the logs (#7695)
This remove a very brittle aspect of flutter drive, whereby it would
assume a known port instead of explicitly finding out what it was.

Fixes #7692 and hopefully fixes the devicelab tests.
2017-01-27 01:37:28 -08:00
Ian Hickson
a4a1392c44 --debug-port is now --observatory-port (#7680) 2017-01-26 18:01:34 -08:00
Ian Hickson
aff4e82891 Technical Debt tracker (#7667) 2017-01-26 17:04:37 -08:00
Yegor
4bec9cd940 update build benchmark to match #7589 (#7600) 2017-01-23 17:49:01 -08:00
Jason Simmons
584dbe3d32 Fix the activity name in the Gallery back button memory metric (#7527) 2017-01-18 14:14:40 -08:00
Yegor
cf61905c04 publish GPU metrics to dashboard; compact transition event printout (#7254) 2016-12-13 15:22:23 -08:00
Jason Simmons
2fd786b567 A devicelab memory test for Android app suspend and resume (#7164) 2016-12-06 10:13:21 -08:00
Jason Simmons
7da3ab634a Devicelab memory usage test that navigates repeatedly to a screen within Gallery (#7137) 2016-12-05 11:08:25 -08:00
Jason Simmons
2563636d90 A devicelab test of memory usage for a minimal app (#7126) 2016-12-02 10:27:43 -08:00
Yegor
45a4f639c9 devicelab: make specifying device OS less confusing (#7132) 2016-12-01 18:35:40 -08:00
Jason Simmons
4da4ca89aa Add a devicelab test that captures memory usage metrics on Android (#7120) 2016-12-01 11:37:16 -08:00
Yegor
80627a5ccd unlock device before running perf test (#6540) 2016-10-25 18:05:53 -07:00
Yegor
33c6526462 fix frame info extraction logic; track frames in gallery test (#6505) 2016-10-25 11:02:18 -07:00
Yegor
8c8be6b5df remove engineEnterTimestampMicros from benchmark list; not useful (#6242) 2016-10-06 22:31:32 -07:00
Yegor
6cdab85d28 Sync test code with Cocoon (#6129)
* fix flutter watch benchmark

Syncs d2d7950ecd

* Split Android/iOS impl behind a unified interface

Syncs db87e10fa5

* Switch from pub get to flutter packages get

Syncs b378005cbb

* "silent" option in test runner; fix analysis errors;
2016-09-29 10:19:15 -07:00
Yegor
1ba1562293 move CI tests into the main repo (#5758) 2016-09-14 13:22:53 -07:00