Commit Graph

464 Commits

Author SHA1 Message Date
Ian Hickson
5e216d44ca
Simplify devicelab logic and fix tests (#139122)
- fix https://github.com/flutter/flutter/issues/53707 by having the test not expect a timeout but instead actually look for the retry message
- simplify the `--task` option to only accept task names rather than also accepting paths
- remove some obsolete options that referred to the manifest which no longer seems to exist
2023-11-29 08:41:27 +00:00
Victoria Ashworth
ddc9b40a98
Fix typo in function name (#136273)
Rename `removeIOSimulator` to `removeIOSSimulator`.
2023-10-12 15:53:00 +00:00
Christopher Fujino
8ca7d7d136
Stop recommending android sdk root (#136296)
Fixes https://github.com/flutter/flutter/issues/117245
2023-10-11 21:04:59 +00:00
Ricardo Amador
4e5e47e76a
Add device ready check (#135526)
*Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.*

*List which issues are fixed by this PR. You must list at least one issue.*
Fixes https://github.com/flutter/flutter/issues/121420

*If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].*
2023-09-30 01:58:52 +00:00
Michael Goderbauer
b0a90aee17
Enable strict-inference (#135043)
Avoids that dynamic accidentally sneaks in, see https://dart.dev/tools/analysis#enabling-additional-type-checks
2023-09-20 19:59:08 +00:00
Andrew Kolos
445e02dd63
fix --exit flag in dev/devicelab/bin/run.dart (#134162)
Fixes #134154 

This PR also changes the default value of the `--exit` flag from `true` to `false`. Effectively, this is not a change in behavior since `--exit` didn't previously work.
2023-09-07 19:36:55 +00:00
Victoria Ashworth
c175cf87a6
Ignore macOS Cocoapods linting failure on DT_TOOLCHAIN_DIR error (#133588)
Xcode 15 introduced an [error](https://github.com/flutter/flutter/issues/132755) into Cocoapods when building macOS apps. 

When `pod lib lint` runs, it under the covers is building the app with `xcodebuild`, which is why this error occurs when linting.

A fix has been made in Cocoapods, but is not in an official release so we can't upgrade Cocoapods yet. This is to temporarily ignore lint failure due to that error.

Fixes https://github.com/flutter/flutter/issues/132980.

Tracking issue to upgrade Cocoapods when fix is in a release: https://github.com/flutter/flutter/issues/133584

Since Xcode 15 isn't in CI, I tested it in a one-off led test:
* [Pre-fix failure](https://chromium-swarm.appspot.com/task?id=6431f228ecf98e10)
* [Post-fix success](https://chromium-swarm.appspot.com/task?id=645ba7ebdab97210)
2023-08-31 14:57:07 +00:00
Jackson Gardner
d8b1e81c1f
Add --experimental-wasm-type-reflection and support newer emscripten builds. (#133084)
This makes two changes to prepare for incoming changes to skwasm in the web engine:
* We will (at least for now) be depending on the `WebAssembly.Function` constructor in `skwasm`, which is hidden behind the `--experimental-wasm-type-reflection` flag. We need to pass that when running skwasm benchmarks.
* We are going to be upgrading the skwasm build to a newer version of emscripten, which exposes the wasm exports via the `wasmExports` property instead of the `asm` property. Make sure to support either, if passed.
2023-08-23 17:07:41 +00:00
Matan Lurey
a7997f606e
Update dev/devicelab/** to provide --local-engine-host. (#132342)
Partial work towards https://github.com/flutter/flutter/issues/132245.

I have to admit I don't totally understand what I've updated, or whether
there are more integration points needed.
2023-08-10 17:25:30 -07:00
Jackson Gardner
1b887c72b5
Skwasm benchmarks. (#129681)
This enables benchmarks for the Skwasm renderer, compiled with
dart2wasm.

Platform views aren't supported in Skwasm yet, so we are skipping those
benchmarks for now.
2023-06-28 17:16:19 -07:00
Victoria Ashworth
cd18c8c02f
Workaround for Dart VM timeout (#127875)
Workaround solution for: https://github.com/flutter/flutter/issues/121231
See https://github.com/flutter/flutter/issues/120808#issuecomment-1551826299 Error Case 2 for more information.

Sometimes the `ios-deploy` process does not return the logs from the application. We've been unable to figure out why. This is a solution to workaround that by using `idevicesyslog` alongside `ios-deploy` as a backup in getting the log for the Dart VM url. As explained in https://github.com/flutter/flutter/issues/120808#issuecomment-1551826299, when error case 2 happens, the `idevicesyslog` does successfully find the Dart VM.

Also, in the comments of the code it mentions `syslog` is not written on iOS 13+, this was added in response to this issue: https://github.com/flutter/flutter/issues/41133.

However, `idevicesyslog` does in fact work (at least for iOS 16), we use it to collect device logs for our CI tests already: 1dc26f80f0/dev/devicelab/lib/framework/devices.dart (L998-L1006)
2023-06-02 17:17:57 +00:00
Phil Quitslund
5bf6318688
Update collection-fors to prefer final (as per updated prefer_final_in_for_each) (#127511)
The newly updated lint will soon flag for-each in collections.

See discussion: https://github.com/dart-lang/linter/pull/4383

/cc @goderbauer
2023-05-26 23:34:36 +00:00
chunhtai
c687dcd56f
Migrates android semanitcs integration to integration test (#127128)
I think the flake is due to setclipboard or semantics update race condition. I migrated the test to use integration test package which relies less on timing

fixes https://github.com/flutter/flutter/issues/124636
2023-05-23 22:33:19 +00:00
Tomasz Gucio
99c7e9f088
Add spaces after flow control statements (#126320) 2023-05-15 11:07:30 +02:00
Harry Terkelsen
daaba8a4aa
Add --local-web-sdk in devicelab runner to make --ab testing work for web (#123825)
This allows us to check for performance differences in local Web SDKs.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#overview
[Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene
[test-exempt]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes
[Discord]: https://github.com/flutter/flutter/wiki/Chat
2023-03-31 14:25:00 -07:00
Michael Goderbauer
fda9ecfef7
Remove 1745 decorative breaks (#123259)
Remove 1745 decorative breaks
2023-03-22 21:12:22 +00:00
yusuf-goog
47dd992266
Adding timestamps to task runner output lines. (#123075)
Adding timestamps to task runner output lines.
2023-03-21 23:25:35 +00:00
Jenn Magder
e2e313ecf6
Take screenshot on devicelab failure (#122249)
Take screenshot on devicelab failure
2023-03-15 02:13:21 +00:00
Jonah Williams
e98c53751e
Revert "[Impeller] Temporary flag flip for devicelab tests to use Impeller redux. (#122340)" (#122362)
Revert "[Impeller] Temporary flag flip for devicelab tests to use Impeller redux."
2023-03-10 16:16:30 +00:00
Jonah Williams
de399511d6
[Impeller] Temporary flag flip for devicelab tests to use Impeller redux. (#122340) 2023-03-09 17:45:17 -08:00
Jonah Williams
4adb299566
Revert "[Impeller] Temporary flag flip for devicelab tests to use Impeller. (#122224)" (#122236)
This reverts commit 21b8b7297f.
2023-03-08 14:04:25 -08:00
Jonah Williams
21b8b7297f
[Impeller] Temporary flag flip for devicelab tests to use Impeller. (#122224)
[Impeller] Temporary flag flip for devicelab tests to use Impeller.
2023-03-08 21:41:51 +00:00
Jenn Magder
a16e620ec2
Funnel devicelab tests through utils process methods (#122161)
Funnel devicelab tests through utils process methods
2023-03-08 19:44:40 +00:00
Ben Konyi
ecd7518df5
Reland "Remove references to Observatory (#118577)" (#121606)
This reverts commit 275ab9c69b.
2023-02-28 11:57:04 -05:00
Michael Goderbauer
275ab9c69b
Revert "Reland "Remove references to Observatory (#118577)" (#121215)" (#121555)
Revert "Reland "Remove references to Observatory (#118577)""
2023-02-27 23:46:53 +00:00
Ben Konyi
fbae472fc3
Reland "Remove references to Observatory (#118577)" (#121215)
This reverts commit 298d8c76ba.
2023-02-27 09:26:43 -05:00
Casey Hillers
298d8c76ba
Revert "Remove references to Observatory (#118577)" (#120929)
This reverts commit 2df140f40d.
2023-02-16 21:28:30 -08:00
Ben Konyi
2df140f40d
Remove references to Observatory (#118577)
Observatory is being deprecated for Dart 3.0 so it should no longer be
referenced in tooling messaging / flags.

See https://github.com/dart-lang/sdk/issues/50233
2023-02-13 14:29:30 -05:00
Michael Goderbauer
ab3c82244e
Remove unnecessary null checks in dev/devicelab (#118842) 2023-01-20 18:56:44 +00:00
Ricardo Amador
f989d551ce
Devicelab android emulator (#113472)
* Testing whether emulator is possible.

* Adding changes to see if emulator can be used from recipe.

* adding emulator support.

* Add the emulator flag for testing.

* Using string for boolean since it cannot be parsed in properties

* Checking to see if these changes are being used.

* Updated bool back to string

* Remove trailing whitespace from file.
2023-01-17 22:01:11 +00:00
Loïc Sharma
cea55d99d0
[Linux] Add a 'flutter run' console output test (#118279)
* Add Linux support for the UI integration test project

* Add Linux run console test
2023-01-13 19:17:12 +00:00
stuartmorgan
2a502363e1
Add native unit tests to iOS and macOS templates (#117147)
* Improve Swift plugin implementation

* Add iOS tests

* Review feedback on structure

* Remove duplicate scheme file

* Add macOS

* Add iOS

* swift test tweaks

* unit tests

* Whitespace

* Add e2e tests
2022-12-21 19:10:22 +00:00
Siva
7c8e171320
Manual Roll of Flutter Engine from 67254d6e4b03 to 8d83b98c55b3 (#116635)
* Roll Flutter Engine from 67254d6e4b03 to 8d83b98c55b3

* Roll Dart SDK from 35a9facce191 to e517487c5679 (Dart 3.0) (#38105)

* Bump SDK versions.

* Bump Dart SDK version constraints

* Update shrine package to 2.0.1 (null safe version)

* Fix more tests.

* Include patches from Jason for min android sdk version

* Fix analyzer warning
2022-12-08 18:03:51 -08:00
Jenn Magder
35afe1bdac
Stop embedding bitcode for iOS in tool (#112831) 2022-10-04 01:41:23 +00:00
Jenn Magder
489094ad98
Quiet iOS native test result zipping (#111434) 2022-09-13 23:08:41 +00:00
Jenn Magder
a4530b7c12
Make module tests pass on Xcode 14 (#110556) 2022-08-30 18:51:20 +00:00
Michael Goderbauer
10a7c9ba22
Unify analysis options (#108462) 2022-07-28 09:07:49 -07:00
Kaushik Iska
f7504f0380
[openpay] Create an OpenPay benchmark (#107838) 2022-07-21 20:01:06 +00:00
Alexandre Ardhuin
ccd33631e3
enable combinators_ordering (#107847) 2022-07-18 22:04:07 +00:00
Alexander Biggs
7db73c0a93
[fuchsia] Change format of ffx call to debug spam. (#105169) 2022-07-10 22:51:06 +00:00
Jenn Magder
3f1f0a8170
Add flutter build macos-framework command (#105242) 2022-06-03 10:13:08 -07:00
Jenn Magder
c6ced845e3
Remove custom unawaited, prefer dart:async version (#103212) 2022-05-07 08:49:04 -07:00
Alexandre Ardhuin
d40ee2149c
remove unnecessary .toString() (#103226) 2022-05-06 16:04:13 -07:00
Alexandre Ardhuin
07f1c20474
add missing trailing commas in list/set/map literals (#102585) 2022-04-27 09:15:35 +02:00
Jenn Magder
88eca6c1ce
Fail integration test if expected bitcode markers not found (#102016) 2022-04-19 13:29:06 -07:00
Casey Hillers
3109073fd9
[devicelab] Remove legacy devicelab manifest logic (#101554) 2022-04-08 09:46:08 -07:00
Jenn Magder
68471bbb74
Remove --enable-web and --enable-windows-desktop from tests (#101208) 2022-04-07 15:57:09 -07:00
keyonghan
647044c969
Use separate metric file names for same tests running on different platforms (#101317)
* use separate file names for same test

* use string buffer
2022-04-05 09:16:04 -07:00
Emmanuel Garcia
16ebbdf336
Remove unintended comment (#101130) 2022-03-31 12:35:05 -07:00
Emmanuel Garcia
5c6918933e
Migrate AGP and Gradle versions to 7.1.2/7.4 (#99723) 2022-03-30 20:30:09 -07:00
Chris Bracken
4b819782fb
[macOS] Add run release test in devicelab (#100526)
Adds a test that invokes flutter run in release mode on macOS desktop,
waits for successful launch and the flutter command list, then sends the
'q' command to quit the running app.

This adds an integration test for https://github.com/flutter/flutter/pull/100504.

Issue: https://github.com/flutter/flutter/issues/100348 (fix)
Issue: https://github.com/flutter/flutter/issues/97978 (partial fix)
Issue: https://github.com/flutter/flutter/issues/97977 (partial fix)
Umbrella issue: https://github.com/flutter/flutter/issues/60113
2022-03-22 14:12:59 -07:00
Emmanuel Garcia
9072a09958
Fix race condition in readJsonResults (#100243) 2022-03-17 13:40:15 -07:00
Jonah Williams
59859df126
Add Windows performance benchmark (#99564) 2022-03-04 15:41:14 -08:00
Emmanuel Garcia
f320d14062
Fix lifecycle for API level 28 (#99433) 2022-03-02 18:30:54 -08:00
Emmanuel Garcia
6c818d772a
Add Android lifecycles test (#99319) 2022-03-01 15:26:21 -08:00
Tomasz Gucio
e4351ff053
Enable use_if_null_to_convert_nulls_to_bools lint (#98753) 2022-02-22 14:39:20 -08:00
Ben Konyi
729f8c83d2
Update regexp for Observatory/VM Service message parsing for device lab (#98935)
Should unblock https://github.com/flutter/flutter/issues/98930
2022-02-22 10:59:53 -08:00
David Iglesias
93c0c043cf
[devicelab] Web benchmarks now run on Chromium 89+ (#98629) 2022-02-16 19:04:14 -08:00
Ben Konyi
2a2f973120
Update flutter_tools to look for new VM service message (#97683)
* Update flutter_tools to look for new VM service message

The Dart SDK will soon move away from the current Observatory message:

"Observatory listening on ..."

To a new message that no longer references Observatory:

"Dart VM Service listening on ..."

This change updates all tests with mocks to check for the new message
and also adds support for the new message in ProtocolDiscovery.

See https://github.com/dart-lang/sdk/issues/46756

* Fix some parsing locations

* Fix analysis failures

* Update message

* Remove extra comment

* Update message

* Add globals prefix
2022-02-15 07:33:57 -08:00
keyonghan
e8bc5c5ace
Support Samsung phone when getting device wakefulness status (#98146) 2022-02-11 12:05:18 -08:00
Jenn Magder
57d5222876
Include -isysroot -arch and -miphoneos-version-min when creating dummy module App.framework (#97689) 2022-02-04 14:50:11 -08:00
Collin Jackson
10fbbafe09
[flutter_tools] Use proper project name in templates (#96373) 2022-02-02 06:50:14 -08:00
Michael Goderbauer
5684b9f69e
Enable no_leading_underscores_for_local_identifiers (#96422) 2022-01-21 14:43:59 -08:00
Ian Hickson
57f097b532
Refactor devicelab bin/run.dart and other cleanup (#96320) 2022-01-10 13:40:17 -08:00
Christopher Fujino
6054eda86b
fix lateinitialization error in devicelab-runner (#94957) 2021-12-28 18:19:16 -08:00
godofredoc
e6971136e4
Adds a home method to device classes. (#95103) 2021-12-13 17:09:09 -08:00
keyonghan
33be68d9bf
Stop sending metrics to builder name (#92989) 2021-11-05 16:28:02 -07:00
Jenn Magder
dc20a6066f
Skip codesigning during native macOS integration tests (#92938) 2021-11-02 16:41:32 -07:00
keyonghan
30d3866ab9
Add extra benchmark metrics to test name in addition to builder name (#92530) 2021-10-27 15:33:06 -07:00
keyonghan
7de92e2653
Add devicelab benchmark tags support (#92141) 2021-10-20 15:03:06 -07:00
Jia Hao
e32201235f
Indicate that only physical iOS devices are supported (#79350) 2021-10-13 10:43:05 -07:00
Ian Hickson
f90b019c68
Enable prefer_relative_imports and fix files. (#91573) 2021-10-11 10:28:07 -07:00
Ian Hickson
7b01346c5c
Enable no_default_cases lint (#91530) 2021-10-11 10:23:04 -07:00
Ian Hickson
b3f63d38ac
Document why some lints aren't enabled and fix some minor issues. (#91527) 2021-10-09 04:03:03 -07:00
Ian Hickson
61a0add286
Enable avoid_redundant_argument_values lint (#91409) (#91462) 2021-10-08 09:25:14 -07:00
Zachary Anderson
b9d2177da0
Revert "Enable avoid_redundant_argument_values lint (#91409)" (#91461)
This reverts commit 5fd259be24.
2021-10-07 21:11:07 -07:00
Ian Hickson
5fd259be24
Enable avoid_redundant_argument_values lint (#91409) 2021-10-07 20:13:02 -07:00
Jenn Magder
be8d29ffc6
Run native_ui_tests_macos in correct directory (#90829) 2021-09-27 17:38:03 -07:00
Jenn Magder
5c79a0c2f4
Run flutter_gallery macOS native tests on presubmit (#89620) 2021-09-22 13:18:07 -07:00
Konstantin Scheglov
cfd0743d10
Fix UNNECESSARY_TYPE_CHECK_TRUE violations. (#89186) 2021-08-31 15:46:05 -07:00
keyonghan
e311e3b3ef
skip staging update (#89137) 2021-08-30 09:26:05 -07:00
keyonghan
db4c104c8e
Use task name when uploading metrics to skia perf (#89004) 2021-08-26 22:31:02 -07:00
keyonghan
b67e264203
Revert "Skip staging test update to cocoon in test runner (#88835)" (#88971)
This reverts commit 884dfc260d.
2021-08-26 09:41:22 -07:00
Casey Hillers
b0950fd567
Revert "Use bucket to check staging test instead of builder name (#88908)" (#88969)
This reverts commit d36f246dcc.
2021-08-26 09:19:50 -07:00
keyonghan
d36f246dcc
Use bucket to check staging test instead of builder name (#88908) 2021-08-25 20:36:02 -07:00
keyonghan
884dfc260d
Skip staging test update to cocoon in test runner (#88835) 2021-08-25 11:36:03 -07:00
keyonghan
b468be3030
Use default value for ResultData when uploading metrics from test runner (#88749) 2021-08-24 12:04:03 -07:00
keyonghan
208d312ff0
Upload devicelab test metrics from test runner (#88447) 2021-08-20 12:57:03 -07:00
yusufm
fe52e56614
Adding a timeout and retry to upload results step. (#87306)
* Adding a timeout and retry to upload results step.

* Adding tests for retries and timeout.

* Adding async to call

* Fixing sleep to trigger the timeout, and adding addition timeout test.

* Fixing minor typo.

* Fixing some minor flutter style issues.

Co-authored-by: yusufm <mohsinally@google.com>
2021-08-03 13:40:28 -07:00
Casey Hillers
79f377dabe
[devicelab] Only upload results on master (#87125) 2021-07-27 12:49:05 -07:00
keyonghan
3aeb794298
Extend test runner command to update test flaky status (#86513) 2021-07-19 09:46:03 -07:00
keyonghan
4d96a3fd50
Rerun devicelab task from test runner (#86394) 2021-07-14 15:23:10 -07:00
William Hesse
e01d327239
Always write devicelab test results to a file if the resultsPath option is present (#86393)
Some runs of DeviceLab performance tests are run without a gitBranch option, but
still must save their results to a local file.
2021-07-13 22:00:26 -07:00
Darren Austin
ef5ffd08f6
Migrate devicelab framework code to null safety. (#86325)
(Attempt to reland #85993)
2021-07-13 12:14:50 -07:00
Darren Austin
08ef7752d5
Revert "Migrate core devicelab framework to null safety. (#85993)" (#86269)
This reverts commit 2175e64e4f.
2021-07-11 23:25:25 -07:00
Darren Austin
2175e64e4f
Migrate core devicelab framework to null safety. (#85993)
Migrate core devicelab framework to null safety.
2021-07-11 18:36:30 -07:00
Dan Field
e36e62e9a6
Revert "Revert "Audit hashCode overrides outside of packages/flutter (#85370)" (#85451)" (#85567)
This reverts commit 2be0d57fa2.
2021-06-30 09:46:54 -07:00
Ren You
2be0d57fa2
Revert "Audit hashCode overrides outside of packages/flutter (#85370)" (#85451)
This reverts commit 9846fa5145.
2021-06-28 09:55:31 -07:00
Dan Field
9846fa5145
Audit hashCode overrides outside of packages/flutter (#85370) 2021-06-26 17:36:04 -07:00
gaaclarke
66fd0b28b0
replaced unawaited with synchronous / await calls where needed (#82839) 2021-06-24 16:22:43 -07:00