Commit Graph

102 Commits

Author SHA1 Message Date
Matan Lurey
5840236ae5
Re-land: "Remove Cocoon from dev/devicelab, keeping Skia perf stats upload" (#165755)
🚫 **Merge after**
https://flutter-review.googlesource.com/c/recipes/+/64300.

---

This reverts commit 0a2d9f5658.
2025-03-22 21:06:17 +00:00
auto-submit[bot]
0a2d9f5658
Reverts "Remove Cocoon from dev/devicelab, keeping Skia perf stats upload. (#165749)" (#165754)
<!-- start_original_pr_link -->
Reverts: flutter/flutter#165749
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: matanlurey
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: Still passing command-line arguments from recipes
that have no effect but cause the runner to crash.
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: matanlurey
<!-- end_original_pr_author -->

<!-- start_reviewers -->
Reviewed By: {jtmcdole}
<!-- end_reviewers -->

<!-- start_revert_body -->
This change reverts the following previous change:
Partial re-land of https://github.com/flutter/flutter/pull/165628:

- Fixes the mistake that the `Cocoon` class did things that well, were
not specific to Cocoon.
- Renamed to `MetricsResultWriter`, as that is all it does now.

---

Closes https://github.com/flutter/flutter/issues/165618.

The `devicelab/bin/test_runner.dart upload-metrics` command use to have
_two_ responsibilities:

- Well, upload test **metrics** (benchmarks) to Skia Perf (it still does
that)
- Upload test **status** to Cocoon (it did until
https://github.com/flutter/flutter/pull/165614)

As https://github.com/flutter/flutter/pull/165614 proved, this API
predated the current LUCI setup, where Cocoon itself receives task
status updates from LUCI, and it turns out this entire time, DeviceLab
was making (at best) NOP calls, and at worst, causing crashes and
corrupt data (https://github.com/flutter/flutter/issues/165610).
<!-- end_revert_body -->

Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
2025-03-22 18:58:47 +00:00
Matan Lurey
05f1980eae
Remove Cocoon from dev/devicelab, keeping Skia perf stats upload. (#165749)
Partial re-land of https://github.com/flutter/flutter/pull/165628:

- Fixes the mistake that the `Cocoon` class did things that well, were
not specific to Cocoon.
- Renamed to `MetricsResultWriter`, as that is all it does now.

---

Closes https://github.com/flutter/flutter/issues/165618.

The `devicelab/bin/test_runner.dart upload-metrics` command use to have
_two_ responsibilities:

- Well, upload test **metrics** (benchmarks) to Skia Perf (it still does
that)
- Upload test **status** to Cocoon (it did until
https://github.com/flutter/flutter/pull/165614)

As https://github.com/flutter/flutter/pull/165614 proved, this API
predated the current LUCI setup, where Cocoon itself receives task
status updates from LUCI, and it turns out this entire time, DeviceLab
was making (at best) NOP calls, and at worst, causing crashes and
corrupt data (https://github.com/flutter/flutter/issues/165610).
2025-03-22 17:48:57 +00:00
auto-submit[bot]
1cf57f1eaa
Reverts "Remove unused Cocoon components from devicelab/bin/test_runner.dart. (#165628)" (#165709)
<!-- start_original_pr_link -->
Reverts: flutter/flutter#165628
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: jtmcdole
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: breaking tree:


https://ci.chromium.org/ui/p/flutter/builders/prod/Linux_mokey%20backdrop_filter_perf__e2e_summary/3833/overview
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: matanlurey
<!-- end_original_pr_author -->

<!-- start_reviewers -->
Reviewed By: {jtmcdole, yjbanov}
<!-- end_reviewers -->

<!-- start_revert_body -->
This change reverts the following previous change:
🚫 **Do not merge** until
https://flutter-review.googlesource.com/c/recipes/+/64220/1 is merged.

---

Closes https://github.com/flutter/flutter/issues/165618.

The `devicelab/bin/test_runner.dart upload-metrics` command use to have
_two_ responsibilities:

- Well, upload test **metrics** (benchmarks) to Skia Perf (it still does
that)
- Upload test **status** to Cocoon (it did until
https://github.com/flutter/flutter/pull/165614)

As https://github.com/flutter/flutter/pull/165614 proved, this API
predated the current LUCI setup, where Cocoon itself receives task
status updates from LUCI, and it turns out this entire time, DeviceLab
was making (at best) NOP calls, and at worst, causing crashes and
corrupt data (https://github.com/flutter/flutter/issues/165610).

In other words, this is removing entirely dead/unused code (though the
recipes have to be updated first).

/cc @jason-simmons as I need Jason's help reviewing the recipes change
and want to provide context.
<!-- end_revert_body -->

Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
2025-03-21 22:30:47 +00:00
Matan Lurey
b06c05c6f4
Remove unused Cocoon components from devicelab/bin/test_runner.dart. (#165628)
🚫 **Do not merge** until
https://flutter-review.googlesource.com/c/recipes/+/64220/1 is merged.

---

Closes https://github.com/flutter/flutter/issues/165618.

The `devicelab/bin/test_runner.dart upload-metrics` command use to have
_two_ responsibilities:

- Well, upload test **metrics** (benchmarks) to Skia Perf (it still does
that)
- Upload test **status** to Cocoon (it did until
https://github.com/flutter/flutter/pull/165614)

As https://github.com/flutter/flutter/pull/165614 proved, this API
predated the current LUCI setup, where Cocoon itself receives task
status updates from LUCI, and it turns out this entire time, DeviceLab
was making (at best) NOP calls, and at worst, causing crashes and
corrupt data (https://github.com/flutter/flutter/issues/165610).

In other words, this is removing entirely dead/unused code (though the
recipes have to be updated first).

/cc @jason-simmons as I need Jason's help reviewing the recipes change
and want to provide context.
2025-03-21 21:27:48 +00:00
Michael Goderbauer
5491c8c146
Auto-format Framework (#160545)
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.

**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.

---------

Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
2024-12-19 20:06:21 +00:00
Ben Konyi
2c84e63ba7
[ Cocoon ] Wait for task results to be received by the task runner before shutting down the task process (#156002)
Prior to this fix, `_TaskRunner.run` would immediately cleanup the
keep-alive port once the task completed, which would result in the
isolate shutting down as soon as the task result was returned from
`ext.cocoonRunTask` callback in the form of a
`ServiceExtensionResponse`. Since the service extension response is
processed by the service isolate, it was possible for the VM to start
shutting down before the service isolate could send the task result data
back to the task runner.

This change introduces a new service extension,
`ext.cocoonTaskResultReceived`, that the task runner invokes after it
receives the task result from `ext.cocoonRunTask`, notifying the task
process that it can close the keep-alive port and shutdown.

Fixes https://github.com/flutter/flutter/issues/155475
2024-10-02 17:01:24 -04:00
Matan Lurey
4b5cf85534
Remove last --disable-dart-dev in flutter/flutter. (#154948)
Work towards https://github.com/flutter/flutter/issues/154268.
2024-09-10 21:37:08 +00:00
Matan Lurey
712cf654d1
Remove --disable-dart-dev across flutter/flutter. (#154280)
As per the Dart CLI team and @bkonyi, this is no longer providing value and we shouldn't cargo-cult it.

Work towards https://github.com/flutter/flutter/issues/154268.
2024-08-29 22:50:04 +00:00
Ian Hickson
5e216d44ca
Simplify devicelab logic and fix tests (#139122)
- fix https://github.com/flutter/flutter/issues/53707 by having the test not expect a timeout but instead actually look for the retry message
- simplify the `--task` option to only accept task names rather than also accepting paths
- remove some obsolete options that referred to the manifest which no longer seems to exist
2023-11-29 08:41:27 +00:00
Michael Goderbauer
b0a90aee17
Enable strict-inference (#135043)
Avoids that dynamic accidentally sneaks in, see https://dart.dev/tools/analysis#enabling-additional-type-checks
2023-09-20 19:59:08 +00:00
Andrew Kolos
445e02dd63
fix --exit flag in dev/devicelab/bin/run.dart (#134162)
Fixes #134154 

This PR also changes the default value of the `--exit` flag from `true` to `false`. Effectively, this is not a change in behavior since `--exit` didn't previously work.
2023-09-07 19:36:55 +00:00
Matan Lurey
a7997f606e
Update dev/devicelab/** to provide --local-engine-host. (#132342)
Partial work towards https://github.com/flutter/flutter/issues/132245.

I have to admit I don't totally understand what I've updated, or whether
there are more integration points needed.
2023-08-10 17:25:30 -07:00
Harry Terkelsen
daaba8a4aa
Add --local-web-sdk in devicelab runner to make --ab testing work for web (#123825)
This allows us to check for performance differences in local Web SDKs.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#overview
[Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene
[test-exempt]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes
[Discord]: https://github.com/flutter/flutter/wiki/Chat
2023-03-31 14:25:00 -07:00
yusuf-goog
47dd992266
Adding timestamps to task runner output lines. (#123075)
Adding timestamps to task runner output lines.
2023-03-21 23:25:35 +00:00
Ben Konyi
ecd7518df5
Reland "Remove references to Observatory (#118577)" (#121606)
This reverts commit 275ab9c69b.
2023-02-28 11:57:04 -05:00
Michael Goderbauer
275ab9c69b
Revert "Reland "Remove references to Observatory (#118577)" (#121215)" (#121555)
Revert "Reland "Remove references to Observatory (#118577)""
2023-02-27 23:46:53 +00:00
Ben Konyi
fbae472fc3
Reland "Remove references to Observatory (#118577)" (#121215)
This reverts commit 298d8c76ba.
2023-02-27 09:26:43 -05:00
Casey Hillers
298d8c76ba
Revert "Remove references to Observatory (#118577)" (#120929)
This reverts commit 2df140f40d.
2023-02-16 21:28:30 -08:00
Ben Konyi
2df140f40d
Remove references to Observatory (#118577)
Observatory is being deprecated for Dart 3.0 so it should no longer be
referenced in tooling messaging / flags.

See https://github.com/dart-lang/sdk/issues/50233
2023-02-13 14:29:30 -05:00
Ricardo Amador
f989d551ce
Devicelab android emulator (#113472)
* Testing whether emulator is possible.

* Adding changes to see if emulator can be used from recipe.

* adding emulator support.

* Add the emulator flag for testing.

* Using string for boolean since it cannot be parsed in properties

* Checking to see if these changes are being used.

* Updated bool back to string

* Remove trailing whitespace from file.
2023-01-17 22:01:11 +00:00
Michael Goderbauer
10a7c9ba22
Unify analysis options (#108462) 2022-07-28 09:07:49 -07:00
Jenn Magder
c6ced845e3
Remove custom unawaited, prefer dart:async version (#103212) 2022-05-07 08:49:04 -07:00
Ben Konyi
729f8c83d2
Update regexp for Observatory/VM Service message parsing for device lab (#98935)
Should unblock https://github.com/flutter/flutter/issues/98930
2022-02-22 10:59:53 -08:00
Ben Konyi
2a2f973120
Update flutter_tools to look for new VM service message (#97683)
* Update flutter_tools to look for new VM service message

The Dart SDK will soon move away from the current Observatory message:

"Observatory listening on ..."

To a new message that no longer references Observatory:

"Dart VM Service listening on ..."

This change updates all tests with mocks to check for the new message
and also adds support for the new message in ProtocolDiscovery.

See https://github.com/dart-lang/sdk/issues/46756

* Fix some parsing locations

* Fix analysis failures

* Update message

* Remove extra comment

* Update message

* Add globals prefix
2022-02-15 07:33:57 -08:00
Ian Hickson
57f097b532
Refactor devicelab bin/run.dart and other cleanup (#96320) 2022-01-10 13:40:17 -08:00
Ian Hickson
f90b019c68
Enable prefer_relative_imports and fix files. (#91573) 2021-10-11 10:28:07 -07:00
keyonghan
4d96a3fd50
Rerun devicelab task from test runner (#86394) 2021-07-14 15:23:10 -07:00
William Hesse
e01d327239
Always write devicelab test results to a file if the resultsPath option is present (#86393)
Some runs of DeviceLab performance tests are run without a gitBranch option, but
still must save their results to a local file.
2021-07-13 22:00:26 -07:00
Darren Austin
ef5ffd08f6
Migrate devicelab framework code to null safety. (#86325)
(Attempt to reland #85993)
2021-07-13 12:14:50 -07:00
Darren Austin
08ef7752d5
Revert "Migrate core devicelab framework to null safety. (#85993)" (#86269)
This reverts commit 2175e64e4f.
2021-07-11 23:25:25 -07:00
Darren Austin
2175e64e4f
Migrate core devicelab framework to null safety. (#85993)
Migrate core devicelab framework to null safety.
2021-07-11 18:36:30 -07:00
Dan Field
5cc7b6c3ea
Add option to stream logs to file for flutter logs and way to use it in devicelab runs (#84008) 2021-06-07 10:33:00 -07:00
Jenn Magder
94a41cde83
Mixed null safety in dev/devicelab (#84017) 2021-06-05 02:29:03 -07:00
gaaclarke
7cdd33fe99
added unawaited function for devicelab and turned on unawaited_futures linter (#82833) 2021-05-18 14:19:03 -07:00
Jonah Williams
4851bd51f9
[devicelab] remove vm_service_client from devicelab (#78559) 2021-03-18 11:51:07 -07:00
Casey Hillers
b0afe3464e
Create BuildTestTask (#77307) 2021-03-04 18:49:03 -08:00
Casey Hillers
c8c681f68e
[devicelab] Create test command to pass task args (#77110) 2021-03-04 11:59:02 -08:00
Casey Hillers
7ed797b053
Revert "[devicelab] Separate build and test from Flutter gallery tests (#76415)" (#77086)
This reverts commit 297c7b5cf2.
2021-03-02 12:17:18 -08:00
Casey Hillers
297c7b5cf2
[devicelab] Separate build and test from Flutter gallery tests (#76415) 2021-03-02 11:49:02 -08:00
Michael Goderbauer
78cbfbff42
Revert "Migrate devicelab to package:vm_service (#71882)" (#72092)
This reverts commit 9d1d0cfeb5.
2020-12-10 12:56:02 -08:00
Dan Field
9d1d0cfeb5
Migrate devicelab to package:vm_service (#71882) 2020-12-10 12:43:04 -08:00
Jonah Williams
e422d5c724
[flutter_tools] reland: --no-android-gradle-daemon in devicelab (#68491)
Remove devicelab specific code for shutting down gradle daemon, add --android-gradle-daemon option to build/run/drive`. Avoids need for un-tested devicelab specific handler. There are also some feature requests for this, so 2 birds one stone.

Example:

flutter build apk --no-android-gradle-daemon will pass --no-daemon on to gradle
2020-10-19 10:27:58 -07:00
Jonah Williams
ed5482de19
Revert "[flutter_tools] add --android-gradle-daemon option, use in devicelab (#68409)" (#68489)
This reverts commit ddab72e79f.
2020-10-19 09:33:52 -07:00
Jonah Williams
ddab72e79f
[flutter_tools] add --android-gradle-daemon option, use in devicelab (#68409)
Remove devicelab specific code for shutting down gradle daemon, add --android-gradle-daemon option to build/run/drive`. Avoids need for un-tested devicelab specific handler. There are also some feature requests for this, so 2 birds one stone.

Example:

flutter build apk --no-android-gradle-daemon will pass --no-daemon on to gradle
2020-10-19 09:12:21 -07:00
Jonah Williams
2321b72dfb
[flutter_tools] move gradle download failure handling into tool (#68140)
This crash affects anyone using flutter, so move it into the tool where it can be tested and documented. Ensure it works on windows devicelab too.
2020-10-15 09:20:52 -07:00
Casey Hillers
9e206a0c93
Refactor devicelab logic to use TaskResult instead of JSON (#67550) 2020-10-07 16:37:09 -07:00
Jenn Magder
a6a8a14d02
Do not cleanup Gradle in non-Android integration tests (#66487) 2020-09-24 18:34:07 -07:00
Tong Wu
abeaf11829
[devicelab] Remove ~/.gradle after each devicelab task run (#65434) 2020-09-09 01:50:03 -07:00
Zachary Anderson
5e0ba2290f
Pass --disable-dart-dev in more places (#62217) 2020-07-24 13:37:08 -07:00