Commit Graph

93 Commits

Author SHA1 Message Date
Kate Lovett
01d1e74af5
Remove abi key permanently from golden file testing (#149858)
We have fiddled with this a bunch, and there is definitively no way to reliably include this, so I am removing it.

Since CI does not cover all possible values, we can't consistently look up images for local testing. We thought removing it from the look up would work fine, and it did for most tests, but there were still some that could not find an image without it.

A brief history on the abi key
- added in https://github.com/flutter/flutter/pull/143621
- disabled in https://github.com/flutter/flutter/pull/148023
- added back in https://github.com/flutter/flutter/pull/148072
  - we thought there was only an issue with alphabetizing keys
- removed from local image look up in https://github.com/flutter/flutter/pull/149696

I updated the docs page to also discuss what makes a good key and what does not based on what we learned here.
2024-06-06 23:05:52 +00:00
Kate Lovett
236b3a716d
Remove abi key from local golden file testing (#149696)
Fixes https://github.com/flutter/flutter/issues/149435

CI does not comprehensively cover all of the possible abi keys, so some folks can't test locally if their machine's abi does not match CI. This just removes it from the local testing look up. The right image will be summoned, and in CI we can still keep track of it.
2024-06-05 23:51:11 +00:00
Ian Hickson
5933e99c6c
Remove hidden dependency on ABI. (#148987)
This is part 17 of a broken down version of the #140101 refactor (though this particular dependency didn't actually exist back then).

This only makes one dependency explicit. Further PRs will do the same for other dependencies, until these APIs have no hidden dependencies.

I also snuck in a minor stylistic change, using interpolation instead of explicit `toString`, to make the code more idiomatic. (I'm actually surprised that that didn't trigger a lint, I thought we had a lint to catch that. I must be thinking of something else though.)

This PR makes no effort to keep the order of parameters reasonable. There is an existing TODO to do a refactor sweep later that does nothing but reorder arguments/parameters/fields to be consistent.
2024-05-30 19:12:22 +00:00
Ian Hickson
2025ebe767
Remove hidden dependencies on HttpClient (#148773)
This is part 16 of a broken down version of the #140101 refactor.

This only makes one dependency explicit. Further PRs will do the same for other dependencies, until these APIs have no hidden dependencies.

This PR makes no effort to keep the order of parameters reasonable. There is an existing TODO to do a refactor sweep later that does nothing but reorder arguments/parameters/fields to be consistent.
2024-05-23 18:20:50 +00:00
Ian Hickson
450b072a21
Remove hidden dependencies on LocalProcessManager (#148096)
This is part 15 of a broken down version of the #140101 refactor.

This only makes one dependency explicit. Further PRs will do the same for other dependencies, until these APIs have no hidden dependencies.

This PR makes no effort to keep the order of parameters reasonable. There is an existing TODO to do a refactor sweep later that does nothing but reorder arguments/parameters/fields to be consistent.
2024-05-21 16:59:16 +00:00
Kate Lovett
4a1e3eaaa2
Fix abi key for local golden file testing (#148072)
Fixes https://github.com/flutter/flutter/issues/148022
2024-05-14 20:58:18 +00:00
Ian Hickson
e1e742ac35
Remove hidden dependencies on the default LocalPlatform (#147342)
This is part 13 of a broken down version of the #140101 refactor.

This only makes one dependency explicit. Further PRs will do the same
for other dependencies, until these APIs have no hidden dependencies.
2024-05-09 15:33:55 -07:00
Kate Lovett
00f40667c1
Exclude abi key in local golden file testing (#148023)
Part of https://github.com/flutter/flutter/issues/148022
2024-05-08 23:39:13 +00:00
Ian Hickson
a2a54fe47f
Remove hidden dependencies on the default goldenFileComparator. (#146956)
This is part 12 of a broken down version of the #140101 refactor.

This only makes one dependency explicit. Further PRs will do the same for other dependencies, until these APIs have no hidden dependencies.
2024-04-24 22:50:58 +00:00
Ian Hickson
1d904b641b
Remove now-redundant tests for isForEnvironment (#146804)
This is part 11 of a broken down version of the #140101 refactor.

These tests are redundant with the more comprehensive tests now in comparator_selection_test.dart.
2024-04-17 19:23:26 +00:00
Ian Hickson
3748b59e44
Make FileSystem dependency explicit througout. (#146008)
This is part 8 of a broken down version of the #140101 refactor.

This only makes one dependency explicit. Further PRs will do the same for other dependencies, until these APIs have no hidden dependencies.

This particular change attempts to be minimal. A future change will apply some formatting cleanup (newlines, reordering parameters and arguments) for clarity.
2024-04-05 23:57:21 +00:00
Ian Hickson
16ec1a8434
Remove state shared across tests (#145281)
This is part 6 of a broken down version of the #140101 refactor.

This change tries to be as minimal as possible to make review as easy as possible. We can further simplify this file in future PRs. This change allows us to re-enable test order randomization for these tests.
2024-03-29 21:19:47 +00:00
Ian Hickson
c7123c96cd
Factor out use of "print" in flutter_goldens (#144846)
This is part 5 of a broken down version of the #140101 refactor.

This PR removes direct use of `print` and replaces it with a callback.
2024-03-17 01:56:19 +00:00
Ian Hickson
dbcce277f3
Rename isAvailableForEnvironment to isForEnvironment (#143176) (#144858)
This is part 4 of a broken down version of the #140101 refactor.

This PR renames isAvailableForEnvironment to isForEnvironment and replaces a regular expression with a simple function. (The latter will change the behaviour for people with branch names like `mainly_refactors` or `chess_master_experiment` or whatever, but I'm pretty sure the old behaviour was not intended.)

----

This is a reland of https://github.com/flutter/flutter/pull/143176 which was speculatively reverted in https://github.com/flutter/flutter/pull/144855 but turned out not to be the cause of the tree redness.
2024-03-10 06:20:38 +00:00
auto-submit[bot]
0432a6941d
Reverts "Rename isAvailableForEnvironment to isForEnvironment (#143176)" (#144855)
Reverts: flutter/flutter#143176
Initiated by: QuncCccccc
Reason for reverting: made tree red.
Original PR Author: Hixie

Reviewed By: {Piinks}

This change reverts the following previous change:
This is part 4 of a broken down version of the #140101 refactor.

This PR renames isAvailableForEnvironment to isForEnvironment and replaces a regular expression with a simple function. (The latter will change the behaviour for people with branch names like `mainly_refactors` or `chess_master_experiment` or whatever, but I'm pretty sure the old behaviour was not intended.)
2024-03-08 23:19:15 +00:00
Ian Hickson
5bd34ef541
Rename isAvailableForEnvironment to isForEnvironment (#143176)
This is part 4 of a broken down version of the #140101 refactor.

This PR renames isAvailableForEnvironment to isForEnvironment and replaces a regular expression with a simple function. (The latter will change the behaviour for people with branch names like `mainly_refactors` or `chess_master_experiment` or whatever, but I'm pretty sure the old behaviour was not intended.)
2024-03-08 20:20:13 +00:00
Jonah Williams
4c0b5ccebc
[gold] Always provide host ABI to gold config (#143621)
This can help us split goldens that are different due to arm non-arm mac, et cetera.

Part of https://github.com/flutter/flutter/issues/143616
2024-02-20 21:06:01 +00:00
Zachary Anderson
8a8616f913
Handle FormatException from SkiaGoldClient (#143755)
Seen in
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20Framework%20Smoke%20Tests/17183/overview
closing the engine tree.
2024-02-20 09:57:23 -08:00
Ian Hickson
77c3807c80
Revert "Make tests more resilient to Skia gold failures and refactor flutter_goldens for extensive technical debt removal (#140101)" (#141814)
Reverts https://github.com/flutter/flutter/pull/140101

That PR somehow made non-matching gold tests not fail at HEAD.

Fixes https://github.com/flutter/flutter/issues/141880
- Blocked by https://github.com/flutter/flutter/issues/140169
  - https://github.com/flutter/flutter/pull/141427
2024-01-19 20:29:11 +00:00
Ian Hickson
62f1594908
Make tests more resilient to Skia gold failures and refactor flutter_goldens for extensive technical debt removal (#140101)
Originally landed in https://github.com/flutter/flutter/pull/139549
Originally reverted in https://github.com/flutter/flutter/pull/140085

- Remove all use of global variables.
- Always pass in all dependencies, only create them in main or in tests.
- Pass in the "print" primitive.
- Make all network traffic retry (except when run locally, when it just auto-passes).
- Enable tests to be run in random order.
- Better error messages
2023-12-21 21:35:07 +00:00
auto-submit[bot]
91877c6345
Reverts "Make tests more resilient to Skia gold failures and refactor flutter_goldens for extensive technical debt removal" (#140085)
Reverts flutter/flutter#139549
Initiated by: Piinks
This change reverts the following previous change:
Original Description:
* Remove all use of global variables.
* Always pass in all dependencies, only create them in main or in tests.
* Pass in the "print" primitive.
* Make all network traffic retry (except when run locally, when it just auto-passes).
* Enable tests to be run in random order.
2023-12-13 20:20:20 +00:00
Ian Hickson
11a9cb7029
Make tests more resilient to Skia gold failures and refactor flutter_goldens for extensive technical debt removal (#139549)
* Remove all use of global variables.
* Always pass in all dependencies, only create them in main or in tests.
* Pass in the "print" primitive.
* Make all network traffic retry (except when run locally, when it just auto-passes).
* Enable tests to be run in random order.
2023-12-13 04:09:13 +00:00
Kate Lovett
91ea024cbc
Don't send images to Gold on release branches (#139706)
Part of fixing https://github.com/flutter/flutter/issues/139673, it will need to be cherry picked into the stable branch to fully resolve.

Originally I tried to come at this from `ci.yaml`, but the syntax does not exist to either conditionally include a dependency based on the branch we are on, or to disable a shard for a given branch (as opposed to enabling it which is supported). I could double every CI shard that uses Gold to try to serve my purpose, but it is already a very large and cumbersome file to keep up to date. Doubling it does not feel like the best solution. Using a RegEx is not my favorite, but I am using the same one we use in our CI, and left a note there to update it if it should ever change. Since there is already a whole infra built around it, I feel it is pretty safe so we can fix the stable tree.

We already had mitigated Gold affecting release branches in the past through flutter/cocoon (https://github.com/flutter/flutter/issues/58814), but https://github.com/flutter/flutter/issues/139673 exposed a rather rare edge case. A change was CP'd into the stable branch that introduced golden file image changes. Typically this would not cause an issue since any change that has landed on the master branch has golden files accounted for. In this case, the CP'd change on master has generated a different image on canvaskit due to another change that was not on stable. So when the CP landed, it generated a new image Gold had never seen before. Gold only tracks the master branch, so we cannot approve the image, and so cannot fix the stable tree.

This would disable the failing check on release branches and fix the tree.
2023-12-07 19:34:05 +00:00
Zachary Anderson
a84e369bd2
Revert "Allow Flutter golden file tests to be flaky (#114450)" (#114902)
This reverts commit 53e6876226.
2022-11-08 09:26:55 -08:00
Kate Lovett
53e6876226
Allow Flutter golden file tests to be flaky (#114450)
* allow marking a golden check as flaky

* add matchesFlutterGolden to analyze.dart; no tags for flutter_goldens/lib

* Pause

* ++

* ++

* Analyzer therapy

* Once more with feeling

* Nits

* Review feedback

* Silly oops

* Test progress

* More tests

* Finish

* Nits

* Analyzer

* Review feedback

Co-authored-by: Yegor Jbanov <yjbanov@google.com>
2022-11-08 01:14:03 +00:00
Kate Lovett
abbc0bedca
Add more logs to diagnose Gold flake (#108930) 2022-08-04 00:11:23 +00:00
Alexandre Ardhuin
56c4e3cdd8
add some exports of public API in foundation/serialization.dart (#104231) 2022-06-08 13:33:12 -07:00
Michael Goderbauer
2b14e993a7
Cleanup random dead test code (#104568) 2022-05-25 10:23:10 -07:00
Yegor
3fe2cbabc4
[web] allow small golden deltas in HTML renderer (#102791) 2022-04-29 11:49:05 -07:00
Alexandre Ardhuin
07f1c20474
add missing trailing commas in list/set/map literals (#102585) 2022-04-27 09:15:35 +02:00
Kate Lovett
324e6d9d4e
Re-enable Gold post-submit fail state (#101528) 2022-04-07 15:36:09 -07:00
Kate Lovett
8e7b361651
Add back Gold passfail (#100576) 2022-03-22 14:00:20 -07:00
Kate Lovett
2386fd90c9
Disable post-submit Gold failures entirely (#100332) 2022-03-17 17:07:02 -05:00
Kate Lovett
7f241ba3fa
Disable post-submit failures (#100308) 2022-03-17 11:58:26 -05:00
Viren Khatri
671aa9e95e
complete migration of flutter repo to Object.hash* (#99505) 2022-03-04 12:56:21 -08:00
Michael Goderbauer
00f3f2b14f
More beautiful linear_gradient sample (#99298) 2022-03-02 16:41:21 -08:00
Kate Lovett
a288bd5e3f
Re-land Assert golden files use the right extension (#99082) 2022-02-24 21:06:19 -08:00
Harry Terkelsen
735f9abfe1
Revert "Assert golden files use the right extension (#99016)" (#99075)
This reverts commit ee86e61f73.
2022-02-24 10:44:12 -08:00
Kate Lovett
ee86e61f73
Assert golden files use the right extension (#99016) 2022-02-23 19:59:21 -08:00
Kate Lovett
895beb04bb
Initialize the Skia client more efficiently (#95055) 2021-12-10 18:54:10 -08:00
Chris Wong
7ef5226799
Migrate to process 4.2.4 (#92630) 2021-11-30 17:14:07 -08:00
Kate Lovett
ea52013895
Close tree if unapproved golden images get in (#93516) 2021-11-15 12:40:32 -08:00
Alexander Dahlberg
f091654fb4
Removed no-shuffle tag and fixed leak in flutter_goldens_test.dart (#92374) 2021-11-11 12:44:32 -08:00
Ian Hickson
9421627324
Enable only_throw_errors (#91567) 2021-10-11 14:13:03 -07:00
Ian Hickson
f25b833f27
Enable avoid_print lint. (#91444) 2021-10-07 16:48:04 -07:00
Mouad Debbar
c2ea78d231
Revert "Enable avoid_print lint. (#91332)" (#91438)
This reverts commit cb378edc9e.
2021-10-07 16:16:17 -04:00
Ian Hickson
cb378edc9e
Enable avoid_print lint. (#91332) 2021-10-07 09:53:03 -07:00
Kate Lovett
65d8dd988d
Update md5 method in flutter_goldens_client (#90154) 2021-09-15 17:42:05 -07:00
Kate Lovett
8587b609a6
Update local gold api (#90072) 2021-09-14 14:53:33 -05:00
Greg Spencer
738ce43d97
Randomize tests, exclude tests that fail with randomization. (#86793)
* Randomize tests, exclude tests that fail with randomization.

* Disable some more tool tests
2021-07-26 23:40:49 -07:00