The goal here is to have a great standalone `android_engine_test` suite
that [replaces
`scenario_app/android`](https://github.com/flutter/flutter/pull/160992).
No test is _functionally_ changed in this PR, but overview of changes:
- Finished renaming the suite `android_engine_tests` instead of
`flutter_driver_android`
- Added instructions and an environment variable for local generation of
golden-files (`UPDATE_GOLDENS=1`)
- Added explanations of the individual tests, where they live, and how
to run them locally
- Added a hybrid-composition (HC, not TLHC, which is already tested)
test
- Renamed "other_smiley" to "surface_texture_smiley" (and renamed the
original to "surface_producer_smiley")
- Removed unnecessary ".android" suffix (we will not run this on
anything but Android)
- Added a `tool/deflake.dart` to run a test suite 10x (or custom) times
locally to try and determine flakiness
After this PR, I'll add flags to let you control variants and name the
screenshots accordingly, i.e.:
- API v34 or v35
- OpenGLES or Vulkan (will require an `AndroidManifest.xml` edit during
the test instrumentation)
The arm64 Macs should have more memory than the Linux machines that are
available in the devicelab. The analyzer benchmark spawns a Dart
analysis server process that has a peak memory consumption of around
7GB, which may be activating the OOM killer on the Linux bots.
See https://github.com/flutter/flutter/issues/161306
Random seed 349875783 is failing half the time.
```
flutter run -v -d IR65S859OJ4HAY6P --profile lib/benchmark_collection.dart --dart-define=seed=349875783
```
I'm concerned that this sometimes passes, which points to some hidden
timing bugs.
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_benchmark flutter_gallery_macos__compile"
}
-->
Issue link: https://github.com/flutter/flutter/issues/160495
Co-authored-by: Chris Bracken <chris@bracken.jp>
This auto-formats all *.dart files in the repository outside of the
`engine` subdirectory and enforces that these files stay formatted with
a presubmit check.
**Reviewers:** Please carefully review all the commits except for the
one titled "formatted". The "formatted" commit was auto-generated by
running `dev/tools/format.sh -a -f`. The other commits were hand-crafted
to prepare the repo for the formatting change. I recommend reviewing the
commits one-by-one via the "Commits" tab and avoiding Github's "Files
changed" tab as it will likely slow down your browser because of the
size of this PR.
---------
Co-authored-by: Kate Lovett <katelovett@google.com>
Co-authored-by: LongCatIsLooong <31859944+LongCatIsLooong@users.noreply.github.com>
I fixed the autoroller in https://github.com/flutter/flutter/pull/160059
but forgot to mark the shard non-bring-up. Note, it's currently failing
in post-submit, but only because bringUp shards run in the staging pool,
which does not have access to the GitHub access token necessary for the
roller to operate.
<!-- start_original_pr_link -->
Reverts: flutter/flutter#159756
<!-- end_original_pr_link -->
<!-- start_initiating_author -->
Initiated by: gmackall
<!-- end_initiating_author -->
<!-- start_revert_reason -->
Reason for reverting: Made the tree red due to some std out.
<!-- end_revert_reason -->
<!-- start_original_pr_author -->
Original PR Author: gmackall
<!-- end_original_pr_author -->
<!-- start_reviewers -->
Reviewed By: {reidbaker}
<!-- end_reviewers -->
<!-- start_revert_body -->
This change reverts the following previous change:
In preparation for changing engine builds to be unstripped by default
https://github.com/flutter/engine/pull/52852, which will allow us to
make progress towards resolving
https://github.com/flutter/flutter/issues/60240.
Tricks AGP in to downloading the NDK when building a flutter app (that
uses the FGP, which to my knowledge is all ways of building flutter
apk/aab/aar).
I want to follow this up by modifying the tool to search for the log
line that the NDK is missing (making it throw an error in that case) as
a safeguard, because that would be the last line of defense before we
accidentally build a bloated app. The safeguard won't work for add to
app, from what I understand, because while they use the FGP (so they
should be forced to be download the NDK) they don't invoke the flutter
tool, and therefore won't invoke the custom error handling of
[`gradle_errors.dart`](https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/android/gradle_errors.dart)
Some details
1. Respects existing values for `externalNativeBuild.cmake.path` for
apps that actually use it.
2. Silences some warnings that would otherwise appear for add to app
builds or builds that manually invoke gradle:
```
C/C++: CMake Warning (dev) in CMakeLists.txt:
C/C++: No project() command is present. The top-level CMakeLists.txt file must
C/C++: contain a literal, direct call to the project() command. Add a line of
C/C++: code such as
C/C++: project(ProjectName)
C/C++: near the top of the file, but after cmake_minimum_required().
C/C++: CMake is pretending there is a "project(Project)" command on the first
C/C++: line.
C/C++: This warning is for project developers. Use -Wno-dev to suppress it.
C/C++: CMake Warning:
C/C++: Manually-specified variables were not used by the project:
C/C++: CMAKE_EXPORT_COMPILE_COMMANDS
C/C++: CMAKE_LIBRARY_OUTPUT_DIRECTORY
C/C++: CMAKE_RUNTIME_OUTPUT_DIRECTORY
```
3. Our ci installs the NDK at an abnormal place that AGP can't find
without help. I've modified all the `build.gradle`s that we have checked
in to point to the pre-installed path. **But some of our tests make a
new app from the templates, and those tests will now start downloading
the NDK** (as they won't be able to find it at it's current path from
templates). We could resolve this by actually fixing
https://github.com/flutter/flutter/issues/136666. This would be a very
significant lift from what I understand - we rely on this hardcoding in
a lot of places in our infra.
Fixes https://github.com/flutter/flutter/issues/155576
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [ ] All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel
on [Discord].
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
<!-- end_revert_body -->
Co-authored-by: auto-submit[bot] <flutter-engprod-team@google.com>
In preparation for changing engine builds to be unstripped by default
https://github.com/flutter/engine/pull/52852, which will allow us to
make progress towards resolving
https://github.com/flutter/flutter/issues/60240.
Tricks AGP in to downloading the NDK when building a flutter app (that
uses the FGP, which to my knowledge is all ways of building flutter
apk/aab/aar).
I want to follow this up by modifying the tool to search for the log
line that the NDK is missing (making it throw an error in that case) as
a safeguard, because that would be the last line of defense before we
accidentally build a bloated app. The safeguard won't work for add to
app, from what I understand, because while they use the FGP (so they
should be forced to be download the NDK) they don't invoke the flutter
tool, and therefore won't invoke the custom error handling of
[`gradle_errors.dart`](https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/android/gradle_errors.dart)
Some details
1. Respects existing values for `externalNativeBuild.cmake.path` for
apps that actually use it.
2. Silences some warnings that would otherwise appear for add to app
builds or builds that manually invoke gradle:
```
C/C++: CMake Warning (dev) in CMakeLists.txt:
C/C++: No project() command is present. The top-level CMakeLists.txt file must
C/C++: contain a literal, direct call to the project() command. Add a line of
C/C++: code such as
C/C++: project(ProjectName)
C/C++: near the top of the file, but after cmake_minimum_required().
C/C++: CMake is pretending there is a "project(Project)" command on the first
C/C++: line.
C/C++: This warning is for project developers. Use -Wno-dev to suppress it.
C/C++: CMake Warning:
C/C++: Manually-specified variables were not used by the project:
C/C++: CMAKE_EXPORT_COMPILE_COMMANDS
C/C++: CMAKE_LIBRARY_OUTPUT_DIRECTORY
C/C++: CMAKE_RUNTIME_OUTPUT_DIRECTORY
```
3. Our ci installs the NDK at an abnormal place that AGP can't find
without help. I've modified all the `build.gradle`s that we have checked
in to point to the pre-installed path. **But some of our tests make a
new app from the templates, and those tests will now start downloading
the NDK** (as they won't be able to find it at it's current path from
templates). We could resolve this by actually fixing
https://github.com/flutter/flutter/issues/136666. This would be a very
significant lift from what I understand - we rely on this hardcoding in
a lot of places in our infra.
Fixes https://github.com/flutter/flutter/issues/155576
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [ ] All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel
on [Discord].
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
---------
Co-authored-by: Gray Mackall <mackall@google.com>
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_benchmark flutter_view_macos__start_up"
}
-->
Issue link: https://github.com/flutter/flutter/issues/159540
Co-authored-by: Chris Bracken <chris@bracken.jp>
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_benchmark hello_world_macos__compile"
}
-->
Issue link: https://github.com/flutter/flutter/issues/159542
Co-authored-by: Chris Bracken <chris@bracken.jp>
Update to the latest definitions of emulator configurations.
Specifically to past https://crrev.com/c/5908894 which added `gpu_mode:
"swangle_indirect"` to the api 35 proto. This should help with emulator
stability on versions past android 13.
Related to b/371020223
Related: https://github.com/flutter/engine/pull/56313
Research notes / my findings for how this system works at
go/flutter-android-ci-emulator
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide] and the [C++,
Objective-C, Java style guides].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I added new tests to check the change I am making or feature I am
adding, or the PR is [test-exempt]. See [testing the engine] for
instructions on writing and running engine tests.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I signed the [CLA].
- [x] All existing and new tests are passing.
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_x64 hot_mode_dev_cycle_macos_target__benchmark"
}
-->
Issue link: https://github.com/flutter/flutter/issues/158568
It would be nice to have somewhere to iterate and experiment with what we can do to either fix frequent timeout problems we have with `flutter build apk` (across platforms, to be clear, though I've just started with Linux) or get more information on why the crashes/timeouts happen.
Open to other ways to doing this (though preferably _not_ LED).
Reverts: flutter/flutter#158242
Initiated by: LouiseHsu
Reason for reverting: Flake rate is low, and the arm64 macs in the staging pools are having difficulties
Original PR Author: fluttergithubbot
Reviewed By: {LouiseHsu}
This change reverts the following previous change:
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_arm64_ios hot_mode_dev_cycle_ios__benchmark"
}
-->
Issue link: https://github.com/flutter/flutter/issues/158241
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_arm64_ios hot_mode_dev_cycle_ios__benchmark"
}
-->
Issue link: https://github.com/flutter/flutter/issues/158241
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_benchmark complex_layout_scroll_perf_macos__timeline_summary"
}
-->
Issue link: https://github.com/flutter/flutter/issues/158251
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Mac_benchmark animated_complex_opacity_perf_macos__e2e_summary"
}
-->
Issue link: https://github.com/flutter/flutter/issues/157423
I've noticed Linux tool_integration_tests 5_5 has come very close to timing out. See https://github.com/flutter/flutter/issues/155317 for more context, I am not 100% confident that increasing the subsharding here will help given that we just very recently increased subsharding from 4 to 5, but the results could be interesting regardless.
<details>
<summary> Pre-launch checklist </summary>
</details>
<!-- meta-tags: To be used by the automation script only, DO NOT MODIFY.
{
"name": "Linux_pixel_7pro service_extensions_test"
}
-->
Issue link: https://github.com/flutter/flutter/issues/157852
Using `Mac_arm64` would require these to go through a `bringup: true`
dance, which would require not running in presubmit. Instead, this PR
just adds the `cpu` property to pin to `arm64` without changing the task
name.
Towards https://github.com/flutter/flutter/issues/157758
Reverts: flutter/flutter#157032
Initiated by: gmackall
Reason for reverting: https://github.com/flutter/flutter/pull/157032#issuecomment-2436336078
Original PR Author: gmackall
Reviewed By: {reidbaker, bartekpacia}
This change reverts the following previous change:
I recently noticed the following log when building an app in verbose mode:
```
This Android Gradle plugin (8.1.0) was tested up to compileSdk = 33 (and compileSdkPreview = "UpsideDownCakePrivacySandbox").
You are strongly encouraged to update your project to use a newer
Android Gradle plugin that has been tested with compileSdk = 35.
```
It looks like AGP would like us to use a newer AGP version if we want to use compileSdk 35 (which we do). This pr upgrades the tests, in advance of updating the templates.
I recently noticed the following log when building an app in verbose mode:
```
This Android Gradle plugin (8.1.0) was tested up to compileSdk = 33 (and compileSdkPreview = "UpsideDownCakePrivacySandbox").
You are strongly encouraged to update your project to use a newer
Android Gradle plugin that has been tested with compileSdk = 35.
```
It looks like AGP would like us to use a newer AGP version if we want to use compileSdk 35 (which we do). This pr upgrades the tests, in advance of updating the templates.
We have a group of tests that verifies that we can build a flutter app across the range of AGP, Kotlin, and Gradle versions we claim to support. They currently only run on postsubmit.
This pr aims to make them run in presubmit as well, but only when changes are made to the templates (not just android templates, see https://github.com/flutter/flutter/pull/157196#discussion_r1807055791 as to why).
Allows applying of `include_flutter.groovy` via the `apply from:` syntax, which allows using a host app that is using the Gradle Kotlin DSL (the default these days when creating an Android app in AS).
Explanation: The `include_flutter.groovy` script is currently not able to be called by Kotlin gradle files, because it is [intended to be invoked with the following lines](https://docs.flutter.dev/add-to-app/android/project-setup#depend-on-the-modules-source-code):
```
setBinding(new Binding([gradle: this])) // new
evaluate(new File( // new
settingsDir.parentFile, // new
'flutter_module/.android/include_flutter.groovy' // new
))
```
`setBinding` isn't part of the Kotlin gradle DSL, and there isn't (that I can find) an easy Kotlin equivalent. If this binding isn't set, the reference to `gradle` in `include_flutter.groovy` is wrong, which breaks the script.
This PR modifies `include_flutter.groovy` to also support being invoked through the standard way of invoking a script via the Gradle Groovy/Kotlin DSLs, which is `apply from:` (or it's slightly different Kotlin syntax). The start of the script identifies which of the two approaches is being used by checking if the binding is set, and then initializes some variables differently depending on the case.
If we land this, I believe we should update the example Gradle files for both the `kts` and `groovy` cases to prefer the `apply from` syntax as I think this is the syntax most developers would be more familiar with already seeing in their Gradle files.
Tells our iOS bots to use the provisioning profile stored in the flutter_internal/mac/mobileprovision CIPD packages tagged `version:to_2025`.
This CIPD package contains an updated provisioning profile which supports both the current development signing certificate expiring in October 2024, and the updated signing cert expiring in August 2025.
CIPD packages can be seen at:
https://chrome-infra-packages.appspot.com/p/flutter_internal/mac/mobileprovision
This is a reland of #155052, except without the removal of the CIPD dependency from the devicelab bots. After landing the original, many devicelab bots (and only devicelab bots) started failing with signing errors. The devicelab bots *should* be getting their certs/provisioning profile via the devicelab Salt configuration server (Googlers, see: go/flutter-salt) but it's possible some tests are fetching it from the CIPD package instead. This new patch simply updates the CIPD package version wherever it was already present. We can attempt to remove the CIPD package dependency from devicelab bots in a followup PR, and switch those over to using the profile installed via Salt (these are the same profile).
Required for: https://g-issues.chromium.org/issues/366034566
Issue: https://github.com/flutter/flutter/issues/152888
Tells our iOS bots to use the provisioning profile stored in the flutter_internal/mac/mobileprovision CIPD packages tagged `version:to_2025`.
This CIPD package contains an updated provisioning profile which supports both the current development signing certificate expiring in October 2024, and the updated signing cert expiring in August 2025.
Also removes the CIPD dependency for devicelab tasks. On devicelab bots, the cert and provisioning profile are deployed via Salt, not via CIPD. We only need this dependency on the Chromium bots.
CIPD packages can be seen at:
https://chrome-infra-packages.appspot.com/p/flutter_internal/mac/mobileprovision
Required for: https://g-issues.chromium.org/issues/366034566
Issue: https://github.com/flutter/flutter/issues/152888
https://github.com/flutter/flutter/pull/151675 bumped module templates to AGP 8.1.
In doing so, I tried to work around a behavior change [that was new in AGP 8.0](https://developer.android.com/build/releases/past-releases/agp-8-0-0-release-notes):
> AGP 8.0 creates no SoftwareComponent by default. Instead AGP creates SoftwareComponents only for variants that are configured to be published using the publishing DSL.
by using AGP's publishing DSL to define which variants to publish in the module's ephemeral gradle files:
```
android.buildTypes.all {buildType ->
if (!android.productFlavors.isEmpty()) {
android.productFlavors.all{productFlavor ->
android.publishing.singleVariant(productFlavor.name + buildType.name.capitalize()) {
withSourcesJar()
withJavadocJar()
}
}
} else {
android.publishing.singleVariant(buildType.name) {
withSourcesJar()
withJavadocJar()
}
}
}
```
The problem is that this doesn't get applied to the plugin projects used by the module, so if a module uses any plugin it breaks. This PR fixes that by applying similar logic, but to each project (not just the module's project).
Tested manually with https://github.com/gmackall/GrayAddToApp, and also re-enabled an old test that tested this use case as a part of the PR.
Fixes: https://github.com/flutter/flutter/issues/154371
This test expects to run on 32-bit hardware, but the devicelab no longer
has 32-bit Android hardware. In particular, the test contains hand-coded
32-bit arm assembly for reading the stack pointer from Dart code via
`dart:ffi`.
This test was added in response to a framework change that caused stack
frames to be larger than expected while building widgets, leading apps
to crash with StackOverflow exceptions unexpectedly. Since then (>3
years ago) this test has not prevented any similar issue, so I believe
deleting it rather than fixing it is a better use of resources.
These same tests but running against API level 34 emulators are remaining as presubmit tests. The API level 35 tests should stay in staging while the emulator stabilizes. So we can collect flake data and continue to debug.
Related: https://github.com/flutter/flutter/issues/153445
These VMs are relatively inexpensive, so there's no reason not to
continue running on the API level 34 image while we work out how to deal
with the apparent instability in the 35 images.
Related: https://github.com/flutter/flutter/issues/152684
These tests were moved to staging when they were flaking at a high rate.
I landed a couple of changes to fix unrelated issues that were masking
the underlying cause. However, these tests are no longer flaking at the
same rate in staging as they were in prod. So, I am moving these back to
prod and will monitor the flake rate.
Related https://github.com/flutter/flutter/issues/152684
In "build tests", the `properties` of the build specify the
configuration of the coordinator, and the `drone_dimensions` specify the
configuration of the devicelab bot. This PR should therefore fix the
infra errors on the `Linux_build_test` tests caused by my last PR at
https://github.com/flutter/flutter/pull/152756. (Unfortunately, a simple
revert won't work since the MotoG4s have already been removed from the
Linux hosts.)
Sets up tests that verify we can build a fresh counter app across our Gradle/AGP/Kotlin support range.
Post submit only, because the suite takes ~30 minutes to run, and I expect it to be _somewhat_ rare that we break only one of these versions (and therefore it doesn't get caught by existing presubmits).
For currently unknown reasons, this test is failing at a rate >10% but only in presubmit. This is blocking many engine -> framework rolls, requiring a lot of manual intervention from the sheriff.
https://github.com/flutter/flutter/issues/150642
