flutter/packages/flutter_tools/test/integration.shard
Reid Baker 2383400fe0
Add Java-Gradle-AGP validation to flutter analyze (#123916)
https://github.com/flutter/flutter/issues/123917

Doc covering a broad set of issues related to android studio updating. 

https://docs.google.com/document/d/1hTXkjbUrBnXgu8NQsth1c3aEqo77rWoEj8CcsQ39wwQ/edit?pli=1#

Specifically this pr: 
- Adds new functions to find a projects AGP, Gradle and java versions,
and tests.
- Adds new functions that take versions and parse if the versions are
compatible with each other, and tests.
- Adds validator for `flutter analyze --suggestions` that evaluates the
java/gradle/agp versions and checks if they are compatible, and
integration test.
- Updates the version of gradle used by
dev/integration_tests/flutter_gallery/ to the minimum supported by java
18 so that the integration tests pass (It is unknown why the java
version is 18.9 instead of 11)
- Moves `isWithinVersionRange` to version.dart, and tests. 
- Adds FakeAndroidStudio to fakes to be used in multiple tests but does
not remove existing copies.

Metrics will be included as part of the definition of done for this bug
but not as part of this cl. It is already too big.

Known work still left in this pr: 
* Understand why analyze integration tests are failing. 


Example output if Java and gradle are not compatible: 
```
┌───────────────────────────────────────────────────────────────────┐
│ General Info                                                      │
│ [✓] App Name: espresso_example                                    │
│ [✓] Supported Platforms: android                                  │
│ [✓] Is Flutter Package: yes                                       │
│ [✓] Uses Material Design: yes                                     │
│ [✓] Is Plugin: no                                                 │
│ [✗] Java/Gradle/Android Gradle Plugin:                            │
│                                                                   │
│ Incompatible Java/Gradle versions.                                │
│                                                                   │
│ Java Version: 17.0.6, Gradle Version: 7.0.2                       │
│                                                                   │
│ See the link below for more information.                          │
│ https://docs.gradle.org/current/userguide/compatibility.html#java │
│                                                                   │
└───────────────────────────────────────────────────────────────────┘
```
Example output if Gradle and AGP are not compatible
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ General Info                                                                │
│ [✓] App Name: espresso_example                                              │
│ [✓] Supported Platforms: android                                            │
│ [✓] Is Flutter Package: yes                                                 │
│ [✓] Uses Material Design: yes                                               │
│ [✓] Is Plugin: no                                                           │
│ [✗] Java/Gradle/Android Gradle Plugin: Incompatible Gradle/AGP versions.    │
│                                                                             │
│ Gradle Version: 7.0.2, AGP Version: 7.4.2                                   │
│                                                                             │
│ Update gradle to at least "7.5".                                            │
│ See the link below for more information:                                    │
│ https://developer.android.com/studio/releases/gradle-plugin#updating-gradle │
│                                                                             │
│ Incompatible Java/Gradle versions.                                          │
│                                                                             │
│ Java Version: 17.0.6, Gradle Version: 7.0.2                                 │
│                                                                             │
│ See the link below for more information:                                    │
│ https://docs.gradle.org/current/userguide/compatibility.html#java           │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```
Example output if Java/Gradle/Agp are not compatible. 
```

┌─────────────────────────────────────────────────────────────────────────────┐
│ General Info                                                                │
│ [✓] App Name: espresso_example                                              │
│ [✓] Supported Platforms: android                                            │
│ [✓] Is Flutter Package: yes                                                 │
│ [✓] Uses Material Design: yes                                               │
│ [✓] Is Plugin: no                                                           │
│ [✗] Java/Gradle/Android Gradle Plugin: Incompatible Gradle/AGP versions.    │
│                                                                             │
│ Gradle Version: 7.0.2, AGP Version: 7.4.2                                   │
│                                                                             │
│ Update gradle to at least "7.5".                                            │
│ See the link below for more information:                                    │
│ https://developer.android.com/studio/releases/gradle-plugin#updating-gradle │
│                                                                             │
│ Incompatible Java/Gradle versions.                                          │
│                                                                             │
│ Java Version: 17.0.6, Gradle Version: 7.0.2                                 │
│                                                                             │
│ See the link below for more information:                                    │
│ https://docs.gradle.org/current/userguide/compatibility.html#java           │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

Commit messages
- Add function to gradle_utils.dart that gets the gradle version from
wrapper or system and add a test for each situation
- Add method to get agp version, add method to validate agp against
gradle version, update documentation, add tests for agp validation.
- Update dart doc for validateGradleAndAgp to describe where the info
came from and corner case behavior, create function to validate java and
gradle and hardcode return to false
- Fill out and test java gradle compatibility function in gradle_utils
- Hook up java gradle evaluateion to hasValidJavaGradleAgpVersions with
hardcoded java version
- Add java --version output parsing and tests
- Add getJavaBinary test
- Update comment in android_sdk for mac behavior with java_home -v

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [ ] All existing and new tests are passing.
2023-04-05 14:36:05 -04:00
..
debug_adapter [flutter_tools] Include mode in app.start event, and forward app.start to DAP clients (#121239) 2023-04-05 05:39:09 +00:00
test_data Fix Gradle warning in a freshly flutter createed Android project (#122290) 2023-03-21 23:41:49 +00:00
analyze_all_templates_test.dart Add lints test for all templates (#120526) 2023-03-26 11:10:23 +00:00
analyze_once_test.dart Bump lower Dart SDK constraints to 3.0 & add class modifiers (#122546) 2023-03-21 20:21:58 +00:00
analyze_size_test.dart fix devtool instructional messages after flutter build ... --analyze-size . Fixes #122229 (#122230) 2023-03-09 22:29:50 +00:00
analyze_suggestions_integration_test.dart Add Java-Gradle-AGP validation to flutter analyze (#123916) 2023-04-05 14:36:05 -04:00
android_e2e_api_test.dart Reland "Upgrade targetSdkVersion and compileSdkVersion to 33" (#116146) 2022-12-01 07:12:47 -08:00
android_plugin_compilesdkversion_mismatch_test.dart Add warning for bumping Android SDK version to match plugins (#92451) 2021-11-09 12:58:07 -08:00
android_plugin_example_app_build_test.dart Reland "Upgrade Gradle and AGP versions to 7.5/7.2 and migrate benchmarks+examples" #108472 (#108510) 2022-07-28 19:38:07 -07:00
android_plugin_ndkversion_mismatch_test.dart Add quotes to Gradle NDK version error message (#109026) 2022-08-07 11:46:08 +00:00
android_plugin_new_output_dir_test.dart let the plugin's output generated in flutter/.android/plugins_build_output/${androidPlugin.name} (#94645) 2022-09-14 15:47:16 +00:00
background_isolate_test.dart Fix hot-restart test by ensuring updated timestmamp is in the future. (#97247) 2022-01-25 14:18:33 -08:00
bash_entrypoint_test.dart [flutter_tools] shard out two integration tests we want to run on macOS arm64 (#101769) 2022-04-13 12:09:11 -07:00
break_on_framework_exceptions_test.dart Bump lower Dart SDK constraints to 3.0 & add class modifiers (#122546) 2023-03-21 20:21:58 +00:00
build_ios_config_only_test.dart fix noop toString() diagnostics (#108836) 2022-08-02 21:51:06 +00:00
build_macos_config_only_test.dart Add build macos --config-only option. (#118649) 2023-01-19 21:29:17 +00:00
cache_test.dart Remove redundant arguments passed to redirecting factory constructors (#115497) 2022-11-17 00:47:14 +00:00
command_output_test.dart flutter_tool: only enable wasm compile in master channel (#121755) 2023-03-02 22:21:50 +00:00
coverage_collection_test.dart migrate some file to null safety (#92957) 2021-12-16 11:39:12 -08:00
daemon_mode_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
debugger_stepping_test.dart Enable no_leading_underscores_for_local_identifiers (#96422) 2022-01-21 14:43:59 -08:00
deferred_components_test.dart remove unnecessary .toString() (#103226) 2022-05-06 16:04:13 -07:00
deprecated_gradle_settings_test.dart Fix typos (#121171) 2023-02-23 19:43:21 +00:00
downgrade_upgrade_integration_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
exit_code_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
expression_evaluation_test.dart Prevent tests from producing dill files alongside the test file (#115075) 2022-11-10 20:00:19 +00:00
flutter_attach_test.dart Stop serving Observatory by default (#122419) 2023-03-23 12:52:30 -04:00
flutter_build_android_app_project_builddir_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
flutter_build_apk_verbose_test.dart Correctly propagate verbosity to subtasks in flutter.gradle (#117897) 2023-01-04 09:54:21 +00:00
flutter_build_config_only_test.dart Create configOnly flag for android (#121904) 2023-03-08 19:46:52 +00:00
flutter_build_preview_sdk.dart Support string compileSdkVersion for preview android SDKs (#104662) 2022-05-31 13:08:09 -07:00
flutter_build_wasm_test.dart flutter_tool: only enable wasm compile in master channel (#121755) 2023-03-02 22:21:50 +00:00
flutter_build_windows_test.dart [Windows] Flow version information to the build output (#106145) 2022-06-27 10:45:56 -07:00
flutter_build_with_compilation_error_test.dart [tool][web] Pass invoker flag to dart2js. (#122344) 2023-03-11 03:24:34 +00:00
flutter_gen_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
flutter_run_test.dart [flutter_tools] Include mode in app.start event, and forward app.start to DAP clients (#121239) 2023-04-05 05:39:09 +00:00
flutter_run_with_error_test.dart Reland "Remove references to Observatory (#118577)" (#121606) 2023-02-28 11:57:04 -05:00
forbidden_imports_test.dart Revert "Allow Flutter golden file tests to be flaky (#114450)" (#114902) 2022-11-08 09:26:55 -08:00
gen_l10n_test.dart ICU Message Syntax Parser (#112390) 2022-11-05 10:26:46 -07:00
gradle_non_android_plugin_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
hot_reload_errors_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
hot_reload_test.dart Simplify null check. (#117026) 2023-01-03 20:51:48 +00:00
hot_reload_with_asset_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
lifetime_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
multidex_build_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
observatory_port_test.dart Reland "Remove references to Observatory (#118577)" (#121606) 2023-02-28 11:57:04 -05:00
overall_experience_test.dart [Focus] Add run key command to dump the focus tree (#123473) 2023-03-28 00:33:39 +00:00
plist_parser_test.dart Add macos project auto migration code for FlutterApplication (#122336) 2023-03-16 01:01:03 +00:00
README.md
shader_compiler_test.dart [flutter_tools] add uint compilation test (#115317) 2022-11-15 01:39:59 +00:00
single_widget_reload_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
stateless_stateful_hot_reload_test.dart Remove test that verifies we can switch to stateless (#120390) 2023-02-09 20:58:57 +00:00
template_manifest_test.dart Simplify null check. (#117026) 2023-01-03 20:51:48 +00:00
test_driver.dart [flutter_tools] Include mode in app.start event, and forward app.start to DAP clients (#121239) 2023-04-05 05:39:09 +00:00
test_test.dart Reland "Serve DevTools when running flutter test (#123607)" (#124058) 2023-04-04 19:44:14 -04:00
test_utils.dart replace some ._() constructors with class modifiers (#122765) 2023-03-23 12:29:18 -07:00
timeline_test.dart [flutter_tools] timeline_test.dart flaky (#116667) 2023-01-03 20:44:53 +00:00
tool_backend_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
unit_coverage_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
variable_expansion_windows_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
variable_expansion_windows.dart Enable avoid_print lint. (#91444) 2021-10-07 16:48:04 -07:00
vmservice_integration_test.dart Adds vmservices for getting iOS build options (#121736) 2023-03-06 22:37:55 +00:00
web_plugin_registrant_test.dart Manual Roll of Flutter Engine from 67254d6e4b03 to 8d83b98c55b3 (#116635) 2022-12-08 18:03:51 -08:00
xcode_backend_test.dart Reland "Remove references to Observatory (#118577)" (#121606) 2023-02-28 11:57:04 -05:00

Integration tests

These tests are not hermetic, and use the actual Flutter SDK. While they don't require actual devices, they run flutter_tester to test Dart VM and Flutter integration.

Use this command to run (from the flutter_tools directory):

../../bin/cache/dart-sdk/bin/pub run test test/integration.shard

You need to have downloaded the Dart SDK in your Flutter clone for this to work. Running ../../bin/flutter will automatically download it.

Coverage exclusion

These tests are expensive to run and do not give meaningful coverage information for the flutter tool (since they are black-box tests that run the tool as a subprocess, rather than being unit tests). For this reason, they are in a separate shard when running on continuous integration and are not run when calculating coverage.