flutter/packages/flutter_tools/test/integration.shard
Christopher Fujino 4ff505c861
[flutter_tools] more debugging for timeouts in break_on_framework_exceptions test (#125584)
Better debugging to investigate: https://github.com/flutter/flutter/issues/125241

When the `test/integration.shard/break_on_framework_exceptions_test.dart` test times out, log out verbose logging to give clues as to why it did not complete.

From one local run it looks like the test runner is failing to load a test file (when I checked the path locally, the file was there, and re-running the `flutter test ...` invocation succeeded--in that the app threw an exception):

```
14:12 +26 -1: breaks when StatefulWidget.build throws [E]
  Timed out launching `flutter test`
  package:matcher                                                        fail
  test/integration.shard/break_on_framework_exceptions_test.dart 623:11  _timeoutAfter.<fn>

250s            Spawning flutter [test, --disable-service-auth-codes, --machine, --start-paused, test/test.dart] in /tmp/flutter_break_on_framework_exceptions.GUJDAZ

251s <=stdout=  {"protocolVersion":"0.1.1","runnerVersion":null,"pid":25763,"type":"start","time":0}

<=stdout=  {"suite":{"id":0,"platform":"vm","path":"/tmp/flutter_break_on_framework_exceptions.GUJDAZ/test/test.dart"},"type":"suite","time":0}

<=stdout=  {"test":{"id":1,"name":"loading /tmp/flutter_break_on_framework_exceptions.GUJDAZ/test/test.dart","suiteID":0,"groupIDs":[],"metadata":{"skip":false,"skipReason":null},"line":null,"column":null,"url":null},"type":"testStart","time":0}

258s <=stdout=  {"testID":1,"error":"Exception: the Dart compiler exited unexpectedly.","stackTrace":"package:flutter_tools/src/base/common.dart 10:3  throwToolExit\npackage:flutter_tools/src/compile.dart 813:13    DefaultResidentCompiler._compile.<fn>\ndart:async/zone.dart 1391:47                     _rootRun\ndart:async/zone.dart 1301:19                     _CustomZone.run\ndart:async/zone.dart 1209:7                      _CustomZone.runGuarded\ndart:async/stream_impl.dart 392:13               _BufferingStreamSubscription._sendDone.sendDone\ndart:async/stream_impl.dart 402:7                _BufferingStreamSubscription._sendDone\ndart:async/stream_impl.dart 291:7                _BufferingStreamSubscription._close\ndart:async/stream_transformers.dart 87:11        _SinkTransformerStreamSubscription._close\ndart:async/stream_transformers.dart 21:11        _EventSinkWrapper.close\ndart:convert/string_conversion.dart 241:11       _StringAdapterSink.close\ndart:convert/line_splitter.dart 141:11           _LineSplitterSink.close\ndart:async/stream_transformers.dart 132:24       _SinkTransformerStreamSubscription._handleDone\ndart:async/zone.dart 1391:47                     _rootRun\ndart:async/zone.dart 1301:19                     _CustomZone.run\ndart:async/zone.dart 1209:7                      _CustomZone.runGuarded\ndart:async/stream_impl.dart 392:13               _BufferingStreamSubscription._sendDone.sendDone\ndart:async/stream_impl.dart 402:7                _BufferingStreamSubscription._sendDone\ndart:async/stream_impl.dart 291:7                _BufferingStreamSubscription._close\ndart:async/stream_transformers.dart 87:11        _SinkTransformerStreamSubscription._close\ndart:async/stream_transformers.dart 21:11        _EventSinkWrapper.close\ndart:convert/string_conversion.dart 241:11       _StringAdapterSink.close\ndart:convert/string_conversion.dart 295:20       _Utf8ConversionSink.close\ndart:convert/chunked_conversion.dart 78:18       _ConverterStreamEventSink.close\ndart:async/stream_transformers.dart 132:24       _SinkTransformerStreamSubscription._handleDone\ndart:async/zone.dart 1391:47                     _rootRun\ndart:async/zone.dart 1301:19                     _CustomZone.run\ndart:async/zone.dart 1209:7                      _CustomZone.runGuarded\ndart:async/stream_impl.dart 392:13               _BufferingStreamSubscription._sendDone.sendDone\ndart:async/stream_impl.dart 402:7                _BufferingStreamSubscription._sendDone\ndart:async/stream_impl.dart...

<=stdout=  {"testID":1,"error":"Failed to load \"/tmp/flutter_break_on_framework_exceptions.GUJDAZ/test/test.dart\": Compilation failed for testPath=/tmp/flutter_break_on_framework_exceptions.GUJDAZ/test/test.dart","stackTrace":"","isFailure":false,"type":"error","time":7518}

<=stdout=  {"testID":1,"result":"error","skipped":false,"hidden":false,"type":"testDone","time":7521}

<=stdout=  {"count":1,"time":7526,"type":"allSuites"}

<=stdout=  {"success":false,"type":"done","time":7529}

259s            Process exited (1)

371s            Expecting test.startedProcess event
[+    95] <=stdout=  {"suite":{"id":0,"platform":"vm","path":"/tmp/flutter_break_on_framework_exceptions.GUJDAZ/test/test.dart"},"type":"suite","time":0}
[+    95] <=stdout=  {"test":{"id":1,"name":"loading /tmp/flutter_break_on_framework_exceptions.GUJDAZ/test/test.dart","suiteID":0,"groupIDs":[],"metadata":{"skip":false,"skipReason":null},"line":null,"column":null,"url":null},"type":"testStart","time":0}
[+  7600] <=stdout=  {"testID":1,"error":"Exception: the Dart compiler exited unexpectedly.","stackTrace":"package:flutter_tools/src/base/common.dart 10:3  throwToolExit\npackage:flutter_tools/src/compile.dart 813:13    DefaultResidentCompiler._compile.<fn>\ndart:async/zone.dart 1391:47                     _rootRun\ndart:async/zone.dart 1301:19                     _CustomZone.run\ndart:async/zone.dart 1209:7                      _CustomZone.runGuarded\ndart:async/stream_impl.dart 392:13               _BufferingStreamSubscription._sendDone.sendDone\ndart:async/stream_impl.dart 402:7                _BufferingStreamSubscription._sendDone\ndart:async/stream_impl.dart 291:7                _BufferingStreamSubscription._close\ndart:async/stream_transformers.dart 87:11        _SinkTransformerStreamSubscription._close\ndart:async/stream_transformers.dart 21:11        _EventSinkWrapper.close\ndart:convert/string_conversion.dart 241:11       _StringAdapterSink.close\ndart:convert/line_splitter.dart 141:11           _LineSplitterSink.close\ndart:async/stream_transformers.dart 132:24       _SinkTransformerStreamSubscription._handleDone\ndart:async/zone.dart 1391:47                     _rootRun\ndart:async/zone.dart 1301:19                     _CustomZone.run\ndart:async/zone.dart 1209:7                      _CustomZone.runGuarded\ndart:async/stream_impl.dart 392:13               _BufferingStreamSubscription._sendDone.sendDone\ndart:async/stream_impl.dart 402:7                _BufferingStreamSubscription._sendDone\ndart:async/stream_impl.dart 291:7                _BufferingStreamSubscription._close\ndart:async/stream_transformers.dart 87:11        _SinkTransformerStreamSubscription._close\ndart:async/stream_transformers.dart 21:11        _EventSinkWrapper.close\ndart:convert/string_conversion.dart 241:11       _StringAdapterSink.close\ndart:convert/string_conversion.dart 295:20       _Utf8ConversionSink.close\ndart:convert/chunked_conversion.dart 78:18       _ConverterStreamEventSink.close\ndart:async/stream_transformers.dart 132:24...

Expecting test.startedProcess event is taking longer than usual...
```
2023-04-27 18:37:59 +00:00
..
debug_adapter [flutter_tools] Include mode in app.start event, and forward app.start to DAP clients (#121239) 2023-04-05 05:39:09 +00:00
test_data Bump the default minSdkVersion to 19 (#125515) 2023-04-27 17:26:28 +00:00
analyze_all_templates_test.dart Add lints test for all templates (#120526) 2023-03-26 11:10:23 +00:00
analyze_once_test.dart Bump lower Dart SDK constraints to 3.0 & add class modifiers (#122546) 2023-03-21 20:21:58 +00:00
analyze_size_test.dart fix devtool instructional messages after flutter build ... --analyze-size . Fixes #122229 (#122230) 2023-03-09 22:29:50 +00:00
analyze_suggestions_integration_test.dart Add Java-Gradle-AGP validation to flutter analyze (#123916) 2023-04-05 14:36:05 -04:00
android_e2e_api_test.dart Reland "Upgrade targetSdkVersion and compileSdkVersion to 33" (#116146) 2022-12-01 07:12:47 -08:00
android_gradle_java_version_test.dart i123643 print java version gradle (#123644) 2023-04-13 18:13:58 -04:00
android_gradle_print_build_variants_test.dart Add vmservice for android build options (#123034) 2023-04-18 18:16:09 +00:00
android_plugin_compilesdkversion_mismatch_test.dart Add warning for bumping Android SDK version to match plugins (#92451) 2021-11-09 12:58:07 -08:00
android_plugin_example_app_build_test.dart Reland "Upgrade Gradle and AGP versions to 7.5/7.2 and migrate benchmarks+examples" #108472 (#108510) 2022-07-28 19:38:07 -07:00
android_plugin_ndkversion_mismatch_test.dart Add quotes to Gradle NDK version error message (#109026) 2022-08-07 11:46:08 +00:00
android_plugin_new_output_dir_test.dart let the plugin's output generated in flutter/.android/plugins_build_output/${androidPlugin.name} (#94645) 2022-09-14 15:47:16 +00:00
background_isolate_test.dart Fix hot-restart test by ensuring updated timestmamp is in the future. (#97247) 2022-01-25 14:18:33 -08:00
bash_entrypoint_test.dart [flutter_tools] shard out two integration tests we want to run on macOS arm64 (#101769) 2022-04-13 12:09:11 -07:00
break_on_framework_exceptions_test.dart [flutter_tools] more debugging for timeouts in break_on_framework_exceptions test (#125584) 2023-04-27 18:37:59 +00:00
build_ios_config_only_test.dart fix noop toString() diagnostics (#108836) 2022-08-02 21:51:06 +00:00
build_macos_config_only_test.dart Add build macos --config-only option. (#118649) 2023-01-19 21:29:17 +00:00
cache_test.dart Remove redundant arguments passed to redirecting factory constructors (#115497) 2022-11-17 00:47:14 +00:00
command_output_test.dart flutter_tool: only enable wasm compile in master channel (#121755) 2023-03-02 22:21:50 +00:00
coverage_collection_test.dart migrate some file to null safety (#92957) 2021-12-16 11:39:12 -08:00
daemon_mode_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
debugger_stepping_test.dart Enable no_leading_underscores_for_local_identifiers (#96422) 2022-01-21 14:43:59 -08:00
deferred_components_test.dart remove unnecessary .toString() (#103226) 2022-05-06 16:04:13 -07:00
deprecated_gradle_settings_test.dart Fix typos (#121171) 2023-02-23 19:43:21 +00:00
downgrade_upgrade_integration_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
exit_code_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
expression_evaluation_test.dart Prevent tests from producing dill files alongside the test file (#115075) 2022-11-10 20:00:19 +00:00
flutter_attach_test.dart Stop serving Observatory by default (#122419) 2023-03-23 12:52:30 -04:00
flutter_build_android_app_project_builddir_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
flutter_build_apk_verbose_test.dart Correctly propagate verbosity to subtasks in flutter.gradle (#117897) 2023-01-04 09:54:21 +00:00
flutter_build_config_only_test.dart Create configOnly flag for android (#121904) 2023-03-08 19:46:52 +00:00
flutter_build_preview_sdk.dart Support string compileSdkVersion for preview android SDKs (#104662) 2022-05-31 13:08:09 -07:00
flutter_build_wasm_test.dart flutter_tool: only enable wasm compile in master channel (#121755) 2023-03-02 22:21:50 +00:00
flutter_build_windows_test.dart [Windows] Flow version information to the build output (#106145) 2022-06-27 10:45:56 -07:00
flutter_build_with_compilation_error_test.dart [tool][web] Pass invoker flag to dart2js. (#122344) 2023-03-11 03:24:34 +00:00
flutter_gen_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
flutter_run_test.dart [flutter_tools] Include mode in app.start event, and forward app.start to DAP clients (#121239) 2023-04-05 05:39:09 +00:00
flutter_run_with_error_test.dart Reland "Remove references to Observatory (#118577)" (#121606) 2023-02-28 11:57:04 -05:00
forbidden_imports_test.dart Revert "Allow Flutter golden file tests to be flaky (#114450)" (#114902) 2022-11-08 09:26:55 -08:00
gen_l10n_test.dart ICU Message Syntax Parser (#112390) 2022-11-05 10:26:46 -07:00
gradle_non_android_plugin_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
hot_reload_errors_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
hot_reload_test.dart Simplify null check. (#117026) 2023-01-03 20:51:48 +00:00
hot_reload_with_asset_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
lifetime_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
multidex_build_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
observatory_port_test.dart Reland "Remove references to Observatory (#118577)" (#121606) 2023-02-28 11:57:04 -05:00
overall_experience_test.dart [flutter_tools] make overall_experience_test resilient to extraneous pub output (#125172) 2023-04-20 17:31:58 +00:00
plist_parser_test.dart Add macos project auto migration code for FlutterApplication (#122336) 2023-03-16 01:01:03 +00:00
README.md Update usage of standalonepub executable in flutter_tools testing docs (#124898) 2023-04-17 23:34:23 +00:00
shader_compiler_test.dart [flutter_tools] add uint compilation test (#115317) 2022-11-15 01:39:59 +00:00
single_widget_reload_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
stateless_stateful_hot_reload_test.dart Remove test that verifies we can switch to stateless (#120390) 2023-02-09 20:58:57 +00:00
template_manifest_test.dart Simplify null check. (#117026) 2023-01-03 20:51:48 +00:00
test_driver.dart [flutter_tools] more debugging for timeouts in break_on_framework_exceptions test (#125584) 2023-04-27 18:37:59 +00:00
test_test.dart Reland "Serve DevTools when running flutter test (#123607)" (#124058) 2023-04-04 19:44:14 -04:00
test_utils.dart replace some ._() constructors with class modifiers (#122765) 2023-03-23 12:29:18 -07:00
timeline_test.dart [flutter_tools] timeline_test.dart flaky (#116667) 2023-01-03 20:44:53 +00:00
tool_backend_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
unit_coverage_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
variable_expansion_windows_test.dart [flutter_tools] migrate some integration tests to null safety (#103560) 2022-05-13 09:39:10 -07:00
variable_expansion_windows.dart Enable avoid_print lint. (#91444) 2021-10-07 16:48:04 -07:00
vmservice_integration_test.dart Adds vmservices for getting iOS build options (#121736) 2023-03-06 22:37:55 +00:00
web_plugin_registrant_test.dart Manual Roll of Flutter Engine from 67254d6e4b03 to 8d83b98c55b3 (#116635) 2022-12-08 18:03:51 -08:00
xcode_backend_test.dart Reland "Remove references to Observatory (#118577)" (#121606) 2023-02-28 11:57:04 -05:00

Integration tests

These tests are not hermetic, and use the actual Flutter SDK. While they don't require actual devices, they run flutter_tester to test Dart VM and Flutter integration.

Use this command to run (from the flutter_tools directory):

../../bin/cache/dart-sdk/bin/dart run test test/integration.shard

You need to have downloaded the Dart SDK in your Flutter clone for this to work. Running ../../bin/flutter will automatically download it.

Coverage exclusion

These tests are expensive to run and do not give meaningful coverage information for the flutter tool (since they are black-box tests that run the tool as a subprocess, rather than being unit tests). For this reason, they are in a separate shard when running on continuous integration and are not run when calculating coverage.