# Flutter's CI Best Practices [flutter.dev/to/ci-yaml](https://flutter.dev/to/ci-yaml) This is a supplemental resource to provide best practices for using Flutter's internal CI/CD system. It is not a comprehensive guide, but rather a collection of tips and tricks to be as efficient as possible. --- Table of Contents - [Flutter's CI Best Practices](#flutters-ci-best-practices) - ["This branch is out-of-date" is not (always) a failure](#this-branch-is-out-of-date-is-not-always-a-failure) - [Prefer the `auto-submit` label over pressing "merge"](#prefer-the-auto-submit-label-over-pressing-merge) - [Prefer the `revert` label over manual reverts of recent commits](#prefer-the-revert-label-over-manual-reverts-of-recent-commits) - [Prefer re-running test suites over re-triggering all tests](#prefer-re-running-test-suites-over-re-triggering-all-tests) - [Cost of Adding New Test Targets](#cost-of-adding-new-test-targets) - [Cost of Renaming/Resharding Tests](#cost-of-renamingresharding-tests) ## "This branch is out-of-date" is not (always) a failure This branch is out-of-date This message is automatically created by GitHub when at least one commit has merged into the ancestor branch of the PR. On Flutter's owned repositories, this is just a warning, and does not prevent merging. There are two edge cases where merging/rebasing is required or recommended: 1. **When the commit your PR was branched from is causing a test to fail**. If a forward fix or revert has since been merged into the ancestor branch, it is recommended to rebase your PR to pick up the fix. 2. **When the commit your PR was branched from is more than a few days old**. If the commit is more than a few days old, it is recommended to rebase your PR in order to make sure that the tests are run against the latest code. ## Prefer the `auto-submit` label over pressing "merge" Adding the auto-submit label The `auto-submit` label is a special label that can be added to a PR to be automatically merged (or queued for merging, in `flutter/flutter`'s case) when the tests are passing. This is the preferred way to merge PRs. If one or more tests fail, and you're confident that the failure is not related to your PR (i.e. is a flake), see [prefer re-running test suites over re-triggering all tests](#prefer-re-running-test-suites-over-re-triggering-all-tests). ## Prefer the `revert` label over manual reverts of recent commits Adding the revert label The `revert` label is a special label that can be added to a PR to revert the commit(s) that are causing a CI failure, and is the preferred way to revert recently merged commits as it bypasses most testing to get the revert landed as fast as possible. Before using this label, You'll need to add a comment on the PR: ```md reason for revert: This breaks the build, see XYZ link. ``` Please coordinate with the Google team that administrates the repository before using the `revert` label, as it may not be appropriate to use in all cases, and could cause confusion if used incorrectly. ## Prefer re-running test suites over re-triggering all tests Failing tests with a re-run button Each full test run in `flutter/flutter` uses between 100 and 200 virtual machines, and takes between 30 and 60 minutes to complete. This is a lot of resources, and should be avoided if possible. Pushing a new commit, or using the built-in merge button in GitHub, will re-trigger all tests in the PR. While sometimes this is necessary, it is a lot more expensive than re-running a single test suite or even a few test suites. ## Cost of Adding New Test Targets The Flutter team has a finite amount of resources to run tests, and adding new test targets, while incrementally useful, can add up to a lot of waiting time for the team. As of 2025/05/23, the presubmit pool for `flutter/flutter` has: - [301 Linux VMs](https://chromium-swarm.appspot.com/botlist?c=id&c=task&c=os&c=status&d=asc&f=os%3AUbuntu&f=pool%3Aluci.flutter.try) - [131 ARM64 Mac VMs](https://chromium-swarm.appspot.com/botlist?f=cpu%3Aarm64&f=os%3AMac-14&f=pool%3Aluci.flutter.try) - [134 Windows VMs](https://chromium-swarm.appspot.com/botlist?f=os%3AWindows-10&f=pool%3Aluci.flutter.try) Resources for on-device testing (device lab) are even more limited. When you add a new test target, it will consume an entire VM for the time it takes to clone the repository, setup the test infrastructure, and run the test suite (often between 15 and 30 minutes, but longer for some integration tests). While it's important to have a good test suite, consider: - Does my test target have to run on multiple platforms? - Is there an existing test target that I can add my test to that is not close to the timeout limit? Consider consulting with the infrastruture team (`team-infra`) before adding a large number of new test targets, or to get advice on how to best create new test targets. ## Cost of Renaming/Resharding Tests _This advice only applies to the `flutter/flutter` repository._ Due to how the test infrastructure works, renaming or resharding tests can be very time-consuming, and should be avoided if possible, or carefully coordinated when required: - New shards must be initially added as `bringup: true`, which means that they will not run in presubmit, and do not trigger tree closures in postsubmit. - The shards will have to be moved to `bringup: false` once they are successfully running, which will require a second PR. - Deleting (or renaming) a shard at tip-of-tree (i.e. `master`) will cause the test to be silently skipped in all release candidate branches, and will require the `.ci.yaml` file to be cherry-picked into each release candidate branch to restore the test coverage. For example, imagine the following existing test configuration (`.ci.yaml`): ```yaml targets: - name: Linux web_tests_1_2 shard: web_tests subshard: "0" - name: Linux web_tests_2_2 shard: web_tests subshard: "1" ``` Behind the scenes, this creates two LUCI builders, one for each test target. Now imagine you want to add a third shard (effectively, renaming the targets): ```yaml targets: - name: Linux web_tests_1_3 shard: web_tests subshard: "0" - name: Linux web_tests_2_3 shard: web_tests subshard: "1" - name: Linux web_tests_3_3 shard: web_tests subshard: "2" ``` This will require the following steps: 1. Add 3 new shards to the `.ci.yaml` file, and set `bringup: true` for each of them. 2. Wait for the new shards to be successfully running. 3. Remove the old shards from the `.ci.yaml` file, and set `bringup: false` for each of them. 4. Cherry-pick the `.ci.yaml` file into each release candidate branch to restore the test coverage. This could require up to 4 PRs, and a lot of coordination with the release team.