AutoRoll Production Manual

General information about the AutoRoller is available in the README.

The AutoRoller requires that gitcookies are in metadata under the key gitcookies_$INSTANCE_NAME.

Alerts

autoroll_failed

The most recent DEPS roll attempt failed. This is usually due to a change in the child repo which is incompatible with the parent and requires some investigation into which bots failed and why. Fixing this usually requires a commit to the child repo, either a revert or a fix. This alert is only enabled for Skia.

no_rolls_24h

There have been no successful rolls landed in the last 24 hours. This alert assumes that at least one commit has landed in the child repo in the last 24 hours; if that is not the case, then this alert can be safely ignored. This alert is only enabled for Skia.

http_latency

The frontend server is taking too long to respond. Check the logs for any obvious cause; if load is high, we may need to bring up more replicas.

error_rate

The AutoRoll server on the given host is logging errors at a higher-than-normal rate. This warrants investigation in the logs. The most common cause of this alert is transient errors while communicating with Git or Gerrit. In that case, there is not much which can be done other than to silence the alert and hope that things improve. If the failure persists, contact the current on-call for the Git service.

If this happened on the Skia->Flutter roller then also take a look at the flutter_license_script_failure section below.

flutter_license_script_failure

The Skia->Flutter roller has failed due to errors from Flutter's license script. Inform the Skia Gardener that the Skia->Flutter roller is failing the license script and that you are investigating.

Take a look at the cloud logs of the roller here. Failures due to license script errors typically look like this: “Failed to transition from “idle” to “active”: Error when running pre-upload step: Error when running dart license script: Command exited with exit status 1:”...

If the license script error looks unrelated to Skia (example), then file a bug to the Flutter team via Github issues with log snippets. Informing liyuqian@ about the issue might expedite the fix.

If the error looks related to Skia, then take a look at the recent unrolled changes to see if you can identify which change caused the license script to fail. Sometimes adding a new directory in third_party without a LICENSE file can cause the script to fail (example). Sometimes a typo in license headers can cause the script to fail (example).

If you need to run the license scripts manually on a clean local checkout, then follow these steps-

  • Checkout the necessary repos:
    • git clone https://github.com/flutter/engine
    • cd engine
    • echo """ solutions = [ { ‘managed’: False, ‘name’: ‘src/flutter’, ‘url’: ‘git@github.com:flutter/engine.git’, ‘custom_deps’: {}, ‘deps_file’: ‘DEPS’, ‘safesync_url’: '', }, ] """ > .gclient
    • gclient sync
  • cd src/flutter
  • Change the Skia rev in the DEPS here if necessary.
  • Run the license scripts:
    • cd tools/licenses
    • ../../../third_party/dart/tools/sdks/dart-sdk/bin/pub get
    • Run dart license script to create new licenses.
      • rm -rf ../../../out/licenses
      • ../../../third_party/dart/tools/sdks/dart-sdk/bin/dart lib/main.dart --src ../../.. --out ../../../out/licenses --golden ../../ci/licenses_golden
    • Copy from out dir to goldens dir. This is required for updating the release file in sky_engine/LICENSE.
      • cp ../../../out/licenses/licenses_skia ../../ci/licenses_golden/licenses_skia
    • Update ../../sky/packages/sky_engine/LICENSE using the dart license script.
      • ../../../third_party/dart/tools/sdks/dart-sdk/bin/dart lib/main.dart --release --src ../../.. --out ../../../out/licenses

Useful documentation links:

  • How to checkout flutter is documented here.
  • License script documentation is here.
  • The code for the pre-upload license step used by the autoroller is here.

Other Troubleshooting Tips

The autoroll page displays “Status: error” and doesn't upload CLs

Some issue is preventing the roller from running normally. You'll need to look through the logs for more information. Normally this is transient, but it can also be caused by mis-configuration of the roller, eg. the configured reviewer is not a committer.

Something is wrong, and I need to shut down the roller ASAP!

Setting the roller mode to “Stopped” should be enough in most cases. If not, you can use kubectl delete -f <file> to kill it.