Android Compile Server Production Manual

General information about the Android compile server is available in the README.

Alerts

queue_too_long

The number of waiting compile tasks on Android Compile Server is too long. Take a look at the pending tasks here. Try to determine if current running tasks are taking too long by port forwarding to skia-corp with kubectl port-forward prometheus-0 9090 9090 and bring up graphs of sync times and compile times. Pending tasks can also be deleted if absolutely necessary here.

mirror_sync_failed

The mirror sync failed. This will likely cause all checkouts to also fail when syncing from the mirror. Fix this by logging into android-compile on skia-corp and running:

android_tree_broken

The Android Compile Bot thinks that the android tree is broken and is allowing Skia CLs to pass because the withpatch and nopatch builds are both red. Verify that the tree is really broken by looking at the android dashboard here. Also look at task logs in the datastore here.

infra_failure

Atleast one compile task failed due to an infra failure. Look for errors in the cloud logs.

If the error appears to be an Android infrastructure issue (eg: sync problems because repository is down) and it does not resolve soon, then make the bot an experimental bot in commit-queue.cfg. Add it back to the regular CQ after the infrastructure problem eventually resolves.