This directory contains scripts to automate the creation, setup and deletion of
Skia's GCE buildbot machines.


Directory Contents
==================

- vm_config.sh
Instantiates constants that are used by the below scripts.

- vm_firewalls_create.sh
Creates the firewalls required for the Skia buildbots. Allows access from Google
machines to port 10115. Allows access to ports 10116 and 10117 to everyone.

- vm_create_masters.sh
Creates the master instances in the specified $ZONE_TAG.

- vm_create_slaves.sh
Creates the slaves instances in the specified $ZONE_TAG.

- vm_delete_masters.sh
Deletes the master instances from the specified $ZONE_TAG.

- vm_delete_slaves.sh
Deletes the slaves instances from the specified $ZONE_TAG.

- vm_setup_image.sh
Installs necessary packages on a vanilla GCE instance. This script setups up the
base image that is used by all Skia GCE instances.

- vm_setup_masters.sh
Sets up the base image with packages and directories necessary for the buildbot
masters.

- vm_setup_slaves.sh
Sets up the base image with packages and directories necessary for the buildbot
slaves.

- vm_setup_utils.sh
Utility functions used by vm_setup_master.sh and vm_setup_slaves.sh.

- vm_status.sh
Outputs the quota and all instances of google.com:skia-buildbots.

- /compute_engine_scripts/monitoring
Setup scripts to create/setup/delete skiamonitor.org, a Graphite server.


How to create a Master Root Disk
================================

Create it with (changing names and zones as required):
gcutil --project=google.com:skia-buildbots adddisk private-master-root-a \
    --source_image=skia-buildbot-image --size_gb=10 --zone=us-central1-a \
    --service_version=v1beta16


Migrating instances before PCRs
===============================

The instances should be moved using the scripts in this directory a few weeks
before a PCR is scheduled. We are allowed instances in the us-central1-a and
us-central1-b zones and will alternate between zone tags 'a' and 'b' during
PCRs.

The scripts should be executed in this order:
* ZONE_TAG=$NEW_ZONE ./vm_create_masters.sh
* ZONE_TAG=$OLD_ZONE ./vm_delete_slaves.sh
* ZONE_TAG=$NEW_ZONE ./vm_create_slaves.sh
* ZONE_TAG=$NEW_ZONE ./vm_setup_slaves.sh
* ZONE_TAG=$NEW_ZONE ./vm_copy_buildbot_history.sh  (Takes about 15-30 mins)
* ZONE_TAG=$OLD_ZONE ./vm_delete_masters.sh  (Do NOT delete the persistent boot disk when prompted to)
* ZONE_TAG=$OLD_ZONE ./vm_delete_rebaseline_servers.sh  (DO delete the persistent boot disk when prompted to)
* ZONE_TAG=$NEW_ZONE ./vm_create_rebaseline_servers.sh
* ZONE_TAG=$NEW_ZONE ./vm_setup_rebaseline_servers.sh

After the above scripts are run, update gce_compile_bots_zone in
global_variables.json to point to the $NEW_ZONE.


Handling Master VM crashes
==========================

On rare occasions Google Compute Engine VMs crash. If the master crashes then
please run the following (this section will be removed after the next PCR
migration because we will then have the ability to automatically recover from
crashes):
* ZONE_TAG=a bash vm_delete_masters.sh  (Do NOT delete the persistent boot disk when prompted to)
* ZONE_TAG=a bash vm_create_masters.sh

For more details, see https://code.google.com/p/skia/issues/detail?id=1676
('Make the buildbot master self-recover if it crashes')


Handling Slave VM crashes
=========================

On rare occasions Google Compute Engine VMs crash. We do not currently have
a way for slaves to automatically come up because they do not have root
persistent disks. You will have to recreate the specific slave(s) that crashed.
Please run the following:
* Edit vm_config.sh and leave only the VM_SLAVE_NAMES and SLAVE_IP_ADDRESSES you
  want to recreate.
* ZONE_TAG=a bash vm_delete_slaves.sh
* ZONE_TAG=a bash vm_create_slaves.sh
* ZONE_TAG=a bash vm_setup_slaves.sh

For more details, see https://code.google.com/p/skia/issues/detail?id=1678
('make GCE buildbot slaves self-recover on boot')