Push

A set of applications and conventions for making it easy to push new versions of applications to servers we manage. Key design targets are easy deploys and easy rollbacks with minimal disruption of our current infrastructure.

There are several components to the system, as seen below:

  +------------------------+
  |                        |
  | Build Release Package  |
  |                        |
  +-----------+------------+
              |
              |
              v
  +-----------------------+      +-------------------+
  |                       |      |                   |
  | Google Storage        |      |   Push Server     |
  |   gs://skia-push/     | <--> |                   |
  |                       |      |                   |
  |                       |      |                   |
  +-------------+---------+      +-------------------+
                |
                |
                v
     +-------------------+
     |                   |
     | +-----------------+-+
     | |                   |
     | |  +----------------+--+
     | |  |                   |
     | |  | GCE Instance      |
     +-+  |   running         |
       +--+   pulld client    |
          |                   |
          +-------------------+

Build Release Package

The process begins with building a Debian package that contains all the assets needed, including a systemd service file if you are deploying an application. Note that not only applications can be deployed this way, for example nginx config files could be deployed by this system also.

The script in ../bash/release.sh builds the debian package and uploads it to the correct spot in Google Storage.

Google Storage

There is a bucket solely for managing push images and configuration and it has the following top level structure:

gs://skia-push
gs://skia-push/debs/
gs://skia-push/server/

Under /debs/ the Debian images produced from the above step are stored. They are stored in this manner:

gs://skia-push/debs/{application name}/{uniquely named Debian package}.deb

Under /server/ there will be a single JSON file for each server that describes the packages that should be installed on that server. The package list is of the actual Debian image names held in /debs/.

gs://skia-push/server/{server name}.json

For example, gs://skia-push/server/skia-push.json looks like this:

[
  "pulld/pulld:jcgregorio@jcgregorio.cnc.corp.google.com:2014-12-09T19:05:02Z:7e0ff6059e653eb75a36efb621d0fd66d1ced433.deb",
  "push/push:jcgregorio@jcgregorio.cnc.corp.google.com:2014-12-09T18:45:59Z:2d9eea8b121f61ac8f1ab853fedc30fc092b3f70.deb",
  "logserver/logserver:jcgregorio@jcgregorio.cnc.corp.google.com:2014-12-09T18:15:56Z:52a8ec0da66d45a58274947080870b742404a92f.deb"
]

Pull Client

On every GCE instance that is managed by the Push Server there is a long running ‘pulld’ process that polls the gs://skia-push/server/{servername}.json file and looks for it to change. When it does change then any new Debian packages are downloaded from gs://skia-push/debs and installed.

GCE Instances

GCE instances that are created with a boot disk built from the ‘skia-pushable-base’ snapshot are fully ready to be push clients. New packages that are part of Debian can't be installed by the push process, for example, nginx. Installing such packages on an instance is done by using a startup script.

Push Server

A web application that makes it easy to choose from the existing releases of each package and choose them to be deployed to each server. It also has one config file that determines which applications are allowed to be deployed in which servers.

skiapush.conf

The server reads all the information from gs://skia-push to build its UI, and on user selection of application versions to deploy it will read and write back a modified file to gs://skia-push/servers/{servername}.json. The push server will also write a new value into some instance level metadata, which will trigger that instance's pulld to look for updated packages. As a failsafe, pulld will look for new packages every 5 minutes.