JSFiddle is a recent service that was launched that demonstrates the above. See go/ for the server code and modules/ and pages/ for the front end.
go.skia.org/infra/go/sklog for logging.
Add flags to your main package like:
port = flag.String("port", ":8002", "HTTP service port (e.g., ':8002')") local = flag.Bool("local", false, "Running locally if true. As opposed to in production.") promPort = flag.String("prom_port", ":20000", "Metrics service address (e.g., ':10110')") resourcesDir = flag.String("resources_dir", "./dist", "The directory to find HTML, JS, and CSS files. If blank the current directory will be used.")
common.InitWithMust([opt], [opt]) in your main function.
go.skia.org/infra/go/login paired with
../../../infra-sk/modules/login.ts (Legacy Polymer apps use
go.skia.org/infra/go/webhook for authentication. When using OAuth, see the secrets section below for including client secrets.
http.Handler (many services use chi.NewRouter() with
go.skia.org/infra/go/httputils.LoggingGzipRequestResponse to provide monitoring and logging of HTTP requests and responses. Then, wrap it in
go.skia.org/infra/go/httputils.HealthzAndHTTPS to add an unlogged /healthz endpoint for use with GKE health monitoring and various HTTPS configuration.
go.skia.org/infra/go/httputils.DefaultClientConfig for HTTP clients, which provides several features:
Write your code with security in mind:
If you add any critical TODOs while you're coding, file a blocking bug for the issue.
If your application requires Puppeteer tests, it should be explicitly opted in by making any necessary changes to //puppeteer-tests/docker/run-tests.sh.
It is customary to have the following commands in a Makefile for the service.
build: Build a development version of the front end.
serve: Run the demo pages of the front end in a “watch” mode. This command is for primary development of front end pages.
core: Build the server components.
watch: Build a development version of the front end in watch mode. This command is for running the server, but also making changes to the front end.
release: Build a Docker container with the front end and backend parts in it (see below).
push: Depends on release, and then pushes to GKE using
Running apps in Docker makes deployment and local testing much much easier. It additionally allows integration with GKE. Some legacy apps are not yet run in Docker, but it is the goal to have everything on GKE+Docker.
Create a Dockerfile for your app in the root of the project folder (e.g.
jsfiddle/Dockerfile). If there are multiple services, put them in a named folder (e.g.
When choosing a base image, consider our light wrappers, found in
kube/*. For example,
kube/basealpine/Dockerfile which can be used by having
FROM gcr.io/skia-public/basealpine:3.9 as the first line in a Dockerfile.
We have a helper script for ‘installing’ an app into a Docker container,
bash/docker_build.sh. A call to this script is customarily put in a bash script which is called by
make release. See
jsfiddle/build_release for an example. To integrate docker_build.sh into the actual container, add a
COPY . / to copy the executable(s) and HTML/JS/CSS from the build context into the container. Legacy apps have a similar set-up, but for building a Debian package instead of a container.
It is customary to include an ENTRYPOINT and CMD with sensible defaults for the app. It's also a best practice to run the app as USER skia unless root is absolutely needed.
Putting all the above together, a bare-bones Dockerfile would look something like:
FROM gcr.io/skia-public/basealpine:3.9 COPY . / USER skia ENTRYPOINT ["/usr/local/bin/my_app_name"] CMD ["--port=:8000", "--resources_dir=/usr/local/share/my_app_name/"]
If your app needs access to a GCS bucket or other similar things, it is recommended you create a new service account for your app. See below for linking it into the container.
Use an existing
create-sa.sh script (e.g.
create-jsfiddle-sa.sh) and tweak the name, committing it into the app's root directory. Run this once to create the service account and create the secrets in GKE.
To run applications locally authenticated as yourself you can run:
gcloud auth application-default login
Which will place credentials at:
that will be picked up by the application.
If you wish to override that behavior and use a different set of credentials then set the Environment Variable
GOOGLE_APPLICATION_CREDENTIALS that points to a different file, such as a
key.json file for a specific service account.
Use of the Git binary itself is strongly discouraged unless it is unavoidable. Please consider an alternative:
gitsyncapp, which also sends PubSub messages for low-latency updates.
The following are valid reasons to use the Git binary itself:
If you do need Git for your app, use the
base-cipd Docker image, which includes a pinned version of Git (as well as other tools). Do not install Git via the package manager in your Docker image.
DESIGN.mdtypically has high level design structures (e.g. where is data stored, how do the pieces of software interact, etc).
PROD.mdhas an overview of the alerts and any other notes for maintaining the service.
app.yamlin k8s-config This controls how your app will be run in GKE. See these docs for more on the schema. Commit this, then run
pushk appnameto make the configuration active.
ports: - containerPort: 20000 name: prom
annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: 'true'
If you need finer grained control over how your pods are started and stopped that can be done by defining a PodDisruptionBudget. CockroachDB defines a PodDisruptionBudget and is a good example of such a budget.
foois the first argument to
spec: automountServiceAccountToken: false ... containers: - name: my-container ... volumeMounts: - name: my-app-sa mountPath: /var/secrets/google env: - name: GOOGLE_APPLICATION_CREDENTIALS value: /var/secrets/google/key.json ... volumes: - name: my-app-sa secret: secretName: my-app
spec: ... containers: - name: my-container ... volumeMounts: - name: skia-org-legacy-login-secrets mountPath: /etc/skia.org/ ... volumes: - name: skia-org-legacy-login-secrets secret: secretName: skia-org-legacy-login-secrets
It is possible to test your service/config without making it publicly visible.
kubectl apply -f app.yaml
kubectl get pods | grep [my-app]where my-app is the name of the new service.
kubectl port-forward my-app-7bf542629-jujzm 8083:8000
If you have simple routing needs, to make your service visible to the public add a
skia.org.domain annotation to your Service YAML with the domain name and deploy your updated yaml with
If your routing is more complicated you can skip the YAML annotation and write the routing rules directly into
Either way you then push a new version of nginx-skia-org:
cd infra/skfe make k8s_push
And watch that the new instances start running:
watch kubectl get pods -lapp=nginx-skia-org
Add prober rules to
probers.json in your application directory.
prober/go/prober/main.goto check the response body if desired.
Add additional stats gathering to your program using
go.skia.org/infra/go/metrics2, e.g. to ensure liveness/heartbeat of any background processes.
Add alert rules to alerts_public. The alerts may link to a production manual,
PROD.md, checked into the application source directory. Examples:
Some general metrics apply to all apps and may not need to be added explicitly for your application, such as: - Too many goroutines. - Free disk space on the instance and any attached disks. - This is also for alerts that apply to skia-public and skia-corp projects.
Check your alert rules by running
make validate in
promk/ (Legacy apps should run that commaind in
Then, after landing your valid alerts, run
make push_config && make push_config_corp in
promk/ (Again, legacy apps should do
make push in
Tell people about your new service.
Be prepared for bug reports. :-)
Some apps are set up to be continuously re-built and re-deployed on every commit of Skia or Skia Infra. To do that, see docker_pushes_watcher/README.md.
When standing up a new CockroachDB cluster the following extra steps need to be done:
resources: requests: cpu: '2' memory: '32Gi' ephemeral-storage: 100M
managed-prometheus-pod-monitoring.ymlbecause CockroachDB hosts metrics at a different path than the expected default of
# CockroachDB hosts metrics at a different path. apiVersion: monitoring.googleapis.com/v1 kind: ClusterPodMonitoring metadata: name: perf-cockroachdb-cluster-pod-monitoring spec: selector: matchLabels: app: perf-cockroachdb endpoints: - port: prom path: _status/vars interval: 15s targetLabels: fromPod: - from: app - from: appgroup
promso it gets scraped:
ports: - containerPort: 26257 name: grpc - containerPort: 8080 name: prom