Delete continuous-deploy app and related alerts.
It has not been working for about ~3 weeks now (since the k8s-config
migration).
It's replacement docker-pushers-watcher has been working and has been
deploying apps continuously. Documentation is at
https://skia.googlesource.com/buildbot/+/master/docker_pushes_watcher/README.md
Bug: skia:9514
Change-Id: Id8b378eb1d8c8a7215a7a1e84fef8b46bb309d31
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/262059
Reviewed-by: Ravi Mistry <rmistry@google.com>
Commit-Queue: Ravi Mistry <rmistry@google.com>
diff --git a/docker_pushes_watcher/README.md b/docker_pushes_watcher/README.md
index 5f84799..fa2ebd6 100644
--- a/docker_pushes_watcher/README.md
+++ b/docker_pushes_watcher/README.md
@@ -45,6 +45,7 @@
* [push_apps_from_skia_image](https://skia.googlesource.com/buildbot/+/master/infra/bots/task_drivers/push_apps_from_skia_image/)
* [push_apps_from_wasm_image](https://skia.googlesource.com/buildbot/+/master/infra/bots/task_drivers/push_apps_from_wasm_image/)
* [push_apps_from_skia_wasm_images](https://skia.googlesource.com/buildbot/+/master/infra/bots/task_drivers/push_apps_from_skia_wasm_images/)
+
The following bots were created using the above task drivers:
* [Housekeeper-PerCommit-PushAppsFromSkiaDockerImage](https://status.skia.org/repo/skia?commit_label=author&filter=search&search_value=PushAppsFromSkiaDockerImage) bot to create and push docker images for fiddler, debugger and api apps.
* [Housekeeper-PerCommit-PushAppsFromWASMDockerImage](https://status.skia.org/repo/skia?commit_label=author&filter=search&search_value=PushAppsFromWASMDockerImage) bot to create and push docker images for jsfiddle, skottie and particle apps.
diff --git a/fiddlek/README.md b/fiddlek/README.md
index dda3bb5..dc4af88 100644
--- a/fiddlek/README.md
+++ b/fiddlek/README.md
@@ -21,9 +21,8 @@
Continuous Deployment of fiddler
--------------------------------
-The fiddler image is continuously deployed as GCP Container Builder succeeds
-in building new gcr.io/skia-public/fiddler images. The app that does the
-deployment is infra/kube/go/continuous-deploy.
+The fiddler image is continuously deployed as new Skia commits come in. See
+documentation at [docker_pushes_watcher/README.md](../docker_pushes_watcher/README.md).
Node Pool
---------
diff --git a/kube/Makefile b/kube/Makefile
index a9d0b08..15e4fba 100644
--- a/kube/Makefile
+++ b/kube/Makefile
@@ -1,4 +1,4 @@
-.PHONY: build test pushk release_configmap_reload release_auth_push release_continuous_deploy
+.PHONY: build test pushk release_configmap_reload release_auth_push
build:
go install ./go/...
@@ -14,10 +14,6 @@
CGO_ENABLED=0 GOOS=linux go install -a ./go/auth-proxy
./build_auth_proxy_release
-release_continuous_deploy:
- CGO_ENABLED=0 GOOS=linux go install -a ./go/continuous-deploy ./go/pushk
- ./build_continuous_deploy_release
-
release_basealpine:
docker pull alpine:3.8
./build_basealpine_release
@@ -27,8 +23,8 @@
./build_reposync_release
pushk repo-sync
-push: release_auth_proxy release_continuous_deploy release_configmap_reload
- pushk auth-proxy configmap-reload continuous-deploy
+push: release_auth_proxy release_configmap_reload
+ pushk auth-proxy configmap-reload
pushk_authproxy: release_auth_proxy
pushk auth-proxy
diff --git a/kube/README.md b/kube/README.md
index 404d1a5..5c5cc9a 100644
--- a/kube/README.md
+++ b/kube/README.md
@@ -36,85 +36,6 @@
When you run pushk it will update the images for all the clusters and then run
`kubectl apply` for each file and for each cluster.
-## Continuous Deployment
-
-Continuous deployment uses three bits on infrastructure:
-
- 1. The same build_foo config files that are used when building from the desktop.
- 2. [GCP Container Builder](https://cloud.google.com/container-builder/).
- 3. The continuous-deploy application.
-
-To do continuous deployment for any application that depends upon the Skia
-repo, such as fiddler, you will need to add two new steps and a new image
-to the `docker/cloudbuild.yaml` file in the Skia repo.
-
-For example:
-
-```
- - name: 'gcr.io/skia-public/infra:prod'
- dir: '/home/skia/golib/src/go.skia.org/infra/fiddlek'
- env:
- - 'ROOT=/workspace/__staging'
- - 'SKIP_BUILD=1'
- args: ['./build_fiddler_release']
- timeout: 600s
-```
-
-As a note, the `gcr.io/skia-public/infra:prod` image that this cloud builder step runs on
-is re-created on each commit to this repo (see [docker/Dockerfile](../docker/Dockerfile)).
-
-This sets the working directory to the one for the app we want to build, then
-runs the `build_fiddler_release` script, but note that we have set the `ROOT`
-and `SKIP_BUILD` environment variables so that the script only builds the
-application and copies the files into the directory w/o calling docker on that
-directory. Also note that we are putting our work product under the /workspace
-directory which is preserved between steps by GCP Container Builder.
-
-Also note that we could add a Makefile target that runs all tests and then
-runs `build_fiddler_release` and calls make instead of `build_fiddler_release`
-directly, which is the preferred method.
-
-Then we add a second step that runs docker on that container to build the
-image:
-
-```
- - name: 'gcr.io/cloud-builders/docker'
- args: ['build', '-t', 'gcr.io/$PROJECT_ID/fiddler:$COMMIT_SHA', '/workspace/__staging']
- timeout: 600s
-```
-
-See [Substituting Variable Values](https://cloud.google.com/container-builder/docs/configuring-builds/substitute-variable-values)
-for more details on `$PROJECT_ID` and `$COMMIT_SHA`.
-
-Finally we add the new image to the list of images that get pushed to
-`gcr.io`:
-
-```
-images:
- - 'gcr.io/$PROJECT_ID/fiddler:$COMMIT_SHA'
- - 'gcr.io/$PROJECT_ID/skia-release:prod'
-
-```
-
-The continuous-deploy application runs in skia-public and listens for PubSub
-messages from GCP Container Builder that is has successfully completed a build
-and in that message it includes a list of images it has uploaded. Update the
-`continuous-deploy.yaml` file to include the short name of the image you want
-continuously deployed as a command-line argument:
-
-```
-containers:
- - name: continuous-deploy
- image: gcr.io/skia-public/continuous-deploy:2018-...
- args:
- - "--logtostderr"
- - "--prom_port=:20000"
- - "fiddler"
-```
-
-Since continuous-deploy runs `pushk`, all of these deployments will be
-recorded in the git repo for skia-public.
-
## Standing up a new cluster in a different project
1. Add a new `__skia_NNN` function to `clusters.sh`.
diff --git a/kube/build_continuous_deploy_release b/kube/build_continuous_deploy_release
deleted file mode 100755
index 3878c97..0000000
--- a/kube/build_continuous_deploy_release
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/bin/bash
-# Builds and uploads a container that proxies web traffic while enforcing IAP.
-set -x -e
-
-APPNAME=continuous-deploy
-
-# Copy files into the right locations in ${ROOT}.
-copy_release_files()
-{
-INSTALL="install -D --verbose"
-${INSTALL} --mode=644 -T ${APPNAME}/Dockerfile ${ROOT}/Dockerfile
-${INSTALL} --mode=755 -T ${GOPATH}/bin/${APPNAME} ${ROOT}/usr/local/bin/${APPNAME}
-${INSTALL} --mode=755 -T ${GOPATH}/bin/pushk ${ROOT}/usr/local/bin/pushk
-${INSTALL} --mode=644 -T ./clusters/config.json ${ROOT}/etc/continuous-deploy/config.json
-}
-
-source ../bash/docker_build.sh
diff --git a/kube/continuous-deploy/Dockerfile b/kube/continuous-deploy/Dockerfile
deleted file mode 100644
index 6891a23..0000000
--- a/kube/continuous-deploy/Dockerfile
+++ /dev/null
@@ -1,15 +0,0 @@
-FROM gcr.io/skia-public/basealpine:3.8
-
-USER root
-
-RUN apk --no-cache add curl git \
- && mkdir -p /usr/local/bin \
- && curl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl \
- && chmod +x /usr/local/bin/kubectl
-
-COPY . /
-
-USER skia
-
-ENTRYPOINT ["/usr/local/bin/continuous-deploy"]
-CMD ["--logtostderr", "fiddler"]
diff --git a/kube/create-continuous-deploy-sa.sh b/kube/create-continuous-deploy-sa.sh
deleted file mode 100755
index 59eb7fd..0000000
--- a/kube/create-continuous-deploy-sa.sh
+++ /dev/null
@@ -1,8 +0,0 @@
-#/bin/bash
-
-# Create the service account that has access to the skia-public-config repo
-# and export a key for it into the kubernetes cluster as a secret.
-
-set -e -x
-
-./secrets/add-service-account.sh skia-public skia-public skia-continuous-deploy "Continous deploy to skia-public."
\ No newline at end of file
diff --git a/kube/go/continuous-deploy/main.go b/kube/go/continuous-deploy/main.go
deleted file mode 100644
index 61c6ffa..0000000
--- a/kube/go/continuous-deploy/main.go
+++ /dev/null
@@ -1,180 +0,0 @@
-// continuous-deploy monitors pubsub events for the GCP Container Builder and
-// pushes updated images when successful images are built.
-
-package main
-
-import (
- "context"
- "encoding/json"
- "flag"
- "fmt"
- "os"
- "regexp"
- "strings"
-
- "cloud.google.com/go/pubsub"
- "go.skia.org/infra/go/auth"
- "go.skia.org/infra/go/common"
- "go.skia.org/infra/go/exec"
- "go.skia.org/infra/go/gitauth"
- "go.skia.org/infra/go/metrics2"
- "go.skia.org/infra/go/sklog"
- cloudbuild "google.golang.org/api/cloudbuild/v1"
- "google.golang.org/api/option"
-)
-
-// flags
-var (
- clusterConfig = flag.String("cluster_config", "", "Absolute filename of the config.json file.")
- local = flag.Bool("local", false, "Running locally if true. As opposed to in production.")
- project = flag.String("project", "skia-public", "The GCE project name.")
- promPort = flag.String("prom_port", ":20000", "Metrics service address (e.g., ':10110')")
-)
-
-var (
- parseImageName *regexp.Regexp
-)
-
-func Init() {
- parseImageName = regexp.MustCompile("^gcr.io/" + *project + "/([^:]+).*$")
-}
-
-// baseImageName returns "fiddler" from "gcr.io/skia-public/fiddler:foo".
-//
-// If the image name doesn't start with "gcr.io" and the project name then "" is returned.
-func baseImageName(s string) string {
- matches := parseImageName.FindStringSubmatch(s)
- if len(matches) != 2 {
- return ""
- } else {
- return matches[1]
- }
-}
-
-// imagesFromInfo parses the incoming PubSub Data 'b' as JSON and then returns
-// the full image names of all the images that match 'shortImageNames'.
-func imagesFromInfo(shortImageNames []string, buildInfo cloudbuild.Build) []string {
- imageNames := []string{}
- for _, im := range buildInfo.Results.Images {
- sklog.Infof("ImageName: %s", im.Name)
- // Is this one of the images we are pushing?
- for _, name := range shortImageNames {
- if baseImageName(im.Name) == name {
- imageNames = append(imageNames, im.Name)
- break
- }
- }
- }
- return imageNames
-}
-
-func main() {
- common.InitWithMust(
- "continuous-deploy",
- common.PrometheusOpt(promPort),
- )
- if len(flag.Args()) == 0 {
- sklog.Fatal("continuous-deploy must be passed in at least one package name to push.")
- }
- Init()
- sklog.Infof("Pushing to: %v", flag.Args())
- ctx := context.Background()
- ts, err := auth.NewDefaultTokenSource(*local, pubsub.ScopePubSub, "https://www.googleapis.com/auth/gerritcodereview")
- if err != nil {
- sklog.Fatal(err)
- }
- if !*local {
- _, err := gitauth.New(ts, "/tmp/git-cookie", true, "skia-continuous-deploy@skia-public.iam.gserviceaccount.com")
- if err != nil {
- sklog.Fatal(err)
- }
- }
- client, err := pubsub.NewClient(ctx, *project, option.WithTokenSource(ts))
- if err != nil {
- sklog.Fatal(err)
- }
- topic := client.Topic("cloud-builds")
- hostname, err := os.Hostname()
- if err != nil {
- sklog.Fatal(err)
- }
- subName := fmt.Sprintf("continuous-deploy-%s", hostname)
- sub := client.Subscription(subName)
- ok, err := sub.Exists(ctx)
- if err != nil {
- sklog.Fatalf("Failed checking subscription existence: %s", err)
- }
- if !ok {
- sub, err = client.CreateSubscription(ctx, subName, pubsub.SubscriptionConfig{
- Topic: topic,
- })
- if err != nil {
- sklog.Fatalf("Failed creating subscription: %s", err)
- }
- }
- pushk := "/usr/local/bin/pushk"
- if *local {
- pushk = "pushk"
- }
- shortImageNames := flag.Args()
-
- pubSubReceive := metrics2.NewLiveness("ci_pubsub_receive", nil)
- for {
- err := sub.Receive(ctx, func(ctx context.Context, msg *pubsub.Message) {
- msg.Ack()
- sklog.Infof("Status: %s", msg.Attributes["status"])
-
- var buildInfo cloudbuild.Build
- if err := json.Unmarshal(msg.Data, &buildInfo); err != nil {
- sklog.Errorf("Failed to decode: %s: %q", err, string(msg.Data))
- return
- }
- repoName := "--unknown--"
- if buildInfo.Source != nil && buildInfo.Source.RepoSource != nil {
- repoName = buildInfo.Source.RepoSource.RepoName
- }
-
- // Record build failures so we can alert on them.
- counter := metrics2.GetCounter("ci_build_failure", map[string]string{"trigger": repoName})
- if msg.Attributes["status"] == "FAILURE" {
- counter.Inc(1)
- } else if msg.Attributes["status"] == "SUCCESS" {
- counter.Reset()
- }
-
- if msg.Attributes["status"] != "SUCCESS" {
- return
- }
- imageNames := imagesFromInfo(shortImageNames, buildInfo)
- if err != nil {
- sklog.Error(err)
- return
- }
- if len(imageNames) == 0 {
- sklog.Infof("No images to push.")
- return
- }
- cfgFile := ""
- if *clusterConfig != "" {
- cfgFile = fmt.Sprintf(" --config-file=%s ", *clusterConfig)
- }
- cmd := fmt.Sprintf("%s --logtostderr %s %s", pushk, cfgFile, strings.Join(imageNames, " "))
- sklog.Infof("About to execute: %q", cmd)
- output, err := exec.RunSimple(ctx, cmd)
- pushFailure := metrics2.GetCounter("ci_push_failure", map[string]string{"trigger": repoName})
- if err != nil {
- sklog.Errorf("Failed to run pushk: %s: %s", output, err)
- pushFailure.Inc(1)
- return
- } else {
- sklog.Info(output)
- }
- pushFailure.Reset()
- pubSubReceive.Reset()
- sklog.Info("Finished push")
- })
- if err != nil {
- sklog.Errorf("Failed receiving pubsub message: %s", err)
- }
- }
-}
diff --git a/kube/go/continuous-deploy/main_test.go b/kube/go/continuous-deploy/main_test.go
deleted file mode 100644
index f459f06..0000000
--- a/kube/go/continuous-deploy/main_test.go
+++ /dev/null
@@ -1,38 +0,0 @@
-package main
-
-import (
- "testing"
-
- "github.com/stretchr/testify/assert"
- "go.skia.org/infra/go/testutils/unittest"
- cloudbuild "google.golang.org/api/cloudbuild/v1"
-)
-
-func TestFindImages(t *testing.T) {
- unittest.SmallTest(t)
- *project = "skia-public"
- Init()
- buildInfo := cloudbuild.Build{
- Results: &cloudbuild.Results{
- Images: []*cloudbuild.BuiltImage{
- {Name: "testing-clang9"},
- {Name: "gcr.io/skia-public/fiddler:prod"},
- {Name: "gcr.io/skia-public/skottie:prod"},
- },
- },
- }
- images := imagesFromInfo([]string{"fiddler", "skottie"}, buildInfo)
- assert.Equal(t, "gcr.io/skia-public/fiddler:prod", images[0])
- assert.Equal(t, "gcr.io/skia-public/skottie:prod", images[1])
-
- images = imagesFromInfo([]string{"skottie"}, buildInfo)
- assert.Equal(t, "gcr.io/skia-public/skottie:prod", images[0])
-}
-
-func TestBaseImageName(t *testing.T) {
- unittest.SmallTest(t)
- assert.Equal(t, "", baseImageName(""))
- assert.Equal(t, "", baseImageName("debian"))
- assert.Equal(t, "fiddler", baseImageName("gcr.io/skia-public/fiddler:prod"))
- assert.Equal(t, "docserver", baseImageName("gcr.io/skia-public/docserver:123456"))
-}
diff --git a/launch.md b/launch.md
index aab031c..a43398f 100644
--- a/launch.md
+++ b/launch.md
@@ -284,4 +284,4 @@
Continuous Deployment
=====================
Some apps are set up to be continuously re-built and re-deployed on every commit of Skia
-or Skia Infra. To do that, see [kube/README.md](./kube/README.md).
+or Skia Infra. To do that, see [docker_pushes_watcher/README.md](./docker_pushes_watcher/README.md).
diff --git a/promk/prometheus/alerts_public.yml b/promk/prometheus/alerts_public.yml
index f4d9ae7..449bc5f 100644
--- a/promk/prometheus/alerts_public.yml
+++ b/promk/prometheus/alerts_public.yml
@@ -25,41 +25,6 @@
https://console.cloud.google.com/logs/viewer?project={{ $labels.project }}&advancedFilter=logName%3D"projects%2F{{ $labels.project }}%2Flogs%2F{{ $labels.job }}"
'
- # Container Builder
- - alert: ContainerBuilderFailure
- expr: ci_build_failure >= 2
- labels:
- category: infra
- severity: warning
- annotations:
- abbr: '{{ $labels.trigger }}'
- description: 'The build with trigger name {{ $labels.trigger }} has failed when rebuilding twice in a row.'
-
- # Continuous Deploy
- - alert: ContinuousDeployLiveness
- expr: liveness_ci_pubsub_receive_s > 60 * 60 * 24 * 2
- for: 5m
- labels:
- category: infra
- severity: critical
- annotations:
- description: 'Continuous deploy has failed to recieve a pubsub event in the last 48 hours.
-
- https://console.cloud.google.com/logs/viewer?project=skia-public&resource=container%2Fcluster_name%2Fskia-public%2Fnamespace_id%2Fdefault&logName=projects%2Fskia-public%2Flogs%2Fcontinuous-deploy&minLogLevel=400
- '
-
- - alert: ContinuousDeployFailures
- expr: ci_push_failure > 2
- for: 5m
- labels:
- category: infra
- severity: critical
- annotations:
- description: 'Continuous deploy has failed to successfully push two or more times in a row.
-
- https://console.cloud.google.com/logs/viewer?project=skia-public&resource=container%2Fcluster_name%2Fskia-public%2Fnamespace_id%2Fdefault&logName=projects%2Fskia-public%2Flogs%2Fcontinuous-deploy&minLogLevel=400
- '
-
# Docker Pushes Watcher
- alert: DockerPushesWatcherLiveness
expr: liveness_docker_watcher_pubsub_receive_s > 60 * 60 * 24 * 2
