[perf] New alert for android_ingest.
Change-Id: Idaaffd77bb14d574b633359d050d8dfcf89821d0
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/303798
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
diff --git a/android_ingest/PROD.md b/android_ingest/PROD.md
index 696cfac..d1c5e71 100644
--- a/android_ingest/PROD.md
+++ b/android_ingest/PROD.md
@@ -4,7 +4,7 @@
Do a git checkout of the target repo:
- git clone https://skia.googlesource.com/git-master-skia
+ git clone https://skia.googlesource.com/git-master-skia
Then point to that directory as the url handed to androidingest as the
--repo_url. This will give you up to date commits, but also doesn't require
@@ -26,11 +26,17 @@
The storing of all uploaded data in the transaction log is failing. Check
GCS permissions and the logs for the errors generated.
-
## bad_files
Visit [android-master-ingest.skia.org](https://android-master-ingest.skia.org/)
and look at the "Recent Bad Requests" section and see why they are failing. A
previous issue has been bad serializing of the data that is POSTed to the server
where all the data was actually just encoded as one long string. If bad data is
-being generated contact http://go/android-ingest-contact.
\ No newline at end of file
+being generated contact http://go/android-ingest-contact.
+
+## liveness
+
+The process of building the git repo is getting behind, check the logs to see
+if it is the androidbuild API or Gerrit.
+
+ $ kubectl logs -lapp=android_ingest | grep Timer:
diff --git a/promk/prometheus/alerts_public.yml b/promk/prometheus/alerts_public.yml
index 67131ef..49af1a4 100644
--- a/promk/prometheus/alerts_public.yml
+++ b/promk/prometheus/alerts_public.yml
@@ -218,6 +218,16 @@
annotations:
description: 'Error rate for processing buildids is too high. See https://github.com/google/skia-buildbot/blob/master/android_ingest/PROD.md#process_failures'
+ - alert: AndroidIngestLiveness
+ expr: liveness_last_successful_add_s > 300
+ labels:
+ category: infra
+ severity: critical
+ owner: jcgregorio@google.com
+ annotations:
+ description: 'Liveness for processing buildids is too high. See https://github.com/google/skia-buildbot/blob/master/android_ingest/PROD.md#liveness'
+
+
- alert: AndroidIngestBadFilesTooHigh
expr: bad_files/uploads > 0.5
labels:
