tree ecc3fa1dcc980eaf94f1e17ea755e8f08529153e
parent 1607e1488356b275a0bb76d3589566bbfea2856f
author Joe Gregorio <jcgregorio@google.com> 1613760177 -0500
committer Joe Gregorio <jcgregorio@google.com> 1613761251 +0000

[perf] Fix Android Group By regression detection.

This previous change broke regression detection in any instance
that uses Group By alerts:

  https://skia-review.googlesource.com/c/buildbot/+/369845

The root of the problem is that the Alert configs produced
by PubSub ingestion driven regression detection were being
expanded across all their GroupBy values, which is exactly
the thing that ingestion driven regression was built to avoid.
So in theory the clusters would have found the regressions
eventually, we would just have needed 100x the number of cores.

This CL:

 - Restores the optimized behavior of ingestion driven
regression detection.
 - Adds in more logging.
 - Sends more info back to the user via Process messages.
 - Add Prometheus alerts to catch such issues in the future.

Change-Id: I8ed956159984ab7a9f15f28f88eb1f89c6c1a5b6
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/372057
Commit-Queue: Joe Gregorio <jcgregorio@google.com>
Reviewed-by: Kevin Lubick <kjlubick@google.com>