Triage

The new perf needs a new triaging page, one that tracks regressions per commit, and allows multiple different queries, such as “source_type=skp” to be run.

Requirements

Right now alerting is run only on the last 50 commits as one monolithic query for skps only. The new triaging page should:

• Run clustering with a window of +/- 5 commits on either side for each commit.
• Only steps that occur at the selected commit will count as a Regression.
• Low and high clusters are stored per commit, per query. (As opposed to by cluster only in the old system).
• Both low and high cluster regressions are stored, if present.
• Each query and low/high cluster needs to be triaged.

Design

The UI will look roughly like this:

+----------------------------------+------------------------------------------+
|            Commit                |               Queries                    |
+-------------------------------------------------------+---------------------+
|                                  | source_type=skp    |   source_type=svg   |
|                                  | sub_result=min_ms  |   sub_result=min_ms |
|                                  +--------+--------------------+------------+
|                                  |   High | Low       |  High  | Low        |
|                                  +--------+--------------------+------------+
| 5166651 Fix FreePageCount alert. |        |           |        |            |
| 669f2da [task scheduler] Get the |        |           |        |            |
| a6ebcd2 Make Status categorize I |        |           |        |            |
| 501bc48 [CQ Watcher] Log when a  |      ✓ |           |        | ?          |
| 1b31bb5 [CT] Reduce number of pa |        |           |        |            |
| ea80add Add ability to specify c |      ✓ | ✗         |        |            |
| c844730 Fix alert queries due to |        |           |        | ✗          |
+----------------------------------+--------+-----------+--------+------------+

• In the above sketch:
  • ‘?’ means an untriaged regression was found.
  • ‘✓’ means this change is acceptable.
  • ‘✗’ means bug.
  • ' ' means sufficient data to cluster, but no regressions found.
• One column for each query, two sub-cols for low and high regressions.
• Queries can be added/removed semi-easily (flags or metadata).
• The queries need to be stored with per-commit data, since they may change over time, i.e. we may add new queries or stop running old queries.
• Each non-empty query/commit/(low|high) cell has:
  • Status:
    • untriaged - Analysis found regression, but not yet triaged.
    • negative - This is a regression.
    • positive - An expected change in behavior, or a noisy cluster.
  • A text comment.
• Each low/high cell pops up a cluster-summary2-sk that allows inspecting the centroid of the regression, the members of the cluster, and triaging the cluster.

The data stored for each commit of the triage page analysis will be:

// map[query]Regression.
map[string]Regression

type Regression struct {
    Low        *cluster2.ClusterSummary, // Can be nil.
    High       *cluster2.ClusterSummary, // Can be nil.
    Frame      dataframe.FrameResponse,
    LowStatus  TriageStatus
    HighStatus TriageStatus
}

type TriageStatus struct {
    Status   string
    Message  string
}

Status is "untriaged", "positive", or "negative".

The alerting analysis will run the following analysis continuously:

The alert system will define a range [last 50 commits].
For each commit:
  For each query:
    * Do clustering.
    * Save results into database for regressions that show up
      for that commit.

UI Model

The map[string]Regression is delivered to the browser as the following struct to make it easier to use Polymer repeat templates:

{
  header: [ "query1", "query2", "query3", ...],
  table: [
    { id: cid1, cols: [ Regression, Regression, Regression, ...], },
    { id: cid2, cols: [ Regression, null,       Regression, ...], },
    { id: cid3, cols: [ Regression, Regression, Regression, ...], },
  ]
}

Note that the list of queries is the union of the current queries being run, and all the queries that appear in all the map[string]Regression's.

The UI for Regression must be able to handle null for a value, which signifies a cell for which no data exists.