perf/TRIAGE.md

Triage
======

The new perf needs a new triaging page, one that tracks regressions per
commit, and allows multiple different queries, such as "source_type=skp" to be
run.

Requirements
------------

Right now alerting is run only on the last 50 commits as one monolithic query
for skps only. The new triaging page should:

  * Run clustering with a window of +/- 5 commits on either side for each
    commit.
  * Only steps that occur at the selected commit will count as a Regression.
  * Low and high clusters are stored per commit, per query. (As opposed to by
    cluster only in the old system).
  * Both low and high cluster regressions are stored, if present.
  * Each query and low/high cluster needs to be triaged.

Design
------

The UI will look roughly like this:

    +----------------------------------+------------------------------------------+
    |            Commit                |               Queries                    |
    +-------------------------------------------------------+---------------------+
    |                                  | source_type=skp    |   source_type=svg   |
    |                                  | sub_result=min_ms  |   sub_result=min_ms |
    |                                  +--------+--------------------+------------+
    |                                  |   High | Low       |  High  | Low        |
    |                                  +--------+--------------------+------------+
    | 5166651 Fix FreePageCount alert. |        |           |        |            |
    | 669f2da [task scheduler] Get the |        |           |        |            |
    | a6ebcd2 Make Status categorize I |        |           |        |            |
    | 501bc48 [CQ Watcher] Log when a  |      ✓ |           |        | ?          |
    | 1b31bb5 [CT] Reduce number of pa |        |           |        |            |
    | ea80add Add ability to specify c |      ✓ | ✗         |        |            |
    | c844730 Fix alert queries due to |        |           |        | ✗          |
    +----------------------------------+--------+-----------+--------+------------+

  * In the above sketch:
    * '?' means an untriaged regression was found.
    * '✓' means this change is acceptable.
    * '✗' means bug.
    * ' ' means sufficient data to cluster, but no regressions found.
  * One column for each query, two sub-cols for low and high regressions.
  * Queries can be added/removed semi-easily (flags or metadata).
  * The queries need to be stored with per-commit data, since they may change
    over time, i.e. we may add new queries or stop running old queries.
  * Each non-empty query/commit/(low|high) cell has:
    * Status:
      * untriaged - Analysis found regression, but not yet triaged.
      * negative - This is a regression.
      * positive - An expected change in behavior, or a noisy cluster.
    * A text comment.
  * Each low/high cell pops up a cluster-summary2-sk that allows inspecting
    the centroid of the regression, the members of the cluster, and triaging
    the cluster.

The data stored for each commit of the triage page analysis will be:

    // map[query]Regression.
    map[string]Regression

    type Regression struct {
        Low        *cluster2.ClusterSummary, // Can be nil.
        High       *cluster2.ClusterSummary, // Can be nil.
        Frame      dataframe.FrameResponse,
        LowStatus  TriageStatus
        HighStatus TriageStatus
    }

    type TriageStatus struct {
        Status   string
        Message  string
    }

    Status is "untriaged", "positive", or "negative".


The alerting analysis will run the following analysis continuously:

    The alert system will define a range [last 50 commits].
    For each commit:
      For each query:
        * Do clustering.
        * Save results into database for regressions that show up
          for that commit.

UI Model
--------

The map[string]Regression is delivered to the browser as the following
struct to make it easier to use Polymer repeat templates:


    {
      header: [ "query1", "query2", "query3", ...],
      table: [
        { id: cid1, cols: [ Regression, Regression, Regression, ...], },
        { id: cid2, cols: [ Regression, null,       Regression, ...], },
        { id: cid3, cols: [ Regression, Regression, Regression, ...], },
      ]
    }

Note that the list of queries is the union of the current queries being run, and
all the queries that appear in all the map[string]Regression's.

The UI for Regression must be able to handle null for a value, which signifies
a cell for which no data exists.