Module: /

High-Level Overview

Pinpoint is a service designed for automated performance and functional bisection on Chromium. Its primary objective is to identify the specific commit that introduced a regression by orchestrating a series of builds and tests across a range of commits. The system manages complex, long-running analysis jobs, integrating with external services like build systems, test schedulers, and data stores to provide a fully automated workflow from job submission to culprit identification.

Project-Specific Terminology

• Bisection: An analysis workflow that iteratively narrows down a range of commits to find a single culprit. It works by testing a commit in the middle of the range, comparing its performance to a known-good baseline, and then deciding which half of the range to continue searching in.
• Pairwise: An analysis workflow that runs tests on two specific commits and statistically compares their performance results to determine if there is a significant difference. This is often used to verify a potential regression or confirm a culprit found by a bisection.
• CombinedCommit: A core data abstraction that represents a specific state of the source code. It goes beyond a simple git hash by also including potential modifications from dependency rolls (DEPS file changes) and in-flight Gerrit patches. This allows Pinpoint to accurately reproduce and test complex code states that are not represented by a single repository commit.
• Temporal: A durable workflow orchestration system used as the backbone of Pinpoint's execution engine. It allows Pinpoint to model long-running bisection jobs as stateful, fault-tolerant processes, managing retries, state persistence, and timeouts externally to the core application logic.
• PGAdapter: A proxy that translates the standard PostgreSQL wire protocol into API calls for a different backend database, such as Google Cloud Spanner. It enables applications written with standard PostgreSQL drivers to connect to and interact with non-PostgreSQL databases without code changes.
• Buganizer: Google's internal issue tracking system. Pinpoint integrates with Buganizer to automatically create, update, and comment on issues related to performance regressions.

Overall Architecture

Pinpoint is architected as a distributed system with a clear separation of concerns between the user interface, the API gateway, the core workflow engine, and its external integrations.

UI (/ui)

The user interface is a single-page application (SPA) built with LitElement and TypeScript. Its design is centered on modularity and a predictable, unidirectional data flow.

• Why LitElement & Component-Based Structure: The UI is composed of reusable web components. The architecture uses a “shell and component” model, where minimal HTML pages (/pages) act as entry points to load root components. This delegates all rendering and logic to the components themselves, promoting reusability and separation of concerns.
• How it Manages Data Flow: Components are divided into two types to enforce a one-way data flow.
  1. Controller Components: These are stateful components responsible for a specific view (e.g., the job list or results page). They fetch data from the backend, manage the view's state, and pass data down to child components via properties.
  2. Presentational Components: These are stateless components that are only responsible for rendering data they receive as properties. They communicate user interactions back to their parent controller by dispatching custom events. This “data down, events up” pattern makes the application's state management easier to reason about.
• How it Communicates with the Backend: All backend communication is centralized in a single API service module (/services/api.ts). This service exports async functions that encapsulate fetch calls and defines TypeScript interfaces that mirror the backend's data structures. This approach decouples UI components from the implementation details of API requests and provides a type-safe contract between the frontend and backend.

Frontend Service (/go/frontend)

This Go service acts as the primary API gateway and web server for the Pinpoint system. It is the component run by the jobsserver container.

• Why it Exists: It serves a dual purpose: it serves the static assets for the SPA and provides the JSON API that the UI consumes. It also acts as a bridge, forwarding requests for complex operations (like scheduling a new job) to the core Pinpoint gRPC service.
• How it Manages Configuration: The service relies on a benchmarks.json file as its single source of truth for all available benchmarks, the bots that can run them, and associated test stories. On startup, this file is parsed and its entire contents are loaded into an in-memory cache. This design ensures that API requests for configuration data (e.g., to populate UI dropdowns) are served with extremely low latency, as they do not require disk or database access.
• How it Connects to the Database: The frontend requires a PostgreSQL-compatible database to store and retrieve job metadata. It achieves this by connecting through PGAdapter. This proxy allows the Go service to use a standard PostgreSQL driver while the underlying data store can be a different system, such as Google Cloud Spanner. This decouples the service from a specific database technology, simplifying ddevelopment and deployment.

Core Backend (/go)

The Go backend is the heart of Pinpoint, containing the business logic for scheduling, executing, and analyzing bisection jobs.

• API Layer (/proto, /go/service): The public interface of the backend is defined using Protocol Buffers (service.proto). This provides a strongly-typed, language-agnostic API contract. The API is exposed via both gRPC for high-performance internal communication and a RESTful JSON interface (auto-generated via gRPC-Gateway) for consumption by the UI. The API design favors explicit, job-specific RPCs (e.g., ScheduleBisection, SchedulePairwise) to ensure type safety and clarity of intent.
• Orchestration Layer (/go/workflows): The core execution logic is implemented as Temporal workflows.
  • Why Temporal: Performance bisections are long-running, stateful processes that can take hours and are susceptible to transient failures in external systems. Using Temporal offloads the responsibility for durability, state management, fault tolerance, and retries from the application code. A Pinpoint job is modeled as a durable workflow execution that can survive process restarts and continue from where it left off.
  • How it Works: Workflows define the high-level sequence of steps in a job (e.g., find midpoint, build, test, compare, repeat). Each individual, short-lived task (e.g., an API call to an external service) is implemented as a Temporal “activity.” A dedicated worker process polls the Temporal cluster for tasks and executes this workflow and activity logic.
• Execution & Analysis Logic: The workflows orchestrate calls to several key modules that contain specialized business logic.
  • /go/midpoint: This module is the bisection strategist. Its key responsibility is to intelligently select the next commit to test. It handles not only linear commit histories but also recursively bisects through dependency (DEPS) changes, which is critical for finding regressions caused by dependency rolls.
  • /go/compare: This is the statistical analysis engine. After a test run, this module receives two sets of performance measurements and determines if they are statistically different. It uses a combination of statistical tests (e.g., Mann-Whitney U) and a dual-threshold system to classify the comparison as definitively Same, definitively Different, or Unknown, triggering more test runs if necessary. This automates the critical decision-making step in a bisection.
  • /go/backends: This module centralizes all communication with external infrastructure. It contains a collection of low-level clients that abstract the APIs for services like Buildbucket (for compiling Chromium), Swarming (for running tests), and Gitiles (for querying git history). This design decouples the core workflow logic from the implementation details of external systems.
• Persistence (/go/sql): While Temporal manages the state of active workflows, a PostgreSQL-compatible SQL database provides long-term storage for job history, metadata, and results. The /go/sql module defines the database schema and provides a data access layer used by the /go/frontend to serve job information via its API.

Buganizer Integration

Pinpoint integrates with Google's IssueTracker (Buganizer) to automate bug management for performance regressions.

• Why it's Needed: This integration allows Pinpoint to automatically create or comment on bugs when a regression is found, linking the analysis directly to the engineering workflow and eliminating the need for manual intervention.
• How it Works: The integration is facilitated by a dedicated service account. This account is granted specific permissions to access internal resources and secrets, such as an IssueTracker API key stored in Secret Manager. To modify issues, the service account must be added as a collaborator to the relevant Buganizer components. This service account is then associated with the Temporal workers, ensuring that any actions performed on Buganizer are done under this specific, audited identity.

Core Workflows

A Performance Bisection Job

1. A user schedules a bisection job through the UI, which sends a request to the Pinpoint Frontend service.
2. The Frontend forwards this request to the core gRPC Service (/go/service).
3. The gRPC service validates the request and initiates a BisectWorkflow execution on the Temporal cluster.
4. A Pinpoint Worker process picks up the task and begins executing the workflow logic.
5. Inside the workflow loop: a. An activity calls the /go/midpoint module to determine the next commit to test. b. Activities then use clients from /go/backends to request a build from Buildbucket and schedule tests on Swarming for that commit. c. Once tests complete, another activity fetches the performance metric values. d. These values are passed to an activity that uses the /go/compare module to statistically determine if the midpoint commit is better or worse than the baseline. e.g. Based on the comparison verdict, the workflow narrows the commit range and the loop continues.
6. Throughout the process, the worker executes activities that update the job's status in the SQL database via the /go/sql layer. The Frontend service can then query this database to display real-time progress in the UI.
7. Once the range is narrowed to a single commit, the workflow completes, marking the commit as the culprit.

Culprit Finder Workflow

The ScheduleCulpritFinder RPC initiates a composite workflow designed for high-confidence culprit identification. It proceeds in three distinct stages:

1. Regression Verification: It first runs a Pairwise job between the provided start and end commits to statistically confirm that a regression actually exists within the range. If no regression is found, the workflow ends early.
2. Bisection: If a regression is confirmed, it proceeds to run a standard Bisection job on the commit range to identify one or more potential culprit commits.
3. Culprit Verification: For each potential culprit found by the bisection, it runs a final Pairwise job comparing the culprit against its immediate parent commit. This step verifies that the specific commit is, by itself, the source of the regression. The workflow only reports culprits that pass this final verification step.

Module: /docs

The /docs module serves as a repository for operational and integration documentation relevant to various services within the project, particularly those supporting the Pinpoint ecosystem. Its primary goal is to provide guidance on how services are configured, interact with external platforms, and manage their data and workflows.

Pinpoint Frontend

The Pinpoint Frontend service (/go/frontend) is the primary user interface and API gateway for the Pinpoint performance regression analysis system. Its core purpose is to provide users with a comprehensive view of Pinpoint jobs, enable them to initiate and manage these jobs, and access crucial configuration data necessary for defining performance experiments. It acts as the critical bridge between user requests and the underlying Pinpoint services, databases, and workflow orchestration.

The service's responsibilities include:

• Providing HTTP endpoints for listing and viewing Pinpoint jobs.
• Serving the Pinpoint UI (landing and results pages).
• Providing configuration data such as available benchmarks, bots, and stories.
• Forwarding relevant requests to an underlying Pinpoint gRPC service for core functionalities like scheduling new jobs.

Key Components and Design Decisions

The Pinpoint Frontend service is designed with several key architectural components to achieve its goals:

• User Interface & API Endpoints: The service serves both a web-based user interface and a set of RESTful JSON API endpoints. These endpoints manage job lifecycle (listing, viewing), and provide dynamic configuration data (e.g., /benchmarks, /bots, /stories). It also acts as a proxy, forwarding requests to the core Pinpoint gRPC service for operations like job scheduling.

• Database Integration with PGAdapter: The Pinpoint Frontend is a Go application that requires a PostgreSQL-compatible database connection. However, it does not directly connect to a PostgreSQL database. Instead, it leverages PGAdapter.

  • Why PGAdapter: PGAdapter acts as an intelligent proxy, translating standard PostgreSQL wire protocol requests into calls for a specific backend database, such as Google Cloud Spanner. This design choice decouples the frontend service from direct database-specific client libraries. It allows the Go service to interact with the database as if it were a standard PostgreSQL instance, significantly simplifying development, testing, and deployment by using familiar database patterns.
  • How it Works: PGAdapter runs as a separate Java process, typically on the same host or network location as the frontend. It requires specific JAR files and configuration (e.g., instance ID, database name, authentication credentials) to establish and maintain the connection between the frontend and the underlying datastore.

• Workflow Orchestration with Temporal: For complex, long-running operations such as initiating Pairwise bisect jobs or canceling ongoing ones, the frontend delegates to a Temporal workflow system.

  • Why Temporal: Temporal provides robust and durable workflow orchestration. This ensures that complex jobs are executed reliably, automatically retried on failures, and their state is consistently maintained across potentially long durations. The frontend itself does not directly perform these complex, stateful operations; instead, it sends signals or initiates workflows on the Temporal cluster.
  • How it Works: The Pinpoint gRPC service (which the frontend forwards requests to) interacts with a Temporal client. This client communicates with a Temporal cluster, which then dispatches tasks to dedicated Temporal workers. These workers are responsible for executing the actual job logic (e.g., fetching data, running bisects). Workers listen on specific namespaces and task queues (e.g., perf-internal, perf.perf-chrome-public.bisect).

• Configuration Management via benchmarks.json: The service relies on benchmarks.json as its single source of truth for critical Pinpoint operational parameters.

  • Why benchmarks.json: This file centrally defines all available performance benchmarks, the bots capable of running them, and the specific test stories associated with each benchmark. This centralized configuration ensures consistency across the system, provides dynamic data for UI elements (like dropdowns), and enables the API endpoints to serve up-to-date configuration.
  • How it Works: Upon service startup, benchmarks.json is parsed and its contents are loaded entirely into an in-memory cache. This approach allows for extremely fast access to configuration data, as subsequent requests to the /benchmarks, /bots, and /stories API endpoints can retrieve information directly from memory without incurring repeated disk I/O, ensuring high performance for frequently queried data.

Configuration Workflow with benchmarks.json

graph TD
    A[Service Startup] --> B[Read benchmarks.json from disk]
    B --> C[Parse JSON & Load into In-Memory Cache]
    C --> D{User/Client sends:}; D -- GET /benchmarks -->
    E[Service retrieves data from In-Memory Cache]
    D -- GET /bots?benchmark=<name> --> E
    D -- GET /stories?benchmark=<name> --> E
    E --> F[Respond to Client with configuration data]

Pinpoint Job Initiation Workflow with Temporal

graph TD
    A[User initiates new Pinpoint job via Frontend UI] --> B{HTTP Request to /pinpoint/new}
    B --> C[Pinpoint Frontend Service]
    C --> D{Internal call to Pinpoint gRPC Service}
    D --> E[Pinpoint gRPC Service]
    E --> F{Calls Temporal Client to start Workflow}
    F --> G[Temporal Cluster]
    G -- Schedules tasks on appropriate queue --> H[Temporal Worker]
    H --> I[Updates database & Notifies Frontend of job status changes]

Buganizer Integration

Automated systems, including Pinpoint, require the ability to interact with Google's IssueTracker (Buganizer) for managing issues. This integration is crucial for workflows that automatically create, update, or comment on bugs in response to detected performance regressions or other significant events, ensuring efficient tracking without manual intervention.

The integration is primarily facilitated through a dedicated service account that acts on behalf of the automated system. This service account requires specific, carefully managed permissions to securely access internal resources and interact with Buganizer.

• Service Account Creation: A new service account is provisioned with a clear and descriptive name, avoiding misleading terms. This is typically managed through infrastructure-as-code changes.
• Permission Grants: The service account is granted several critical roles:
  • Read access to CAS (Credential Access System) for retrieving necessary secrets.
  • The roles/cloudtrace.agent for distributed tracing and observability.
  • Read access to internal code repositories, often involving a separate approval process.
• IssueTracker API Key Access: To interact with the IssueTracker API, the service account needs to read the IssueTracker secret key, which is stored in Secret Manager. This is a sensitive operation requiring an administrator to temporarily elevate their own permissions using breakglass to grant the service account the “Secret Manager Secret Accessor” role specifically for the IssueTracker key. This ensures tight control over access to sensitive credentials.
• Ticket Collaboration: For the service account to be able to modify tickets (e.g., add comments, change status), it must either be added as a collaborator on the relevant Buganizer components or be granted the “issue editor” role by a Component Admin. This provides the automated system with the necessary authority to update issues as required by workflow logic.
• Deployment: The configured service account is then associated with the Pinpoint workers, specifically for Temporal workflows, ensuring these workers operate under the appropriate identity when initiating or updating Buganizer issues.

Module: /go

This module contains the complete Go-based backend for Pinpoint, a service for performance and functional bisection on Chromium. It orchestrates complex, long-running analysis jobs by integrating with various external services like build systems, test schedulers, and data stores.

Core Principles

• Durable Orchestration with Temporal: The system's core logic is implemented as Temporal workflows. This design choice provides fault tolerance, state persistence, and observability for long-running bisection jobs, offloading complex state management and retry logic from the application code.
• Separation of Concerns: The architecture distinctly separates the public API, the workflow orchestration logic, and low-level clients for external services. This modularity simplifies maintenance and testing.
• Configuration as Code: Bot and test configurations are embedded directly into the binary. This simplifies deployment and guarantees that the application and its configuration are always in sync.

Key Modules and Responsibilities

• /go/service: The public API layer. It exposes a gRPC service (and a JSON equivalent via grpc-gateway) for clients to schedule, query, and cancel jobs. Its primary responsibility is to validate incoming requests and translate them into Temporal workflow executions, acting as the main entry point into the system.

• /go/workflows: The heart of Pinpoint's execution engine. This module defines the Temporal workflows and activities that implement the business logic for bisection, pairwise testing, and culprit finding.

  • Why: It models a Pinpoint job as a durable, stateful process. A BisectWorkflow, for example, orchestrates the entire process of finding a midpoint, building, testing, and comparing results, with Temporal ensuring the process can survive failures and continue where it left off.
  • How: Workflows define the high-level steps, while activities execute the short-lived, potentially failing tasks like making an API call to an external service. A dedicated worker process executes this logic.

• /go/backends: A collection of low-level clients for all external services.

  • Why: To centralize and abstract interactions with external infrastructure like Buildbucket (for builds), Swarming (for tests), Gitiles (for git history), and BigQuery (for data storage). This decouples the core workflow logic from the implementation details of external APIs.
  • How: Each submodule provides a dedicated client (e.g., buildbucket.go, swarming.go) that handles authentication, request formation, and response parsing for a specific service.

• /go/midpoint: The bisection strategist.

  • Why: To intelligently select the next commit to test. The core challenge is handling not just linear commit history but also changes rolled in through dependencies (DEPS).
  • How: When bisecting a commit range, it finds the median commit. If the range is already narrowed to two adjacent commits, it parses their DEPS files, identifies the first modified dependency, and recursively continues the bisection within that dependency's repository.

• /go/compare: The statistical analysis engine.

  • Why: To provide automated, data-driven decisions on whether a change introduced a regression. This is the critical decision point in each bisection step.
  • How: It uses a combination of statistical tests (Kolmogorov-Smirnov, Mann-Whitney U) and a dual-threshold system. This allows it to quickly classify two sets of measurements as definitively Same, definitively Different, or Unknown (requiring more test runs), optimizing the speed and accuracy of the bisection.

• /go/bot_configs: The central configuration provider.

  • Why: To manage the vast and complex configurations for different bots, benchmarks, and tests without hardcoding them in the business logic.
  • How: It loads configuration from embedded JSON and YAML files. It provides lookup functions that resolve the correct build parameters and test targets for any given bot and benchmark combination, including handling aliases and regex-based matching.

• /go/sql and /go/frontend: The persistence and presentation layers.

  • Why: To store and display job metadata. While Temporal manages active workflow state, the SQL database provides long-term storage for job history and results.
  • How: /go/sql defines the database schema using Go structs and provides a jobs_store for data access. /go/frontend is a lightweight web service that queries this store to provide a simple UI and JSON API for viewing job status and results.

Core Workflow: A Performance Bisection Job

The following diagram illustrates how the key modules interact to execute a typical bisection job.

graph TD
    A["Client"] -- "1. ScheduleBisect" --> B["gRPC Service (/go/service)"]
    B -- "2. Starts Workflow" --> C["Temporal Server"]
    C -- "3. Dispatches Task" --> D["Pinpoint Worker (/go/workflows/worker)"]

    subgraph "Pinpoint Worker Logic"
        subgraph "Bisection Workflow"
            D -- Starts --> E{"Loop: While range > 1"}
            E -- "4. Find Midpoint" --> F["Activity: Midpoint Logic (/go/midpoint)"]
            F --> G["Activity: Build & Test"]
            G -- "5. Trigger Build/Test" --> H["Clients (/go/backends)"]
            H --> I["Buildbucket & Swarming"]
            I -- "6. Results (CAS digests)" --> H
            H --> G
            G -- "7. Values" --> J["Activity: Read & Compare"]
            J -- "Fetches data from CAS" --> H
            J -- "8. Compares samples" --> K["Statistical Logic (/go/compare)"]
            K -- "9. Verdict (Same/Different)" --> J
            J -- "Narrows commit range" --> E
        end

        subgraph "Job Status Persistence (Parallel)"
            D -- "Updates Job Status" --> M["Activity: Update Datastore"]
            M --> N["Job Store (/go/sql)"]
            O["Frontend (/go/frontend)"] -- "Reads from" --> N
        end
    end

    E -- "Culprit Found" --> L["Workflow Complete"]
    L -- "10. Returns Result" --> C
    C -- "11. Result available to" --> A

Module: /go/backends

This module consolidates clients and logic for interacting with various external services critical to Pinpoint's bisection workflow. It provides a standardized and abstracted layer for tasks such as triggering builds, managing test execution, fetching commit information, storing data, and reporting results, minimizing direct external API dependencies across the Pinpoint codebase.

Key Components and Responsibilities

• bigquery.go: Manages interactions with Google BigQuery. It abstracts data ingestion and table management, enabling Pinpoint to store bisection results, telemetry, or other structured data efficiently.
• buildbucket.go: Provides the primary interface for Buildbucket, the system for building Chrome. Its responsibilities include:
  • Triggering and Managing Builds: Scheduling new Chrome builds for specific commits, with optional DEPS overrides or Gerrit patches, and cancelling ongoing builds.
  • Build Status and Artifacts: Querying build status and retrieving CAS (Content Addressable Storage) references for build outputs. This is crucial for passing artifacts to Swarming for testing.
  • Build Reuse: Intelligently searching for and identifying existing builds (Pinpoint try builds or CI waterfall builds) that match specific criteria (commit, DEPS, patches) to avoid redundant work. This is achieved via GetSingleBuild, GetBuildWithDeps, GetBuildWithPatches, and GetBuildFromWaterfall, leveraging the PinpointWaterfall mapping (defined in waterfall_map.go).
• crrev.go: Facilitates conversion between various Chromium commit identifiers (e.g., commit positions, Git hashes) and canonical Git hashes, ensuring consistent commit references across different backend systems.
• gitiles.go: Offers a client for Gitiles, primarily used to access Git repository information (e.g., chromium/src). It provides an authenticated interface for fetching repository details and commit history, essential for bisection range determination.
• issuetracker.go: Integrates with Google IssueTracker (Buganizer). Its core responsibility is to automate the reporting of bisection results (e.g., identified culprits) as comments on bug reports. It uses embedded templates (culprit_detected.tmpl) for structured, informative messages.
• swarming.go: Manages interactions with Swarming, the distributed task execution system for running tests. Key functions include:
  • Task Triggering: Scheduling Swarming tasks, often using CAS references from Buildbucket builds to execute performance benchmarks or other tests.
  • Task Monitoring: Retrieving task status, start times, and CAS output references (for results or subsequent tasks).
  • Bot Management: Querying available bots based on specific builder configurations to ensure tasks run on appropriate hardware.
  • Task Interruption Detection: Identifying if other tasks ran on a bot between two specific bisection tasks (GetBotTasksBetweenTwoTasks) to detect potential interference.

Core Workflow (Build-and-Test Bisection Example)

graph TD
    A["Pinpoint Bisection Request"] --> B{"Commit Range, Builder, Test"}
    B --> C["Crrev: Resolve Commit Hashes"]
    C --> D{"For each candidate commit in range"}
    D --> E["Buildbucket: Search for Existing Build (incl. Waterfall)"]
    E --> F{"Build Found?"}
    F -- Yes --> I["Get CAS Reference"]
    F -- No --> G["Buildbucket: Start New Build"]
    G --> H{"Build Completed?"}
    H -- No --> H
    H -- Yes --> I["Get CAS Reference"]
    I --> J["Swarming: Trigger Test Task with CAS"]
    J --> K{"Swarming Task Completed?"}
    K -- No --> K
    K -- Yes --> L["Swarming: Get Task Results"]
    L --> M["Analyze Test Results"]
    M --> N{"Bisection Complete?"}
    N -- No --> D
    N -- Yes --> O["Identify Culprit(s)"]
    O --> P["IssueTracker: Report Culprit(s)"]
    O --> Q["BigQuery: Log Bisection Data"]

Module: /go/backends/mocks

go/backends/mocks

This module provides generated mock clients for Pinpoint's external backend services. It facilitates unit and integration testing by decoupling tested components from actual external dependencies.

Mocks are generated from interfaces defined in go.skia.org/infra/pinpoint/go/backends using the mockery tool. This approach ensures mock consistency with interface contracts and enables robust, isolated testing without live service reliance or API call overhead.

Each file, such as BigQueryClient.go or BuildbucketClient.go, contains a mock implementation for a specific backend client interface. These mocks leverage github.com/stretchr/testify/mock to define expected method calls, return values, and side effects, simplifying test setup and enabling precise control over simulated external service behavior.

Module: /go/bot_configs

This module centralizes and manages configuration data for Pinpoint bots, dictating how Chrome builds are specified in LUCI and how performance tests are executed on Swarming. It also maps bots and benchmarks to specific build targets (isolates).

Bot Configurations

Bot configurations define parameters for building Chrome and running tests.

• bot_configs.go: Defines the BotConfig structure and provides the primary interface (GetBotConfig) for retrieving bot configurations. It also includes validation logic to ensure data integrity.
  • Separation of external.json and internal.json: Public configurations (external.json) are separated from internal or experimental ones (internal.json). internal.json augments or overrides external.json entries.
  • Embedded JSON: Configurations are embedded directly into the Go binary (_embed directive), ensuring runtime availability and simplifying deployment.
  • Lazy Loading: Configurations are unmarshaled from JSON only once (sync.Once) upon first access, optimizing performance.
  • Aliases: The BotConfig.Alias field allows bots with identical configurations to refer to a common alias, reducing duplication. GetBotConfig transparently resolves these aliases.
  • Validation: The validate() function enforces data integrity, checking for duplicate bot definitions, nested aliases, and completeness of configuration fields.

Isolate Targets

Isolate targets specify the executable artifact to run for performance tests on Swarming.

• isolate_targets.go: Maps bot names and benchmark names to specific Swarming isolate targets.
  • Embedded YAML: Isolate target mappings are embedded from isolate_targets.yaml into the binary.
  • Tiered Lookup: GetIsolateTarget resolves targets using a priority order:
  • Benchmark-specific overrides.
  • Exact bot name matches.
  • Regex pattern matches within the bot name.
  • A default performance_test_suite if no other match is found. This hierarchy provides flexible control over target selection.

Key Workflow: Retrieving a Bot Configuration

graph TD
    A["GetBotConfig(bot, externalOnly)"] --> B{externalOnly?}
    B -- Yes --> C[Load external.json]
    B -- No --> D[Load external.json + internal.json]
    C --> E[Check 'bot' in loaded configs]
    D --> E
    E -- Not Found --> F[Return error]
    E -- Found --> G{BotConfig has Alias?}
    G -- Yes --> H[Resolve Alias to final BotConfig]
    G -- No --> I[Return BotConfig]
    H --> I

Key Workflow: Retrieving an Isolate Target

graph TD
    A["GetIsolateTarget(bot, benchmark)"] --> B["Validate 'bot' exists in bot configs"]
    B -- Error --> C["Return error"]
    B -- OK --> D["Load isolate_targets.yaml"]
    D --> E{"BenchmarkTargets[benchmark] exists?"}
    E -- Yes --> F["Return Benchmark Target"]
    E -- No --> G{"BotExactTargets[bot] exists?"}
    G -- Yes --> H["Return Exact Bot Target"]
    G -- No --> I{"BotRegexTargets matches bot?"}
    I -- Yes --> J["Return Regex Match Target"]
    I -- No --> K["Return default performance_test_suite"]

Module: /go/cbb

The /go/cbb module provides essential utilities for the Continuous Browser Benchmarking (CBB) infrastructure. It automates the distribution of Safari Technology Preview (STP) releases for CBB devices and facilitates the ingestion of historical CBB v3 performance data into the modern CBB v4 system (Skia Perf dashboard).

download_stp

This utility ensures CBB devices always have access to the latest Safari Technology Preview (STP) releases for benchmarking. Apple does not provide a direct API for STP downloads. Therefore, download_stp parses Apple's Safari resources page to dynamically discover the latest STP builds and extract their .dmg installer URLs.

Downloaded .dmg installers are then packaged and uploaded to CIPD (infra/chromeperf/cbb/safari_technology_preview) for reliable and centralized distribution across the CBB infrastructure. Packages are tagged with specific CIPD references (e.g., stable, canary, latest, [release]-macos[version]), enabling controlled rollouts and precise version targeting for CBB devices. The process is idempotent, checking for existing packages and references before performing redundant uploads.

graph TD
    A["Start"] --> B{"Fetch & Parse Safari Resources Page"}
    B --> C{"Extract Latest Release Number & Download Links"}
    C --> D{"Is STP [Release]-macos15 in CIPD?"}
    D -- Yes --> E["Exit"]
    D -- No --> F["Download STP .dmg (Sequoia)"]
    F --> G["Create CIPD Package (Sequoia)"]
    G --> H["Apply Refs: stable, canary, latest, Release-macos15"]
    H --> I["Download STP .dmg (Tahoe)"]
    I --> J["Create CIPD Package (Tahoe)"]
    J --> K["Apply Ref: Release-macos26"]
    K --> L["End"]

upload_v3_data

This utility backfills historical CBB v3 performance metrics, sourced from a specific Google3 CSV export, into the Skia Perf dashboard. This enables modern CBB v4 analyses to include a comprehensive historical context from legacy data.

The conversion process, orchestrated by main.go, consumes a single cbb_v3_data.csv file and produces multiple Skia Perf JSON files. Key responsibilities include:

• Data Ingestion and Normalization: Reads the CSV and normalizes inconsistent Chrome version strings (e.g., 137.0.7151.070 to 137.0.7151.70) to conform to Skia Perf's expected formats.
• Commit Position Mapping: A critical step that translates CBB v3 browser versions (Chrome, Safari, Edge across channels and devices) into equivalent Skia Perf “commit positions.” This mapping, leveraging hardcoded configurations for known versions and a heuristic fallback for others, is essential for correctly plotting historical data along the Skia Perf x-axis.
• Skia Perf Format Generation: Transforms CBB v3 data points into individual Skia Perf format.Format JSON structures. This involves mapping CBB v3 identifiers (device, browser, benchmark) to Skia Perf bot, benchmark, and test trace keys, and explicitly marking the data as “Legacy Data.”
• Output Management: Writes the generated JSON files, structured for subsequent bulk upload to the chrome-perf-non-public GCS bucket, to ~/cbb.

graph TD
    subgraph Google3 Environment
        A[Original CBB v3 Data] --> B(Blaze tool: download_f1);
        B -- Exports as --> C(cbb_v3_data.csv in ~/cbb);
    end

    subgraph Skia Infra Toolchain
        C --> D(upload_v3_data tool);
        D -- Reads, Normalizes & Maps --> E{Browser Version to Commit Position};
        E -- Transforms to --> F(Skia Perf JSON Format);
        F -- Generates multiple --> G(Output .json files in ~/cbb);
    end

    subgraph Skia Perf System
        G --> H(gsutil: Upload to GCS bucket);
        H -- Ingested by --> I(Skia Perf Dashboard);
        I -- Visualizes legacy data as --> J(CBB v4 Historical Traces);
    end

Module: /go/cbb/download_stp

This module automates the distribution of Safari Technology Preview (STP) releases for Continuous Browser Benchmarking (CBB). It ensures CBB devices always have access to the latest STP versions by packaging them into CIPD.

Design and Implementation:

• STP Discovery: Apple does not provide a direct API for STP downloads. The tool parses the Safari Technology Preview resources page to extract the latest release number and download URLs for various macOS versions. This approach is necessary to dynamically discover new STP builds.
• CIPD Packaging: STP .dmg installers are downloaded and uploaded as CIPD packages (infra/chromeperf/cbb/safari_technology_preview). CIPD is used for reliable and centralized distribution of artifacts across the infrastructure.
• Reference Management: Packages are tagged with specific CIPD references (e.g., stable, canary, latest, [release]-macos[version]). These references allow CBB devices to consume specific STP versions, enabling controlled rollouts and easy version targeting.
• Idempotency: Before creating new CIPD packages, the tool checks if the current STP release (identified by its version and macOS) already exists in CIPD with the necessary references. This prevents redundant uploads and ensures efficiency.

Key Workflow:

graph TD
    A["Start"] --> B{"Fetch & Parse Safari Resources Page"}
    B --> C{"Extract Latest Release Number & Download Links"}
    C --> D{"Is STP [Release]-macos15 in CIPD?"}
    D -- Yes --> E["Exit"]
    D -- No --> F["Download STP .dmg (Sequoia)"]
    F --> G["Create CIPD Package (Sequoia)"]
    G --> H["Apply Refs: stable, canary, latest, Release-macos15"]
    H --> I["Download STP .dmg (Tahoe)"]
    I --> J["Create CIPD Package (Tahoe)"]
    J --> K["Apply Ref: Release-macos26"]
    K --> L["End"]

Module: /go/cbb/upload_v3_data

Converts legacy Cross-Browser Benchmarking (CBB) v3 data, sourced from a specific Google3 CSV export, into the Skia Perf dashboard's JSON ingestion format. This enables backfilling historical CBB v3 performance metrics into the modern CBB v4 system.

Design and Implementation

The utility operates as a one-shot conversion process, consuming a single CSV file and producing multiple Skia Perf JSON files.

Key Components and Responsibilities

• main.go: Orchestrates the entire conversion.
  • Data Ingestion and Normalization: Reads cbb_v3_data.csv from the user's ~/cbb directory. It normalizes inconsistent Chrome version strings (e.g., 137.0.7151.070 to 137.0.7151.70) to match standard formats expected by Skia Perf.
  • Commit Position Mapping: Translates CBB v3 browser versions (Chrome, Safari, Edge across stable/dev channels and devices) into equivalent Skia Perf “commit positions.” This mapping is crucial for correctly plotting historical data on the Skia Perf x-axis. A set of hardcoded maps (chromeStableCP, safariTPCP, etc.) provides precise mappings for known versions. A heuristic fallback calculates approximate commit positions for Chrome versions not explicitly listed, catering to the historical nature of the data.
  • Skia Perf Format Generation: Transforms raw CBB v3 data points into individual Skia Perf format.Format JSON structures. This includes mapping CBB v3 device, browser, and benchmark names to their corresponding Skia Perf bot, benchmark, and test trace keys, and explicitly marking them as “Legacy Data.”
  • Output Management: Writes the generated JSON files to ~/cbb, structuring them for subsequent bulk upload to the chrome-perf-non-public GCS bucket.

graph TD
    subgraph Google3 Environment
        A[Original CBB v3 Data] --> B(Blaze tool: download_f1);
        B -- Exports as --> C(cbb_v3_data.csv in ~/cbb);
    end

    subgraph Skia Infra Toolchain
        C --> D(upload_v3_data tool);
        D -- Reads, Normalizes & Maps --> E{Browser Version to Commit Position};
        E -- Transforms to --> F(Skia Perf JSON Format);
        F -- Generates multiple --> G(Output .json files in ~/cbb);
    end

    subgraph Skia Perf System
        G --> H(gsutil: Upload to GCS bucket);
        H -- Ingested by --> I(Skia Perf Dashboard);
        I -- Visualizes legacy data as --> J(CBB v4 Historical Traces);
    end

Module: /go/clients

The /go/clients module provides abstracted interfaces for interacting with external services essential to Pinpoint, such as build systems and data storage. It centralizes and simplifies communication, ensuring Pinpoint's core logic remains decoupled from specific service APIs and configurations.

Design Principles and Implementation Choices

1. Service Abstraction:

   • Why: To decouple Pinpoint's business logic from the specifics of underlying external services. This allows for easier maintenance, potential future integration of alternative services, and a consistent API for Pinpoint workflows.
   • How: Generic Go interfaces (e.g., BuildClient, UploadClient) define common operations for distinct service types.

2. Project-Specific Implementations:

   • Why: External services often have unique requirements, configurations, or operational heuristics specific to a project (e.g., Chrome/Chromium). Abstracting these details allows the generic interface to remain clean.
   • How: Concrete client implementations (e.g., buildChromeClient, uploadChromeDataClient) translate generic Pinpoint requests into service-specific API calls and data structures, including complex project-specific logic or heuristics where necessary.

3. Delegation to Backend Clients:

   • Why: To separate the logic of what operations to perform and how to map parameters (handled by clients) from the low-level mechanics of making authenticated API requests and managing data serialization/deserialization.
   • How: clients implementations embed and utilize clients from the go/backends module, which are responsible for direct, authenticated interactions with service APIs (e.g., Buildbucket, BigQuery).

Key Components

• build submodule: Offers the BuildClient interface for managing Chrome/Chromium builds. The buildChromeClient implementation encapsulates the detailed logic for interacting with Buildbucket, including builder resolution, constructing request properties, and specific build search heuristics (e.g., differentiating “try” vs. “waterfall” builds).
• upload submodule: Provides the UploadClient interface for ingesting data into storage systems. The uploadChromeDataClient implementation handles operations for Google BigQuery, including deriving table schemas from Go structs and inserting data rows.

Module: /go/clients/build

The /go/clients/build module provides an abstraction layer for interacting with build systems, primarily Buildbucket, to manage Chrome/Chromium builds within Pinpoint. It enables other Pinpoint components to perform build-related operations (find, start, get status, retrieve artifacts, cancel) without direct knowledge of the underlying build system's API or project-specific configurations.

Design Principles and Implementation Choices

1. Build System Agnostic Interface (build_client.go):

   • Why: To decouple Pinpoint's core logic from specific build platform details (e.g., Buildbucket, other potential future systems). A unified interface simplifies interaction and allows for future expansion.
   • How: The BuildClient interface defines generic operations. A factory function (NewBuildClient) provides authenticated client instances; currently, it exclusively returns a Chrome-specific client but is designed for future multi-project support.

2. Chrome/Chromium Specific Implementation (build_chrome.go):

   • Why: Chromium builds have unique requirements: specific builders, buckets (try, ci), custom request properties (e.g., git_repo, deps_revision_overrides), and nuanced logic for finding existing builds (e.g., handling patches or DEPS rolls differently for “try” vs. “waterfall” builds).
   • How: buildChromeClient encapsulates this logic. It maps Pinpoint's generic workflows.BuildParams to Buildbucket-specific requests, leveraging bot_configs for device-to-builder translation. Its FindBuild method implements a precise search strategy: first for Pinpoint-initiated “try” builds, then conditionally falling back to “waterfall” (CI) builds if no patches or DEPS rolls are present.

3. Generic Request/Response Wrappers (types.go):

   • Why: To maintain the BuildClient interface's independence from concrete Buildbucket protobufs or other API-specific data structures.
   • How: Simple structs like FindBuildRequest and StartBuildRequest use any to hold the actual, system-specific request/response payloads. Concrete client implementations then perform type assertions.

4. Delegation to Backend Client:

   • Why: To separate the module's logic (mapping Pinpoint concepts to build system operations) from the low-level concerns of making API calls.
   • How: buildChromeClient embeds backends.BuildbucketClient to handle the actual HTTP requests and protobuf serialization for Buildbucket.

Key Components

• build_client.go: Defines the BuildClient interface, which is the primary API contract for build operations. It also contains NewBuildClient for creating authenticated instances.
• build_chrome.go: Provides the concrete buildChromeClient implementation for Chrome/Chromium projects. This file contains the detailed logic for constructing Buildbucket requests, translating Pinpoint parameters, and implementing the specific build search heuristics.
• types.go: Contains lightweight, generic request and response structs that wrap system-specific payloads, enabling the BuildClient interface to remain generic.

Build Request Workflow

graph TD
    A["Pinpoint Workflow needs a build"] --> B{"Call BuildClient.FindBuild or
    StartBuild"}

    subgraph "Find Build"
        B --> C{"build_chrome.go: Create ChromeFindBuildRequest"}
        C --> D{"Resolve Device to Buildbucket Builder"}
        D --> E{"Attempt BuildbucketClient.GetSingleBuild
        <br>(Pinpoint-tracked builds)"}
        E -- "No Build & No Patches/DEPS" -->
        F{"Attempt BuildbucketClient.GetBuildFromWaterfall<br>(CI builds)"}
        E -- "Build Found or Waterfall Found" --> G["Return Build Info"]
        F -- "No Build Found" --> H["Return No Build Found"]
        E -- "No Build & Has Patches/DEPS" --> H
    end

    subgraph "Start Build"
        B --> I{"build_chrome.go: Create Buildbucket ScheduleBuildRequest"}
        I --> J{"Inject Chrome-specific Properties<br>(e.g., git_repo, revision,
        deps_overrides)"}
        J --> K{"Inject Pinpoint Tags"}
        K --> L{"Call BuildbucketClient.StartBuild"}
        L --> M["Return New Build Info"]
    end

Module: /go/clients/upload

This module offers a unified client for uploading data to backend storage, specifically Google BigQuery for Chrome performance data.

The UploadClient interface (upload_client.go) abstracts the underlying storage technology, decoupling consumers from specific backends (e.g., BigQuery, Cloud SQL). This design enables future flexibility for different storage solutions.

Key Components

• upload_client.go: Defines the module's public API.

  • UploadClient interface: Specifies CreateTableFromStruct (to define table schema from a Go struct using tags) and Insert (to upload data rows). This is the primary interaction point.
  • NewUploadClient factory: Returns a concrete UploadClient implementation, currently uploadChromeDataClient.

• upload_chrome_data.go: Implements UploadClient for BigQuery-specific operations.

  • uploadChromeDataClient: Manages BigQuery interactions for a given project, dataset, and table. It leverages go/backends.BigQueryClient to perform actual BigQuery API calls.
  • How it works: NewUploadClient instantiates uploadChromeDataClient, which establishes a connection to BigQuery. Subsequent calls to CreateTableFromStruct or Insert translate generic requests into BigQuery-specific operations, utilizing Go struct tags (e.g., bigquery:"column_name") for schema mapping and data insertion.

Workflow

graph TD
    A["Application"] --> B{"Configure UploadClientConfig"}
    B --> C["Call NewUploadClient(config)"]
    C --> D{"Get UploadClient instance"}
    D -- "(Optional) Create Table Schema" -->
    E["Call CreateTableFromStruct(DefinitionStruct)"]
    E -- "Delegates to" --> F["go/backends.BigQueryClient.CreateTable"]
    D -- "Upload Data Rows" --> G["Call Insert(DataRowsStructs)"]
    G -- "Delegates to" --> H["go/backends.BigQueryClient.Insert"]

Module: /go/common

The /go/common module defines core data structures and utilities for CombinedCommit objects. These objects encapsulate a main repository commit alongside overridden dependency commits, crucial for defining specific test environments where a main project is tested against particular versions of its dependencies.

Design Decisions & Implementation Choices

The CombinedCommit type extends the pinpoint_proto.CombinedCommit protobuf message, adding domain-specific logic and helper methods. This keeps the protobuf definition pure for serialization while providing rich functionality for business operations.

• Dependency Management: The ModifiedDeps field (a slice of pinpoint_proto.Commit) allows flexible specification of multiple dependency overrides. This enables testing a main project (e.g., Chromium) with specific versions of its transitive dependencies (e.g., V8, Skia), offering granular control over the build environment.
• Identity and Hashing: The Key() method generates a unique FNV hash for a CombinedCommit. This serves as a lightweight, non-cryptographic identifier, suitable for caching and map indexing, facilitating efficient tracking and comparison of build requests.
• Dynamic Updates: The UpsertModifiedDep() method efficiently adds or updates dependency commits. Despite its linear time complexity, this design is pragmatic, as the number of modified dependencies is expected to remain small, preventing performance bottlenecks.
• Convenience Constructors: Functions like NewChromiumCommit() and NewCommit() simplify pinpoint_proto.Commit instance creation, abstracting common setup patterns.

Key Components & Responsibilities

The combined_commit.go file implements the CombinedCommit type and its methods. It is responsible for:

• Representing Complex Commits: Encapsulating a main repository commit with its specific dependency overrides.
• Commit Manipulation: Providing methods to add, update, or retrieve dependency information (UpsertModifiedDep, DepsToMap, GetLatestModifiedDep).
• Cloning and Identity: Offering Clone() for creating independent copies and Key() for generating unique identifiers.

Workflow: Building a Combined Commit

graph TD
    A[Start with Main Repository Commit] --> B{Add/Update a specific Dependency?};
    B -- Yes --> C[Specify Dependency Repository and GitHash];
    C --> D[Upsert Dependency to CombinedCommit];
    B -- No / More Dependencies? --> B;
    B -- Done --> E[Resulting CombinedCommit Object];
    E --> F[Can be used for Bisection/Testing];

Module: /go/compare

The compare module provides statistical methods to determine if two sets of measurements are significantly different, similar, or inconclusive. It is crucial for automated systems like Pinpoint to identify performance regressions or functional failures efficiently.

Design and Implementation Choices

• Robust Statistical Testing: To account for varying data characteristics and sensitivity requirements, the module employs both the Kolmogorov-Smirnov (KS) test, which is effective for detecting differences in distribution shapes, and the Mann-Whitney U (MWU) test, which is robust for detecting shifts in medians. The minimum p-value from these two tests is used as the overall significance measure.
• Optimized Decision-Making with Dual Thresholds: Instead of a single statistical significance level, the module utilizes a dual-threshold system (provided by pinpoint/go/compare/thresholds). A LowThreshold identifies statistically Different samples, while a HighThreshold allows for early classification of Same samples, thereby conserving computational resources in iterative bisection processes. P-values falling between these thresholds result in an Unknown verdict, suggesting the need for more data.
• Contextual Comparison Logic: The comparison logic adapts based on the type of data being analyzed:
  • Performance Bisections: Handle rational, non-negative numbers typical of benchmark results. They incorporate a “small difference” check to avoid bisecting on trivial changes and normalize magnitudes based on the interquartile range (IQR) for threshold calculation.
  • Functional Bisections: Focus on failure rates (represented as 0s and 1s). They leverage specialized thresholds tailored for binary data, generally assuming an “improvement” means a decrease in failure rate.
• Pairwise Comparison for Related Samples: For comparisons involving paired observations (e.g., before-and-after measurements), the ComparePairwise function is used. This delegates to the Wilcoxon Signed-Rank Test (provided by pinpoint/go/compare/stats), which supports data transformations (e.g., logarithmic) and provides confidence intervals relevant for percentage changes, making it suitable for analysis of relative effects.

Key Components

• compare.go: This file defines the core comparison orchestration. It introduces Verdict (e.g., Same, Different, Unknown) and ImprovementDir (e.g., Up, Down, UnknownDir) enums. It exposes high-level functions like CompareFunctional, ComparePerformance, and ComparePairwise which encapsulate the specific comparison logic, threshold application, and data pre-processing necessary for each context.
• kolmogorov_smirnov.go: Implements the 2-sample Kolmogorov-Smirnov (KS) test. It calculates the maximum difference between the empirical cumulative distribution functions (CDFs) of two independent samples to determine their statistical similarity.
• mann_whitney_u.go: Implements the Mann-Whitney U (MWU) test. This non-parametric test determines if two independent samples are drawn from the same distribution by comparing their medians through rank analysis, robustly handling tied values.
• pinpoint/go/compare/stats: This submodule provides the Wilcoxon Signed-Rank Test, primarily used for ComparePairwise. It supports exact and asymptotic calculations, data transformations (LogTransform, NormalizeResult), and confidence interval computation.
• pinpoint/go/compare/thresholds: This submodule manages the dual-threshold system (LowThreshold, HighThreshold). It provides pre-computed high thresholds that dynamically adjust based on sample size and expected magnitude, optimizing the bisection process by enabling quicker decisions.

Main Comparison Workflow

graph TD
    A["Input: valuesA, valuesB, Context (Performance/Functional,
    Direction, Magnitude)"] --> B{"Determine if small difference
    (Performance only)?"}
    B -- Yes --> C["Verdict: Same (IsTooSmall)"]
    B -- No --> D{"Calculate p-values for KS and MWU tests"}
    D --> E["PValue = min(PValueKS, PValueMWU)"]
    E --> F{"Retrieve LowThreshold and context-specific HighThreshold"}
    F --> G{"PValue <= LowThreshold?"}
    G -- Yes --> H["Verdict: Different"]
    G -- No --> I{"PValue <= HighThreshold?"}
    I -- Yes --> J["Verdict: Unknown (collect more data)"]
    I -- No --> K["Verdict: Same"]
    H --> L["Return CompareResults"]
    J --> L
    K --> L
    C --> L

Module: /go/compare/stats

This module provides a Go implementation of the Wilcoxon Signed-Rank Test, primarily for pairwise statistical analysis in performance benchmarking, as used in projects like Pinpoint. It enables non-parametric comparison of related samples, offering precise p-values and confidence intervals.

Key Components and Responsibilities

• wilcoxon_signed_rank.go: Core implementation of the Wilcoxon Signed-Rank Test.
  • WilcoxonSignedRankedTest: General function for one-sample or paired-sample tests.
  • PairwiseWilcoxonSignedRankedTest: Specialized function for Pinpoint's pairwise job context, providing LogTransform, NormalizeResult, and OriginalResult options for data preprocessing and result interpretation (e.g., percentage change).
• sign_rank.go: Provides the exact probability distribution functions (pSignRank for CDF, qSignRank for quantile) for the Wilcoxon Signed-Rank statistic, crucial for small sample sizes.
• zeroin.go: Implements Brent's root-finding algorithm (zeroin). This is used by wilcoxon_signed_rank.go to numerically compute confidence interval boundaries when the normal approximation is applied.

Design and Implementation Choices

• Wilcoxon Signed-Rank Test: Chosen for its robustness in comparing related samples without assuming normal distribution, which is suitable for many performance metrics. The implementation is a Go rewrite of an R equivalent, ensuring quantitative consistency.
• Exact vs. Asymptotic Methods: The module dynamically switches between exact calculations (for sample sizes less than 50, providing higher precision) and normal approximation (for larger samples, enhancing computational efficiency).
• Data Transformation Support: Offers LogTransform (for metrics with multiplicative effects or skewed distributions), NormalizeResult (for expressing changes as percentages relative to a baseline), and OriginalResult (for raw differences). This adapts the test to various data characteristics encountered in performance analysis.
• Confidence Intervals: Calculations include confidence intervals (CI), providing an estimated range of plausible values for the true median difference, which complements p-values for a more complete statistical inference.
• Handling Ties and Zero Differences: The rank function and subsequent logic correctly account for tied values and zero differences within the samples, maintaining statistical validity.

Workflow: WilcoxonSignedRankedTest

graph TD
    A["WilcoxonSignedRankedTest(x, y, alt)"] --> B{"Calculate Differences: d = x - y"}
    B --> C{"Filter non-zero differences: d_non_zero"}
    C --> D{"Is Sample Size < 50 AND no Ties/Zeroes?"}
    D -- Yes --> E["Compute Exact P-Value (sign_rank.go: newWilcoxonDistribution, pSignRank)"]
    E --> F["Compute Exact CI (getDiffs, sign_rank.go: qSignRank)"]
    D -- No (Normal Approx) --> G["Compute Z-statistic & Sigma (accounting for ties)"]
    G --> H["Compute Asymptotic P-Value (Standard Normal CDF)"]
    H --> I["Compute Asymptotic CI (zeroin.go: zeroin for asymptoticW)"]
    F --> J["Return WilcoxonSignedRankedTestResult"]
    I --> J

Module: /go/compare/thresholds

The thresholds module optimizes hypothesis testing for iterative systems like Pinpoint bisection. It provides a dual-threshold system to efficiently determine if two data samples are statistically different, similar, or require more data, thereby minimizing computational resources.

Design and Implementation

• Dual Threshold Approach: Beyond the standard LowThreshold (0.05) for detecting significant differences, a dynamic HighThreshold is introduced. This high threshold enables early conclusions of “no significant difference” when the p-value is sufficiently high, accelerating decisions and reducing sampling costs in bisection processes.
• Pre-computed Lookups: High thresholds are stored in static tables (highThresholdsFunctional, highThresholdsPerformance) instead of being calculated on-the-fly. This design ensures rapid, predictable lookups, offloading computationally intensive statistical derivations to external generation scripts.
• Contextual Adaptation: Thresholds adjust based on normalized_magnitude and sample_size. Different magnitude normalizations are provided for functional (failure rate) and performance (interquartile range) comparisons, ensuring appropriate statistical sensitivity across diverse measurement types.

Key Components

• thresholds.go: Defines the LowThreshold constant and provides HighThresholdPerformance and HighThresholdFunctional functions. These functions query the pre-calculated tables to return the relevant high threshold for a given magnitude and sample size.

Decision Workflow

graph TD
    A["Hypothesis Test P-value"] --> B{"P-value < LowThreshold?"}
    B -- Yes --> C["Result: Different Samples (Reject Null)"]
    B -- No --> D{"P-value > HighThreshold?"}
    D -- Yes --> E["Result: Similar Samples (Reject Alternative)"]
    D -- No --> F["Result: Inconclusive (Need More Data)"]

Module: /go/frontend

The /go/frontend module is the public-facing HTTP service for the Pinpoint job management system. It provides a web-based user interface and a set of HTTP APIs for managing Pinpoint jobs, retrieving benchmark configurations, and initiating job executions via an internal Pinpoint core service. It orchestrates various backend components (database, core Pinpoint service) to deliver a cohesive user experience.

Design Principles

• Delegated Core Logic: Actual Pinpoint job execution and management are delegated to an internal pinpoint_service. This abstracts workflow complexities, allowing frontend to focus on presentation, data aggregation, and user interaction.
• Hybrid Data Sourcing: Job metadata is stored and retrieved from a jobs_store (PostgreSQL) for persistence and querying. Static configurations (benchmarks, stories, bots) are embedded in benchmarks.json, simplifying deployment and ensuring consistent configuration.
• Separation of Concerns: Application bootstrapping, infrastructure setup (e.g., database connectivity, HTTP server), and static asset serving are isolated in /go/frontend/cmd. Core business logic, API definition, and UI rendering are handled by /go/frontend/service, promoting modularity and maintainability.
• Templated UI: Basic HTML templates are served directly by the Go application, providing a functional user interface without requiring a separate frontend application framework.

Responsibilities and Key Components

/go/frontend/cmd This module is the executable entry point, responsible for bootstrapping the application and launching the HTTP server.

• main.go:
  • Database Connectivity: Establishes and manages a PostgreSQL connection pool using pgxpool for efficient, centralized database access.
  • Service Initialization: Instantiates jobs_store for data persistence and initializes the /go/frontend/service module, which encapsulates the application's core logic and registers its HTTP API handlers.
  • HTTP Server Setup: Configures the chi router, registers API and static asset routes, and starts the HTTP server, making the Pinpoint frontend accessible.

/go/frontend/service This module provides the HTTP API and user interface for the Pinpoint system, orchestrating data display and core service interactions.

• Service struct (jobs.go): The central orchestrator, holding references to the jobs_store, HTML templates, benchmark configurations, and the pinpoint_service's HTTP handler. Its RegisterHandlers method defines all public API and UI endpoints.
• HTTP Handlers (jobs.go):
  • Job Management: ListJobsHandler and GetJobHandler provide API endpoints for querying and retrieving detailed job information from the jobs_store, including validation for pagination and date ranges.
  • Configuration Discovery: ListBenchmarksHandler, ListBotConfigurationsHandler, and ListStoriesHandler expose the embedded benchmarks.json data as JSON APIs, supporting dynamic UI population and input validation.
  • UI Pages: Dedicated templateHandler instances render landing-page.html and results-page.html for user interaction.
  • Pinpoint Core Proxy: The pinpointHandler forwards requests to /pinpoint/* directly to an HTTP adapter for the internal pinpoint_service, allowing the frontend to trigger and monitor actual Temporal-backed Pinpoint workflow executions.
• benchmarks.json: An embedded static JSON file defining available benchmarks, their stories, and supported bots, used for configuration APIs and UI validation.

Workflows

Application Initialization

graph TD
    A["Start Application (main.go)"] --> B{"Parse Flags & Context"}
    B --> C["Configure & Connect to PostgreSQL (pgxpool)"]
    C --> D["Initialize JobStore (jobs_store)"]
    C --> D["Initialize JobStore (jobs_store)"]
    D --> E["Initialize Frontend Service (frontend/service)"]
    E --> F["Register HTTP Handlers (chi.Router)"]
    F --> G["Serve Static Assets"]
    G --> H["Start HTTP Server (ListenAndServe)"]

Request Handling

graph TD
    A["Client/User Interface"] -->|HTTP Request| B("Pinpoint Frontend Service")

    subgraph "Pinpoint Frontend Service"
        B -- "HTTP GET /json/jobs/list" --> D("Retrieve Jobs")
        D --> E["JobStore (SQL)"]
        B -- "HTTP GET /benchmarks" --> F("Retrieve Benchmarks")
        F --> G["Embedded benchmarks.json"]
        B -- "HTTP POST /pinpoint/job/create" --> H("Proxy to Core Service")
        H --> I["Pinpoint Core Service (Temporal)"]
        B -- "Render HTML page" --> J["HTML Templates"]
    end

    E -- "Job data" --> D
    G -- "Benchmark Config" --> F
    I -- "Workflow response" --> H
    D -- "JSON Response" --> A
    F -- "JSON Response" --> A
    H -- "JSON Response" --> A
    J -- "HTML page" --> A

Module: /go/frontend/cmd

The /go/frontend/cmd module is the executable entry point for the Pinpoint frontend service. It bootstraps the application, establishes database connectivity, initializes core services, and starts the HTTP server to handle API requests and serve static web assets.

This module acts as the primary orchestrator, assembling various backend components into a cohesive web service. Its design separates application bootstrapping and infrastructure concerns from the core business logic, promoting modularity. It ensures the application is correctly configured with database access and that the business logic from frontend/service is exposed via HTTP.

Responsibilities and Key Components

• main.go: The application's starting point.
  • Database Connectivity: Parses the connection string and establishes a PostgreSQL connection pool using pgxpool. This centralizes database setup and optimizes connection management.
  • Service Initialization: Instantiates the jobs_store for data persistence and initializes the frontend/service. The frontend/service encapsulates core business logic and registers its HTTP API handlers.
  • HTTP Server Setup: Configures and starts the HTTP server with chi for routing. It also handles serving static UI assets from a specified production directory, making the web interface accessible alongside API endpoints.

graph TD
    A["Start Application (main.go)"] --> B{"Parse Flags & Context"}
    B --> C["Configure & Connect to PostgreSQL (pgxpool)"]
    C --> D["Initialize JobStore (jobs_store)"]
    D --> E["Initialize Frontend Service (frontend/service)"]
    E --> F["Register HTTP Handlers (chi.Router)"]
    F --> G["Serve Static Assets"]
    G --> H["Start HTTP Server (ListenAndServe)"]

Module: /go/frontend/service

This module provides the HTTP API and user interface for the Pinpoint job management system. It acts as a gateway, exposing job metadata, benchmark configurations, and routing core job creation/management requests to an internal Pinpoint service.

Design Principles and Implementation Choices

This service is designed for clear separation of concerns, aggregating data from various sources to serve both programmatic clients and human users:

• Delegated Core Logic: Actual Pinpoint job execution and management is delegated to an internal pinpoint_service. This underlying service, primarily gRPC-based and interacting with Temporal workflows, is exposed via an HTTP/JSON handler. This ensures the frontend module remains focused on presentation and data aggregation, abstracting the complexities of workflow orchestration.
• Hybrid Data Sourcing:
  • Persistent Job Data: Pinpoint job metadata (e.g., status, details, history) is stored and retrieved from a jobs_store (SQL database). This enables scalable querying, filtering, and pagination of job data through dedicated API endpoints.
  • Embedded Static Configuration: Benchmark definitions, available stories, and supported bots are loaded from an embedded benchmarks.json file. Embedding this static data ensures it's always present with the application binary, simplifying deployment and providing consistent configuration for UI elements and input validation.
• Templated UI: HTML templates (landing-page.html, results-page.html) are loaded and served dynamically, providing a basic web-based user interface for interacting with the Pinpoint system without requiring a separate frontend application framework.

Key Components

• Service struct (jobs.go): The central component, orchestrating all functionality. It holds references to the jobs_store, HTML templates, benchmark configurations, and the pinpoint_service's HTTP handler. Its RegisterHandlers method defines all public API and UI endpoints.
• HTTP Handlers (jobs.go):
  • Job Management: ListJobsHandler and GetJobHandler provide API endpoints for querying and retrieving detailed information about Pinpoint jobs from the jobs_store. These handlers include robust validation for pagination and date range parameters.
  • Configuration Discovery: ListBenchmarksHandler, ListBotConfigurationsHandler, and ListStoriesHandler expose the embedded benchmark configuration data as JSON APIs. These are critical for dynamically populating user interfaces with valid options for creating new Pinpoint jobs.
  • UI Pages: Dedicated templateHandler instances serve the static HTML pages, providing the user-facing interface.
  • Pinpoint Core Proxy: Requests to /pinpoint/* are forwarded to the pinpointHandler, which is an HTTP adapter for the internal pinpoint_service instance. This allows the frontend to trigger and monitor actual Temporal-backed Pinpoint workflow executions.
• benchmarks.json: An embedded JSON file that provides static configuration for available benchmarks, their associated stories, and supported bots. This data underpins the frontend's configuration APIs and guides UI input validation.

Workflow

graph TD
    A["Client/User Interface"] -->|HTTP Request| B("Pinpoint Frontend Service")

    subgraph "Pinpoint Frontend Service"
        B -- "HTTP GET /json/jobs/list" --> D("Retrieve Jobs")
        D --> E["JobStore (SQL)"]
        B -- "HTTP GET /benchmarks" --> F("Retrieve Benchmarks")
        F --> G["Embedded benchmarks.json"]
        B -- "HTTP POST /pinpoint/job/create" --> H("Proxy to Core Service")
        H --> I["Pinpoint Core Service (Temporal)"]
        B -- "Render HTML page" --> J["HTML Templates"]
    end

    E -- "Job data" --> D
    G -- "Benchmark Config" --> F
    I -- "Workflow response" --> H
    D -- "JSON Response" --> A
    F -- "JSON Response" --> A
    H -- "JSON Response" --> A
    J -- "HTML page" --> A

Module: /go/midpoint

The /go/midpoint module provides the core logic for identifying the next candidate commit in a bisection process. Pinpoint uses bisection to efficiently locate a regression‘s root cause by repeatedly narrowing down a range of commits. This module’s central role is to calculate the “midpoint” within a given commit range.

Responsibilities and Key Components

The midpoint module's primary responsibility is to determine the next commit to test, handling both direct repository changes and changes introduced via dependency rolls (DEPS).

• MidpointHandler: This is the main orchestrator for midpoint calculations. It manages Gitiles clients for interacting with various repositories and coordinates the recursive search for the midpoint.
• FindMidCombinedCommit: The module's public entry point. It takes two common.CombinedCommit objects (which represent a main repository commit combined with its modified dependencies) and returns the next midpoint. This function is designed to:
  • Bisect the main repository: If no modified dependencies are involved, it finds the median commit directly within the main repository's history.
  • Handle DEPS rolls: If the two main commits are adjacent, it assumes a DEPS roll. It then parses the DEPS files for both commits, identifies the first differing git-based dependency, and recursively bisects within that dependency's history.
  • Manage CombinedCommit state: It ensures consistent comparison of CombinedCommit objects by dynamically filling in implied dependency hashes through fillModifiedDeps. This is crucial when one CombinedCommit explicitly references a dependency change, and the other implicitly uses its parent's version of that dependency.

Design and Implementation Choices

The module's design revolves around the common.CombinedCommit structure, which allows bisection to traverse across different repositories - starting with a main repository (like Chromium) and then “diving” into its git-based dependencies when a DEPS roll is detected.

• Recursive search for culprits: When two commits are found to be adjacent, the system's design anticipates a DEPS roll. Instead of terminating, it shifts the bisection focus to the dependencies defined in the DEPS file. This allows Pinpoint to pinpoint regressions caused by changes in upstream libraries.
• fillModifiedDeps for state consistency: Bisection paths can involve CombinedCommit objects with varying levels of explicit dependency information. fillModifiedDeps ensures that both ends of a comparison have their ModifiedDeps arrays consistently populated from their respective DEPS files. This normalization is vital for accurate comparison and subsequent bisection steps, allowing {C@1} to be correctly compared against {C@1, V8@4} by implicitly defining V8@3 for C@1.
• Support for git-based DEPS only: The implementation currently focuses on git-based dependencies, filtering out CIPD-based dependencies during DEPS parsing. This simplifies the initial scope but means CIPD-related regressions cannot be bisected this way.
• Single-layer DEPS roll focus: A current limitation is that the module only identifies and bisects the first differing git-based dependency in a DEPS roll. It does not handle scenarios where multiple dependencies are rolled simultaneously or where a rolled dependency itself contains another roll that needs further bisection when adjacent. This is a pragmatic choice to simplify the initial implementation.

Workflow: Finding a Midpoint

graph TD
    A["Start: FindMidCombinedCommit(startCC, endCC)"] --> B{"startCC == endCC?"}
    B -- Yes --> C["Error: Identical commits"]
    B -- No --> D{"Any ModifiedDeps?"}

    D -- Yes --> E["Fill ModifiedDeps for consistency"]
    E --> F["Identify latest differing ModifiedDep: startDep, endDep"]
    F --> G{"startDep.GitHash == endDep.GitHash?"}
    G -- Yes --> H["Handle edge cases: DEPS roll boundaries"]
    G -- No --> I["Find midpoint within ModifiedDep: findMidCommit(startDep, endDep)"]
    I --> J["Return new CombinedCommit with updated ModifiedDep"]

    D -- No --> K["No ModifiedDeps: Bisect Main commit"]
    K --> L["Find midpoint in Main: findMidCommit(startCC.Main, endCC.Main)"]
    L --> M{"Midpoint is in Main?"}
    M -- Yes --> N["Return new CombinedCommit with updated Main"]
    M -- No --> O["Midpoint is a new dep: Add to ModifiedDeps"]
    O --> J

    subgraph "findMidCommit(startCommit, endCommit)"
        P["Try findMidpoint(startCommit, endCommit)"] --> Q{"Result is startCommit (adjacent)?"}
        Q -- No --> R["Return found midpoint"]
        Q -- Yes --> S["Assume DEPS roll: Call findMidCommitInDEPS(startCommit, endCommit)"]
        S --> T{"DEPS roll midpoint found?"}
        T -- No --> U["Return startCommit (no further mid)"]
        T -- Yes --> V["Return DEPS midpoint"]
    end

Module: /go/read_values

The read_values module provides an API to retrieve and process performance benchmark results stored in RBE Content-Addressable Storage (CAS) for Pinpoint workflows.

Core Responsibility

This module is responsible for fetching raw performance sample values and applying optional data aggregations (e.g., mean, min) for specified benchmarks and charts from a list of CAS digests.

Design and Implementation

• CAS Abstraction: The CASProvider interface decouples the core logic from specific CAS backend implementations. This design promotes testability and allows for future extensions beyond RBE without modifying the perfCASClient.
• RBE Integration: rbeProvider implements CASProvider for RBE-CAS, utilizing cabe/go/backends for efficient data retrieval. DialRBECAS centralizes the connection to RBE, accommodating Pinpoint's use of multiple Swarming instances to store CAS results. This ensures correct CAS client selection for specific data.
• Data Processing: perfCASClient encapsulates the logic to fetch raw perfresults.PerfResults from CAS digests, filter them by benchmark and chart, and apply specified aggregation methods. This transforms low-level CAS content into structured performance metrics required by Pinpoint.
• Aggregation: Integration with perfresults.AggregationMapping allows users to select statistical aggregation methods. This enables the module to return summarized performance data, which is vital for analysis, trend detection, and reducing data volume, instead of large sets of raw samples.

Key Components

• read_values.go: Implements the CASProvider interface, the perfCASClient responsible for fetching and processing performance data, and public methods for reading values by chart or for all charts, including aggregation logic.
• read_values_test.go: Contains unit tests that validate the data fetching, filtering, and aggregation functionalities using a mocked CASProvider to ensure correctness and isolation.

Workflow

graph TD
    A[Request Benchmark Data] --> B{Call DialRBECAS};
    B --> C[Initialize perfCASClient for specific Swarming instance];
    C --> D{Call ReadValuesByChart or ReadValuesForAllCharts};
    D --> E[For each CAS Digest in request];
    E --> F[Fetch raw PerfResults from CAS via rbeProvider];
    F -- On Success --> G[Filter results by Benchmark/Chart];
    G --> H{Aggregation Method Specified?};
    H -- Yes --> I[Apply aggregation to samples];
    H -- No --> J[Collect all raw samples];
    I --> K[Append processed values];
    J --> K;
    K --> E;
    E -- All Digests Processed --> L[Return TestResults];
    F -- On Error --> M[Return Error];
    D -- On Error --> M;
    B -- On Error --> M;

Module: /go/run_benchmark

The run_benchmark module orchestrates performance benchmark executions as Swarming tasks for Pinpoint. It dynamically constructs benchmark commands based on specified parameters, schedules these tasks on Swarming, and offers utilities to monitor and interpret task statuses.

Core Components and Responsibilities

• run_benchmark.go: Manages the lifecycle of Swarming tasks for benchmark execution.

  • How: The Run function is the primary entry point, initiating Swarming tasks. It leverages the BenchmarkTest interface to decouple task scheduling from the specifics of command generation.
  • Why: The State type abstracts raw Swarming statuses into meaningful states (e.g., IsTaskSuccessful, IsTaskTerminalFailure), simplifying status interpretation and providing a consistent API for downstream services.

• benchmark_test_factory.go: Dynamically creates the appropriate benchmark test executor.

  • How: The NewBenchmarkTest function acts as a factory, using bot_configs to determine the specific isolate target (e.g., performance_test_suite) for a given benchmark and bot. Based on this, it returns an implementation of the BenchmarkTest interface.
  • Why: This design choice promotes extensibility; new benchmark types can be integrated by implementing the BenchmarkTest interface without altering the core task scheduling logic in run_benchmark.go.

• telemetry.go: Implements the BenchmarkTest interface for Telemetry and Crossbench performance tests.

  • How: The GetCommand method constructs the precise command-line arguments needed to execute a Telemetry or Crossbench test on a Swarming bot, including handling different types of benchmarks (gtests, standard telemetry benchmarks, crossbench) and various parameters like story, story tags, and extra arguments.
  • Why: Centralizes the complex logic of converting Pinpoint‘s abstract benchmark parameters into concrete executable commands compatible with Chromium’s performance testing infrastructure. Maps and specific argument generation functions ensure alignment with Chromium's perf waterfall.

• swarming_helpers.go: Utility functions for building Swarming API requests.

  • How: Contains functions like createSwarmingRequest, generateProperties, convertDimensions, and generateTags to assemble a complete apipb.NewTaskRequest object, populating all necessary fields for a Pinpoint-initiated Swarming task.
  • Why: Decouples the low-level details of Swarming request construction from the higher-level orchestration in run_benchmark.go, improving clarity and maintainability. It also centralizes Swarming-specific configurations like timeouts and service accounts.

Workflow

graph TD
    A[Run Benchmark Request] --> B{Resolve Bot Config & Isolate Target};
    B --> C{Factory: Create BenchmarkTest Instance};
    C -- GetCommand() --> D[Generate Benchmark Execution Command];
    D --> E[Build Swarming Task Request];
    E --> F[Trigger Swarming Task];
    F --> G[Receive Swarming Task IDs];

    subgraph "Benchmark Command Details (telemetry.go)"
        C -- (e.g., TelemetryTest) --> D1{Is it a Waterfall GTest?};
        D1 -- Yes --> D2[Add GTest-specific args];
        D1 -- No, Is it Crossbench? --> D3[Add Crossbench-specific args];
        D3 -- No --> D4[Add Telemetry args];
        D2 & D3 & D4 --> D5[Include Story/StoryTag filters];
        D5 --> D;
    end

    subgraph "Swarming Request Details (swarming_helpers.go)"
        E --> E1[Convert Dimensions];
        E --> E2[Generate Task Properties];
        E --> E3[Generate Tags];
        E1 & E2 & E3 --> E;
    end

Module: /go/service

The /go/service module provides the gRPC API for Pinpoint, enabling users to schedule and query performance analysis jobs. It acts as the orchestration layer, translating user requests into Temporal workflow executions.

Design and Implementation Rationale

The module‘s core design relies on Temporal workflows for executing long-running, stateful performance analysis jobs. This approach leverages Temporal’s fault tolerance, retry mechanisms, and state management capabilities, offloading complex job lifecycle management from the service. The tpr_client.TemporalProvider (go.skia.org/infra/temporal/go/client) abstracts the Temporal client, ensuring testability and adaptability to different Temporal environments.

The service uses gRPC for efficient communication and grpc-gateway (github.com/grpc-ecosystem/grpc-gateway/v2) to automatically generate a RESTful JSON API. This provides flexibility, allowing clients to interact with Pinpoint via either gRPC or standard HTTP/JSON.

A local rate limiter (golang.org/x/time/rate) is integrated to temporarily control the influx of new job requests. This is a pragmatic decision to manage system load, particularly during migration phases or to prevent resource exhaustion, acting as a temporary safeguard.

Workflow execution options consistently set MaximumAttempts: 1 for all scheduled jobs. This design assumes that once a Pinpoint job workflow begins, any non-recoverable failure should not trigger a top-level workflow retry. Instead, retries and error handling are expected to occur within the individual activities composing the workflow.

Input validation (validation.go) is a critical “fail-fast” mechanism. Requests are thoroughly checked before initiating any Temporal workflows, preventing invalid jobs from consuming resources and providing immediate, clear feedback to users.

Key Components

service_impl.go

Implements the Pinpoint gRPC API, defining methods for scheduling and querying jobs, and for job cancellation.

• server struct: Encapsulates the rate.Limiter for request throttling and the tpr_client.TemporalProvider for Temporal interactions.
• New function: The constructor, injecting the Temporal client provider and rate limiter.
• NewJSONHandler: Exposes the gRPC service over HTTP/JSON using grpc-gateway.
• RPC Methods (e.g., ScheduleBisection, ScheduleCulpritFinder, SchedulePairwise, QueryPairwise, CancelJob):
  • Apply rate limiting.
  • Perform initial request validation.
  • Connect to Temporal via tpr_client.TemporalProvider.
  • Generate unique workflow IDs.
  • Configure and execute Temporal workflows with specific timeouts and retry policies.
  • Convert Temporal workflow results or status into gRPC responses.

validation.go

Houses the logic for validating incoming gRPC requests.

• updateFieldsForCatapult, updateCulpritFinderFieldsForCatapult: Temporary functions for migration compatibility, translating legacy Statistic fields to AggregationMethod.
• validateBisectRequest, validateCulpritFinderRequest, validatePairwiseRequest, validateQueryPairwiseRequest: Contains specific validation rules for each job type, ensuring all required parameters are present, valid, and conform to expected formats (e.g., non-empty git hashes, supported aggregation methods, valid UUIDs for job IDs).

Key Workflow: Scheduling a Job

graph TD
    A[Client Sends Schedule Request] --> B{Is Request Rate-Limited?};
    B -- Yes --> C[Return Resource Exhausted Error];
    B -- No --> D{Validate Request Parameters?};
    D -- No (Invalid) --> E[Return Invalid Argument Error];
    D -- Yes --> F[Connect to Temporal Service];
    F -- Error --> G[Return Internal Server Error];
    F -- Success --> H[Start New Temporal Workflow];
    H -- Error --> G;
    H -- Success --> I[Return Job ID to Client];

Key Workflow: Querying a Job

graph TD
    A[Client Sends Query Request] --> B{Validate Request Parameters?};
    B -- No (Invalid) --> C[Return Invalid Argument Error];
    B -- Yes --> D[Connect to Temporal Service];
    D -- Error --> E[Return Internal Server Error];
    D -- Success --> F[Describe Temporal Workflow Execution];
    F -- Error --> E;
    F -- Success --> G{What is Workflow Status?};
    G -- COMPLETED --> H[Fetch Workflow Results];
    G -- FAILED/TIMED_OUT/TERMINATED --> I[Return FAILED Status];
    G -- CANCELED --> J[Return CANCELED Status];
    G -- RUNNING/CONTINUED_AS_NEW --> K[Return RUNNING Status];
    G -- Other --> E;
    H -- Error --> E;
    H -- Success --> L[Return Detailed Results];

Module: /go/sql

This module manages Pinpoint's persistent job data, defining its database schema, generating Spanner SQL DDL, and providing a data access layer for interacting with job records.

Why and How

• Schema as Go Structs: The database schema is defined as Go structs (in schema) annotated with sql tags. This provides type safety, ensures consistency between application code and the database, and enhances maintainability compared to raw SQL DDL.
• Automated DDL Generation: The tosql module programmatically converts these Go struct definitions into Spanner-compatible CREATE TABLE DDL. This guarantees schema alignment, centralizes management, and simplifies controlled evolution by embedding the generated DDL directly into schema/spanner/spanner.go at compile-time.
• Flexible Data Storage: Complex, evolving job parameters and results are stored in JSONB columns within the Jobs table. This design choice provides schema flexibility, minimizing the need for frequent database migrations when new fields or data types are introduced in job definitions or results.
• Decoupled Data Access: The jobs_store module provides a standardized data access layer, decoupling job execution logic from the underlying Spanner database. This ensures job states, parameters, and results are reliably recorded and accessible, supporting the complete lifecycle of a performance experiment.

Responsibilities and Key Components

• schema: Defines the authoritative Go-native data structures, primarily JobSchema, that model the Jobs Spanner table. It uses sql tags to specify column types and constraints, including flexible JSONB fields for dynamic data. The spanner submodule within schema contains the generated Spanner DDL.
• tosql: An executable responsible for generating the Spanner SQL DDL. It reads the Go struct definitions from schema, utilizes the go/sql/exporter utility to translate them into DDL, and writes the output to schema/spanner/spanner.go.
• jobs_store: Manages all persistence and retrieval operations for Pinpoint performance analysis jobs. Its responsibilities include initial job creation (AddInitialJob), status updates (UpdateJobStatus), detailed run data storage (AddCommitRuns), analysis results persistence (AddResults), error reporting (SetErrors), and flexible job retrieval for dashboards (GetJob, ListJobs with filtering and pagination).

Workflows

Schema Definition and Generation

graph TD
    A["Go Structs in schema/schema.go (with 'sql' tags)"] --> B{"tosql executable"}
    B --> C["go/sql/exporter library"]
    C --> D["Generated Spanner DDL (written to schema/spanner/spanner.go)"]
    D --> E["Pinpoint Spanner Database (Schema applied)"]

Job Lifecycle Persistence

graph TD
    A["User Schedules Job"] --> B{"jobs_store.AddInitialJob"}
    B --> C["Job Stored: Pending"]
    C --> D["Job Execution Workflow"]
    D -- "Updates Status" --> E{"jobs_store.UpdateJobStatus"}
    D -- "Stores Run Data" --> F{"jobs_store.AddCommitRuns"}
    D -- "Stores Analysis Results" --> G{"jobs_store.AddResults"}
    D -- "Records Errors" --> H{"jobs_store.SetErrors"}
    E & F & G & H --> I["Job Data Persisted"]
    I --> J{"jobs_store.GetJob / jobs_store.ListJobs"}
    J --> K["Dashboard / Client Retrieval"]

Module: /go/sql/jobs_store

The jobs_store module manages the persistence and retrieval of Pinpoint performance analysis jobs. It provides a robust, standardized interface for other Pinpoint components to interact with job data, decoupling job execution logic from the underlying storage mechanism. This design ensures job states, parameters, and results are reliably recorded and accessible, supporting the complete lifecycle of a performance experiment.

Responsibilities and Key Components

• Job Data Persistence (jobs_store.go): The core JobStore interface and its jobStoreImpl implementation handle all database operations related to jobs.

  • Initial Job Creation: AddInitialJob stores the complete SchedulePairwiseRequest as the foundational record for a new job. This captures all user-defined parameters at the job's inception.
  • Job State Management: UpdateJobStatus tracks the progression of a job through its lifecycle, updating its status and, upon completion, recording its total execution duration.
  • Detailed Run Data Storage: AddCommitRuns stores comprehensive information about the builds and test runs executed for both the start and end commits of a pairwise comparison. This data is critical for understanding the experimental context.
  • Analysis Result Storage: AddResults persists the final statistical analysis results, such as Wilcoxon test outcomes, providing the quantitative summary of the job.
  • Error Reporting: SetErrors captures and stores any errors encountered during job execution, aiding in debugging and user communication.
  • Job Retrieval: GetJob fetches all details for a specific job, while ListJobs supports dashboard views by allowing filtered and paginated retrieval of job summaries.
  • Why structured JSON columns? The module utilizes flexible JSONB columns (e.g., additional_request_parameters, metric_summary) to store complex, evolving data structures (like schema.AdditionalRequestParametersSchema and pinpointpb.PairwiseExecution_WilcoxonResult). This approach avoids frequent schema migrations when new fields or data types are introduced in the job definition or results.

• Dashboard Integration (jobs_store.go): DashboardJob and ListJobsOptions are tailored data structures to efficiently support a user-facing dashboard. ListJobsOptions allows for flexible filtering by criteria such as job name, benchmark, date range, bot, and user, along with pagination, directly addressing the “how” of presenting a manageable list of jobs to users.

Job Lifecycle Workflow

graph TD
    A["User Schedules Job"] --> B{"AddInitialJob"}
    B --> C["Job Stored: Pending"]
    C --> D{"Job Execution Workflow"}
    D -- "Updates Status" --> E{"UpdateJobStatus"}
    D -- "Stores Commit & Run Data" --> F{"AddCommitRuns"}
    D -- "Stores Analysis Results" --> G{"AddResults"}
    D -- "Records Errors" --> H{"SetErrors"}
    E --> I["Job Status: Running/Completed/Failed"]
    F --> J["Additional Parameters (CommitRuns) Updated"]
    G --> K["Metric Summary Updated"]
    H --> L["Error Message Updated"]
    I & J & K & L --> M{"GetJob / ListJobs"}
    M --> N["Dashboard / Client"]

Module: /go/sql/schema

Overview

This module defines the Go-native data structures that represent the schema for Pinpoint's persistent job data, primarily for the Jobs Spanner table. It acts as the programmatic interface for interacting with job records in the database.

Design and Implementation

The module's core design revolves around providing a type-safe, declarative representation of the database schema in Go.

• Go Structs as Schema Definition: schema.go uses Go structs (e.g., JobSchema, AdditionalRequestParametersSchema) with sql:"..." tags to define the database schema. This ensures the application's data models align directly with the database schema, enabling compile-time type checking and reducing runtime mismatches.

• Flexible Data Storage with JSONB: Critical job parameters and results (e.g., AdditionalRequestParameters, MetricSummary) are stored as JSONB fields in the database. This design choice provides flexibility, allowing the addition of new parameters or changes to result formats without requiring schema migrations on the main Jobs table.

• Integration with Core Types: The schema integrates types from other Pinpoint modules, such as go/workflows (for build/test run data) and pinpoint/proto/v1 (for metric summary results). This ensures data consistency and a unified data model across the Pinpoint ecosystem.

• Automated DDL Generation: The actual Spanner DDL (found in the spanner submodule) is automatically generated from these Go structs by a dedicated exporter tool. This centralizes schema management, guarantees consistency, and simplifies controlled schema evolution by preventing manual DDL errors.

Key Components

• schema.go:

  • JobSchema: The primary Go struct modeling the Jobs table. It defines core job metadata like JobID (primary key), JobName, JobStatus, SubmittedBy, Benchmark, and BotName, alongside nested JSONB fields for flexible data.
  • AdditionalRequestParametersSchema: Encapsulates diverse parameters required for a job, such as commit hashes, story details, and aggregation methods. This struct is serialized into a JSONB column within JobSchema.
  • CommitRunData / CommitRuns: Nested structures within AdditionalRequestParametersSchema to store detailed build and test run information for comparative analyses.

• spanner (Submodule): This submodule contains the declarative Spanner SQL DDL for the Jobs table, specifically the Schema constant within spanner.go. This DDL is generated from the Go structs defined in schema.go and includes a TTL policy for data retention.

Workflow: Schema Definition and Deployment

graph TD
    A["Go Structs in schema.go"] --> B{"Annotated with 'sql' tags"}
    B --> C["go/sql/exporter/ Tool"]
    C --> D["go/sql/schema/spanner/spanner.go (Generated CREATE TABLE DDL)"]
    D --> E["Pinpoint Spanner Database (Schema Applied)"]

Module: /go/sql/schema/spanner

This module provides the declarative Spanner SQL schema for the Jobs table.

The schema is automatically generated by //go/sql/exporter/, centralizing schema management and ensuring consistency across the system by avoiding manual definition errors. This also facilitates controlled schema evolution.

The Jobs table is central to Pinpoint, storing comprehensive details about each performance testing job. This includes critical metadata like job_id, job_status, submitted_by, and benchmark. JSONB fields (additional_request_parameters, metric_summary) are used to store flexible, unstructured data, accommodating diverse job configurations and results without rigid schema changes. A TTL (Time-To-Live) policy is applied to automatically purge old job records (after 1095 days), managing data volume and retention.

The sole key component is the Schema constant within spanner.go, which contains the complete CREATE TABLE IF NOT EXISTS Jobs DDL statement.

graph TD
    A["Schema Definition Source (e.g., ORM models)"] --> B["go/sql/exporter/ tool"]
    B --> C["go/sql/schema/spanner/spanner.go (Generated 'Schema' constant)"]
    C --> D["Pinpoint Spanner Database (Schema Applied)"]

Module: /go/sql/tosql

This module generates Spanner SQL DDL from Go struct definitions, ensuring consistency between application data models and the database.

Why and How: The database schema's source of truth is defined as Go structs, specifically pinpoint/go/sql/schema.JobSchema. This approach provides several benefits over directly writing raw SQL DDL:

• Type Safety: Leverages Go's type system for schema definitions.
• Consistency: The same Go structs can be used for ORM, validation, and schema generation, eliminating divergence.
• Maintainability: Easier for Go developers to read, understand, and modify.

The tosql executable uses the go/sql/exporter library to introspect these Go structs and convert them into Spanner-compatible DDL. This DDL is then embedded as a string into a generated Go file (schema/spanner/spanner.go). This makes the DDL available at compile time, simplifying deployment and ensuring the application always uses the correct schema.

Responsibilities:

• Schema Generation: Programmatically creates Spanner DDL from Go struct definitions.
• Schema Consistency: Ensures the database schema is directly derived from and aligned with the application's Go data models.

Key Components:

• main.go: The executable that identifies the Go structs to be exported (e.g., pinpoint/go/sql/schema.JobSchema), invokes the go/sql/exporter library, and writes the generated Spanner DDL to schema/spanner/spanner.go.
• go/sql/exporter: A generic library responsible for converting Go struct definitions into SQL DDL for various database backends, used here for Spanner schema generation.

Workflow:

graph LR
    A["pinpoint/go/sql/schema.JobSchema Go Struct"] --> B("tosql executable (main.go)")
    B --> C{"go/sql/exporter library"}
    C --> D["Spanner DDL as Go String"]
    D --> E["schema/spanner/spanner.go file"]

Module: /go/workflows

This module is the central hub for defining and implementing Pinpoint's backend logic using the Temporal workflow engine. It provides durable, fault-tolerant, and observable execution for complex, long-running tasks like performance bisection and A/B testing. The design separates the public interface of workflows from their underlying implementation and execution environment.

Key Components

• workflows.go: Defines the public contract for all Pinpoint workflows.

  • Why: To decouple clients from the implementation. A client only needs to know a workflow's name (e.g., perf.bisect) and its parameter struct (BisectParams) to start a job, without importing the implementation code.
  • How: It provides constants for workflow names and Go structs for parameters and results. These structs ensure type safety and include helper methods for safely parsing and sanitizing user-provided inputs (e.g., GetMagnitude, GetInitialAttempt).

• internal submodule: Contains the core business logic and implementation of all primary workflows.

  • Why: This module encapsulates the complex orchestration of building Chrome, running benchmarks, and analyzing results.
  • How: It implements key workflows like BisectWorkflow, which uses a concurrent strategy to analyze multiple commit ranges in parallel, and PairwiseCommitsRunnerWorkflow, which runs tests back-to-back on the same bot to minimize environmental noise. It interacts with external services like Swarming and Buildbucket through dedicated activities.

• catapult submodule: A compatibility layer to support the legacy Catapult UI.

  • Why: Ensures a smooth transition from the old Catapult backend by allowing the legacy frontend to work with the modern, Temporal-based system. It is designed for eventual deprecation.
  • How: It acts as an adapter. Workflows in this module, like CatapultBisectWorkflow, orchestrate the core internal workflows and then transform their results into the legacy data format expected by Catapult's datastore and UI.

• worker submodule: The executable that runs the workflow and activity logic.

  • Why: To separate the business logic from the runtime environment. The same workflow code can be executed by different worker configurations for development, staging, or production.
  • How: The worker binary connects to the Temporal server, registers all the workflow and activity functions defined in internal and catapult, and polls a task queue for jobs to execute.

• sample and experiment submodules: Command-line tools for invoking workflows.

  • Why: To provide developers with a direct way to trigger and debug workflows, and to run specialized experiments without needing the full Pinpoint UI.
  • How: These are simple client applications that use the Temporal Go SDK and the public contracts from workflows.go to start workflows with specific parameters from the command line.

High-Level Workflow Architecture

The diagram below illustrates how a client's request is processed. The client uses the public definitions to start a workflow, which is then picked up and executed by a worker process.

graph TD
    subgraph "Client"
        A["Pinpoint API / `sample` CLI"]
    end
    subgraph "Temporal_Server"
        B["Task Queue"]
    end
    subgraph "Worker_Process"
        C["Pinpoint Worker"]
    end

    A -- "1. ExecuteWorkflow (uses `workflows.go` definitions)" --> B
    C -- "2. Polls for tasks" --> B
    B -- "3. Dispatches task" --> C
    C -- "4. Executes implementation (from `internal`, `catapult`)" -->
    D["External Services <br/> (Swarming, Buildbucket)"]

Module: /go/workflows/catapult

The /go/workflows/catapult module acts as a compatibility layer, enabling Pinpoint's modern, Skia-based bisection and culprit-finding logic to integrate with the legacy Catapult UI and datastore. This ensures continuity for existing Catapult users during the transition to a new Pinpoint backend, with the module designed for eventual deprecation.

The module's design focuses on:

• Backwards Compatibility: Bridging the data model and API differences between modern Pinpoint and legacy Catapult.
• Orchestration: Utilizing Temporal workflows to manage the multi-step process of bisection, data transformation, and persistence.
• Data Transformation: Converting complex Pinpoint bisection results into the specific pinpoint_proto.LegacyJobResponse format expected by Catapult's frontend.
• Regression Verification: Implementing a “sandwich verification” pattern to confirm regressions and accurately identify culprits, minimizing false positives.

Key Components

• CatapultBisectWorkflow (in catapult_bisect.go): Orchestrates the execution of a Pinpoint bisection for Catapult. It invokes the core internal.BisectWorkflow for bisection logic, then delegates to ConvertToCatapultResponseWorkflow for data transformation, and finally uses WriteBisectToCatapultActivity to persist results to the Catapult API.
• ConvertToCatapultResponseWorkflow (in catapult_bisect.go): A child workflow that maps modern Pinpoint bisection results and request parameters into the legacy pinpoint_proto.LegacyJobResponse structure. This includes structuring “states,” “attempts,” and “comparisons” as required by the Catapult UI.
• CulpritFinderWorkflow (in culprit_finder.go): Implements the “sandwich verification” process. It first uses PairwiseWorkflow to confirm an initial regression, then CatapultBisectWorkflow to find potential culprits, and finally re-verifies each culprit with PairwiseWorkflow to confirm its significance. Optionally, it can trigger external culprit processing.
• parsers.go: Contains comprehensive logic for translating data between Pinpoint‘s internal models and Catapult’s legacy proto. This includes extracting and formatting commit details (author, message, commit position), run data (values, bot IDs, CAS references), and comparison results.
• activites.go: Provides Temporal activities for interacting with external systems:
  • FetchCommitActivity: Retrieves rich commit metadata from Gitiles.
  • FetchTaskActivity: Gathers detailed Swarming task information.
  • WriteBisectToCatapultActivity: Executes the final write operation to the legacy Catapult API.
• write.go: Defines the CatapultClient and its WriteBisectToCatapult method, responsible for authenticated HTTP POST requests to the legacy Catapult API endpoints to store bisection results.

Core Workflow: Catapult Bisection

graph TD
    A["CatapultBisectWorkflow Start"] --> B{"Generate Workflow ID"}
    B --> C["Execute Child Workflow: internal.BisectWorkflow<br/>
    (Perform Skia-based Bisection)"]
    C -- "BisectExecution" --> D["Execute Child Workflow:
    ConvertToCatapultResponseWorkflow<br/>
    (Transform to Legacy Catapult Format)"]
    D -- "LegacyJobResponse" --> E["Execute Activity:
    WriteBisectToCatapultActivity<br/>(Persist to Catapult Datastore)"]
    E -- "DatastoreResponse" --> F["Log Datastore Response"]
    F --> G["CatapultBisectWorkflow End"]

Core Workflow: Culprit Finder (Sandwich Verification)

graph TD
    A["CulpritFinderWorkflow Start"] --> B{"Execute Child Workflow:
    PairwiseWorkflow<br/>(Verify Initial Regression)"}
    B -- "PairwiseExecution (initial regression)" --> C{"Is Regression
    Significant?"}
    C -- No --> D["End (No Regression Detected)"]
    C -- Yes --> E["Execute Child Workflow: CatapultBisectWorkflow<br/>
    (Find Potential Culprits)"]
    E -- "BisectExecution (potential culprits)" --> F{"Are Culprits Found?"}
    F -- No --> G["End (Regression Detected, No Culprits Found)"]
    F -- Yes --> H["Parallel: For Each Culprit:<br/>Run Culprit Verification
    (PairwiseWorkflow)"]
    H -- "Verified Culprits" --> I["Invoke Culprit Processing Workflow
    (Optional)"]
    I --> J["End (Regression Detected, Culprits Verified)"]

Module: /go/workflows/experiment

The experiment module is a command-line interface for initiating Chrome performance experiments through Temporal. It abstracts experiment setup and execution by delegating to Temporal workflows, ensuring durable and observable execution.

Its primary role is to translate user-defined experiment parameters (benchmark, commit, bot, iterations) into a request for the workflows.TestAndExport Temporal workflow. The TestAndExport workflow then orchestrates Chrome builds, triggers Swarming tasks, and exports task IDs to BigQuery. This design separates experiment initiation from durable, fault-tolerant execution.

Key Component:

• main.go: Contains the executable logic: parses flags, connects to Temporal, constructs internal.TestAndExportParams, and invokes the workflows.TestAndExport workflow. The client initiates the workflow with a unique ID and a single-attempt retry policy, meaning it won't auto-retry starting the workflow if the initial call fails.

Workflow:

graph TD
    A["User executes `experiment` CLI"] --> B{"Parses Flags"}
    B --> C["Connects to Temporal Service"]
    C --> D["Constructs TestAndExportParams"]
    D --> E["Triggers workflows.TestAndExport Temporal Workflow"]
    E -- "Exports Swarming Task IDs to BQ" --> F["External System (e.g.,
    `collectResults`) for result collection"]

Module: /go/workflows/internal

This module contains the core implementation of Pinpoint's Temporal workflows. It orchestrates complex tasks such as performance bisection, A/B testing (pairwise), and continuous benchmarking for the Chrome for Benchmarking (CBB) project. The design separates concerns into distinct workflows and activities for building Chrome, running tests, analyzing results, and interacting with external services like Swarming, Buildbucket, and Gerrit.

Bisection Workflows

The primary goal is to automatically find the single commit that introduced a performance regression within a given commit range.

• bisect.go: Implements the main BisectWorkflow. It employs a concurrent bisection strategy using a workflow.Selector to manage multiple benchmark runs and comparisons simultaneously. This non-blocking approach allows the workflow to efficiently explore different sub-ranges of the original commit range in parallel. If a comparison between two commits is statistically inconclusive, the workflow dynamically increases the number of test runs to gather more data before making a decision.
• bisect_run.go: Provides the BisectRun struct, which abstracts the state of benchmark runs for a single commit. This is crucial because a commit's runs may be initiated by different bisection sub-problems or increased in number over time. It decouples the management of test runs from the main bisection logic.
• midpoint.go: Contains the FindMidCommitActivity, which is essential for the bisection algorithm. Its logic is more complex than a simple (start+end)/2 because it must correctly handle dependency rolls within the Chromium DEPS file, ensuring the true midpoint is found even when the regression is in a dependency like V8 or Skia.

graph TD
    A[Start Bisect] --> B{Initial Pair Runs};
    B --> C{Compare Results};
    C -- Different --> D{Find Midpoint};
    C -- Same --> E[No Culprit Found];
    C -- Unknown --> F{Schedule More Runs};
    F --> C;
    D --> G{Is Range a Single Commit?};
    G -- Yes --> H[Report Culprit];
    G -- No --> I{Create New Sub-Ranges};
    I --> J[Run Benchmark on Midpoint];
    J --> C;

Pairwise Comparison Workflows

These workflows provide a statistically rigorous A/B comparison between two specific commits or builds. The design prioritizes minimizing environmental noise.

• pairwise.go: The main PairwiseWorkflow orchestrates the comparison, persists job status and results to a database via database_activities.go, and handles the overall lifecycle of a pairwise job.
• pairwise_runner.go: Implements the PairwiseCommitsRunnerWorkflow. Its key design choice is to execute benchmark pairs back-to-back on the same physical bot, alternating the run order (A then B, B then A). This minimizes noise from machine-to-machine variations. The workflow also includes logic to balance the dataset by discarding pairs if one of the runs fails, preserving the statistical integrity of the analysis.
• compare.go: Defines activities that perform the statistical analysis. It strategically separates functional analysis (e.g., crash rates) from performance analysis. A significant functional difference often indicates a clear regression and can conclude the analysis early without needing to evaluate performance metrics.

Benchmark Execution and Data Collection

These are the foundational components that perform the actual work of building and testing.

• commits_runner.go: The SingleCommitRunner workflow is a reusable unit that orchestrates the entire process for one commit: building, running a set of benchmarks, and collecting all results. It is the primary building block for both bisection and pairwise workflows.
• build_workflow.go: Encapsulates all interaction with the build system (Buildbucket). It intelligently finds existing builds to reuse or triggers new ones, waits for completion, and fetches the resulting build artifact (CAS reference).
• run_benchmark.go: Manages the execution of a single benchmark test on Swarming. It includes robust polling and retry logic, especially for NO_RESOURCE errors, which is critical for reliability. The RunBenchmarkPairwiseWorkflow specifically handles scheduling two tasks back-to-back on the same bot.
• fetch_and_process.go: Provides a data pipeline to post-process results. It can query a BigQuery log of all Swarming tasks for a workflow, retrieve detailed sample values for each task, and upload the processed data to a separate BQ table for analysis.

Chrome for Benchmarking (CBB) Workflows

This is a specialized system for continuously monitoring the performance of major browsers.

• cbb_new_release_detector.go: The top-level workflow that periodically checks for new official releases of Chrome, Edge, and Safari.
• chrome_releases_info.go: Fetches the latest browser version information (e.g., from Chromium Dash). It compares this with historical data stored in GCS (gcs_store.go) to detect new versions. Upon detecting a new release, it commits updated metadata files into the Chromium repository using the git.go activities.
• cbb_runner.go: Triggered after a new browser release is committed. It runs a standard suite of CBB benchmarks (e.g., Speedometer3, Jetstream2). A key responsibility is to format the raw benchmark output into the standard Perf ingestion format and upload it to a GCS bucket, where it can be picked up by the Skia Perf dashboard.
• stp_downloader.go: A utility workflow to automatically download new versions of Safari Technology Preview and package them for deployment to test devices via CIPD.

Supporting Components

• database_activities.go: Provides a clean interface for workflows to interact with a SQL database for persisting job state, parameters, and results. This supports the Pinpoint UI.
• bug_update.go: Contains activities to interact with an issue tracker, enabling the system to automatically post comments on bugs with the results of a bisection or pairwise run.
• options.go: Centralizes all Temporal retry policies and timeouts. This separation makes it easy to configure and maintain the reliability and execution constraints for different types of workflows and activities (e.g., short local activities vs. long-running builds).
• testdata: Contains data for reproducible tests of the complex bisection logic, including DEPS files with specific commit hashes and a recorded Temporal event history.

Module: /go/workflows/internal/testdata

This module contains sample data and configurations essential for testing the go/workflows bisection logic.

DEPS files (e.g., LatestChromiumDEPS, NMinusOneV8DEPS) define specific, fixed commit hashes for external dependencies like V8 and Skia. These are used to create reproducible test scenarios for the bisection algorithm, representing “good” and “bad” states or distinct points in a commit range. This ensures that the bisection process can be consistently evaluated against known commit references.

bisect_event_history_20240627.json captures a complete perf.bisect workflow execution as a Temporal event history. It serves as a golden test case to validate the workflow's intricate logic and state transitions, demonstrating how the system iteratively narrows down a problematic commit. This file is crucial for verifying that the bisection correctly orchestrates child workflows and activities to pinpoint performance regressions.

The bisection workflow iteratively tests commit ranges:

graph TD
    A["perf.bisect Workflow Started"] --> B{"FindAvailableBots"}
    B --> C1["Run Single Commit (Lower Bound)"]
    B --> C2["Run Single Commit (Higher Bound)"]
    C1_RES("Lower Results") --> D{"Collect Data for Comparison"}
    C2_RES("Higher Results") --> D
    D --> E{"CompareActivity"}
    E -- "Significant Difference" --> F{"FindMidCommit"}
    F --> G{"Check If Range Is Single Commit"}
    G -- "Not Single Commit" --> H["Run Single Commit (Mid Commit)"]
    H --> H_RES("Mid Results")
    H_RES --> D
    G -- "Single Commit" --> I["Report Bisected Commit"]

Module: /go/workflows/sample

sample: A CLI for Triggering Pinpoint Workflows

The sample module is a command-line tool for developers to manually trigger and test various Pinpoint workflows. It serves as a sample client implementation, demonstrating how to programmatically start workflows like bisection, culprit finding, and pairwise runs using the Temporal Go SDK. Its primary use is for development and debugging.

Design and Implementation

The tool is designed as a simple, flag-driven application for ease of use during development.

• Why: This approach provides a straightforward way to test individual workflows with specific parameters without requiring the full Pinpoint frontend or API. It consolidates example invocations for all major workflows into a single, executable file.
• How: The main function parses command-line flags. A dedicated boolean flag (e.g., --bisect, --single-commit) selects which workflow to run. The application establishes a connection to the Temporal server, and based on the selected flag, calls a corresponding trigger... function.

Key Components

main.go is the single file containing all logic.

• Workflow Invocation: Its primary responsibility is to translate command-line arguments into a specific workflow execution request. Each trigger... function (e.g., triggerBisectWorkflow, triggerCulpritFinderWorkflow) is tailored to a single workflow. It populates the workflow‘s parameter struct from the command-line flags and uses the Temporal client’s ExecuteWorkflow method to start it.
• Client Connection: It initializes and manages the Temporal client connection (client.Dial), which is necessary to communicate with the Temporal server.
• Result Handling: After triggering a workflow, it synchronously waits for the result using the Get method on the workflow handle and prints the outcome to the console for immediate feedback.

Workflow Execution Process

The following diagram illustrates the typical flow when a user runs the sample tool.

graph TD
    A["User runs ./sample with flags, e.g., --bisect"] --> B{"Parse Flags"}
    B --> C["Connect to Temporal Server"]
    C --> D{"Select trigger function based on flag"}
    D -- "e.g., --bisect" --> E["triggerBisectWorkflow"]
    subgraph E ["triggerBisectWorkflow"]
        direction LR
        E1["Build BisectParams from flags"] --> E2["client.ExecuteWorkflow"]
        E2 --> E3["Wait for result"]
    end
    E3 --> F["Print workflow result to console"]

Module: /go/workflows/worker

The Pinpoint Worker is the executable that runs the business logic for all Pinpoint jobs, such as performance bisections and pairwise analyses. It operates as a Temporal worker, connecting to a Temporal cluster to poll for and execute tasks.

Design and Implementation

The worker is designed to be a scalable and resilient processor for long-running, stateful operations. Using the Temporal framework abstracts away the complexities of distributed systems, allowing developers to focus on the core business logic of a job.

The worker's lifecycle and operational model is as follows:

1. Initialization: On startup, it parses command-line flags to configure its connection to the Temporal server (--hostPort, --namespace) and the specific task queue it will listen to (--taskQueue). This flag-based configuration allows the same binary to be used in different environments (local, staging, production).
2. Registration: It registers all the workflow and activity functions it is capable of executing. This acts as a contract with the Temporal server, indicating which tasks this worker can handle. For example, it registers internal.BisectWorkflow with the name workflows.Bisect.
3. Execution: The worker enters a polling loop, continuously asking the Temporal server for tasks from its designated queue. When it receives a task, it executes the corresponding registered function.

A key design choice is the use of the --databaseWriteback feature flag. This conditionally registers a set of activities that write job results to a new PostgreSQL database. This enables incremental migration and testing of a new data backend without disrupting existing functionality.

graph TD
    A[Start ./worker] --> B[Connect to Temporal Server];
    B --> C[Listen on a specific Task Queue];
    C --> D[Register all Workflow and Activity implementations];
    D --> E{Poll for tasks};
    E -- Receives Task --> F[Execute corresponding Workflow/Activity code];
    F -- Task Complete --> E;

Key Components

• main.go: This is the application's entry point. Its primary responsibility is to initialize and configure the Temporal worker. It brings together all the disparate workflow and activity implementations from sub-packages (e.g., internal, catapult) and registers them with the core worker engine. It also handles the setup of essential services like tracing and metrics.

• BUILD.bazel: Defines the rules to build the worker Go binary and package it into a skia_app_container named bisect_workflow. This container is the deployable artifact for the Pinpoint backend.

Module: /proto

This module defines the Pinpoint service's public API using Protocol Buffers. It provides strongly-typed data structures for performance analysis workflows like bisection and pairwise comparison. The API is accessible via both gRPC and a RESTful JSON interface (via gRPC-Gateway).

API Design

The API is defined in a single source of truth, service.proto, from which server stubs, client libraries, and the REST gateway are auto-generated. This approach ensures consistency across all interfaces.

The design favors explicit, job-specific RPCs (ScheduleBisection, SchedulePairwise) over a generic one. This improves type safety and makes the API's intent clear, as each request message contains only parameters relevant to its specific workflow.

A key abstraction is the CombinedCommit message. It models a specific code state not just by a git hash, but also by including potential DEPS modifications and Gerrit patches. This is crucial for accurately reproducing and testing complex changes, such as dependency rolls or in-flight CLs.

To ensure a smooth migration from the previous system, LegacyJobRequest and LegacyJobResponse mirror the legacy Catapult Pinpoint API, providing backward compatibility for existing clients.

Key Files

• service.proto: The core API definition. It contains all RPC service methods, message structures, and the HTTP annotations that map gRPC services to RESTful endpoints.
• service.pb.go, service_grpc.pb.go, service.pb.gw.go: Auto-generated Go language bindings for the protobuf messages, gRPC interfaces, and REST gateway handlers. These files should not be modified directly.
• generate.go: Uses //go:generate directives to document and automate the commands for regenerating Go files from the .proto definition, ensuring a repeatable process.

Culprit Finder Workflow

The ScheduleCulpritFinder RPC initiates a composite workflow to identify the specific commit that caused a performance regression.

graph TD
    A["Client sends ScheduleCulpritFinderRequest"] --> B{"Pinpoint Service"}
    B --> C("1. Regression Verification <br><i>Pairwise job between start/end
    commits</i>")
    C -- "Regression Confirmed" --> D("2. Bisection <br><i>Bisect job to find
    potential culprits</i>")
    C -- "No Regression" --> E["Workflow Ends: No culprits"]
    D -- "Culprit(s) Found" --> F("3. Culprit Verification
    <br><i>Pairwise job between each culprit and its parent</i>")
    D -- "No Culprit Found" --> E
    F -- "Verified" --> G["Workflow Ends: Culprit(s) reported"]
    F -- "Not Verified" --> H["Workflow Ends: Bisection error or false positive"]

Module: /proto/v1

This module defines the public API for the Pinpoint service using Protocol Buffers. It provides strongly-typed data structures and service endpoints for performance analysis tasks like bisection and pairwise comparison. The API is designed to be accessed via both gRPC and a RESTful JSON interface, which is enabled by gRPC-Gateway.

API Design

The API is defined in service.proto and serves as the single source of truth. This choice enables auto-generation of server stubs, client libraries, and the REST gateway, ensuring consistency across different interfaces.

The design favors explicit, job-specific RPCs (ScheduleBisection, SchedulePairwise, ScheduleCulpritFinder) over a single generic endpoint. This improves type safety and makes the API's intent clear, as each request message contains only the parameters relevant to its specific workflow.

A key abstraction is the CombinedCommit message. It models a specific state of the code not just by a git hash, but also by including potential DEPS modifications (modified_deps) and Gerrit patches (patch). This is crucial for accurately reproducing and testing complex changes, such as those involving dependency rolls or in-flight CLs.

To ensure a smooth migration from the previous system, the API includes LegacyJobRequest and LegacyJobResponse. These messages mirror the structure of the legacy Catapult Pinpoint API, providing backward compatibility for existing clients like the Pinpoint UI.

Key Files

• service.proto: The core API definition file. It contains all RPC service methods, message structures (requests, responses, and data models like Commit), and enums (SwarmingStatus, PairwiseJobStatus). It also includes HTTP annotations that map gRPC services to RESTful endpoints.
• service.pb.go, service_grpc.pb.go, service.pb.gw.go: These files are auto-generated from service.proto. They provide the Go language bindings for the protobuf messages, the gRPC client and server interfaces, and the reverse-proxy handlers for the RESTful gateway, respectively. Application code should import and use these generated packages, not modify them directly.
• generate.go: This file uses //go:generate directives to document and automate the commands needed to regenerate the Go files from the service.proto definition. This ensures the generation process is repeatable and consistent.

Culprit Finder Workflow

The ScheduleCulpritFinder RPC initiates an end-to-end workflow to identify the specific commit that caused a performance regression between two points in history. This is a composite workflow that internally leverages other Pinpoint capabilities.

graph TD
    A["Client sends ScheduleCulpritFinderRequest"] --> B{"Pinpoint Service"}
    B --> C("1. Regression Verification <br><i>Pairwise job between start/end
    commits</i>")
    C -- "Regression Confirmed" --> D("2. Bisection <br><i>Bisect job to find
    potential culprits</i>")
    C -- "No Regression" --> E["Workflow Ends: No culprits"]
    D -- "Culprit(s) Found" --> F("3. Culprit Verification
    <br><i>Pairwise job between each culprit and its parent</i>")
    D -- "No Culprit Found" --> E
    F -- "Verified" --> G["Workflow Ends: Culprit(s) reported"]
    F -- "Not Verified" --> H["Workflow Ends: Bisection error or false
    positive"]

Module: /ui

Overview

The Pinpoint UI is a single-page application built with LitElement and TypeScript. Its architecture is designed for modularity and a clear separation of concerns, organized into three primary areas:

• pages: Minimal HTML files that serve as entry points for different application views.
• modules: A collection of reusable LitElement web components that form the building blocks of the UI.
• services: A centralized API client for all backend communication.

Architectural Principles

The UI is built on a few key design principles to ensure maintainability and a predictable data flow.

Component-Based Structure

The application follows a “shell and component” model. Each file in /pages is a minimal HTML shell whose sole purpose is to load a single root web component from /modules. This design delegates all rendering, state management, and user interaction logic to the components, keeping the page entry points clean and simple.

Unidirectional Data Flow

Components are divided into two main categories to enforce a clear, one-way data flow:

1. Controller Components (e.g., pinpoint-landing-page-sk, pinpoint-results-page-sk): These are stateful components that manage a view. They are responsible for fetching data from the API service, managing the view's state (e.g., filters, pagination), and passing data down to child components via properties.
2. Presentational Components (e.g., jobs-table-sk, commit-run-overview-sk): These are stateless (“dumb”) components focused solely on rendering data. They receive data as properties and communicate user interactions (like a button click or sort request) back up to their parent controller by dispatching custom events. They contain no business logic.

This pattern makes components more reusable and the application easier to reason about, as data flows down and events flow up.

Centralized API Service

All communication with the Pinpoint backend is consolidated in /services/api.ts. This file serves as the single source of truth for API interactions.

• Why: Centralizing the API client decouples UI components from the specifics of HTTP requests. Any changes to backend endpoints only need to be updated in this one file.
• How: It exports async functions (e.g., listJobs, getJob) that handle making fetch calls. It also defines TypeScript interfaces (Job, Commit) that mirror the Go structs on the backend, creating a strong, type-safe contract between the frontend and backend.

Key Workflows and Components

Job List and Filtering

The landing page demonstrates the core unidirectional data flow pattern. The pinpoint-scaffold-sk provides the global UI, including filter controls. When a user applies a filter, the scaffold dispatches an event. The pinpoint-landing-page-sk (the controller) listens for this event, updates its state and the URL, fetches the filtered data using the API service, and passes the new list of jobs down to the jobs-table-sk for rendering.

sequenceDiagram
    actor User
    participant Scaffold as pinpoint-scaffold-sk
    participant LandingPage as pinpoint-landing-page-sk
    participant PinpointAPI as services/api.ts
    participant JobsTable as jobs-table-sk

    User->>Scaffold: Changes filter and clicks "Apply"
    Scaffold->>LandingPage: Dispatches 'filters-changed' event
    LandingPage->>LandingPage: Updates internal state & browser URL
    LandingPage->>PinpointAPI: listJobs(new filter state)
    PinpointAPI-->>LandingPage: Returns filtered jobs
    LandingPage->>JobsTable: Passes new 'jobs' property
    JobsTable-->>User: Renders updated table

Creating a New Job

The pinpoint-new-job-sk component encapsulates the entire workflow for scheduling a new job. It is a modal dialog that contains the form, validation logic, and the call to the schedulePairwise function in the API service. This keeps the complexity of job creation isolated within a single component.

Viewing Job Results

The pinpoint-results-page-sk acts as a controller for the results view. It reads the job ID from the URL, fetches the corresponding job data, and composes various presentational components (commit-run-overview-sk, wilcoxon-results-sk) to display different aspects of the results. It conditionally renders components based on the job's type and status.

Module: /ui/modules

Overview

The /ui/modules directory contains the LitElement-based web components that constitute the Pinpoint user interface. The architecture is designed around a clear separation between stateful “page” (controller) components and stateless “presentational” components.

This design promotes reusability and a unidirectional data flow. Page components manage application state, fetch data from the API, and orchestrate the UI. Presentational components receive data via properties and communicate user interactions back to their parent via custom events, without containing any business logic themselves.

Component Architecture

Page and Controller Components

These components manage the primary views and application state.

• pinpoint-landing-page-sk: The main entry point for the jobs list. Its core responsibility is managing state, including filters, sorting, and pagination, and synchronizing it with the browser's URL. It fetches data and passes it down to presentational components like jobs-table-sk.
• pinpoint-results-page-sk: Displays the detailed results of a single Pinpoint job. It fetches all necessary data based on the Job ID in the URL and composes other components (commit-run-overview-sk, wilcoxon-results-sk) to render different aspects of the job result. It conditionally displays content based on the job's status and type.

Structural and Modal Components

These components provide the overall application structure and on-demand dialogs.

• pinpoint-scaffold-sk: Provides the main application shell, including a consistent header with global actions like search and filtering. It uses a <slot> to allow page components to inject their content. It is designed to be decoupled from the page's content, communicating user actions (e.g., filter changes) via custom events.
• pinpoint-new-job-sk: A modal dialog for scheduling new jobs. It encapsulates the form logic, client-side validation, and API calls needed to create a job. It offers both a simplified and a detailed view to cater to different user needs.
• job-overview-sk: A simple modal used to display the formatted configuration parameters of a job.

Presentational Components

These are “dumb” components that focus solely on rendering data.

• jobs-table-sk: Renders a list of jobs in a table. It is stateless regarding data fetching. User actions, such as clicking a column header to sort, trigger custom events (sort-changed). The parent component is responsible for handling these events, re-sorting the data, and passing it back down.
• commit-run-overview-sk: Provides a compact, visual summary of test runs for a single commit. Its key design decision is to define a “successful” run as one that produces metric data, visually marking runs with no values as failures (red).
• wilcoxon-results-sk: Renders the statistical results from a completed pairwise job. It translates the raw statistical data into a color-coded table, making it easy to interpret whether a change caused a significant performance improvement or regression.

Core Workflow: Viewing and Filtering Jobs

The interaction between the scaffold, landing page, and table demonstrates the application's event-driven architecture.

sequenceDiagram
    actor User
    participant Scaffold as pinpoint-scaffold-sk
    participant LandingPage as pinpoint-landing-page-sk
    participant JobsTable as jobs-table-sk
    participant PinpointAPI

    User->>Scaffold: Changes filter and clicks "Apply"
    Scaffold->>LandingPage: Dispatches 'filters-changed' event
    LandingPage->>LandingPage: Updates state & URL
    LandingPage->>PinpointAPI: listJobs(new filter state)
    PinpointAPI-->>LandingPage: Returns filtered jobs
    LandingPage->>JobsTable: Passes new 'jobs' property
    JobsTable-->>User: Renders updated table

Module: /ui/modules/commit-run-overview-sk

The commit-run-overview-sk module provides a UI component to visualize the results of a build and its associated test runs for a single commit.

Design and Implementation

The primary goal of this component is to offer a compact, at-a-glance summary of a performance test execution for a given commit. This is particularly useful in contexts like bisection, where users need to quickly assess the outcome at each step.

It is implemented as a self-contained LitElement (commit-run-overview-sk.ts), making it a reusable presentation component. It's designed to be stateless regarding data fetching; it receives the necessary CommitRunData and JobSchema objects via properties. This separation of concerns keeps the component focused solely on rendering.

A key implementation detail is how it determines the success or failure of a test run. Instead of relying on a status field, the component's isEmptyValues method checks for the presence of metric values. A run that completes but produces no metric data is visually marked as a “failure” (a red box). This design choice reflects the fact that for performance testing, a run without results is not a successful run.

To maintain a clean and uncluttered main view, detailed information for each test run is presented on-demand in a modal dialog (md-dialog).

Key Components

• commit-run-overview-sk.ts: This file contains the complete implementation of the CommitRunOverviewSk web component. Its responsibilities include:
  • Rendering the high-level build information, such as the commit hash and a link to the build log.
  • Displaying a grid of small, colored squares, where each square represents a single test iteration (TestRun). The square's color (green for success, red for failure) is determined by the logic described above.
  • Handling user interaction. A click on any run's square triggers a dialog that shows detailed metadata for that specific run, including links to the Swarming task and CAS output.

Workflow

The component follows a simple interactive workflow for displaying detailed run information.

graph TD
    A["Component receives CommitRunData"] --> B{"Render View"}
    B --> C["Display build info"]
    B --> D["Display grid of colored run boxes"]
    D -- "Click a run box" --> E{"showRunDetails()"}
    E --> F["Update state with selected run"]
    F --> G["Show md-dialog with run details"]
    G -- "Click 'Close'" --> H["Hide dialog"]

Module: /ui/modules/job-overview-sk

Module: job-overview-sk

This module provides a UI component, <job-overview-sk>, designed to display the configuration parameters of a Pinpoint job in a modal dialog. Its purpose is to offer users a clear, formatted, and easily accessible summary of a job's inputs without cluttering the primary interface.

Design and Implementation

The component is implemented as a LitElement that wraps a Material Web <md-dialog>. This choice provides a consistent, modern UI for presenting contextual information on-demand.

The core responsibility of job-overview-sk is to transform a raw JobSchema object into a human-readable format. This is achieved through several key design choices:

• Data Aggregation: It combines top-level properties (e.g., Benchmark, BotName) with nested parameters from AdditionalRequestParameters into a single, unified list. This provides a comprehensive view of all relevant job arguments.
• Selective Filtering: To maintain clarity, certain parameters deemed internal or less relevant for an overview (like commit_runs and duration) are explicitly filtered out.
• User-Friendly Formatting:
  • Technical keys (e.g., start_commit_githash) are mapped to more descriptive labels (e.g., “Start Commit”) via the formatKey method. This makes the information more accessible to a wider audience.
  • Complex values, such as nested JSON objects, are automatically pretty-printed using renderParameterValue, ensuring they are readable within the dialog.

Key Components

• job-overview-sk.ts: The LitElement component definition. It accepts a job property of type JobSchema. The component remains hidden until its public show() method is invoked, which then displays the dialog with the formatted job data.

Workflow

The component is designed to be controlled by a parent element. The parent is responsible for passing the job data and triggering the dialog's visibility.

sequenceDiagram
    participant Parent as Parent Component
    participant User
    participant E as job-overview-sk

    Parent->>E: Sets [job] property with JobSchema data
    User->>Parent: Clicks a "View Arguments" button
    Parent->>E: Calls show() method
    E-->>User: Displays dialog with formatted job parameters
    User->>E: Clicks "Close" button
    E-->>User: Dialog closes

Module: /ui/modules/jobs-table-sk

Module: /ui/modules/jobs-table-sk

Overview

jobs-table-sk is a presentational web component that displays a list of Pinpoint jobs in a sortable, interactive table. Its primary purpose is to render job data provided to it, without managing any application state or data fetching itself.

Design and Implementation

The component is designed to be “stateless” or “dumb”. It receives all necessary data, including the list of jobs and the current sorting state (sortBy, sortDir), via properties. This design decouples the component from business logic, making it highly reusable and straightforward to test.

Instead of directly modifying data or calling APIs, the component communicates user actions to its parent via custom events.

• Sorting: When a user clicks a column header, jobs-table-sk emits a sort-changed event. It does not re-sort the data internally. The parent component is responsible for listening to this event, re-fetching or re-sorting the job list, and then passing the updated data back down to the table.
• Cancellation: A “Cancel” button is conditionally rendered for jobs that are ‘Pending’ or ‘Running’. Clicking it fires a cancel-job-clicked event, delegating the responsibility of making the API call to cancel the job to the parent.

This event-driven, unidirectional data flow simplifies the component's responsibility to just rendering and user input translation.

Key Components

• jobs-table-sk.ts: This file contains the complete definition of the JobsTableSk Lit element. It defines the component's properties (jobs, sortBy, sortDir), renders the table structure, handles click events on headers to dispatch sorting events, and conditionally displays action buttons.

Workflow: Sorting Jobs

The following diagram illustrates the interaction between the user, the jobs-table-sk component, and its parent when sorting the table.

sequenceDiagram
    participant User
    participant jobs-table-sk
    participant Parent Component

    User->>jobs-table-sk: Clicks a column header
    jobs-table-sk->>jobs-table-sk: Determines new sort key and direction
    jobs-table-sk-->>Parent Component: Dispatches 'sort-changed' event
    Parent Component->>Parent Component: Updates state, fetches/sorts data
    Parent Component-->>jobs-table-sk: Passes new 'jobs', 'sortBy', and 'sortDir' properties
    jobs-table-sk->>jobs-table-sk: Re-renders with sorted data

Module: /ui/modules/pinpoint-landing-page-sk

Overview

The pinpoint-landing-page-sk element is the main entry point for the Pinpoint application. It orchestrates the display of a paginated, filterable, and sortable list of Pinpoint jobs. Its primary design goal is to manage the application's state and reflect it in the URL, allowing users to share and bookmark specific views of the job list.

Design and Implementation

The component‘s state, including filters, search terms, sorting preferences, and the current page, is managed using stateReflector. This synchronizes the component’s internal state with the browser's URL query parameters. When the page loads, it initializes its state from the URL; conversely, any change to the state (e.g., applying a filter) updates the URL.

This component acts as a controller, delegating rendering and specific UI interactions to child components:

• pinpoint-scaffold-sk: Provides the overall page structure, header, and user controls for searching and filtering. When a user applies a filter, this component emits an event which pinpoint-landing-page-sk handles by re-fetching the job list with the new filter parameters.
• jobs-table-sk: Responsible for rendering the list of jobs in a table. It emits events when a user clicks a column header to sort the data or clicks a button to cancel a job.

Key Responsibilities and Workflows

• Data Fetching: On initialization or when filters change, the component calls the listJobs API endpoint to retrieve a page of jobs. It handles loading and error states during this process.
• Pagination: It fetches a fixed number of jobs per page. The “Next” button is enabled if the API returns a full page of results, which is a simple heuristic for determining if more data is available. Navigation updates the page number in the state and triggers a new API call with the appropriate offset.
• Sorting: Sorting is performed client-side. When the jobs-table-sk emits a sort-changed event, this component updates its sorting state and re-sorts only the currently visible page of jobs before re-rendering. The sorting parameters are not sent to the backend API.
• Job Cancellation: When a user initiates a job cancellation from the jobs-table-sk, this component displays a confirmation dialog. Upon submission with a required reason, it calls the cancelJob API and refreshes the job list.

sequenceDiagram
    participant User
    participant URL
    participant pinpoint-landing-page-sk as LandingPage
    participant pinpoint-scaffold-sk as Scaffold
    participant jobs-table-sk as JobsTable
    participant PinpointAPI as API

    User->>URL: Navigates to page (or refreshes)
    URL-->>LandingPage: stateReflector reads initial state
    LandingPage->>API: listJobs(state)
    API-->>LandingPage: Returns list of jobs
    LandingPage->>Scaffold: Renders with filter values
    LandingPage->>JobsTable: Renders with job list

    User->>Scaffold: Changes a filter
    Scaffold->>LandingPage: Emits 'filters-changed' event
    LandingPage->>URL: Updates URL via stateReflector
    LandingPage->>API: listJobs(new filter state)
    API-->>LandingPage: Returns filtered jobs
    LandingPage->>JobsTable: Re-renders with new jobs

    User->>JobsTable: Clicks column header to sort
    JobsTable->>LandingPage: Emits 'sort-changed' event
    LandingPage->>URL: Updates URL via stateReflector
    Note over LandingPage: Performs client-side sort on current page of jobs
    LandingPage->>JobsTable: Re-renders with sorted jobs

Module: /ui/modules/pinpoint-new-job-sk

pinpoint-new-job-sk Module

The pinpoint-new-job-sk module provides a modal dialog for scheduling new Pinpoint performance testing jobs. Pinpoint is used to diagnose performance regressions or compare performance between two Chrome commits (a “try job”).

Design and Implementation

The component is designed to accommodate both novice and expert users by offering two distinct views: “Simplified” and “Detailed”.

• Simplified View: This view streamlines job creation for common use cases. It pre-populates the form with a standard test configuration (e.g., speedometer3.crossbench on mac-m1-pro-perf) and only requires the user to input the start and end commit hashes. This reduces complexity for users who need a quick, standard performance check.
• Detailed View: This provides full control over all job parameters, including the specific benchmark, device (bot), story, iteration count, and an optional bug ID for tracking. This view is for users with specific testing needs.

The form's options are populated dynamically. When a user selects a benchmark in the detailed view, the component fetches the corresponding lists of compatible devices and stories from the Pinpoint API. This ensures that users can only select valid configurations, preventing backend errors.

Client-side validation is performed before submission to provide immediate feedback on required fields and valid value ranges, such as the iteration count. User feedback on the job submission status is provided via a non-blocking toast-sk element, which on success includes a direct link to the newly created job.

Key Components

• pinpoint-new-job-sk.ts: The core LitElement that defines the modal‘s UI and behavior. It manages the component’s state, handles user input, orchestrates API calls to fetch data and schedule jobs, and renders either the detailed or simplified view.
• pinpoint-new-job-sk relies on the //pinpoint/ui/services:api_ts_lib module for all communication with the Pinpoint backend. This includes fetching benchmarks, bots, and stories, as well as submitting the final job request via the schedulePairwise function.

Workflows

New Job Creation Workflow

The following diagram illustrates the process of creating a new Pinpoint job using this component.

sequenceDiagram
    participant User
    participant PinpointNewJobSk as Modal Component
    participant PinpointAPI as API Service

    User->>+Modal Component: Opens modal (show())
    Modal Component->>+PinpointAPI: listBenchmarks()
    PinpointAPI-->>-Modal Component: Returns benchmarks
    Modal Component->>+PinpointAPI: listBots()
    PinpointAPI-->>-Modal Component: Returns bots
    Note over User,Modal Component: User fills in job details
    User->>+Modal Component: Clicks "Start"
    Modal Component->>Modal Component: Validates form input
    alt Form is valid
        Modal Component->>+PinpointAPI: schedulePairwise(request)
        PinpointAPI-->>-Modal Component: Success (jobId)
        Modal Component->>User: Shows success toast with job link
        Modal Component->>Modal Component: Dispatches 'pinpoint-job-started' event
        Modal Component->>Modal Component: close()
    else Form is invalid
        Modal Component->>User: Shows error toast
    end

Module: /ui/modules/pinpoint-results-page-sk

Module: pinpoint-results-page-sk

Overview

pinpoint-results-page-sk is a full-page Lit element responsible for displaying the detailed results of a single Pinpoint job. It serves as the primary view for users to analyze job outcomes, compare performance between commits, and access statistical analysis. The component is designed to be self-contained; it fetches all necessary data based on a Job ID provided in the URL.

Design

The core design is centered around providing a comprehensive, state-aware view of a job.

• Data-Driven Rendering: The component's lifecycle begins by parsing the Job ID from the URL. It then fetches the corresponding job data from the API. The entire UI is reactive to the state of this fetch operation, displaying loading, error, or data-rich views accordingly.
• Component Composition: Instead of implementing all UI elements within this module, it composes several specialized child components. It uses commit-run-overview-sk to display details for the base and experimental commits side-by-side, and job-overview-sk to show the job's full configuration parameters in a dialog. This separation of concerns keeps the results page focused on layout and state management.
• Conditional Logic for Results: A key design choice is how it handles different job types and statuses. The statistical analysis, rendered by wilcoxon-result-sk, is only shown for completed Pairwise jobs. For other statuses (e.g., Running, Failed, Canceled), the component renders distinct status boxes. This provides clear, immediate feedback to the user about the job's state without cluttering the UI with irrelevant or unavailable information.

Key Components

• pinpoint-results-page-sk.ts: The main element. Its primary responsibilities include:
  • Extracting the Job ID from the browser's URL.
  • Orchestrating the API call to fetch job data.
  • Managing the page's state (e.g., loading, error, job).
  • Rendering the overall page layout, including the header with job metadata and the main content area.
  • Composing and passing data down to child components (commit-run-overview-sk, wilcoxon-result-sk, job-overview-sk).

Workflows

The primary workflow is fetching and displaying job data based on the current URL.

graph TD
    A["Page Loads"] --> B{"Extract Job ID from URL"}
    B --> C{"Job ID exists?"}
    C -- "No" --> D["Show 'No Job ID' error"]
    C -- "Yes" --> E["Fetch job data via API"]
    E --> F{"Fetch successful?"}
    F -- "No" --> G["Show fetch error"]
    F -- "Yes" --> H["Render job header & commit details"]
    H --> I{"Job is 'Pairwise' and 'Completed'?"}
    I -- "Yes" --> J["Render 'wilcoxon-result-sk' with statistics"]
    I -- "No" --> K["Render relevant status box (e.g., Pending, Failed)"]

Module: /ui/modules/pinpoint-scaffold-sk

The pinpoint-scaffold-sk module provides the main layout and user interface shell for the Pinpoint application. It creates a consistent header with global actions like searching, filtering, and creating new jobs, while allowing different page content to be displayed within its main body.

Design and Implementation

The primary design goal is to decouple the application's overall structure (the scaffold) from the specific content it displays (e.g., a list of jobs).

This is achieved through two main web component patterns:

1. Content Projection: The scaffold uses a <slot> element. This allows parent components to insert their own content into the main area of the scaffold, making the scaffold a reusable container.
2. Event-Driven Communication: Instead of directly manipulating its child content, the scaffold communicates user actions via custom events. When a user types in the search bar or applies filters, the component dispatches search-changed and filters-changed events, respectively. A parent component listens for these events and is responsible for updating the content shown in the <slot>. This keeps the scaffold unaware of the specific data being displayed.

Upon initialization, the component fetches the available benchmarks and bots from the Pinpoint API to populate the filter dropdown menus.

Key Components

• pinpoint-scaffold-sk.ts: This is the core file defining the <pinpoint-scaffold-sk> Lit element. It contains the HTML template for the header and main content area, the logic for handling user input in the search and filter menus, and the code for dispatching events to parent components. It also includes the logic for showing the <pinpoint-new-job-sk> modal.

Workflows

Filtering Content

The following diagram illustrates how the scaffold interacts with a parent component to filter content.

sequenceDiagram
    actor User
    participant Scaffold as pinpoint-scaffold-sk
    participant ParentComponent as e.g., Job List Page

    User->>Scaffold: Clicks the filter icon
    Scaffold-->>User: Shows filter menu
    User->>Scaffold: Changes filter values and clicks "Apply"
    Scaffold->>ParentComponent: Dispatches "filters-changed" event with new
    filter parameters
    ParentComponent->>ParentComponent: Receives event, fetches new data, and
    updates its content inside the scaffold's <slot>

Module: /ui/modules/wilcoxon-results-sk

Overview

The wilcoxon-results-sk module is a UI component designed to present the statistical results from a pairwise Wilcoxon signed-rank test. It visualizes the performance comparison between a “control” and a “treatment” group from a Pinpoint job, making it easy to determine if a change resulted in a statistically significant improvement or regression.

Design and Implementation

The component is implemented as a LitElement, wilcoxon-result-sk. Its primary design goal is to translate complex statistical data from a JobSchema object into a concise and easily interpretable table.

It operates by processing the MetricSummary field of a given Pinpoint job. For each metric found, it calculates and displays:

• Median Difference: The percentage change between the control and treatment medians.
• Confidence Interval: The 95% confidence interval for the median difference, providing a range for the true effect size.
• P-value: The probability value to assess statistical significance.

A key feature is the use of color-coding to provide at-a-glance interpretation. Cells are colored based on the outcome:

• Green (improvement): A statistically significant change in the desired direction (e.g., latency going down).
• Red (regression): A statistically significant change in the opposite of the desired direction.
• Grey (neutral): The change is not statistically significant.

This logic depends on the metric's “improvement direction” (whether higher or lower values are better). Currently, this is a known limitation as the direction is hardcoded to ‘UP’ (higher is better) and awaits a backend implementation.

Key Components

• wilcoxon-results-sk.ts: Defines the WilcoxonResultSk custom element. It receives a JobSchema object as a property. Its main responsibility is to render the results table. It contains the logic for formatting the statistical values into human-readable strings (e.g., percentages) and applying the appropriate CSS classes (improvement, regression, neutral) to visually communicate the test's outcome.

Module: /ui/pages

The pages module provides the top-level HTML entry points for the Pinpoint web application. Each page corresponds to a distinct user view or workflow.

Design Philosophy

The design prioritizes a clean separation between page structure and application logic. Each page is a minimal HTML shell that loads a single root custom element. The corresponding TypeScript file's sole responsibility is to import this element.

This “shell and component” approach delegates all complex UI rendering, state management, and user interaction to the custom elements (defined in other modules). This keeps the page definitions simple and focused on bootstrapping the correct user experience.

Key Pages

• landing-page: The main entry point for the application. It loads the <pinpoint-landing-page-sk> element, which contains the interface for configuring and starting new Pinpoint jobs.
• results-page: Displays the outcome of a specific job. It loads the <pinpoint-results-page-sk> element, which fetches and renders the detailed results for a given job ID.

Module: /ui/services

This module serves as the API client for the Pinpoint frontend. It abstracts all communication with the backend, providing a centralized and type-safe way for UI components to fetch data and trigger actions.

Design and Implementation

• Centralization: All backend interactions are consolidated here to simplify maintenance and ensure consistency. Changes to backend API endpoints only require updates in this module, not across various UI components.
• Type Safety: The module uses TypeScript interfaces (e.g., Job, JobSchema, Commit) that mirror the Go structs in the backend. This creates a strong data contract between the frontend and backend, preventing data-related bugs at compile time.
• API Interaction: Functions like schedulePairwise, listJobs, and getJob use the standard fetch API to make asynchronous HTTP requests. They handle request formatting (URL parameters, JSON bodies) and basic error handling.

Key Components

• api.ts: This file is the single source of truth for frontend-backend communication.
  • Data Models: Defines TypeScript interfaces (Job, JobSchema, Commit, etc.) that match the backend's data structures. This ensures that data received from or sent to the API conforms to an expected shape.
  • API Functions: Exposes a set of async functions (schedulePairwise, listJobs, getJob, etc.) that wrap fetch calls to specific backend endpoints. Each function is responsible for constructing the correct request and parsing the response.

Example Workflow: Scheduling a Job

The following diagram shows how a UI component uses this service to schedule a new pairwise job.

sequenceDiagram
    participant UI Component
    participant services/api.ts
    participant Pinpoint Backend

    UI Component->>services/api.ts: Calls schedulePairwise(request)
    services/api.ts->>Pinpoint Backend: POST /pinpoint/v1/pairwise with JSON body
    Pinpoint Backend-->>services/api.ts: Returns job execution details (JSON)
    services/api.ts-->>UI Component: Returns Promise<PairwiseExecution>