| --- |
| title: 'Skia Gold' |
| linkTitle: 'Skia Gold' |
| --- |
| |
| ## Overview |
| |
| Gold is a web application that compares the images produced by our bots against |
| known baseline images. |
| |
| Key features: |
| |
| - Baselines are managed in Gold outside of Git, but in lockstep with Git |
| commits. |
| - Each commit creates >500k images. |
| - Deviations from the baseline are triaged after a CL lands and images are |
| triaged as either `positive` or `negative`. 'Positive' means the diff is |
| considered acceptable. 'Negative' means the diff is considered unacceptable |
| and requires a fix. If a CL causes Skia to break it is reverted or an |
| additional CL is landed to fix the problem. |
| - We test across a range of dimensions, e.g.: |
| |
| - OS (Windows, Linux, Mac, Android, iOS) |
| - Architectures (Intel, ARM) |
| - Backends (CPU, OpenGL, Vulkan etc.) |
| - etc. |
| |
| - Written in Go, Polymer and deployed on the Google Cloud. The code is in the |
| [Skia Infra Repository](https://github.com/google/skia-buildbot). |
| |
| ## Recommended Workflows |
| |
| ### How to best use Gold for commonly faced problems |
| |
| These instructions will refer to various views which are accessible via the left |
| navigation on [gold.skia.org](https://gold.skia.org/). |
| |
| View access is public, triage access is granted to Skia contributors. You must |
| be logged in to triage. |
| |
| ## Problem #1: As Skia Gardener, I need to triage and “assign” many incoming new images. |
| |
| Solution today: |
| |
| - Access the By Blame view to see digests needing triage and associated |
| owners/CLs |
| - Only untriaged digests will be shown by default |
| - Blame is not sorted in any particular order |
| - Digests are clustered by runs and the most minimal set of blame |
| |
| <img src=../BlameView.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |
| |
| - Select digests for triage |
| - Digests will be listed in order with largest difference first |
| - Click to open the digest view with detailed information |
| |
| <img src=../Digests.png style="margin-left:40px" align="left" width="780"/> |
| <br clear="left"> |
| |
| - Open bugs for identified owner(s) |
| - The digest detail view has a link to open a bug from the UI |
| - Via the Gold UI or when manually entering a bug, copy the full URL of single |
| digest into a bug report |
| - The URL reference to the digest in Issue Tracker will link the bug to the |
| digest in Gold |
| |
| <img src="../IssueHighlight.png" style="margin-left:60px" align="left" width="720" border=1/> |
| <br clear="left"> |
| |
| <br> |
| |
| Future improvements: |
| |
| - Smarter, more granular blamelist |
| |
| <br> |
| |
| ## Problem #2: As a developer, I need to land a CL that may change many images. |
| |
| To find your results: |
| |
| - Immediately following commit, access the By Blame view to find untriaged |
| digest groupings associated with your ID |
| - Click on one of the clusters including your CL to triage |
| - Return to the By Blame view to walk through all untriaged digests involving |
| your change |
| - Note: It is not yet implemented in the UI but possible to filter the view by |
| CL. Delete hashes in the URL to only include the hash for your CL. |
| |
| <img src=../BlameView.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |
| |
| To rebaseline images: |
| |
| - Access the Ignores view and create a new, short-interval (hours) ignore for |
| the most affected configuration(s) |
| |
| <img src=../Ignores.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |
| |
| - Click on the Ignore to bring up a search view filtered by the affected |
| configuration(s) |
| - Mark untriaged images as positive (or negative if appropriate) |
| - Follow one of two options for handling former positives: |
| - Leave former positives as-is and let them fall off with time if there is low |
| risk of recurrence |
| - Mark former positives as negative if needed to verify the change moving |
| forward |
| |
| Future improvements: |
| |
| - Trybot support prior to commit, with view limited to your CL |
| - Pre-triage prior to commit that will persist when the CL lands |
| |
| <br> |
| |
| ## Problem #3: As a developer or infrastructure engineer, I need to add a new or updated config. |
| |
| (ie: new bot, test mode, environment change) |
| |
| Solution today: |
| |
| - Follow the process for rebaselining images: |
| - Wait for the bot/test/config to be committed and show up in the Gold UI |
| - Access the Ignores view and create a short-interval ignore for the |
| configuration(s) |
| - Triage the ignores for that config to identify positive images |
| - Delete the ignore |
| |
| Future improvements: |
| |
| - Introduction of a new or updated test can make use of try jobs and pre-triage. |
| - New configs may be able to use these features as well. |
| |
| <br> |
| |
| ## Problem #4: As a developer, I need to analyze the details of a particular image digest. |
| |
| Solution: |
| |
| - Access the By Test view |
| |
| <img src=../ByTest.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |
| |
| - Click the magnifier to filter by configuration |
| - Access the Cluster view to see the distribution of digest results |
| - Use control-click to select and do a direct compare between data points |
| - Click on configurations under “parameters” to highlight data points and |
| compare |
| |
| <img src=../ClusterConfig.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |
| |
| - Access the Grid view to see NxN diffs |
| |
| <img src=../Grid.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |
| |
| - Access the Dot diagram to see history of commits for the trace |
| - Each dot represents a commit |
| - Each line represents a configuration |
| - Dot colors distinguish between digests |
| |
| <img src=../DotDiagram.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |
| |
| <br> |
| |
| Future improvements: |
| |
| - Large diff display of image vs image |
| |
| <br> |
| |
| ## Problem #5: As a developer, I need to find results for a particular configuration. |
| |
| Solution: |
| |
| - Access the Search view |
| - Select any parameters desired to search across tests |
| |
| <img src=../Search.png style="margin-left:30px" align="left" width="800"/> |
| <br clear="left"> |