The fuzzer will consist of four mostly independent components:
The randomizer continuously generates standalone Skia programs. It then builds and runs these programs against the raster, GPU, and PDF backends of skia. Each such program will be referred to as a “fuzz”.
We are not so much interested in the graphical output of a fuzz (although we will save it), but rather whether the fuzz successfully executed or not.
Each fuzz‘s source code and generated images / PDF will be stored in Google storage. Metadata will also be maintained to record the fuzz’s time of creation, runtime exit status, and system architecture.
Note that the upload to google storage will be optional; users will be able to run the randomizer as a standalone command-line tool on their desktop for testing purposes. In that case, all output will be dumped to a local directory structure, and metadata will be written to the filesystem as well.
A “good citizen” who runs the fuzzer on their workstation to consume spare cycles can also upload the results to google storage if their account is authorized to do so; this is why the system architecture should be recorded in case we uncover any crashes that are system-specific.
TODO(humper): Discuss the design of the actual random program generation. You know, the interesting part :)
The aggregator will periodically count the number of fuzzes in google storage that have a failing exit status and upload that metric to the skia monitoring server. This will allow us to create alerts for failing fuzzes.
The aggregator will run quite frequently, probably at least once a minute.
Because we don't need to maintain an infinite backlog of working programs, the sanitizer will periodically purge fuzzes from google storage that are marked as successful and older than some threshold.
Skia developers should be able to visually browse the history of the fuzzer and get quick access to any failing tests.