tree: 039ef93e4cc046bd8aba42eb504365a32bae1655 [path history] [tgz]
  1. go/
  2. res/
  3. sys/
  4. templates/
  5. .bowerrc
  6. bower.json
  7. bucket-lifecycle-config.json
  8. build_backup_release
  9. build_internal_release
  10. build_release
  11. build_staging_release
  12. create-task-scheduler-internal-sa.sh
  13. create-task-scheduler-sa.sh
  14. elements.html
  15. Makefile
  16. package.json
  17. probers.json5
  18. PROD.md
  19. README.md
  20. setup-bigtable.sh
  21. skiapush.json5
  22. startup-script-internal.sh
  23. startup-script-production.sh
  24. startup-script-staging.sh
  25. vm.go
task_scheduler/README.md

Skia Task Scheduler

This directory contains code for a custom Swarming task scheduler used by Skia's bots.

Motivation

Skia‘s automated testing involves sets of tasks which depend on one another, for example compiling code and then running tests on a particular platform. These tasks are performed on Skia’s bots via Swarming. We need a way to intelligently schedule these Swarming tasks to optimally utilize the machines in our test lab. In particular, we want to keep up with incoming commits to the Skia repo by testing multiple commits as part of a single task, and then during idle time run tests at commits which were previously batched to increase the granularity of our test data. Additionally, the scheduler must keep track of the directed acyclic graph of tasks for each commit.

Design

At a high level, the scheduler first generates a set of all tasks which could possibly be scheduled, filters out tasks which cannot run (eg. due to unsatisfied dependencies) or should not run (eg. we've already run it), then assigns a score for each candidate task based on the value added by running that task (eg. how many new commits it tests, or how large a batch it bisects). It then sorts the tasks by score to form a queue. When a swarming bot is free, the scheduler triggers the highest-scoring task candidate which matches the Swarming dimensions of the bot. Detailed design can be found here: https://docs.google.com/document/d/12DzzmeDBDomNxTWWtHCRIfj6MoB8Yvw4v5horGuJPek/edit and here: https://docs.google.com/document/d/1tKlBi0reIKo6ActxN8TQY-4t80uQCJXv_CW9WVWG5w8/edit