[task scheduler] Add opt-in support for task deduplication in Swarming

Bug: skia:8032
Change-Id: I446d3f464f43a160a2799a0fb62006aed30e4c56
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/220976
Reviewed-by: Ben Wagner aka dogben <benjaminwagner@google.com>
Commit-Queue: Eric Boren <borenet@google.com>
6 files changed
tree: e5b38fc20684c4734fc5753c2f5bd383d88d3f18
  1. .gitattributes
  2. .gitignore
  3. DATASTORE.md
  4. DEPS
  5. LICENSE
  6. Makefile
  7. PRESUBMIT.py
  8. PRIVACY_POLICY.md
  9. README.md
  10. STYLEGUIDE.md
  11. am/
  12. android_compile/
  13. android_ingest/
  14. android_stats/
  15. api/
  16. appengine_scripts/
  17. autoroll/
  18. bash/
  19. blamer/
  20. build_infra_prod.sh
  21. chromeextensions/
  22. cmd/
  23. codereview.settings
  24. comments/
  25. common-sk/
  26. contestk/
  27. cq_watcher/
  28. ct/
  29. datahopper/
  30. debugger-assets/
  31. debugger/
  32. docker/
  33. docserverk/
  34. ds/
  35. fiddlek/
  36. firestore/
  37. fuzzer/
  38. get_service_account/
  39. git_cookie_authdaemon/
  40. gitsync/
  41. go.mod
  42. go.sum
  43. go/
  44. gold-client/
  45. golden/
  46. grafana/
  47. html-template-minifier/
  48. infra-sk/
  49. infra/
  50. jsdoc/
  51. jsfiddle/
  52. k8s_checker/
  53. kube/
  54. launch.md
  55. leasing/
  56. licenses/
  57. logmetrics/
  58. make/
  59. monitoring/
  60. named-fiddles/
  61. notifier/
  62. particles/
  63. perdiff/
  64. perf/
  65. periodic-trigger/
  66. power/
  67. prober/
  68. proberk/
  69. prometheus/
  70. promk/
  71. proxy/
  72. pulld/
  73. push/
  74. res/
  75. run_unittests
  76. run_unittests.go
  77. scripts/
  78. sheriff_emails/
  79. skfe/
  80. skolo/
  81. skottie/
  82. status/
  83. task_driver/
  84. task_scheduler/
  85. tools/
  86. tracedb/
  87. velero/
  88. webtools/
  89. whitespace.txt
README.md

Skia-Buildbot Repository

This repo contains infrastructure code for Skia.

Getting the Source Code

The main source code repository is a Git repository hosted at https://skia.googlesource.com/buildbot.git. It is possible to check out this repository directly with git clone or via go get:

$ cd ${WORKDIR}
$ git clone https://skia.googlesource.com/buildbot.git

or

$ go get -u -t go.skia.org/infra/...

The latter fetches the repository into your $GOPATH directory along with all the Go dependencies, while the former allows you to work in whatever directory you want. If you're working within GOPATH, you probably want to set this variable:

export GO111MODULE=on

Note: go.skia.org is a custom import path and will only work if used like the examples here.

Install Node.js (not as root) and add the bin dir to your path. Optionally run npm install npm -g, as suggested by the npm getting started doc.

Install other dependencies:

$ sudo apt-get install python-django
$ go get -u github.com/kisielk/errcheck \
  golang.org/x/tools/cmd/goimports \
  go.chromium.org/luci/client/cmd/isolate
$ npm install -g polylint bower

Build ~everything:

$ make all

Generated Code

Some code is generated using go generate with external binaries. First, install the version of protoc referenced in the asset creation script and ensure it is on your PATH before other versions of protoc.

Install the necessary go packages:

$ go get -u \
  github.com/golang/protobuf/protoc-gen-go \
  golang.org/x/tools/cmd/stringer \
  google.golang.org/grpc \
  github.com/vektra/mockery/...

To generate code run in this directory:

$ go generate ./...

Running unit tests

Install Cloud SDK.

The installed python-django version must be >= 1.7. Run the following to update:

$ sudo pip install Django --upgrade

Use this command to run the presubmit tests:

$ ./run_unittests --small