[gold] Add gitilesfollower

This is a simplified version of gitsync that works directly
with the SQL database and gitiles. The idea here is that
this process will make a mapping of commit_id and git_hash
in the GitCommits table, so when we see those git_hash
during ingestion, we can derive a commit_id and use that
for the actual ordering of data.

This version takes a lesson from Perf by only querying data
from the most recent commit in the DB and the latest on
the tree to make Gold resilient to merged/changed
history (e.g. go/skia-infra-pm-007)

As a result, I've made the commit_id field required for
the GitCommits table.

Suggested review order:
 - sql.go and tables.go
 - gitilesfollower_manual_test.go
 - gittilesfollower_test.go
 - gitilesfollower.go
 - Everything else

Bug: skia:10582, skia:11367
Change-Id: I0b6d3e89c91ff38aeef99f11b02775f644ed819d
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/375664
Reviewed-by: Leandro Lovisolo <lovisolo@google.com>
12 files changed
tree: 2d0dd657622df73788edee43a68808965cc59bcc
  1. .bazelignore
  2. .bazelrc
  3. .eslintrc.js
  4. .gitattributes
  5. .gitignore
  6. .mocharc.json
  7. BUILD.bazel
  8. DATASTORE.md
  9. DEPS
  10. LICENSE
  11. Makefile
  12. PRESUBMIT.py
  13. PRIVACY_POLICY.md
  14. README.md
  15. STYLEGUIDE.md
  16. WORKSPACE
  17. am/
  18. android_ingest/
  19. android_stats/
  20. api/
  21. autoroll/
  22. bash/
  23. bazel/
  24. blamer/
  25. bugs-central/
  26. build_infra_prod.sh
  27. chromeextensions/
  28. cipd.ensure
  29. codereview-watcher/
  30. codereview.settings
  31. comments/
  32. contestk/
  33. cq_watcher/
  34. ct/
  35. datahopper/
  36. debugger-app/
  37. demos/
  38. docker/
  39. docker_pushes_watcher/
  40. docs/
  41. docserverk/
  42. dot/
  43. ds/
  44. elements-sk-scss.bzl
  45. external/
  46. fiddlek/
  47. firestore/
  48. get_service_account/
  49. git_cookie_authdaemon/
  50. gitsync/
  51. go.mod
  52. go.sum
  53. go/
  54. go_repositories.bzl
  55. gold-client/
  56. golden/
  57. hashtag/
  58. html-template-minifier/
  59. infra-sk/
  60. infra/
  61. jsdoc/
  62. jsfiddle/
  63. k8s-checker/
  64. kube/
  65. launch.md
  66. leasing/
  67. licenses/
  68. machine/
  69. make/
  70. modules/
  71. named-fiddles/
  72. new_element/
  73. package-lock.json
  74. package.json
  75. particles/
  76. perdiff/
  77. perf/
  78. periodic-trigger/
  79. power/
  80. proberk/
  81. promk/
  82. pulld/
  83. puppeteer-tests/
  84. push/
  85. run_unittests
  86. run_unittests.go
  87. scrap/
  88. scripts/
  89. shaders/
  90. sk8s/
  91. skbug/
  92. skfe/
  93. skolo/
  94. skottie/
  95. status/
  96. task_driver/
  97. task_scheduler/
  98. tools.go
  99. tools/
  100. tree_status/
  101. trybot_updater/
  102. tsconfig.json
  103. velero/
  104. whitespace.txt
README.md

Skia-Buildbot Repository

This repo contains infrastructure code for Skia.

Getting the Source Code

The main source code repository is a Git repository hosted at https://skia.googlesource.com/buildbot.git. It is possible to check out this repository directly with git clone or via go get.

Using git clone allows you to work in whatever directory you want. You will still need to set GOPATH in order to build some apps (recommended to put this in a cache dir). E.g.:

$ cd ${WORKDIR}
$ git clone https://skia.googlesource.com/buildbot.git
$ export GOPATH=${HOME}/.cache/gopath/$(basename ${WORKDIR})
$ mkdir $GOPATH
$ cd buildbot

Using go get will fetch the repository into your GOPATH directory along with all the Go dependencies. You will need to set GOPATH and GO111MODULE=on. E.g.:

$ export GOPATH=${WORKDIR}
$ export GO111MODULE=on
$ go get -u -t go.skia.org/infra/...
$ cd ${GOPATH}/src/go.skia.org/infra/

Note: go.skia.org is a custom import path and will only work if used like the examples here.

Install Node.js (not as root) and add the bin dir to your path. Optionally run npm install npm -g, as suggested by the npm getting started doc.

Install other dependencies:

$ sudo apt-get install python-django
$ go get -u github.com/kisielk/errcheck \
  golang.org/x/tools/cmd/goimports \
  go.chromium.org/luci/client/cmd/isolate

Build ~everything:

$ make all

Generated Code

Some code is generated using go generate with external binaries. First, install the version of protoc referenced in the asset creation script and ensure it is on your PATH before other versions of protoc.

Install the necessary go packages:

$ go get -u \
  github.com/golang/protobuf/protoc-gen-go \
  golang.org/x/tools/cmd/stringer \
  google.golang.org/grpc \

Get mockery and add it to your PATH.

To generate code run in this directory:

$ go generate ./...

Running unit tests

Install Cloud SDK.

Use this command to run the presubmit tests:

$ ./run_unittests --small