[perf] Optimize Spanner database and Git setup in tests

sql_tracestore tests started to timeout after recent 4th migration. This
CL speeds unit tests up.

- Fix connection and session leaks in Spanner emulator tests by configuring the default `sqltest.NewSpannerDBForTests` helper to automatically register cleanup and close the connection pool.
- Optimize `sqltracestore_test.go` by reusing a single shared database connection pool across all tests, significantly reducing Spanner emulator database creation and schema migration runs.
- Implement table clearing (`clearDatabase`) in `sqltracestore_test.go` to maintain test isolation between runs while reusing the `commits` table.
- Optimize `GitBuilder` to support batch commits without pushing (via `DisableAutoPush`), pushing only once at the end of setup. This reduces process execution overhead in `gittest.NewForTest` from ~9s to ~1s.
- Share the git repository setup in `sqltracestore_test.go` via a single initialization, making the git setup phase instant (sub-microsecond) for subsequent tests.

Change-Id: Ic3ee4f3b6561ee9d781cd513987505684a5294b7
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/1292096
Reviewed-by: Sergei Rudenkov <sergeirudenkov@google.com>
Commit-Queue: Marcin Mordecki <mordeckimarcin@google.com>
6 files changed
tree: 27cc052db9b1d3e088a0fefb01b9ce532910d6db
  1. .agents/
  2. am/
  3. android_ingest/
  4. android_stats/
  5. api/
  6. attest/
  7. autogardener/
  8. autoroll/
  9. bash/
  10. bazel/
  11. blamer/
  12. bugs-central/
  13. cabe/
  14. cbb/
  15. cd/
  16. cdn/
  17. cherrypick-watcher/
  18. cmd/
  19. codesize/
  20. comment_rag/
  21. comments/
  22. cq_watcher/
  23. ct/
  24. datahopper/
  25. debugger-app/
  26. demos/
  27. docker/
  28. docs/
  29. docsyserver/
  30. ds/
  31. elements-sk/
  32. experimental/
  33. external/
  34. fiddlek/
  35. firestore/
  36. get_service_account/
  37. gitsync/
  38. go/
  39. gold-client/
  40. golden/
  41. helloworld/
  42. infra/
  43. infra-sk/
  44. jsdoc/
  45. jsfiddle/
  46. k8s-checker/
  47. k8s-deployer/
  48. kube/
  49. leasing/
  50. licenses/
  51. machine/
  52. make/
  53. mcp/
  54. modules/
  55. named-fiddles/
  56. new_element/
  57. npm-audit-mirror/
  58. perdiff/
  59. perf/
  60. periodic-trigger/
  61. pinpoint/
  62. proberk/
  63. promk/
  64. puppeteer-tests/
  65. rag/
  66. sa-keys-checker/
  67. scrap/
  68. scripts/
  69. shaders/
  70. sk/
  71. skbug/
  72. skcq/
  73. skfe/
  74. skolo/
  75. skottie/
  76. static_server/
  77. status/
  78. task_driver/
  79. task_scheduler/
  80. temporal/
  81. test-service/
  82. tool/
  83. tools/
  84. tree_status/
  85. trybot_updater/
  86. .bazelignore
  87. .bazelrc
  88. .bazelversion
  89. .eslintrc.js
  90. .gitattributes
  91. .gitignore
  92. .golangci.yml
  93. .mockery.yaml
  94. .npmrc
  95. .prettierignore
  96. .prettierrc.json
  97. .puppeteerrc.js
  98. .pylintrc
  99. .stylelintrc.js
  100. .swcrc
  101. .vpython
  102. BAZEL_CHEATSHEET.md
  103. BUILD.bazel
  104. build_infra_prod.sh
  105. CDB.md
  106. codereview.settings
  107. DATASTORE.md
  108. demopage.sh
  109. DEPS
  110. elements.d.ts
  111. go.mod
  112. go.sum
  113. karmatest.sh
  114. launch.md
  115. LICENSE
  116. LINT.md
  117. Makefile
  118. MODULE.bazel
  119. MODULE.bazel.lock
  120. OWNERS
  121. package-lock.json
  122. package.json
  123. pnpm-lock.yaml
  124. PRESUBMIT.py
  125. PRIVACY_POLICY.md
  126. puppeteerrc.bzl
  127. puppeteerrc.js.tpl
  128. README.md
  129. requirements.txt
  130. run_lint.sh
  131. STYLEGUIDE.md
  132. tools.go
  133. tsconfig.json
  134. whitespace.txt
  135. workflowcheck.yaml
README.md

Skia-Buildbot Repository

This repo contains infrastructure code for Skia.

Supported Infrastucture Platforms

The infrastructure code is generally built to run on x86 linux. Running on other platforms may be possible but is not officially supported and success will vary depending on the command.

Getting the Source Code

The main source code repository is a Git repository hosted at https://skia.googlesource.com/buildbot.git. It is possible to check out this repository directly with git clone or via go get.

Using git clone allows you to work in whatever directory you want. You will still need to set GOPATH in order to build some apps (recommended to put this in a cache dir). E.g.:

$ cd ${WORKDIR}
$ git clone https://skia.googlesource.com/buildbot.git
$ export GOPATH=${HOME}/.cache/gopath/$(basename ${WORKDIR})
$ mkdir $GOPATH
$ cd buildbot

Install dependencies

Almost all applications are built with Bazel, and bazelisk is the recommended tool to ensure you have the right version of bazel installed:

go install github.com/bazelbuild/bazelisk@latest
go install github.com/bazelbuild/buildtools/buildifier@latest
go install github.com/kisielk/errcheck@latest
go install golang.org/x/tools/cmd/goimports@latest
go install github.com/mikefarah/yq/v4@latest
go install go.chromium.org/luci/client/cmd/...@latest

Add bazelisk to path

export PATH=$PATH:$(go env GOPATH)/bin

Install Node.js and npm

You will need Node.js and npm installed to run web infrastructure tests and linters. We recommend using nvm to manage Node versions.

After installing Node.js, run the following command to install repository dependencies (including linter tools):

npm install

Install other dependencies:

sudo apt-get install jq

Build ~everything

bazelisk build --config=mayberemote //...

Test everything

Set up environ variables

This step might be an optional step, but some test requires these enviornment variables.

Runs

./scripts/run_emulators/run_emulators start

The following are example of environment variables.

Emulators started. Set environment variables as follows:
export DATASTORE_EMULATOR_HOST=localhost:8891
export BIGTABLE_EMULATOR_HOST=localhost:8892
export PUBSUB_EMULATOR_HOST=localhost:8893
export FIRESTORE_EMULATOR_HOST=localhost:8894
export COCKROACHDB_EMULATOR_HOST=localhost:8895

And stores these environment variables to ~/.bashrc file.

Execute tests

bazelisk test --config=mayberemote //...

Generated Code

To update generated code run the following in any directory:

go generate ./...

Running unit tests

Install Cloud SDK.

Use this command to run the presubmit tests:

./run_unittests --small

AI Code Review

Getting AI code review for a local commit:

bazelisk run //cmd/autoreview

Read more at cmd/autoreview/README.md