Migrate repository to a single package.json file.

File //package.json now contains the union of all other package.json files, with most NPM packages updated to their latest versions.

Files //html-template-minifier/package.json and //jsdoc/package.json remain unchanged. All other package.json files are deleted.

Minor changes to some JS/TS files were necessary after updating the NPM packages.

I added a //make/npm.mk file that defines an npm-ci make target that can be added as a prerequisite to any targets that depend on the node_modules directory.

Added some apps to CI (see changes in //Makefile).

Nearly all changes to BUILD.bazel file are automatic (Gazelle).

Steps I took to generate //package.json:

    # Compute the list of target package.json files.
    $ allpackagejson=`find | grep "package\.json$" | grep -v html-template-minifier | grep -v jsdoc | grep -v node_modules | grep -v bower_components | grep -v .git | sort`

    # Collect all NPM packages packages from all package.json files.
    $ allpkgs=`jq -r '(.dependencies // {}) + (.devDependencies // {}) | keys | .[]' $allpackagejson | sort | uniq`

    # Uninstall all NPM packages from //package.json.
    $ npm uninstall `jq -r '(.dependencies // {}) + (.devDependencies // {}) | keys | .[]' package.json`

    # Delete all other package.json and package-lock.json files.
    $ git rm `find | grep "package\.json$" | grep -v ^./package.json$ | grep -v html-template-minifier | grep -v jsdoc | grep -v node_modules | grep -v bower_components | grep -v .git`
    $ git rm `find | grep package-lock.json | grep -v ^./package-lock.json$ | grep -v html-template-minifier | grep -v jsdoc | grep -v node_modules | grep -v bower_components | grep -v .git`

    # Install all collected packages at their most recent version.
    $ npm install $allpkgs

    # Reinstall some packages at specific versions.
    $ npm install webpack@^4.0.0 @types/webpack@^4.0.0
    $ npm install webpack-dev-middleware@^3.0.0 @types/webpack-dev-middleware@^3.0.0
    $ npm install webpack-dev-server@^3.0.0 @types/webpack-dev-server@^3.0.0
    $ npm install webpack-cli@^3.0.0
    $ npm install mini-css-extract-plugin@^0.9.0 @types/mini-css-extract-plugin@^0.9.0
    $ npm install html-webpack-plugin@^4.0.0
    $ npm install copy-webpack-plugin@^5.0.0 @types/copy-webpack-plugin@^5.0.0
    $ npm install node-sass@^4.0.0 sass-loader@^7.0.0
    $ npm install postcss-loader@^2.0.0
    $ npm install autoprefixer@^9.0.0
    $ npm install html-loader@^0.5.5
    $ npm install cssnano@^3.0.0
    $ npm install karma@^4.0.0 @types/karma@^4.0.0
    $ npm install karma-webpack@^4.0.0
    $ npm install sinon@^7.0.0 @types/sinon@^7.0.0
    $ npm install d3-selection@^1.0.0
    $ npm install @bazel/rollup@~2.0.0 @bazel/terser@~2.0.0 @bazel/typescript@~2.0.0
    $ npm install @rollup/plugin-commonjs@~15.0.0 @rollup/plugin-node-resolve@~9.0.0
    $ npm install terser@^5.0.0

    # Uninstall unneeded packages.
    $ npm uninstall @bundled-es-modules/fetch-mock @types/fetch-mock
    $ npm uninstall terser-webpack-plugin

Bug: skia:10614
Change-Id: I342f0c950af313752025087e70fd2b2d6fa6a709
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/395916
Commit-Queue: Leandro Lovisolo <lovisolo@google.com>
Reviewed-by: Ravi Mistry <rmistry@google.com>
Reviewed-by: Joe Gregorio <jcgregorio@google.com>
Reviewed-by: Eric Boren <borenet@google.com>
Reviewed-by: Kevin Lubick <kjlubick@google.com>
202 files changed
tree: 1b3d00108ed2fd6a68a9dfa47f85a6add0d0227c
  1. am/
  2. android_ingest/
  3. android_stats/
  4. api/
  5. autoroll/
  6. bash/
  7. bazel/
  8. blamer/
  9. bugs-central/
  10. codereview-watcher/
  11. comments/
  12. contestk/
  13. cq_watcher/
  14. ct/
  15. datahopper/
  16. debugger-app/
  17. demos/
  18. docker/
  19. docker_pushes_watcher/
  20. docs/
  21. docsyserver/
  22. dot/
  23. ds/
  24. external/
  25. fiddlek/
  26. firestore/
  27. get_service_account/
  28. git_cookie_authdaemon/
  29. gitsync/
  30. go/
  31. gold-client/
  32. golden/
  33. hashtag/
  34. html-template-minifier/
  35. infra/
  36. infra-sk/
  37. jsdoc/
  38. jsfiddle/
  39. k8s-checker/
  40. kube/
  41. leasing/
  42. licenses/
  43. machine/
  44. make/
  45. modules/
  46. named-fiddles/
  47. new_element/
  48. particles/
  49. perdiff/
  50. perf/
  51. periodic-trigger/
  52. power/
  53. proberk/
  54. promk/
  55. pulld/
  56. puppeteer-tests/
  57. push/
  58. scrap/
  59. scripts/
  60. shaders/
  61. sk/
  62. sk8s/
  63. skbug/
  64. skfe/
  65. skolo/
  66. skottie/
  67. status/
  68. task_driver/
  69. task_scheduler/
  70. tools/
  71. tree_status/
  72. trybot_updater/
  73. velero/
  74. .bazelignore
  75. .bazelrc
  76. .eslintrc.js
  77. .gitattributes
  78. .gitignore
  79. .mocharc.json
  80. BUILD.bazel
  81. build_infra_prod.sh
  82. cipd.ensure
  83. codereview.settings
  84. DATASTORE.md
  85. DEPS
  86. elements-sk-scss.bzl
  87. go.mod
  88. go.sum
  89. go_repositories.bzl
  90. launch.md
  91. LICENSE
  92. Makefile
  93. package-lock.json
  94. package.json
  95. PRESUBMIT.py
  96. PRIVACY_POLICY.md
  97. README.md
  98. run_unittests
  99. run_unittests.go
  100. STYLEGUIDE.md
  101. tools.go
  102. tsconfig.json
  103. whitespace.txt
  104. WORKSPACE
README.md

Skia-Buildbot Repository

This repo contains infrastructure code for Skia.

Getting the Source Code

The main source code repository is a Git repository hosted at https://skia.googlesource.com/buildbot.git. It is possible to check out this repository directly with git clone or via go get.

Using git clone allows you to work in whatever directory you want. You will still need to set GOPATH in order to build some apps (recommended to put this in a cache dir). E.g.:

$ cd ${WORKDIR}
$ git clone https://skia.googlesource.com/buildbot.git
$ export GOPATH=${HOME}/.cache/gopath/$(basename ${WORKDIR})
$ mkdir $GOPATH
$ cd buildbot

Using go get will fetch the repository into your GOPATH directory along with all the Go dependencies. You will need to set GOPATH and GO111MODULE=on. E.g.:

$ export GOPATH=${WORKDIR}
$ export GO111MODULE=on
$ go get -u -t go.skia.org/infra/...
$ cd ${GOPATH}/src/go.skia.org/infra/

Note: go.skia.org is a custom import path and will only work if used like the examples here.

Install Node.js (not as root) and add the bin dir to your path. Optionally run npm install npm -g, as suggested by the npm getting started doc.

Install other dependencies:

$ sudo apt-get install python-django
$ go get -u github.com/kisielk/errcheck \
  golang.org/x/tools/cmd/goimports \
  go.chromium.org/luci/client/cmd/isolate

Build ~everything:

$ make all

Generated Code

Some code is generated using go generate with external binaries. First, install the version of protoc referenced in the asset creation script and ensure it is on your PATH before other versions of protoc.

Install the necessary go packages:

$ go get -u \
  github.com/golang/protobuf/protoc-gen-go \
  golang.org/x/tools/cmd/stringer \
  google.golang.org/grpc \

Get mockery and add it to your PATH.

To generate code run in this directory:

$ go generate ./...

Running unit tests

Install Cloud SDK.

Use this command to run the presubmit tests:

$ ./run_unittests --small