[gold] Puppeteer tests: Dockerized testing infrastructure, and example test.
This CL lays the groundwork for running general-purpose, Dockerized tests that use Puppeteer, and optionally produce some kind of output. Its main intended uses are screenshot-grabbing tests (which are then uploaded to Gold) and integration tests that will use Puppeteer to drive the webapp served by the skiacorrectness binary.
The example test in this CL (puppeteer_test.js) will be removed once we have a few real tests in place.
Next steps:
- Write a test for triagelog-page-sk that grabs a few screenshots.
- JS code to upload any screenshots to Gold after the test finishes (possibly by invoking goldctl).
Bug: skia:9608
Change-Id: I46a86cd66e1fa41e075bd9c202e4c57ff864efc2
Reviewed-on: https://skia-review.googlesource.com/c/buildbot/+/253379
Commit-Queue: Leandro Lovisolo <lovisolo@google.com>
Reviewed-by: Kevin Lubick <kjlubick@google.com>
diff --git a/golden/.gitignore b/golden/.gitignore
index 378eac2..64a2f96 100644
--- a/golden/.gitignore
+++ b/golden/.gitignore
@@ -1 +1,2 @@
build
+puppeteer-tests/output
diff --git a/golden/Makefile b/golden/Makefile
index f60e4b6..e4fa326 100644
--- a/golden/Makefile
+++ b/golden/Makefile
@@ -19,7 +19,7 @@
go test ./go/... -bench=. -run=NONE
.PHONY: test
-test: go-test js-test
+test: go-test js-test puppeteer-test
true
.PHONY: go-test
@@ -34,6 +34,27 @@
js-test-debug:
npx karma start --no-single-run
+.PHONY: puppeteer-test
+puppeteer-test: build-puppeteer-tests-docker-image clean-puppeteer-tests-output
+ docker run -it --rm -v `pwd`/puppeteer-tests/output:/out gold-puppeteer-tests
+
+.PHONY: build-puppeteer-tests-docker-image
+build-puppeteer-tests-docker-image:
+ puppeteer-tests/docker/build-image.sh
+
+.PHONY: puppeteer-test-nodocker
+puppeteer-test-nodocker: clean-puppeteer-tests-output
+ npx mocha puppeteer-tests/test
+
+.PHONY: puppeteer-test-debug
+puppeteer-test-debug: clean-puppeteer-tests-output
+ npx mocha puppeteer-tests/test --debug-brk
+
+.PHONY: clean-puppeteer-tests-outout
+clean-puppeteer-tests-output:
+ rm -rf puppeteer-tests/output
+ mkdir -p puppeteer-tests/output
+
node_modules: package.json
npm install
diff --git a/golden/package.json b/golden/package.json
index e292777..8c3dcc5 100644
--- a/golden/package.json
+++ b/golden/package.json
@@ -28,6 +28,7 @@
"native-promise-only": "~0.8.1",
"postcss-cli": "~6.1.3",
"pulito": "~4.4.0",
+ "puppeteer": "2.0.0",
"sinon": "^7.4.2",
"uglify-js": "~3.6.0",
"vulcanize": "~1.16.0",
diff --git a/golden/puppeteer-tests/README.md b/golden/puppeteer-tests/README.md
new file mode 100644
index 0000000..1ab5d81
--- /dev/null
+++ b/golden/puppeteer-tests/README.md
@@ -0,0 +1,64 @@
+# Puppeteer Tests
+
+This directory contains JS tests that make use of [Puppeteer](https://pptr.dev).
+Puppeteer is a Node.js library that provides an API to instantiate and control a
+headless Chromium browser. Most things that can be done manually in the browser
+can be done using Puppeteer.
+
+Examples of such tests might include:
+
+ - Screenshot-grabbing tests. For example, in the case of lit-html components,
+ such a test might perform the following steps:
+ 1. Load the component's demo page on a headless Chromium instance.
+ 2. Perform some basic assertions on the structure of the webpage to make sure
+ it loads correctly.
+ 3. Grab screenshots.
+ 4. Upload those screenshots to the Skia Infra Gold instance.
+ <!-- TODO(lovisolo): Set up a Gold instance for Skia Infra.
+ Add a link to it here. -->
+
+ - Integration tests. Example steps:
+ 1. Fire up a Go web server configured to use fake/mock instances of its
+ dependencies.
+ - e.g. use the Firestore emulator instead of hitting GCP.
+ - Ideally in a
+ [hermetic](https://testing.googleblog.com/2012/10/hermetic-servers.html)
+ way for increased speed and reduced flakiness.
+ 2. Drive the app with Puppeteer. Make assertions along the way.
+ 3. Optionally grab screenshots and upload them to Gold.
+
+Tests under this directory use the [Mocha](https://mochajs.org/) test runner.
+
+Any output files generated from these tests (e.g. screenshots) will be found in
+`$SKIA_INFRA_ROOT/golden/puppeteer-tests/output`.
+
+### Docker container
+
+Puppeteer tests run inside a Docker container. This provides a more stable
+testing environment and reduces screenshot flakiness.
+
+The corresponding `Dockerfile` can be found in
+`$SKIA_INFRA_ROOT/golden/puppeteer-tests/docker`.
+
+## Usage
+
+Run `make puppeteer-test` from `$SKIA_INFRA_ROOT/golden`. This will build and
+run a Docker container that executes the Mocha test runner.
+
+If you wish to run these tests outside of Docker, try
+`make puppeteer-test-nodocker`. Or equivalently, `cd` into
+`$SKIA_INFRA_ROOT/golden/puppeteer-tests/test` and run `npx mocha`.
+
+## Debugging
+
+The two options below run the tests outside of Docker.
+
+ - `cd` into `$SKIA_INFRA_ROOT/golden/puppeteer-tests/test` and run
+ `npx mocha debug`. This will start the tests and immediately drop into the
+ Node.js inspector.
+ - Alternatively, `cd` into `$SKIA_INFRA_ROOT/golden/puppeteer-tests/test` and
+ run `npx mocha --inspect-brk`. This will start the tests and wait until a
+ debugger is attached. Attach any debgger of your liking, e.g. Chrome Dev
+ Tools, VSCode, IntelliJ, etc. See
+ [here](https://mochajs.org/#-debug-inspect-debug-brk-inspect-brk-debug-inspect)
+ for more.
diff --git a/golden/puppeteer-tests/docker/Dockerfile b/golden/puppeteer-tests/docker/Dockerfile
new file mode 100644
index 0000000..536dd6c
--- /dev/null
+++ b/golden/puppeteer-tests/docker/Dockerfile
@@ -0,0 +1,34 @@
+FROM node:12.13
+
+# Install Chrome and fonts to support major charsets. This installs the
+# necessary libraries to make the bundled version of Chromium that Puppeter
+# installs work. Copied verbatim from:
+# https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md#running-puppeteer-in-docker
+RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
+ && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
+ && apt-get update \
+ && apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf \
+ --no-install-recommends \
+ && rm -rf /var/lib/apt/lists/*
+
+# Output directory. Screenshots and other test output will be stored here.
+RUN mkdir /out
+
+WORKDIR /src
+
+# This runs "npm ci" only if package{-lock}.json changed.
+COPY package.json package-lock.json ./
+
+# See https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable
+RUN npm ci
+
+# Any dependencies should be explicitly copied into the image.
+COPY webpack.config.js ./
+COPY modules modules/
+COPY pages pages/
+COPY puppeteer-tests puppeteer-tests/
+
+WORKDIR puppeteer-tests
+
+# Runs all tests under directory "puppeteer-tests".
+CMD ["npx", "mocha"]
diff --git a/golden/puppeteer-tests/docker/build-image.sh b/golden/puppeteer-tests/docker/build-image.sh
new file mode 100755
index 0000000..e2e4c89
--- /dev/null
+++ b/golden/puppeteer-tests/docker/build-image.sh
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# This script takes ~2s to complete after the Docker image is built for the
+# first time.
+
+cd "$(dirname "$0")" # Set working directory to this script's directory.
+
+cp dockerignore ../../.dockerignore
+docker build -t gold-puppeteer-tests -f Dockerfile ../.. --quiet # Remove --quiet for debugging.
+rm ../../.dockerignore
diff --git a/golden/puppeteer-tests/docker/dockerignore b/golden/puppeteer-tests/docker/dockerignore
new file mode 100644
index 0000000..d12493d
--- /dev/null
+++ b/golden/puppeteer-tests/docker/dockerignore
@@ -0,0 +1,3 @@
+node_modules
+frontend
+build
diff --git a/golden/puppeteer-tests/test/puppeteer_test.js b/golden/puppeteer-tests/test/puppeteer_test.js
new file mode 100644
index 0000000..aedbcf9
--- /dev/null
+++ b/golden/puppeteer-tests/test/puppeteer_test.js
@@ -0,0 +1,64 @@
+// This is just a sample test to verify that Puppeteer works inside Docker. It
+// shows how to use Puppeteer to query the DOM and take screenshots.
+//
+// TODO(lovisolo): Remove after we have a few real tests.
+
+const expect = require('chai').expect;
+const express = require('express');
+const fs = require('fs');
+const path = require('path');
+const puppeteer = require('puppeteer');
+
+describe('puppeteer', function() {
+ let browser, page, server;
+
+ before(async () => {
+ server = await startTestServer();
+ browser = await launchBrowser();
+ });
+
+ after(async () => {
+ await browser.close();
+ await server.close();
+ });
+
+ beforeEach(async () => { page = await browser.newPage(); });
+ afterEach(async () => { await page.close(); });
+
+ it('queries the DOM', async () => {
+ await page.goto(`http://localhost:${server.address().port}`);
+ expect(await page.$eval('h1', (el) => el.innerText)).to.equal('hello');
+ expect(await page.$eval('p', (el) => el.innerText)).to.equal('world');
+ });
+
+ it('takes screenshots', async () => {
+ await page.goto(`http://localhost:${server.address().port}`);
+ await page.screenshot({path: path.join(outputDir(), 'screenshot.png')});
+ });
+});
+
+// Starts an Express server on a random, unused port. Serves a test page.
+const startTestServer = () => {
+ const app = express();
+ app.get('/', (_, res) => {
+ res.send('<html><body><h1>hello</h1><p>world</p></body></html>');
+ });
+ return new Promise((resolve) => {
+ const server = app.listen(0, () => resolve(server));
+ });
+};
+
+// TODO(lovisolo): Extract out the functions below into a file named e.g.
+// "testbed.js" under directory "puppeteer-tests".
+
+const inDocker = () => fs.existsSync('/.dockerenv');
+
+const launchBrowser = () => puppeteer.launch(inDocker() ? {
+ args: ['--disable-dev-shm-usage', '--no-sandbox'],
+} : {});
+
+const outputDir =
+ () => inDocker()
+ ? '/out'
+ // Resolves to $SKIA_INFRA_ROOT/golden/puppeteer-tests/output.
+ : path.join(__dirname, '..', 'output');
