SkQP: consolidate cut_release scripts
replace the following scripts: cut_release, get_gold_results.py,
goldgetter.py, make_rendertests_list.py, and upload_model with a single
program: cut_release. Still depends on three C++ programs: jitter_gms,
list_gpu_unit_tests, and make_skqp_model.
Change-Id: I28f59bc1f0caedc05d6ce2c4cc11bbd66cfb9784
Reviewed-on: https://skia-review.googlesource.com/c/skia/+/209171
Reviewed-by: Ben Wagner <bungeman@google.com>
Commit-Queue: Hal Canary <halcanary@google.com>
diff --git a/tools/skqp/README_GENERATING_MODELS.md b/tools/skqp/README_GENERATING_MODELS.md
index b36ba54..b852564 100644
--- a/tools/skqp/README_GENERATING_MODELS.md
+++ b/tools/skqp/README_GENERATING_MODELS.md
@@ -8,21 +8,19 @@
COMMIT=origin/master
-1. Get the positively triaged results from Gold:
+ Or use the script to find the best one:
+
+ cd SKIA_SOURCE_DIRECTORY
+ git fetch origin
+ COMMIT=$(python tools/skqp/find_commit_with_best_gold_results.py \
+ origin/master ^origin/skqp/dev)
+
+1. Get the positively triaged results from Gold and generate models:
cd SKIA_SOURCE_DIRECTORY
git fetch origin
git checkout "$COMMIT"
- python tools/skqp/get_gold_results.py "${COMMIT}~10" "$COMMIT"
-
- This will produce a file `meta_YYYMMMDDD_HHHMMMSS_COMMIT_COMMIT.json` in
- the current directory.
-
-2. From a checkout of Skia's master branch, execute:
-
- cd SKIA_SOURCE_DIRECTORY
- git checkout "$COMMIT"
- tools/skqp/cut_release META_JSON_FILE
+ python tools/skqp/cut_release.py HEAD~10 HEAD
This will create the following files:
@@ -32,9 +30,9 @@
These three files can be commited to Skia to create a new commit. Make
`origin/skqp/dev` a parent of this commit (without merging it in), and
- push this new commit to `origin/skqp/dev`:
+ push this new commit to `origin/skqp/dev`, using this script:
- tools/skqp/branch_skqp_dev.sh
+ sh tools/skqp/branch_skqp_dev.sh
Review and submit the change:
@@ -47,20 +45,18 @@
(Optional) Test the SkQP APK:
- adb uninstall org.skia.skqp
- tools/skqp/test_apk.sh LOCATION/skqp-universal-debug.apk
+ tools/skqp/test_apk.sh (LOCATION)/skqp-universal-debug.apk
(Once changes land) Upload the SkQP APK.
- tools/skqp/upload_apk LOCATION/skqp-universal-debug.apk
+ tools/skqp/upload_apk HEAD (LOCATION)/skqp-universal-debug.apk
-`tools/skqp/cut_release`
-------------------------
+`tools/skqp/cut_release.py`
+---------------------------
-This tool will call `make_gmkb.go` to generate the `m{ax,in}.png` files for
-each render test. Additionaly, a `models.txt` file enumerates all of the
-models.
+This tool will call `make_skqp_model` to generate the `m{ax,in}.png` files for
+each render test.
Then it calls `jitter_gms` to see which render tests pass the jitter test.
`jitter_gms` respects the `bad_gms.txt` file by ignoring the render tests
@@ -70,7 +66,7 @@
Next, the `skqp/rendertests.txt` file is created. This file lists the render
tests that will be executed by SkQP. These are the union of the tests
enumerated in the `good.txt` and `bad.txt` files. If the render test is found
-in the `models.txt` file and the `good.txt` file, its per-test threshold is set
+in the `good.txt` file and the model exists, its per-test threshold is set
to 0 (a later CL can manually change this, if needed). Otherwise, the
threshold is set to -1; this indicated that the rendertest will be executed (to
verify that the driver will not crash), but the output will not be compared
diff --git a/tools/skqp/cut_release b/tools/skqp/cut_release
deleted file mode 100755
index a6b199c..0000000
--- a/tools/skqp/cut_release
+++ /dev/null
@@ -1,26 +0,0 @@
-#! /bin/sh
-# Copyright 2018 Google LLC.
-# Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.
-
-if [ -z "$1" ]; then
- echo "Usage: $0 META.JSON" >&2
- exit 1
-fi
-
-set -x
-set -e
-META_JSON="$1"
-cd "$(dirname "$0")/../.."
-env GIT_SYNC_DEPS_QUIET=1 python tools/git-sync-deps
-O='out/ndebug'
-mkdir -p $O
-bin/gn gen $O --args='cc="clang" cxx="clang++" is_debug=false'
-ninja -C $O jitter_gms list_gpu_unit_tests make_skqp_model
-GMKB='platform_tools/android/apps/skqp/src/main/assets/gmkb'
-python tools/skqp/goldgetter.py "$META_JSON" "$GMKB" $O/make_skqp_model
-$O/jitter_gms tools/skqp/bad_gms.txt
-python tools/skqp/make_rendertests_list.py
-rm 'bad.txt' 'good.txt' "$GMKB"/models.txt
-sh tools/skqp/upload_model
-$O/list_gpu_unit_tests \
- > platform_tools/android/apps/skqp/src/main/assets/skqp/unittests.txt
diff --git a/tools/skqp/cut_release.py b/tools/skqp/cut_release.py
new file mode 100755
index 0000000..fba4f6c
--- /dev/null
+++ b/tools/skqp/cut_release.py
@@ -0,0 +1,168 @@
+#! /usr/bin/env python
+# Copyright 2018 Google LLC.
+# Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.
+
+import json
+import md5
+import multiprocessing
+import os
+import shutil
+import sys
+import tempfile
+import urllib
+import urllib2
+
+from subprocess import check_call, check_output
+
+assert '/' in [os.sep, os.altsep] and os.pardir == '..'
+
+ASSETS = 'platform_tools/android/apps/skqp/src/main/assets'
+BUCKET = 'skia-skqp-assets'
+
+def make_skqp_model(arg):
+ name, urls, exe = arg
+ tmp = tempfile.mkdtemp()
+ for url in urls:
+ urllib.urlretrieve(url, tmp + '/' + url[url.rindex('/') + 1:])
+ check_call([exe, tmp, ASSETS + '/gmkb/' + name])
+ shutil.rmtree(tmp)
+ sys.stdout.write(name + ' ')
+ sys.stdout.flush()
+
+def goldgetter(meta, exe):
+ assert os.path.exists(exe)
+ jobs = []
+ for rec in meta:
+ urls = [d['URL'] for d in rec['digests']
+ if d['status'] == 'positive' and
+ (set(d['paramset']['config']) & set(['vk', 'gles']))]
+ if urls:
+ jobs.append((rec['testName'], urls, exe))
+ pool = multiprocessing.Pool(processes=20)
+ pool.map(make_skqp_model, jobs)
+ sys.stdout.write('\n')
+ return set((n for n, _, _ in jobs))
+
+def gold(first_commit, last_commit):
+ c1, c2 = (check_output(['git', 'rev-parse', c]).strip()
+ for c in (first_commit, last_commit))
+ f = urllib2.urlopen('https://public-gold.skia.org/json/export?' + urllib.urlencode([
+ ('fbegin', c1),
+ ('fend', c2),
+ ('query', 'config=gles&config=vk&source_type=gm'),
+ ('pos', 'true'),
+ ('neg', 'false'),
+ ('unt', 'false')
+ ]))
+ j = json.load(f)
+ f.close()
+ return j
+
+def gset(path):
+ s = set()
+ if os.path.isfile(path):
+ with open(path, 'r') as f:
+ for line in f:
+ s.add(line.strip())
+ return s
+
+def make_rendertest_list(models, good, bad):
+ assert good.isdisjoint(bad)
+ do_score = good & models
+ no_score = bad | (good - models)
+ to_delete = models & bad
+ for d in to_delete:
+ path = ASSETS + '/gmkb/' + d
+ if os.path.isdir(path):
+ shutil.rmtree(path)
+ results = dict()
+ for n in do_score:
+ results[n] = 0
+ for n in no_score:
+ results[n] = -1
+ return ''.join('%s,%d\n' % (n, results[n]) for n in sorted(results))
+
+def get_digest(path):
+ m = md5.new()
+ with open(path, 'r') as f:
+ m.update(f.read())
+ return m.hexdigest()
+
+def upload_cmd(path, digest):
+ return ['gsutil', 'cp', path, 'gs://%s/%s' % (BUCKET, digest)]
+
+def upload_model():
+ bucket_url = 'gs://%s/' % BUCKET
+ extant = set((u.replace(bucket_url, '', 1)
+ for u in check_output(['gsutil', 'ls', bucket_url]).splitlines() if u))
+ cmds = []
+ filelist = []
+ for dirpath, _, filenames in os.walk(ASSETS + '/gmkb'):
+ for filename in filenames:
+ path = os.path.join(dirpath, filename)
+ digest = get_digest(path)
+ if digest not in extant:
+ cmds.append(upload_cmd(path, digest))
+ filelist.append('%s;%s\n' % (digest, os.path.relpath(path, ASSETS)))
+ tmp = tempfile.mkdtemp()
+ filelist_path = tmp + '/x'
+ with open(filelist_path, 'w') as o:
+ for l in filelist:
+ o.write(l)
+ filelist_digest = get_digest(filelist_path)
+ if filelist_digest not in extant:
+ cmds.append(upload_cmd(filelist_path, filelist_digest))
+
+ pool = multiprocessing.Pool(processes=20)
+ pool.map(check_call, cmds)
+ shutil.rmtree(tmp)
+ return filelist_digest
+
+def remove(x):
+ if os.path.isdir(x) and not os.path.islink(x):
+ shutil.rmtree(x)
+ if os.path.exists(x):
+ os.remove(x)
+
+def main(first_commit, last_commit):
+ check_call(upload_cmd('/dev/null', get_digest('/dev/null')))
+
+ os.chdir(os.path.dirname(__file__) + '/../..')
+ remove(ASSETS + '/files.checksum')
+ for d in [ASSETS + '/gmkb', ASSETS + '/skqp', ]:
+ remove(d)
+ os.mkdir(d)
+
+ check_call([sys.executable, 'tools/git-sync-deps'],
+ env=dict(os.environ, GIT_SYNC_DEPS_QUIET='T'))
+ build = 'out/ndebug'
+ check_call(['bin/gn', 'gen', build,
+ '--args=cc="clang" cxx="clang++" is_debug=false'])
+ check_call(['ninja', '-C', build,
+ 'jitter_gms', 'list_gpu_unit_tests', 'make_skqp_model'])
+
+ models = goldgetter(gold(first_commit, last_commit), build + '/make_skqp_model')
+
+ check_call([build + '/jitter_gms', 'tools/skqp/bad_gms.txt'])
+
+ with open(ASSETS + '/skqp/rendertests.txt', 'w') as o:
+ o.write(make_rendertest_list(models, gset('good.txt'), gset('bad.txt')))
+
+ remove('good.txt')
+ remove('bad.txt')
+
+ with open(ASSETS + '/skqp/unittests.txt', 'w') as o:
+ o.write(check_output([build + '/list_gpu_unit_tests']))
+
+ with open(ASSETS + '/files.checksum', 'w') as o:
+ o.write(upload_model() + '\n')
+
+ sys.stdout.write(ASSETS + '/files.checksum\n')
+ sys.stdout.write(ASSETS + '/skqp/rendertests.txt\n')
+ sys.stdout.write(ASSETS + '/skqp/unittests.txt\n')
+
+if __name__ == '__main__':
+ if len(sys.argv) != 3:
+ sys.stderr.write('Usage:\n %s C1 C2\n\n' % sys.argv[0])
+ sys.exit(1)
+ main(sys.argv[1], sys.argv[2])
diff --git a/tools/skqp/get_gold_results.py b/tools/skqp/get_gold_results.py
deleted file mode 100755
index ebf23f2..0000000
--- a/tools/skqp/get_gold_results.py
+++ /dev/null
@@ -1,42 +0,0 @@
-#! /usr/bin/env python
-
-# Copyright 2018 Google LLC.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-import os
-import subprocess
-import sys
-import time
-import urllib
-
-def gold_export_url(first_commit, last_commit):
- query = [
- ('fbegin', first_commit),
- ('fend', last_commit),
- ('query', 'config=gles&config=vk&source_type=gm'),
- ('pos', 'true'),
- ('neg', 'false'),
- ('unt', 'false')
- ]
- return 'https://public-gold.skia.org/json/export?' + urllib.urlencode(query)
-
-def git_rev_parse(rev):
- return subprocess.check_output(['git', 'rev-parse', rev]).strip()
-
-def main(args):
- if len(args) != 2:
- sys.stderr.write('Usage:\n %s FIRST_COMMIT LAST_COMMIT\n' % __file__)
- sys.exit(1)
- c1 = git_rev_parse(args[0])
- c2 = git_rev_parse(args[1])
- now = time.strftime("%Y%m%d_%H%M%S", time.gmtime())
- url = gold_export_url(c1, c2)
- sys.stdout.write(url + '\n')
- filename = 'meta_%s_%s_%s.json' % (now, c1[:16], c2[:16])
- urllib.urlretrieve(url, filename)
- sys.stdout.write('\n' + filename + '\n')
-
-if __name__ == '__main__':
- main(sys.argv[1:])
-
diff --git a/tools/skqp/goldgetter.py b/tools/skqp/goldgetter.py
deleted file mode 100755
index ba58eb9..0000000
--- a/tools/skqp/goldgetter.py
+++ /dev/null
@@ -1,48 +0,0 @@
-#! /usr/bin/env python
-# Copyright 2019 Google LLC.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-import json
-import multiprocessing
-import os
-import shutil
-import subprocess
-import sys
-import tempfile
-import urllib
-
-def make_skqp_model(arg):
- name, urls, dst_dir, exe = arg
- tmp = tempfile.mkdtemp()
- for url in urls:
- urllib.urlretrieve(url, tmp + '/' + url[url.rindex('/') + 1:])
- subprocess.check_call([exe, tmp, dst_dir + '/' + name])
- shutil.rmtree(tmp)
- sys.stdout.write(name + ' ')
- sys.stdout.flush()
-
-def main(meta, dst, exe):
- assert os.path.exists(exe)
- jobs = []
- with open(meta, 'r') as f:
- for rec in json.load(f):
- urls = [d['URL'] for d in rec['digests']
- if d['status'] == 'positive' and
- (set(d['paramset']['config']) & set(['vk', 'gles']))]
- if urls:
- jobs.append((rec['testName'], urls, dst, exe))
- if not os.path.exists(dst):
- os.mkdir(dst)
- pool = multiprocessing.Pool(processes=20)
- pool.map(make_skqp_model, jobs)
- sys.stdout.write('\n')
- with open(dst + '/models.txt', 'w') as o:
- for n, _, _, _ in jobs:
- o.write(n + '\n')
-
-if __name__ == '__main__':
- if len(sys.argv) != 4:
- sys.stderr.write('Usage:\n %s META.JSON DST_DIR MAKE_SKQP_MODEL_EXE\n\n' % sys.argv[0])
- sys.exit(1)
- main(sys.argv[1], sys.argv[2], sys.argv[3])
diff --git a/tools/skqp/make_rendertests_list.py b/tools/skqp/make_rendertests_list.py
deleted file mode 100755
index 54d9203..0000000
--- a/tools/skqp/make_rendertests_list.py
+++ /dev/null
@@ -1,49 +0,0 @@
-#! /usr/bin/env python
-
-# Copyright 2018 Google LLC.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-import csv
-import os
-import shutil
-import sys
-
-def gset(path):
- s = set()
- if os.path.isfile(path):
- with open(path, 'r') as f:
- for line in f:
- s.add(line.strip())
- return s
-
-def main():
- assert '/' in [os.sep, os.altsep]
- assets = os.path.join(os.path.dirname(__file__), os.pardir, os.pardir,
- 'platform_tools/android/apps/skqp/src/main/assets')
- models = gset(assets + '/gmkb/models.txt')
- good = gset('good.txt')
- bad = gset('bad.txt')
- assert good.isdisjoint(bad)
- do_score = good & models
- no_score = bad | (good - models)
- to_delete = models & bad
- for d in to_delete:
- path = assets + '/gmkb/' + d
- if os.path.isdir(path):
- shutil.rmtree(path)
- results = dict()
- for n in do_score:
- results[n] = 0
- for n in no_score:
- results[n] = -1
- skqp = assets + '/skqp'
- if not os.path.isdir(skqp):
- os.mkdir(skqp)
- with open(skqp + '/rendertests.txt', 'w') as o:
- for n in sorted(results):
- o.write('%s,%d\n' % (n, results[n]))
-
-if __name__ == '__main__':
- main()
-
diff --git a/tools/skqp/upload_model b/tools/skqp/upload_model
deleted file mode 100755
index b1058e6..0000000
--- a/tools/skqp/upload_model
+++ /dev/null
@@ -1,62 +0,0 @@
-#! /bin/sh
-
-# Copyright 2018 Google Inc.
-# Use of this source code is governed by a BSD-style license that can be
-# found in the LICENSE file.
-
-set -e
-
-BASE_DIR='platform_tools/android/apps/skqp/src/main/assets'
-PATH_LIST='gmkb'
-BUCKET=skia-skqp-assets
-
-EXTANT="$(mktemp "${TMPDIR:-/tmp}/extant.XXXXXXXXXX")"
-gsutil ls gs://$BUCKET/ | sed "s|^gs://$BUCKET/||" > "$EXTANT"
-
-upload() {
- MD5=$(md5sum < "$1" | head -c 32)
- if ! grep -q "$MD5" "$EXTANT"; then
- URL="gs://${BUCKET}/$MD5"
- gsutil cp "$1" "$URL" > /dev/null 2>&1 &
- fi
- echo $MD5
-}
-
-size() { gsutil du -s gs://$BUCKET | awk '{print $1}'; }
-
-cd "$(dirname "$0")/../../$BASE_DIR"
-
-rm -f files.checksum
-
-FILES="$(mktemp "${TMPDIR:-/tmp}/files.XXXXXXXXXX")"
-
-: > "$FILES"
-
-COUNT=$(find $PATH_LIST -type f | wc -l)
-INDEX=1
-SHARD_COUNT=32
-
-SIZE=$(size)
-find $PATH_LIST -type f | sort | while IFS= read -r FILENAME; do
- printf '\r %d / %d ' "$INDEX" "$COUNT"
- if ! [ -f "$FILENAME" ]; then
- echo error [${FILENAME}] >&2;
- exit 1;
- fi
- case "$FILENAME" in *\;*) echo bad filename: $FILENAME >&2; exit 1;; esac
- MD5=$(upload "$FILENAME")
- printf '%s;%s\n' "$MD5" "$FILENAME" >> "$FILES"
-
- if [ $(($INDEX % $SHARD_COUNT)) = 0 ]; then
- wait
- fi
- INDEX=$(( $INDEX + 1))
-done
-printf '\rdone \n'
-upload "$FILES" > files.checksum
-wait
-
-D=$(( $(size) - $SIZE ))
-printf 'Added %d bytes to %s, %d%%\n' $D $BUCKET $(( $D * 100 / $SIZE ))
-
-rm "$EXTANT"
