nanostat compares statistics about nanobench results.
You need to have Go installed. Then run:
go get -u go.skia.org/infra/perf/nanostat
If you have $GOPATH
or $GOBIN
set then nanostat
should appear in your path, otherwise it will be installed at $HOME/go/bin/nanostat
.
Each input file should be a JSON output file from nanobench.
Invoked on a pair of input files, nanostat computes statistics for each file and a column showing the percent change in mean from the first to the second file. Next to the percent change, nanostat shows the p-value and sample sizes from a test of the two distributions of nanobench results.
For example in the results below, ‘p’ equals 0.001 or 0.1%, and the analysis was done with 10 samples from the first file and 8 samples from the second file.
old new delta stats name 2.15 ± 5% 2.00 ± 2% -7% (p=0.001, n=10+ 8) tabl_digg.skp
Small p-values indicate that the two distributions are significantly different. If the test indicates that there was no significant change between the two benchmarks (defined as p > alpha), nanostat displays a single ~ instead of the percent change.
Suppose we collect benchmark results from running
out/Release/nanobench --config gl 8888 --outResultsFile old.json
Then make some changes to the code, recompile nanobench and run:
out/Release/nanobench --config gl 8888 --outResultsFile new.json
Then nanostat summarizes the differences between the old and new runs:
$ nanostat --iqrr old.json new.json old new delta s name 0.78 ± 3% 0.72 ± 3% -8% (p=0.000, n=10+10) desk_wowwiki.skp 2.15 ± 5% 2.00 ± 2% -7% (p=0.001, n=10+ 8) tabl_digg.skp 3.08 ± 2% 2.96 ± 3% -4% (p=0.001, n= 9+10) desk_facebook.skp 0.71 ± 2% 0.69 ± 3% -3% (p=0.028, n= 9+10) desk_ebay.skp 4.59 ± 1% 4.46 ± 1% -3% (p=0.000, n=10+ 8) desk_linkedin.skp 1.40 ± 1% 1.39 ± 0% -1% (p=0.011, n= 9+ 9) desk_css3gradients.skp $
usage: nanostat [options] old.json new.json options: -all If true then include insignificant changes in output. -alpha float Consider a change significant if p < α. Must be > 0. (default 0.05). -iqrr If true then remove outliers in the samples using the Interquartile Range Rule. -sort order Sort by order: [-]delta, [-]name (default "delta") -test string The type of test to do, 'utest' for Mann-Whitney U test, and 'ttest' for a Two Sample Welch T test. (default "utest")
To get help:
$ nanostat -h