skcms: Increase transform op capacity and add bounds checks

Complex color transforms (e.g. A2B to B2A with 4-channel curve tables,
matrices, format and space conversion) can require ~40 ops, which
exceeded the previous hardcoded limit of 32 ops and resulted in a
stack buffer overflow.

Increase SKCMS_MAX_PROGRAM_OPS to 64, providing ample padding while
keeping stack overhead small. Harden add_op and add_op_ctx to avoid
writing past the program buffer during compilation, and add runtime
checks in both skcms_Transform and run_program to gracefully reject
programs that exceed the maximum capacity.

This limit is only hit when doing an A2B to B2A conversion, which
Chromium and Android do not do.

Fixed: b/530238257
TAG=agy
CONV=56f173c3-5496-4fd5-ad6b-34aebcff742d
Change-Id: I1868e43405dcd109c01c007ee0ade4d0aebe3198
Reviewed-on: https://skia-review.googlesource.com/c/skcms/+/1351256
Reviewed-by: Florin Malita <fmalita@google.com>
Commit-Queue: Christopher Cameron <ccameron@google.com>
3 files changed
tree: 34a9e173217000b9a16bd2cb98085d6d9b3cb00c
  1. bazel/
  2. fuzz/
  3. infra/
  4. ninja/
  5. profiles/
  6. src/
  7. toolchain/
  8. .bazelrc
  9. .bazelversion
  10. .gitignore
  11. BAZEL.md
  12. bench.c
  13. BUILD.bazel
  14. build.ninja
  15. codereview.settings
  16. iccdump.c
  17. LICENSE
  18. MODULE.bazel
  19. MODULE.bazel.lock
  20. msvs.ninja
  21. OWNERS
  22. README.chromium
  23. README.md
  24. skcms.cc
  25. skcms.gni
  26. skcms.h
  27. test_only.c
  28. test_only.h
  29. tests.c
  30. whitespace.txt
README.md

To build with ninja, simply run ninja from the root directory. This will produce a library in /out.

There are also Bazel rules, outlined in BAZEL.md