Huff dec: Fix non-deterministic output w/bad input
Referring to https://bugzilla.mozilla.org/show_bug.cgi?id=1050342,
there are certain very rare circumstances under which a malformed JPEG
image can cause different Huffman decoder output to be produced,
depending on the size of the source manager's I/O buffer. (More
specifically, the fast Huffman decoder didn't handle invalid codes in
the same manner as the slow decoder, and since the fast decoder requires
all data to be memory-resident, the buffering strategy determines
whether or not the fast decoder can be used on a particular MCU block.)
After extensive experimentation, the Mozilla and Chrome developers and I
determined that this truly was an innocuous issue. The patch that both
browsers adopted as a workaround caused a performance regression with
32-bit code, which is why it was not accepted into libjpeg-turbo. This
commit fixes the problem in a less disruptive way with no performance
regression.
diff --git a/jdhuff.h b/jdhuff.h
index ac6e0e5..60d8e8b 100644
--- a/jdhuff.h
+++ b/jdhuff.h
@@ -4,7 +4,7 @@
* This file was part of the Independent JPEG Group's software:
* Copyright (C) 1991-1997, Thomas G. Lane.
* libjpeg-turbo Modifications:
- * Copyright (C) 2010-2011, 2015-2016, D. R. Commander.
+ * Copyright (C) 2010-2011, 2015-2016, 2021, D. R. Commander.
* Copyright (C) 2018, Matthias Räncker.
* For conditions of distribution and use, see the accompanying README.ijg
* file.
@@ -234,7 +234,10 @@
s |= GET_BITS(1); \
nb++; \
} \
- s = htbl->pub->huffval[(int)(s + htbl->valoffset[nb]) & 0xFF]; \
+ if (nb > 16) \
+ s = 0; \
+ else \
+ s = htbl->pub->huffval[(int)(s + htbl->valoffset[nb])]; \
}
/* Out-of-line case for Huffman code fetching */
