Fix unsigned int overflow in libjpeg memory mgr.

When attempting to decode a malformed JPEG image (refer to
https://bugzilla.mozilla.org/show_bug.cgi?id=1295044) with dimensions
61472 x 32800, the maximum_space variable within the
realize_virt_arrays() function will exceed the maximum value of a 32-bit
integer and will wrap around.  The memory manager subsequently fails
with an "Insufficient memory" error (case 4, in alloc_large()), so this
commit simply causes that error to be triggered earlier, before UBSan
has a chance to complain.

Note that this issue did not ever represent an exploitable security
threat, because the POSIX-based memory manager that we use doesn't ever
do anything meaningful with the value of maximum_space.
jpeg_mem_available() simply sets avail_mem = maximum_space, so the
subsequent behavior of the memory manager is the same regardless of
whether maximum_space is correct or not.  This commit simply removes a
UBSan warning in order to make it easier to detect actual security
issues.
2 files changed
tree: a7d3dfdc0c2eea40fd03c4005fcfe8300cfda281
  1. .gitignore
  2. BUILDING.md
  3. CMakeLists.txt
  4. ChangeLog.md
  5. LICENSE.md
  6. Makefile.am
  7. README.ijg
  8. README.md
  9. acinclude.m4
  10. bmp.c
  11. bmp.h
  12. cderror.h
  13. cdjpeg.c
  14. cdjpeg.h
  15. change.log
  16. cjpeg.1
  17. cjpeg.c
  18. cmakescripts/
  19. coderules.txt
  20. configure.ac
  21. djpeg.1
  22. djpeg.c
  23. doc/
  24. doxygen-extra.css
  25. doxygen.config
  26. example.c
  27. jaricom.c
  28. java/
  29. jcapimin.c
  30. jcapistd.c
  31. jcarith.c
  32. jccoefct.c
  33. jccolext.c
  34. jccolor.c
  35. jcdctmgr.c
  36. jchuff.c
  37. jchuff.h
  38. jcinit.c
  39. jcmainct.c
  40. jcmarker.c
  41. jcmaster.c
  42. jcomapi.c
  43. jconfig.h.in
  44. jconfig.txt
  45. jconfigint.h.in
  46. jcparam.c
  47. jcphuff.c
  48. jcprepct.c
  49. jcsample.c
  50. jcstest.c
  51. jctrans.c
  52. jdapimin.c
  53. jdapistd.c
  54. jdarith.c
  55. jdatadst-tj.c
  56. jdatadst.c
  57. jdatasrc-tj.c
  58. jdatasrc.c
  59. jdcoefct.c
  60. jdcoefct.h
  61. jdcol565.c
  62. jdcolext.c
  63. jdcolor.c
  64. jdct.h
  65. jddctmgr.c
  66. jdhuff.c
  67. jdhuff.h
  68. jdinput.c
  69. jdmainct.c
  70. jdmainct.h
  71. jdmarker.c
  72. jdmaster.c
  73. jdmaster.h
  74. jdmerge.c
  75. jdmrg565.c
  76. jdmrgext.c
  77. jdphuff.c
  78. jdpostct.c
  79. jdsample.c
  80. jdsample.h
  81. jdtrans.c
  82. jerror.c
  83. jerror.h
  84. jfdctflt.c
  85. jfdctfst.c
  86. jfdctint.c
  87. jidctflt.c
  88. jidctfst.c
  89. jidctint.c
  90. jidctred.c
  91. jinclude.h
  92. jmemmgr.c
  93. jmemnobs.c
  94. jmemsys.h
  95. jmorecfg.h
  96. jpeg_nbits_table.h
  97. jpegcomp.h
  98. jpegint.h
  99. jpeglib.h
  100. jpegtran.1
  101. jpegtran.c
  102. jquant1.c
  103. jquant2.c
  104. jsimd.h
  105. jsimd_none.c
  106. jsimddct.h
  107. jstdhuff.c
  108. jutils.c
  109. jversion.h
  110. libjpeg.map.in
  111. libjpeg.txt
  112. md5/
  113. rdbmp.c
  114. rdcolmap.c
  115. rdgif.c
  116. rdjpgcom.1
  117. rdjpgcom.c
  118. rdppm.c
  119. rdrle.c
  120. rdswitch.c
  121. rdtarga.c
  122. release/
  123. sharedlib/
  124. simd/
  125. structure.txt
  126. testimages/
  127. tjbench.c
  128. tjbenchtest.in
  129. tjbenchtest.java.in
  130. tjexampletest.in
  131. tjunittest.c
  132. tjutil.c
  133. tjutil.h
  134. transupp.c
  135. transupp.h
  136. turbojpeg-jni.c
  137. turbojpeg-mapfile
  138. turbojpeg-mapfile.jni
  139. turbojpeg.c
  140. turbojpeg.h
  141. usage.txt
  142. win/
  143. wizard.txt
  144. wrbmp.c
  145. wrgif.c
  146. wrjpgcom.1
  147. wrjpgcom.c
  148. wrppm.c
  149. wrppm.h
  150. wrrle.c
  151. wrtarga.c