Commit 2f3bb8e4 for libheif
commit 2f3bb8e48c24ea42a0dbc6fc180f44f0d745d811
Author: Dirk Farin <dirk.farin@gmail.com>
Date: Sun Sep 6 00:21:48 2026 +0200
Let each color-conversion operator declare its own bit-depth limit
An operator that cannot handle a bit depth should not offer itself to the
pipeline for it. Op_YCbCr_to_RGB already worked that way, but most operators
only bounded the depth from below: '(bits_per_pixel > 8) != hdr' and
'bits_per_pixel <= 8 -> reject' both accept everything above 8, and
Op_mono_to_YCbCr420 stated no bit-depth condition at all, which is how a
64-bit 'unci' component reached it and shifted an 'int' by 56.
Add has_samples_wider_than_16bit() next to ColorState, which carries the
shared reason in one place (operators access samples through uint8_t* or
uint16_t*, so nothing wider than 16 bits is reachable, colour or alpha), and
call it from the operators that were open-ended:
Op_mono_to_YCbCr420, Op_flatten_alpha_plane, Op_adjust_alpha_bit_depth,
Op_to_sdr_planes, Op_RRGGBBaa_swap_endianness, Op_RGB_HDR_to_RRGGBBaa_BE,
Op_RRGGBBaa_BE_to_RGB_HDR, Op_RGB_to_YCbCr, Op_RRGGBBxx_HDR_to_YCbCr420,
Op_YCbCr420_to_RRGGBBaa, and the four chroma_sampling operators.
Op_YCbCr_to_RGB's own '> 16' check becomes the helper so that its alpha plane,
which it copies through at the plane's own depth, is bounded too.
Op_RGB24_32_to_YCbCr and Op_RGB24_32_to_YCbCr444_GBR index the interleaved
input as bytes and hard-code an 8-bit output, so they now say '!= 8'.
Operators that already stated a tighter or exact range are left alone:
Op_Any_RGB_to_YCbCr_420_Sharp (an explicit {8,10,12,16} list),
Op_bayer_bilinear_to_RGB24_32 (<= 16 with an else-reject), Op_to_hdr_planes
(a bounded target plus a per-channel runtime guard), and the several operators
that pin an exact 8 bits. Op_drop_alpha_plane needs no limit: it copies whole
planes and reads no samples.
With this, pipeline construction refuses a wider-than-16-bit input before the
catch-all added in 8bfecb52 is reached; verified by disabling that check and
confirming the sequence_fuzzer testcase stays clean under UBSan and the test
suite still passes. The catch-all stays as a backstop until an operator really
supports more than 16 bits, as does its comment saying so.
diff --git a/libheif/color-conversion/alpha.cc b/libheif/color-conversion/alpha.cc
index edcab7ac..0d2478e0 100644
--- a/libheif/color-conversion/alpha.cc
+++ b/libheif/color-conversion/alpha.cc
@@ -143,6 +143,10 @@ Op_flatten_alpha_plane<Pixel>::state_after_conversion(const ColorState& input_st
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
if (input_state.has_alpha && input_state.get_alpha_bits_per_pixel() != input_state.bits_per_pixel) {
return {};
}
@@ -366,6 +370,12 @@ Op_adjust_alpha_bit_depth::state_after_conversion(const ColorState& input_state,
return {};
}
+ // Rewrites the alpha plane from its own bit depth to the colour bit depth, so both
+ // ends have to be accessible as 8- or 16-bit samples.
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
std::vector<ColorStateWithCost> states;
ColorState output_state = input_state;
diff --git a/libheif/color-conversion/chroma_sampling.cc b/libheif/color-conversion/chroma_sampling.cc
index 5b63f733..3d0a9cd1 100644
--- a/libheif/color-conversion/chroma_sampling.cc
+++ b/libheif/color-conversion/chroma_sampling.cc
@@ -49,6 +49,10 @@ Op_YCbCr444_to_YCbCr420_average<Pixel>::state_after_conversion(const ColorState&
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
if (input_state.nclx.get_matrix_coefficients() == 0) {
return {};
}
@@ -273,6 +277,10 @@ Op_YCbCr444_to_YCbCr422_average<Pixel>::state_after_conversion(const ColorState&
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
if (input_state.nclx.get_matrix_coefficients() == 0) {
return {};
}
@@ -475,6 +483,10 @@ Op_YCbCr420_bilinear_to_YCbCr444<Pixel>::state_after_conversion(const ColorState
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
if (input_state.nclx.get_matrix_coefficients() == 0) {
return {};
}
@@ -756,6 +768,10 @@ Op_YCbCr422_bilinear_to_YCbCr444<Pixel>::state_after_conversion(const ColorState
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
if (input_state.nclx.get_matrix_coefficients() == 0) {
return {};
}
diff --git a/libheif/color-conversion/colorconversion.cc b/libheif/color-conversion/colorconversion.cc
index 3e33998c..eee9333b 100644
--- a/libheif/color-conversion/colorconversion.cc
+++ b/libheif/color-conversion/colorconversion.cc
@@ -628,12 +628,12 @@ Result<std::shared_ptr<HeifPixelImage>> convert_colorspace(const std::shared_ptr
// 56 (OSS-Fuzz 5154611212910592). A nop conversion is handled above and still hands
// the image through untouched, so wide components stay accessible to the caller.
//
- // This is a catch-all. The constraint properly belongs in each operator's
- // state_after_conversion(), where Op_YCbCr_to_RGB already declines an input wider
- // than 16 bits: an operator that cannot handle a bit depth should not offer itself
- // to the pipeline for it. Several operators currently bound the depth from below
- // only (e.g. '(bits_per_pixel > 8) != hdr'). Remove this check once an operator
- // actually supports more than 16 bits per component.
+ // This is a backstop, not the primary defence. The constraint belongs in each
+ // operator's state_after_conversion(), and every operator now declares it there
+ // (most through has_samples_wider_than_16bit()), so construct_pipeline() above
+ // already fails for a wider input and a real conversion never reaches this loop.
+ // Keep it until an operator actually supports more than 16 bits per component,
+ // then remove it together with that operator's call to the helper.
for (heif_channel channel : channels) {
if (input->get_bits_per_pixel(channel) > 16) {
diff --git a/libheif/color-conversion/colorconversion.h b/libheif/color-conversion/colorconversion.h
index 54e94e0d..e1a6d8a6 100644
--- a/libheif/color-conversion/colorconversion.h
+++ b/libheif/color-conversion/colorconversion.h
@@ -53,6 +53,27 @@ struct ColorState
std::ostream& operator<<(std::ostream& ostr, const ColorState& state);
+
+// True if 'state' has a colour or alpha component wider than 16 bits.
+//
+// Every conversion operator reads and writes sample data through uint8_t* or uint16_t*
+// and derives shift amounts from the bit depth, so none of them can handle a wider
+// component. Images with wider components do exist: 'unci' components may be up to 256
+// bits and we store up to 128 of them (64-bit integers, 32/64-bit floats, complex
+// numbers) so that they can be read through the component API.
+//
+// An operator that cannot handle a bit depth must not offer itself to the pipeline for
+// it, so each operator states its own supported range in state_after_conversion(). This
+// helper spells out the upper bound they currently all share; operators with a tighter
+// or different range (an exact 8 bits, or an explicit list) say so themselves instead.
+// When an operator gains support for wider samples it simply stops calling this, and the
+// catch-all in convert_colorspace() can go away.
+inline bool has_samples_wider_than_16bit(const ColorState& state)
+{
+ return state.bits_per_pixel > 16 ||
+ (state.has_alpha && state.get_alpha_bits_per_pixel() > 16);
+}
+
// These are some integer constants for typical color conversion Op speed costs.
// The integer value is the speed cost. Any other integer can be assigned to the speed cost.
enum SpeedCosts
diff --git a/libheif/color-conversion/hdr_sdr.cc b/libheif/color-conversion/hdr_sdr.cc
index ca70ef24..34031724 100644
--- a/libheif/color-conversion/hdr_sdr.cc
+++ b/libheif/color-conversion/hdr_sdr.cc
@@ -148,6 +148,12 @@ Op_to_sdr_planes::state_after_conversion(const ColorState& input_state,
return {};
}
+ // Every channel, alpha included, is read through a uint16_t* and shifted down by
+ // (bits_per_pixel - 8).
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
std::vector<ColorStateWithCost> states;
ColorState output_state;
diff --git a/libheif/color-conversion/monochrome.cc b/libheif/color-conversion/monochrome.cc
index 4ec5b708..afdebdb0 100644
--- a/libheif/color-conversion/monochrome.cc
+++ b/libheif/color-conversion/monochrome.cc
@@ -33,6 +33,10 @@ Op_mono_to_YCbCr420::state_after_conversion(const ColorState& input_state,
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
std::vector<ColorStateWithCost> states;
ColorState output_state;
diff --git a/libheif/color-conversion/rgb2rgb.cc b/libheif/color-conversion/rgb2rgb.cc
index 29db93f3..777021d7 100644
--- a/libheif/color-conversion/rgb2rgb.cc
+++ b/libheif/color-conversion/rgb2rgb.cc
@@ -164,6 +164,10 @@ Op_RGB_HDR_to_RRGGBBaa_BE::state_after_conversion(const ColorState& input_state,
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
if (input_state.has_alpha && input_state.get_alpha_bits_per_pixel() != input_state.bits_per_pixel) {
return {};
}
@@ -454,6 +458,10 @@ Op_RRGGBBaa_BE_to_RGB_HDR::state_after_conversion(const ColorState& input_state,
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
std::vector<ColorStateWithCost> states;
ColorState output_state;
@@ -671,6 +679,12 @@ Op_RRGGBBaa_swap_endianness::state_after_conversion(const ColorState& input_stat
return {};
}
+ // Swaps the two bytes of each component, which is only meaningful for components
+ // that are stored in 16 bits.
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
std::vector<ColorStateWithCost> states;
ColorState output_state;
diff --git a/libheif/color-conversion/rgb2yuv.cc b/libheif/color-conversion/rgb2yuv.cc
index 9a746504..12678128 100644
--- a/libheif/color-conversion/rgb2yuv.cc
+++ b/libheif/color-conversion/rgb2yuv.cc
@@ -40,6 +40,10 @@ Op_RGB_to_YCbCr<Pixel>::state_after_conversion(const ColorState& input_state,
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
// TODO: add support for <8 bpp
if (input_state.bits_per_pixel < 8) {
return {};
@@ -340,6 +344,10 @@ Op_RRGGBBxx_HDR_to_YCbCr420::state_after_conversion(const ColorState& input_stat
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
int matrix = target_state.nclx.get_matrix_coefficients();
if (matrix == 0 || matrix == 8 || matrix == 11 || matrix == 14) {
return {};
@@ -534,6 +542,12 @@ Op_RGB24_32_to_YCbCr::state_after_conversion(const ColorState& input_state,
return {};
}
+ // The interleaved input is indexed as bytes and the output is hard-coded to 8 bits,
+ // so this operator handles 8-bit input only.
+ if (input_state.bits_per_pixel != 8) {
+ return {};
+ }
+
if (target_state.chroma != heif_chroma_420 &&
target_state.chroma != heif_chroma_422 &&
target_state.chroma != heif_chroma_444) {
@@ -831,6 +845,12 @@ Op_RGB24_32_to_YCbCr444_GBR::state_after_conversion(const ColorState& input_stat
return {};
}
+ // The interleaved input is indexed as bytes and the output is hard-coded to 8 bits,
+ // so this operator handles 8-bit input only.
+ if (input_state.bits_per_pixel != 8) {
+ return {};
+ }
+
if (target_state.nclx.get_matrix_coefficients() != 0) {
return {};
}
diff --git a/libheif/color-conversion/yuv2rgb.cc b/libheif/color-conversion/yuv2rgb.cc
index 9cd8c1cc..625e01e0 100644
--- a/libheif/color-conversion/yuv2rgb.cc
+++ b/libheif/color-conversion/yuv2rgb.cc
@@ -75,7 +75,8 @@ Op_YCbCr_to_RGB<Pixel>::state_after_conversion(const ColorState& input_state,
return {};
}
- if (input_state.bits_per_pixel > 16) {
+ // Also covers the alpha plane, which is copied through at its own bit depth below.
+ if (has_samples_wider_than_16bit(input_state)) {
return {};
}
@@ -597,6 +598,10 @@ Op_YCbCr420_to_RRGGBBaa::state_after_conversion(const ColorState& input_state,
return {};
}
+ if (has_samples_wider_than_16bit(input_state)) {
+ return {};
+ }
+
if (input_state.has_alpha && input_state.get_alpha_bits_per_pixel() != input_state.bits_per_pixel) {
return {};
}
diff --git a/tests/uncompressed_wide_component_colorconv.cc b/tests/uncompressed_wide_component_colorconv.cc
index 7dae5f4f..2a9c1127 100644
--- a/tests/uncompressed_wide_component_colorconv.cc
+++ b/tests/uncompressed_wide_component_colorconv.cc
@@ -36,12 +36,13 @@
// '128 << (bit_depth - 8)', shifting an 'int' by 56: undefined behaviour
// (UndefinedBehaviorSanitizer abort in Op_mono_to_YCbCr420::convert_colorspace).
//
-// The fix rejects a real (non-nop) color conversion of any image with a channel
-// wider than 16 bits, at the single entry point of the pipeline in
-// convert_colorspace(). This test builds such a file, confirms the decoder path
-// itself is unaffected (the untransformed decode still returns the 64-bit
-// monochrome plane), and requires the conversions that used to run the
-// unsupported operators to fail cleanly instead.
+// Each operator now declines a bit depth it cannot access, so an operator that
+// cannot handle a wide sample never offers itself to the pipeline and pipeline
+// construction fails; convert_colorspace() keeps a backstop check behind that.
+// This test builds such a file, confirms the decoder path itself is unaffected
+// (the untransformed decode still returns the 64-bit monochrome plane), and
+// requires the conversions that used to run the unsupported operators to fail
+// cleanly instead.
#include "catch_amalgamated.hpp"
#include "libheif/heif.h"
@@ -215,12 +216,20 @@ TEST_CASE("unci with a 64-bit component refuses color conversion instead of shif
// Converting to YCbCr 4:2:0 selects Op_mono_to_YCbCr420, which computed the
// chroma midpoint as '128 << (bit_depth - 8)'. With a 64-bit component that
// shifted an 'int' by 56. The conversion must now be refused cleanly.
+ //
+ // Op_mono_to_YCbCr420::state_after_conversion() declines a sample wider than 16
+ // bits, so pipeline construction fails before the catch-all in
+ // convert_colorspace() is reached: the subcode is Unsupported_color_conversion.
+ // Should the operators ever stop declining wide samples, the catch-all answers
+ // with Unsupported_bit_depth instead. Either refusal is correct here; what
+ // matters is that the conversion does not run and nothing shifts out of range.
{
heif_image* img = nullptr;
err = heif_decode_image(handle, &img, heif_colorspace_YCbCr, heif_chroma_420, nullptr);
INFO("YCbCr decode error (" << err.code << "/" << err.subcode << "): " << err.message);
REQUIRE(err.code == heif_error_Unsupported_feature);
- REQUIRE(err.subcode == heif_suberror_Unsupported_bit_depth);
+ REQUIRE((err.subcode == heif_suberror_Unsupported_color_conversion ||
+ err.subcode == heif_suberror_Unsupported_bit_depth));
REQUIRE(img == nullptr);
if (img != nullptr) {