Commit dfb8cd5b for libheif

commit dfb8cd5bc45b560e0fa04deb3722f98350e8af66
Author: Dirk Farin <dirk.farin@gmail.com>
Date:   Mon Oct 5 12:28:36 2026 +0200

    Decline targets without a planar YCbCr chroma format in Op_RGB_to_YCbCr

    Op_RGB_to_YCbCr copied the chroma format of the target state into the
    YCbCr state that it declares in state_after_conversion(), without
    checking that it is a chroma format of planar YCbCr. For a monochrome
    target it declared a YCbCr state with a Y plane only, and for an
    interleaved RGB target a YCbCr state with R, G and B. convert_colorspace()
    then returned an image with Y, Cb and Cr planes, which contradicts the
    declared state. Both instantiations (8-bit and 16-bit samples) did this.

    No conversion was affected. No operation leads from such a state to a
    target, so these states were dead ends in the search for the pipeline,
    and no pipeline contained such a step.

    The operation now declines a target whose chroma format is not 4:2:0,
    4:2:2 or 4:4:4, like Op_RGB24_32_to_YCbCr does.

    Converting 369 kinds of input images to every combination of colorspace
    and chroma format with five bit depths, five nclx targets and five sets
    of options (1.38 million conversions through convert_colorspace()) gives
    the same results with and without this change.

diff --git a/libheif/color-conversion/rgb2yuv.cc b/libheif/color-conversion/rgb2yuv.cc
index 3bc6f79f..aed3b5fc 100644
--- a/libheif/color-conversion/rgb2yuv.cc
+++ b/libheif/color-conversion/rgb2yuv.cc
@@ -56,6 +56,17 @@ Op_RGB_to_YCbCr<Pixel>::state_after_conversion(const ColorState& input_state,
     return {};
   }

+  // The output is planar YCbCr, and its chroma format is taken from the target state below.
+  // A target with another chroma format (monochrome or an interleaved RGB format) would be
+  // declared as a YCbCr state with that chroma format, which no image can have:
+  // convert_colorspace() then returns an image with Y, Cb and Cr planes that contradicts
+  // the declared state.
+  if (target_state.chroma != heif_chroma_420 &&
+      target_state.chroma != heif_chroma_422 &&
+      target_state.chroma != heif_chroma_444) {
+    return {};
+  }
+
   int matrix = target_state.nclx.get_matrix_coefficients();
   if (matrix == 11 || matrix == 14) {
     return {};