Commit 611bcc793f for ffmpeg

commit 611bcc793fd8fce743fc6b3f784c0b9b8cf0d961
Author: Martin Mirchev <mmirchev@comp.nus.edu.sg>
Date:   Fri Nov 6 11:40:23 2026 +0800

    avcodec/vc2enc: reject wavelet types the encoder does not implement

    The wavelet_type option is declared with the numeric range [0,
    VC2_TRANSFORMS_NB], but ff_vc2enc_init_transforms() only installs four of
    the transform function pointers (9_7, 5_3, Haar and Haar-with-shift). The
    other in-range values leave the corresponding vc2_subband_dwt[] entry NULL
    (13_7, Fidelity, Daubechies 9_7), and VC2_TRANSFORMS_NB itself is one past
    the end of the array.

    dwt_plane() then calls t->vc2_subband_dwt[s->wavelet_idx](), so e.g.
    "-c:v vc2 -wavelet_type 2" calls through a NULL pointer and
    "-wavelet_type 7" reads a function pointer out of bounds and calls it,
    crashing the encoder.

    Reject an unsupported wavelet type at init with AVERROR(EINVAL) instead.

    Found with an LLM-assisted static analysis tool and verified manually.

    Signed-off-by: Martin Mirchev <mmirchev@comp.nus.edu.sg>

diff --git a/libavcodec/vc2enc.c b/libavcodec/vc2enc.c
index 2c155c01a1..8cb4a05675 100644
--- a/libavcodec/vc2enc.c
+++ b/libavcodec/vc2enc.c
@@ -1093,6 +1093,14 @@ static av_cold int vc2_encode_init(AVCodecContext *avctx)
         return AVERROR(EINVAL);
     }

+    if (s->wavelet_idx != VC2_TRANSFORM_9_7  &&
+        s->wavelet_idx != VC2_TRANSFORM_5_3  &&
+        s->wavelet_idx != VC2_TRANSFORM_HAAR &&
+        s->wavelet_idx != VC2_TRANSFORM_HAAR_S) {
+        av_log(avctx, AV_LOG_ERROR, "Unsupported wavelet type %d\n", s->wavelet_idx);
+        return AVERROR(EINVAL);
+    }
+
     if (s->base_vf <= 0) {
         if (avctx->strict_std_compliance < FF_COMPLIANCE_STRICT) {
             s->strict_compliance = s->base_vf = 0;