Commit dad02dc2d7 for ffmpeg
commit dad02dc2d775b1ef1014bc6c4c53ac25f4e177e2
Author: Ruk Doe <info@doany.io>
Date: Sun Nov 1 15:13:43 2026 +0900
avcodec/vaapi_encode_av1: enable loop filter and CDEF for CQP
In CQP mode the encoder never enabled CDEF in the sequence header and
left loop_filter_level[] at 0, so the output had no deblocking and no
CDEF at all. With bitrate control the driver writes these parameters
into the packed frame header at the bit offsets we pass. In CQP mode
va_enc_av1.h says the offsets "should be set to 0 and ignored by
driver", so the driver does not patch the packed header and the values
are up to the application. The iHD driver filters with the
filter_level and cdef_* picture parameters in that case.
Choose them from base_q_idx:
- loop_filter_level[0..1] = min((base_q_idx - 32)^2 * 7 / 4096, 63)
(0 below 32), and half of that, rounded up, for U and V. The
deblocking thresholds scale with the level, the blocking scales with
the quantizer step, and the step grows faster than linearly with
base_q_idx. GStreamer's vaav1enc also derives the level from the q
index with a table; the quadratic stays within a few levels of its
luma table and gives the same 15/8 luma/chroma levels at base_q_idx
128 (also the fixed values of the libva-utils av1encode sample).
Sharpness stays 0.
- CDEF with cdef_bits = 2, damping 3 + base_q_idx / 64 (as in
GStreamer's vaav1enc) and four
{ primary strength, secondary strength code } candidates scaled by
p = base_q_idx / 32: { p, 1 }, off, { p / 2, 1 } and { 2p, 2 }, i.e.
secondary strength codes 1 and 2 (strengths 1 and 2). The hardware
can choose one per 64x64 block, including turning CDEF off for
detailed areas, as in the libva-utils and GStreamer encoders. Below
base_q_idx 32 all strengths are 0.
The parameters only depend on base_q_idx, so the per frame type offsets
from i_qfactor/b_qfactor carry over to the filters. No options are
added; the values only replace the previous all-off settings. Other
rate control modes are unchanged.
VAConfigAttribEncAV1.support_cdef_channel_strength ("Allows setting
cdef_y_strengths and cdef_uv_strengths in PPS") is not checked: drivers
read it differently and none of them needs it here. iHD reports 0 on
DG2 but still programs the hardware with the strengths from the picture
parameters as long as cdef_uv_strengths equals cdef_y_strengths
(CheckCDEF()), and reports 1 from MTL on. radeonsi reports 1 only on
VCN5, the only generation that takes explicit strengths. Mesa's d3d12
driver reports it when the D3D12 driver supports CDEF at all.
GStreamer's vaav1enc reads the bit but does not use it, and passes
equal luma and chroma strengths. Luma and chroma get the same strengths
here, so DG2 accepts them; gating on the bit would drop CDEF on DG2,
and bitrate control already enables CDEF without checking it.
On radeonsi the firmware still writes the frame header and picks the
loop filter level; CQP now enables CDEF there as well, with the
strengths chosen by the firmware before VCN5. VCN5 uses the strengths
above when cdef_bits > 0; below base_q_idx 32 cdef_bits stays 0 and it
falls back to the firmware defaults. As with bitrate control, a D3D12
driver without CDEF support would now reject the configuration in CQP
mode too (no such driver is known); VA-API has no capability bit for
CDEF itself.
Tested on an Intel Arc A310 (iHD driver) with a 1440x1080 interlaced
broadcast recording, deinterlaced to 29.97 fps, two 60 s segments, CQP
at qindex 70-120 and ICQ at global_quality 24-27. Quality is XPSNR as
(6 * Y + U + V) / 8 of the per-plane values from the xpsnr filter.
Before this change CQP needed 47.8% and 66.2% more bitrate than ICQ for
the same XPSNR; changing b_qfactor did not help. Against the old CQP,
the BD-rate is -33.5% and -42.0% (luma only: -30.5% and -37.4%), which
brings CQP close to ICQ (+2.8% and +0.4%). The VMAF (v0.6.1) BD-rate is
+1.1% and +1.4%; VMAF already rated the unfiltered CQP output above ICQ
(-3.8% and -8.7%), so it does not reflect the blocking the filters
remove. Encoding speed is unchanged (about 165-170 fps).
ICQ used base_q_idx about 82/87/108 for I/P/B frames with luma loop
filter levels about 4/5/10 (4/5/9 with this change) and
cdef_y_strengths values (primary * 4 + secondary code) of 11-16. These
numbers are from that one GPU only.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XNTXZLNBnfhFULMqQoQmmY
diff --git a/libavcodec/vaapi_encode_av1.c b/libavcodec/vaapi_encode_av1.c
index b70f17580f..0745379fbc 100644
--- a/libavcodec/vaapi_encode_av1.c
+++ b/libavcodec/vaapi_encode_av1.c
@@ -446,11 +446,10 @@ static int vaapi_encode_av1_init_sequence_params(AVCodecContext *avctx)
vseq->ip_period = base_ctx->b_per_p + 1;
vseq->seq_fields.bits.enable_order_hint = sh->enable_order_hint;
+ vseq->seq_fields.bits.enable_cdef = sh->enable_cdef = 1;
- if (!(ctx->va_rc_mode & VA_RC_CQP)) {
+ if (!(ctx->va_rc_mode & VA_RC_CQP))
vseq->bits_per_second = ctx->va_bit_rate;
- vseq->seq_fields.bits.enable_cdef = sh->enable_cdef = 1;
- }
ret = vaapi_encode_av1_add_obu(avctx, obu, AV1_OBU_SEQUENCE_HEADER, &priv->sh);
if (ret < 0)
@@ -465,6 +464,59 @@ end:
return ret;
}
+/**
+ * With VA_RC_CQP the driver does not patch the packed frame header, so
+ * choose the in-loop filter parameters here from base_q_idx and pass the
+ * same values in the picture parameters, keeping the two in sync for
+ * drivers which use the application-provided header. Drivers which
+ * generate the frame header themselves may choose their own values. In
+ * other modes the driver writes them into the header at the bit offsets
+ * we pass.
+ */
+static void vaapi_encode_av1_set_cqp_filters(AV1RawFrameHeader *fh,
+ VAEncPictureParameterBufferAV1 *vpic)
+{
+ const int q = fh->base_q_idx;
+ /*
+ * Deblocking thresholds scale with the filter level, and the
+ * blocking to remove scales with the quantizer step size, which
+ * grows faster than linearly with base_q_idx, so use a quadratic
+ * of base_q_idx. It reaches the maximum level 63 at 224.
+ */
+ const int level = q > 32 ? FFMIN((q - 32) * (q - 32) * 7 >> 12, 63) : 0;
+ const int pri = q >> 5;
+ /*
+ * Four { primary strength, secondary strength code } CDEF candidates;
+ * the hardware picks one per 64x64 block. Put the main candidate
+ * first, and "off" keeps fine texture from being smoothed.
+ */
+ const int cdef[4][2] = {
+ { pri, 1 },
+ { 0, 0 },
+ { pri >> 1, 1 },
+ { pri << 1, 2 },
+ };
+
+ fh->loop_filter_level[0] = vpic->filter_level[0] = level;
+ fh->loop_filter_level[1] = vpic->filter_level[1] = level;
+ fh->loop_filter_level[2] = vpic->filter_level_u = (level + 1) >> 1;
+ fh->loop_filter_level[3] = vpic->filter_level_v = (level + 1) >> 1;
+
+ /* low base_q_idx: CDEF gains little, keep strengths at 0 */
+ if (!pri)
+ return;
+
+ fh->cdef_damping_minus_3 = vpic->cdef_damping_minus_3 = q >> 6;
+ fh->cdef_bits = vpic->cdef_bits = 2;
+ /* iHD rejects different luma and chroma CDEF strengths on DG2 */
+ for (int i = 0; i < 4; i++) {
+ fh->cdef_y_pri_strength[i] = fh->cdef_uv_pri_strength[i] = cdef[i][0];
+ fh->cdef_y_sec_strength[i] = fh->cdef_uv_sec_strength[i] = cdef[i][1];
+ vpic->cdef_y_strengths[i] = vpic->cdef_uv_strengths[i] =
+ (cdef[i][0] << 2) | cdef[i][1];
+ }
+}
+
static int vaapi_encode_av1_init_picture_params(AVCodecContext *avctx,
FFHWBaseEncodePicture *pic)
{
@@ -602,6 +654,9 @@ static int vaapi_encode_av1_init_picture_params(AVCodecContext *avctx,
if (fh->frame_type == AV1_FRAME_KEY || fh->error_resilient_mode)
fh->primary_ref_frame = AV1_PRIMARY_REF_NONE;
+ if (ctx->va_rc_mode & VA_RC_CQP)
+ vaapi_encode_av1_set_cqp_filters(fh, vpic);
+
vpic->base_qindex = fh->base_q_idx;
vpic->frame_width_minus_1 = fh->frame_width_minus_1;
vpic->frame_height_minus_1 = fh->frame_height_minus_1;