Commit 418fe97b5 for llama.cpp

commit 418fe97b5f01b11eeb89f30d8a0459433edbf1d2
Author: cwriter <silvan.niederer@bluewin.ch>
Date:   Sat Nov 21 04:37:40 2026 +0100

    sycl: Improve Grouped XMX GEMM by avoiding SLM staging (#31428)

diff --git a/docs/backend/SYCL.md b/docs/backend/SYCL.md
index ca0b6d7bc..4a1d16aad 100644
--- a/docs/backend/SYCL.md
+++ b/docs/backend/SYCL.md
@@ -827,6 +827,7 @@ User can use the device management in [docs/multi-gpu.md](https://github.com/ggm
 | GGML_SYCL_XMX_MAX_COLS | non-negative integer, 80 (default) | Most activation columns (batch size) the `GGML_SYCL_ENABLE_XMX_ESIMD` kernels take; wider batches use the other mat-mul kernels. A larger value keeps XMX for bigger batches (as repeated launches), `0` never uses it. |
 | GGML_SYCL_XMX_GLU_MAX_COLS | 0 to 16, 16 (default) | Most columns the fused gate and up XMX kernel takes; wider batches run gate and up as separate mat-vecs. 16 is the most the kernel is built for, larger values are clamped to 16 and `0` turns the fused kernel off. |
 | GGML_SYCL_XMX_GATHER_TYPES | decimal bitmask, all bits set (default) | Weight formats that may use the XMX dequant-GEMM paths, which dequantize weights straight into the XMX tiles. This speeds up prompt processing of MoE models on GPUs with XMX units (Arc A- and B-series, Arc Pro, Data Center GPU Max), for example pp512 of Qwen3-30B-A3B UD-IQ3_XXS by about 50% on an Arc Pro B60. Bits:<br>* 1: IQ4_NL, 2: IQ3_S, 4: IQ4_XS, 8: IQ3_XXS, 16: IQ2_XXS, 32: IQ2_XS, 64: IQ2_S, 128: IQ1_S, 256: IQ1_M<br>* 512: Q8_0, 1024: Q4_K, 2048: Q5_K, 4096: Q6_K, 8192: Q2_K, 16384: Q3_K (MoE `MUL_MAT_ID` only)<br>Add values to combine them, for example `3` for IQ4_NL and IQ3_S; `0` disables the paths. A set bit does not force the path: batches of more than 64 tokens per expert use the library GEMM unless the device schedule width heuristic is overridden with `GGML_SYCL_MMID_SCHED=3`. Row lengths must be a multiple of 256 (32 for IQ4_NL and Q8_0). |
+| GGML_SYCL_MMID_ESIMD | 0, 1 (default), 2 | The `MUL_MAT_ID` grouped XMX dequant-GEMM as an ESIMD kernel that dequantizes the weights straight into the `dpas` register operands, with no shared local memory image of them, from either weight layout (the per-expert reorder or the canonical blocks). q8_0, iq4_nl, iq4_xs, iq3_s, iq3_xxs and q2_K to q6_K. It stands in for the 8x16x16 sub-group-16 combination of f16 or bf16 (Xe2, Xe-HPC) and for the 8x8x16 sub-group-8 one (Xe-HPG: Arc A-series, ARL-H; there it works on 16 rows a thread and reads with 1D block loads, since that architecture has half the register file and no 2D block loads); while it is on, a grouped call whose weight type it covers is moved to that combination from any other shape of its type and sub-group size, whatever `GGML_SYCL_XMX_GATHER_SHAPES` allows, because the shape bits choose among `joint_matrix` tiles and the ESIMD kernel tiles on its own. A call it declines runs that combination's `joint_matrix` kernel. 0 uses the `joint_matrix` kernel, 1 the ESIMD one, 2 the ESIMD one summing K in the `joint_matrix` kernel's order (bit-identical output for f16; slower). An AOT build gets its 256-register file through `-ze-opt-enable-auto-large-GRF-mode` (set by the CMake build). Not built with `GGML_SYCL_SUPPORT_XMX=OFF`. |
 | GGML_SYCL_MMID_SCHED | decimal bitmask, 1 (default) | Build the grouped MoE routing and tile schedule on the GPU, avoiding the expert-ID host readback and CPU sort.<br>* 1: enable the device schedule.<br>* 2: override the fallback for models with few routed experts by disabling the performance heuristic that rejects batches averaging more than 64 routed rows per expert. This may reduce performance; type, shape, precision and device capability checks still apply.<br>Use `3` to enable the device schedule and override this heuristic, or `0` to use the host schedule. Bit 2 alone does not enable the device schedule. Higher bits are ignored. |
 | GGML_SYCL_XMX_GATHER_SHAPES | decimal bitmask, 255 (default) | XMX `joint_matrix` combinations the paths of `GGML_SYCL_XMX_GATHER_TYPES` may use; the operand type comes from `GGML_SYCL_DYNAMIC_PRECISION` and the best supported combination is picked automatically (logged as `fg_pick_combo`). Bits:<br>* Xe2, Xe3, Xe-HPC: 1: f16 8x16x16, 2: f16 16x16x16, 4: f16 32x64x16, 8: f16 32x64x32, 32: tf32 8x16x8, 64: bf16 8x16x16<br>* Xe-HPG (Arc A770, ARL-H): 16: f16 8x8x16, 128: bf16 8x8x16<br>Clear a bit to exclude a combination, or set a single bit to force one for testing. |
 | GGML_SYCL_DYNAMIC_PRECISION | `F16` (default with `GGML_SYCL_F16=ON`), `BF16`, `TF32` or `F32` (default otherwise) | Operand type of the XMX dequant-GEMM paths (`GGML_SYCL_XMX_GATHER_TYPES`); accumulation is always f32. `F16` is the fastest, but activations above 65504 overflow. `BF16` keeps the f32 range at a 7-bit mantissa, `TF32` keeps the range and the f16 mantissa but is about 30% slower and needs Xe2, Xe3 or Xe-HPC, and `F32` turns the XMX paths off. Ops that request a higher src1 precision ([TAG_GGML_PREC]) get it regardless of this setting. |
diff --git a/ggml/src/ggml-sycl/CMakeLists.txt b/ggml/src/ggml-sycl/CMakeLists.txt
index 8312a8305..0a5aede0b 100644
--- a/ggml/src/ggml-sycl/CMakeLists.txt
+++ b/ggml/src/ggml-sycl/CMakeLists.txt
@@ -218,6 +218,8 @@ endif()

 if (GGML_SYCL_TARGET STREQUAL "INTEL")
     add_compile_definitions(GGML_SYCL_WARP_SIZE=16)
+    # With AOT the option goes to ocloc inside -options (see GGML_SYCL_DEVICE_ARCH below): a bare
+    # -Xs hands it to ocloc as a command-line flag, which fails the device link.
     if (NOT GGML_SYCL_DEVICE_ARCH)
         target_link_options(ggml-sycl PRIVATE -Xs -ze-intel-greater-than-4GB-buffer-required)
     else()
@@ -248,10 +250,18 @@ endif()

 if (GGML_SYCL_DEVICE_ARCH)
     message(STATUS "GGML_SYCL_DEVICE_ARCH=${GGML_SYCL_DEVICE_ARCH} (AOT via spir64_gen)")
+    # AOT (oneAPI 2026.2) ignores the grf_size<256> kernel property on ESIMD kernels and builds them
+    # with 128 registers, which made the ESIMD MoE GEMM 1.7-3.9x slower than under JIT; SPMD kernels
+    # keep honouring the property. A module-wide -doubleGRF also moves the ESIMD kernels that ask for
+    # 128 to 256, and a server built with it hung on its first request, so IGC picks the size per
+    # kernel from its register pressure instead (ESIMD kernels too; -ze-opt-large-grf-kernel does not
+    # reach them).
+    set(GGML_SYCL_AOT_BACKEND_OPTIONS
+        "-device ${GGML_SYCL_DEVICE_ARCH} -options '-ze-intel-greater-than-4GB-buffer-required -ze-opt-enable-auto-large-GRF-mode'")
     target_compile_options(
         ggml-sycl PRIVATE
         -fsycl-targets=spir64_gen
-        "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH} -options -ze-intel-greater-than-4GB-buffer-required\""
+        "SHELL:-Xsycl-target-backend=spir64_gen \"${GGML_SYCL_AOT_BACKEND_OPTIONS}\""
     )

     # Pass through parallel job (process) count for parallelising the
@@ -266,7 +276,7 @@ if (GGML_SYCL_DEVICE_ARCH)
     target_link_options(
         ggml-sycl PRIVATE
         -fsycl-targets=spir64_gen
-        "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH} -options -ze-intel-greater-than-4GB-buffer-required\""
+        "SHELL:-Xsycl-target-backend=spir64_gen \"${GGML_SYCL_AOT_BACKEND_OPTIONS}\""
         -fsycl-max-parallel-link-jobs=${GGML_SYCL_MAX_PARALLEL_LINK_JOBS}
     )

diff --git a/ggml/src/ggml-sycl/common.hpp b/ggml/src/ggml-sycl/common.hpp
index 75e783d12..ae156acca 100644
--- a/ggml/src/ggml-sycl/common.hpp
+++ b/ggml/src/ggml-sycl/common.hpp
@@ -72,6 +72,7 @@ extern int g_ggml_sycl_debug;
 extern int g_ggml_sycl_enable_optimize;
 extern int g_ggml_sycl_enable_fusion;
 extern int g_ggml_sycl_enable_esimd;
+extern int g_ggml_sycl_mmid_esimd;
 extern int g_ggml_sycl_mmvq_wide;
 extern int g_ggml_sycl_enable_xmx_esimd;
 extern int g_ggml_sycl_xmx_max_cols;
diff --git a/ggml/src/ggml-sycl/fused-gemm-esimd.cpp b/ggml/src/ggml-sycl/fused-gemm-esimd.cpp
new file mode 100644
index 000000000..37f87e377
--- /dev/null
+++ b/ggml/src/ggml-sycl/fused-gemm-esimd.cpp
@@ -0,0 +1,1160 @@
+#include "fused-gemm-esimd.hpp"
+
+#include <sycl/ext/intel/esimd.hpp>
+#include <sycl/ext/intel/esimd/xmx/dpas.hpp>
+#include <sycl/ext/intel/experimental/grf_size_properties.hpp>
+
+// One thread owns MR rows of one expert's weight and one BN-wide tile of the schedule, over the
+// whole K. Per k step (32 values) it loads the rows' stored bits, dequantizes them in registers to
+// the f16 A operand of dpas (RC = 8 rows x 16 k, row major) and multiplies it with the packed B,
+// which is already the dpas VNNI operand: [K/2][Npad][2] halves, NW columns a dpas.
+//
+// The joint_matrix kernel dequantizes into SLM and reads it back in the same sub-group, which is
+// staging, not sharing; dpas takes registers, so that round trip and its barriers are gone here.
+// Its K split over GGML_SYCL_GROUPED_GEMM_KSPLIT sub-groups is reproduced as sequential K ranges summed in
+// the same ascending order as its tile_c reduction, and every format below rounds A exactly as its
+// joint_matrix A stage does, so the output is the same bits.
+//
+// A weight format is a grouped_gemm_esimd_a<block, MR> with: raw, the stored bits of the MR rows for one k
+// step; load(), which fetches them from the expert's slice in either layout (grouped_gemm_esimd::slice); and
+// decode(), the f16 A of one row group.
+// It reads the same fields as the joint_matrix A stage of its format.
+//
+// The kernel is built for one grouped_gemm_arch: the dpas width NW (the columns of B one dpas takes, the
+// sub-group size of the joint_matrix combination it stands in for), the rows of one thread, and
+// whether 2D block loads exist. Xe2 and Xe-HPC: 16 wide, 32 rows, 2D loads. Xe-HPG (Arc A, ARL-H):
+// 8 wide, no 2D block messages, and 256 registers of 32 bytes, half of Xe2's file, so 16 rows: its
+// 32-row accumulators alone would take half the registers.
+static constexpr int GGML_SYCL_GROUPED_GEMM_KSPLIT = FG_KSPLIT;
+static constexpr int GGML_SYCL_GROUPED_GEMM_BN     = GGML_SYCL_FG_BN;
+static constexpr int GROUPED_GEMM_MR               = 32;  // rows of one thread on Xe2: one 2D load covers them
+static constexpr int GROUPED_GEMM_RC               = 8;   // rows of one dpas
+static constexpr int GROUPED_GEMM_QK               = 32;  // values of one k step
+
+template <int NW_, int MR_, bool L2D_> struct grouped_gemm_arch {
+    static constexpr int  NW  = NW_;
+    static constexpr int  MR  = MR_;
+    static constexpr bool L2D = L2D_;
+};
+
+using grouped_gemm_xe2   = grouped_gemm_arch<16, GROUPED_GEMM_MR, true>;
+using grouped_gemm_xehpg = grouped_gemm_arch<8, 16, false>;
+
+// An AOT build compiles each kernel only for the targets of its dpas width: GGML_SYCL_XMX_AOT_SG for
+// spir64_gen, FG_AOT_SG (fused-gemm.hpp) for intel_gpu_* targets. A build without XMX (GGML_SYCL_SUPPORT_XMX
+// off) compiles none of them
+template <typename A> static constexpr bool grouped_gemm_built() {
+#if !defined(GGML_SYCL_SUPPORT_XMX)
+    return false;
+#elif defined(GGML_SYCL_XMX_AOT_SG)
+    return A::NW == GGML_SYCL_XMX_AOT_SG;
+#elif defined(FG_AOT_SG)
+    return A::NW == FG_AOT_SG;
+#else
+    return true;
+#endif
+}
+
+namespace grouped_gemm_esimd {
+using namespace sycl::ext::intel::esimd;
+
+// f(std::integral_constant<int, I>) for I in [0, N): register selects at a runtime offset become
+// indirect accesses, so loops whose index picks registers are spelled out at compile time
+template <int I, int N> struct static_for {
+    template <typename Fn> static ESIMD_INLINE void run(Fn & f) {
+        if constexpr (I < N) {
+            f(std::integral_constant<int, I>{});
+            static_for<I + 1, N>::run(f);
+        }
+    }
+};
+
+// One field of a format's block. In the reorder (SoA) layout it is a stream of its own, `size` bytes
+// a block, after `pre` bytes a block of the streams before it; in the canonical (AoS) layout it is
+// `aos` bytes into the block.
+struct field {
+    int pre;
+    int size;
+    int aos;
+};
+
+// Where one thread's rows sit in an expert's slice, in either layout. nblocks counts the stored blocks
+// of the slice, bpr those of one row, bs is the size of one block.
+template <int MR, bool L2D> struct slice {
+    const uint8_t * x;
+    int             M;
+    int             bpr;
+    int             nblocks;
+    int             m0;
+    bool            aos;
+    int             bs;
+
+    // per row, the byte offset from x of byte `inner` of field f of block ib
+    ESIMD_INLINE simd<uint32_t, MR> at(const field & f, const int ib, const int inner) const {
+        const int             stride = aos ? bs : f.size;
+        simd<uint32_t, MR> o(0, (uint32_t) (bpr * stride));
+        return o + (uint32_t) ((aos ? f.aos : nblocks * f.pre) + (m0 * bpr + ib) * stride + inner);
+    }
+
+    // a field's bytes at offset `inner` start on a dword in every block (the AoS block and offset
+    // are multiples of 4); the SoA streams always do
+    ESIMD_INLINE bool dword_aligned(const field & f, const int inner) const {
+        return !aos || (bs % 4 == 0 && (f.aos + inner) % 4 == 0);
+    }
+
+    ESIMD_INLINE simd<uint16_t, MR> u16(const field & f, const int ib, const int inner) const {
+        return gather<uint16_t, MR>((const uint16_t *) x, at(f, ib, inner));
+    }
+
+    ESIMD_INLINE simd<uint32_t, MR> u32(const field & f, const int ib, const int inner) const {
+        if (dword_aligned(f, inner)) {
+            return gather<uint32_t, MR>((const uint32_t *) x, at(f, ib, inner));
+        }
+        return rows<1>(f, ib, inner);
+    }
+
+    // W dwords of each of the MR rows from byte `inner` (a multiple of 4) of field f of block ib, as
+    // [row][W]. SoA with 2D block loads (L2D): one message over the stream. Otherwise the rows' dwords
+    // are gathered, as [W][row], at offsets rounded down to a dword (every field starts on an even
+    // byte; the SoA streams are dword aligned), shifted by 16 bits where an AoS row's block starts on
+    // 2 mod 4, and transposed with strided reads.
+    template <int W> ESIMD_INLINE simd<uint32_t, MR * W> rows(const field & f, const int ib, const int inner) const {
+        if constexpr (L2D) {
+            if (!aos) {
+                const unsigned pitch = (unsigned) (bpr * f.size);
+                return load_2d<uint32_t, W, MR>((const uint32_t *) (x + (size_t) nblocks * f.pre), pitch - 1, M - 1,
+                                                pitch - 1, (ib * f.size + inner) / 4, m0);
+            }
+        }
+        const simd<uint32_t, MR> o = at(f, ib, inner);
+        simd<uint32_t, MR * W>   g;
+        if (dword_aligned(f, inner)) {
+            g = gather<uint32_t, MR * W, W>((const uint32_t *) x, o);
+        } else {
+            // per row: sh is 16 where the run starts on 2 mod 4, else 0; hm keeps the next dword's low
+            // half only where it is needed
+            const simd<uint32_t, MR> oa = o & ~3u;
+            const simd<uint32_t, MR> sh = (o & 2u) << 3;
+            const simd<uint32_t, MR> hm = 0u - (sh >> 4);
+            simd<uint32_t, MR * W>   lo = gather<uint32_t, MR * W, W>((const uint32_t *) x, oa);
+            // the dword after the run holds its last 2 bytes where a row needs it; a row that does not
+            // reads its first dword again rather than past the field
+            const simd<uint32_t, MR> tail = gather<uint32_t, MR>((const uint32_t *) x, oa + ((4u * W) & hm));
+            auto shift = [&](auto vc) SYCL_ESIMD_FUNCTION {
+                constexpr int         v = decltype(vc)::value;
+                simd<uint32_t, MR> a = lo.template select<MR, 1>(v * MR);
+                simd<uint32_t, MR> b;
+                if constexpr (v + 1 < W) {
+                    b = lo.template select<MR, 1>((v + 1) * MR);
+                } else {
+                    b = tail;
+                }
+                g.template select<MR, 1>(v * MR) = (a >> sh) | ((b << 16) & hm);
+            };
+            static_for<0, W>::run(shift);
+        }
+        if constexpr (W == 1) {
+            return g;
+        } else {
+            simd<uint32_t, MR * W> t;
+            auto row = [&](auto rc) SYCL_ESIMD_FUNCTION {
+                constexpr int r = decltype(rc)::value;
+                t.template select<W, 1>(r * W) = g.template select<W, MR>(r);
+            };
+            static_for<0, MR>::run(row);
+            return t;
+        }
+    }
+};
+
+// A decode returns its row group's A in the order the two dpas of a k step read it, [k half][row][16
+// k]: both iq formats store k and k + 16 in the same byte or grid dword half, so that order needs
+// only reads with a stride, while [row][32 k] needed strided writes, which split into SIMD4.
+using a_halves = simd<sycl::half, GROUPED_GEMM_RC * GROUPED_GEMM_QK>;
+
+// f16 of the f32 product code * scale, each row's scale over its 16 values of a k half: the
+// expression the joint_matrix stages evaluate before E::make rounds it
+ESIMD_INLINE simd<sycl::half, GROUPED_GEMM_RC * 16> scale_half(simd<float, GROUPED_GEMM_RC * 16> v,
+                                                               simd<float, GROUPED_GEMM_RC>      d) {
+    v *= d.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+    return v;
+}
+
+// The iq4 codebook by byte, in SLM: entry b holds the values of its low and its high nibble as the
+// two 16-bit halves of one dword, as T (half, or int16 for an f32 product). A read-only table every
+// lane indexes, which is what SLM is for: register-indirect iselect() over the same lanes cost
+// 7.3 of 12.7 ms, a 16-entry SLM table per nibble about 0.4.
+template <typename T> ESIMD_INLINE void iq4_pair_table_to_slm() {
+    static_assert(sizeof(T) == 2, "two values a dword");
+    simd<uint32_t, 16> lo;
+#pragma unroll
+    for (int i = 0; i < 16; ++i) {
+        lo[i] = sycl::bit_cast<uint16_t>((T) kvalues_iq4nl[i]);
+    }
+#pragma unroll
+    for (int hi = 0; hi < 16; ++hi) {
+        slm_block_store<uint32_t, 16>(hi * 64, lo | ((uint32_t) sycl::bit_cast<uint16_t>((T) kvalues_iq4nl[hi]) << 16));
+    }
+}
+
+// 16 packed bytes a row; byte j holds k = j in its low and k = 16 + j in its high nibble (the order
+// of fg_decode_iq4_xs and of the iq4_nl stage). One gather per byte; the low halves are k half 0,
+// the high halves k half 1, each [row][16].
+template <typename T>
+ESIMD_INLINE simd<T, GROUPED_GEMM_RC * GROUPED_GEMM_QK> iq4_codes(simd<uint32_t, GROUPED_GEMM_RC * 4> q) {
+    simd<uint32_t, GROUPED_GEMM_RC * 16>       off = q.template bit_cast_view<uint8_t>();
+    simd<uint32_t, GROUPED_GEMM_RC * 16>       pr  = slm_gather<uint32_t, GROUPED_GEMM_RC * 16>(off * 4);
+    auto                       pv  = pr.template bit_cast_view<T>();
+    simd<T, GROUPED_GEMM_RC * GROUPED_GEMM_QK> v;
+    v.template select<GROUPED_GEMM_RC * 16, 1>(0)                    = pv.template select<GROUPED_GEMM_RC * 16, 2>(0);
+    v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) = pv.template select<GROUPED_GEMM_RC * 16, 2>(1);
+    return v;
+}
+// A 2D surface: rows `pitch` bytes apart, 16-byte aligned and at least 64 bytes
+constexpr bool surface_ok(const int pitch) {
+    return pitch % 16 == 0 && pitch >= 64;
+}
+
+// byte i of each lane's dword, i a runtime index: a shift by a scalar, not a register pick
+template <int MR> ESIMD_INLINE simd<uint32_t, MR> byte_of(simd<uint32_t, MR> w, const int i) {
+    return (w >> (8 * i)) & 0xffu;
+}
+
+// one of four per-lane dwords by a runtime index, as uniform branches rather than a register pick
+template <int MR> ESIMD_INLINE simd<uint32_t, MR> dword_of(simd<uint32_t, MR> * w, const int i) {
+    simd<uint32_t, MR> r = w[0];
+    if (i == 1) {
+        r = w[1];
+    } else if (i == 2) {
+        r = w[2];
+    } else if (i == 3) {
+        r = w[3];
+    }
+    return r;
+}
+
+// a row group's 32 k-ordered code bytes a row as f32, in [k half][row][16 k]
+ESIMD_INLINE simd<float, GROUPED_GEMM_RC * GROUPED_GEMM_QK> byte_codes(simd<uint32_t, GROUPED_GEMM_RC * 8> q) {
+    auto bm = q.template bit_cast_view<uint8_t, GROUPED_GEMM_RC, GROUPED_GEMM_QK>();
+    simd<float, GROUPED_GEMM_RC * GROUPED_GEMM_QK> v;
+    v.template select<GROUPED_GEMM_RC * 16, 1>(0) = bm.template select<GROUPED_GEMM_RC, 1, 16, 1>(0, 0);
+    v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) =
+        bm.template select<GROUPED_GEMM_RC, 1, 16, 1>(0, 16);
+    return v;
+}
+
+// f16 of d * code - m in f32, each row's d and m over its 16 values of a k half: the expression of
+// the K-quant stages with a min
+ESIMD_INLINE simd<sycl::half, GROUPED_GEMM_RC * 16> scale_min_half(simd<float, GROUPED_GEMM_RC * 16> v,
+                                                                   simd<float, GROUPED_GEMM_RC>      d,
+                                                                   simd<float, GROUPED_GEMM_RC>      m) {
+    v = v * d.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0) -
+        m.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+    return v;
+}
+
+// the two k halves of a row group, the code bytes times one (d, m) per row and half
+ESIMD_INLINE a_halves halves_dm(simd<uint32_t, GROUPED_GEMM_RC * 8> q,
+                                simd<float, GROUPED_GEMM_RC>        d0,
+                                simd<float, GROUPED_GEMM_RC>        m0,
+                                simd<float, GROUPED_GEMM_RC>        d1,
+                                simd<float, GROUPED_GEMM_RC>        m1,
+                                const float                         bias = 0.0f) {
+    simd<float, GROUPED_GEMM_RC * GROUPED_GEMM_QK> c = byte_codes(q);
+    if (bias != 0.0f) {
+        c -= bias;
+    }
+    a_halves v;
+    v.template select<GROUPED_GEMM_RC * 16, 1>(0) =
+        scale_min_half(c.template select<GROUPED_GEMM_RC * 16, 1>(0), d0, m0);
+    v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) =
+        scale_min_half(c.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16), d1, m1);
+    return v;
+}
+
+// a row group's 32 values from grid dwords (byte b of entry e is k = 4e + b) and a sign dword per
+// row (bit k negates k), times d: the expression of the iq3 stages
+ESIMD_INLINE a_halves grid_signed(simd<uint32_t, GROUPED_GEMM_RC * 8> ge,
+                                  simd<uint32_t, GROUPED_GEMM_RC>     signs,
+                                  simd<float, GROUPED_GEMM_RC>        d) {
+    auto                                 gm = ge.template bit_cast_view<uint32_t, GROUPED_GEMM_RC, 8>();
+    simd<uint32_t, GROUPED_GEMM_RC * 16> sg = signs.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+    simd<uint32_t, GROUPED_GEMM_RC * 16> k(0, 1);
+    k &= 15;
+    a_halves v;
+#pragma unroll
+    for (int h = 0; h < 2; ++h) {
+        simd<uint32_t, GROUPED_GEMM_RC * 4> eh = gm.template select<GROUPED_GEMM_RC, 1, 4, 1>(0, 4 * h);
+        simd<float, GROUPED_GEMM_RC * 16>   f  = eh.template bit_cast_view<uint8_t>();
+        f.merge(-f, ((sg >> (k + 16 * h)) & 1) != 0);
+        v.template select<GROUPED_GEMM_RC * 16, 1>(h * GROUPED_GEMM_RC * 16) = scale_half(f, d);
+    }
+    return v;
+}
+}  // namespace grouped_gemm_esimd
+
+template <typename block_q_t, int MR = GROUPED_GEMM_MR> struct grouped_gemm_esimd_a {
+    static constexpr bool supported = false;
+};
+
+// [qs 32][d], one block a k step
+template <int MR> struct grouped_gemm_esimd_a<block_q8_0, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK8_0;
+    static constexpr bool whole     = true;  // decode_all() for the whole tile
+    static constexpr int  slm_bytes = 0;
+    static ESIMD_INLINE void init_slm() {}
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+        grouped_gemm_esimd::simd<uint16_t, MR>     d;
+    };
+
+    static constexpr grouped_gemm_esimd::field QS = { 0, QK8_0, 2 }, D = { QK8_0, 2, 0 };
+    static bool ok(const int K, const bool aos) { return K % qk == 0 && (aos || K >= 64); }
+    struct sb_t {};
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t &, const Sl &, const int) {}
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t &, const Sl & s, const int kb, const int) {
+        r.q = s.template rows<8>(QS, kb, 0);
+        r.d = s.u16(D, kb, 0);
+    }
+    // all MR x 32 quants at once: int8 -> half, times the row's scale. q * d needs at most 19
+    // significant bits, so the f32 product the joint_matrix stage rounds to half is exact and one
+    // correctly rounded half multiply gives the same bits.
+    static ESIMD_INLINE grouped_gemm_esimd::simd<sycl::half, MR * GROUPED_GEMM_QK> decode_all(raw & r) {
+        using namespace grouped_gemm_esimd;
+        simd<sycl::half, MR * GROUPED_GEMM_QK> av = r.q.template bit_cast_view<int8_t>();
+        simd<sycl::half, MR>         dh = r.d.template bit_cast_view<sycl::half>();
+        av *= dh.template replicate_vs_w_hs<MR, 1, GROUPED_GEMM_QK, 0>(0);
+        return av;
+    }
+};
+
+// [qs 16][d], one block a k step
+template <int MR> struct grouped_gemm_esimd_a<block_iq4_nl, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK4_NL;
+    static constexpr bool whole     = false;
+    static constexpr int  slm_bytes = 256 * sizeof(uint32_t);
+
+    static ESIMD_INLINE void init_slm() { grouped_gemm_esimd::iq4_pair_table_to_slm<sycl::half>(); }
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 4> q;
+        grouped_gemm_esimd::simd<sycl::half, MR>   d;
+    };
+
+    static constexpr grouped_gemm_esimd::field QS = { 0, QK4_NL / 2, 2 }, D = { QK4_NL / 2, 2, 0 };
+    static bool ok(const int K, const bool aos) { return K % qk == 0 && (aos || K / 2 >= 64); }
+    struct sb_t {};
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t &, const Sl &, const int) {}
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t &, const Sl & s, const int kb, const int) {
+        using namespace grouped_gemm_esimd;
+        r.q                     = s.template rows<4>(QS, kb, 0);
+        simd<uint16_t, MR> d = s.u16(D, kb, 0);
+        r.d                     = d.template bit_cast_view<sycl::half>();
+    }
+    // the codebook values are integers below 128, exact in half, and d * value has at most 18
+    // significant bits: one correctly rounded half multiply is the stage's f32 product rounded
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        a_halves v = iq4_codes<sycl::half>(r.q.template select<GROUPED_GEMM_RC * 4, 1>(g * GROUPED_GEMM_RC * 4));
+        simd<sycl::half, GROUPED_GEMM_RC>      dg = r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+        simd<sycl::half, GROUPED_GEMM_RC * 16> dr = dg.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+        v.template select<GROUPED_GEMM_RC * 16, 1>(0) *= dr;
+        v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) *= dr;
+        return v;
+    }
+};
+
+// [qs 128][d 2][scales_h 2][scales_l 4] per 256 values; k step ib8 of a superblock is sub-block ib8
+template <int MR> struct grouped_gemm_esimd_a<block_iq4_xs, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    // int16 table and f32 products: d * (scale - 32) * value can need 24 significant bits, more
+    // than a half multiply keeps
+    static constexpr int  slm_bytes = 256 * sizeof(uint32_t);
+
+    static ESIMD_INLINE void init_slm() { grouped_gemm_esimd::iq4_pair_table_to_slm<int16_t>(); }
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 4> q;
+        grouped_gemm_esimd::simd<float, MR>        d;
+    };
+    struct sb_t {
+        grouped_gemm_esimd::simd<float, MR>    d;
+        grouped_gemm_esimd::simd<uint32_t, MR> sh;  // scales_h
+        grouped_gemm_esimd::simd<uint32_t, MR> sl;  // scales_l, 4 bytes
+    };
+
+    static constexpr grouped_gemm_esimd::field      QS = { 0, QK_K / 2, 8 }, D = { QK_K / 2, 2, 0 },
+                                                    SH = { QK_K / 2 + 2, 2, 2 }, SL = { QK_K / 2 + 4, 4, 4 };
+    static bool ok(const int K, const bool) { return K % qk == 0; }
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+        using namespace grouped_gemm_esimd;
+        simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+        sb.d  = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+        sb.sh = s.u16(SH, ib, 0);
+        sb.sl = s.u32(SL, ib, 0);
+    }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        r.q = s.template rows<4>(QS, kb / 8, 16 * j);
+        // d * (6-bit scale - 32), in f32 as the stage computes it
+        simd<int32_t, MR> sc = simd<int32_t, MR>(((sb.sl >> (4 * j)) & 0xf) | (((sb.sh >> (2 * j)) & 3) << 4)) - 32;
+        r.d = sb.d * simd<float, MR>(sc);
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        simd<int16_t, GROUPED_GEMM_RC * GROUPED_GEMM_QK> c =
+            iq4_codes<int16_t>(r.q.template select<GROUPED_GEMM_RC * 4, 1>(g * GROUPED_GEMM_RC * 4));
+        simd<float, GROUPED_GEMM_RC> dg = r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+        a_halves                     v;
+        v.template select<GROUPED_GEMM_RC * 16, 1>(0) = scale_half(c.template select<GROUPED_GEMM_RC * 16, 1>(0), dg);
+        v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) =
+            scale_half(c.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16), dg);
+        return v;
+    }
+};
+
+// [qs 64][qh 8][signs 32][d 2, scales 4] per 256 values. Value k of sub-block ib8 is byte k % 4 of
+// grid entry k / 4, the entry index being qs[k / 4] with bit 8 from qh bit k / 4, negated where bit
+// k of the sub-block's sign dword is set; times d * (1 + 2 * 4-bit scale)
+template <int MR> struct grouped_gemm_esimd_a<block_iq3_s, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    // the 512-entry grid in SLM, read by every lane: the global gather of it cost 2.4x the kernel
+    static constexpr int  slm_bytes = 512 * sizeof(uint32_t);
+    static ESIMD_INLINE void init_slm() {
+        using namespace grouped_gemm_esimd;
+#pragma unroll
+        for (int i = 0; i < 512; i += 64) {
+            slm_block_store<uint32_t, 64>(i * 4, block_load<uint32_t, 64>(iq3s_grid + i));
+        }
+    }
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 2> q;      // [row][8 qs bytes]
+        grouped_gemm_esimd::simd<uint32_t, MR>     qh;     // the sub-block's qh byte
+        grouped_gemm_esimd::simd<uint32_t, MR>     signs;  // the sub-block's 32 sign bits
+        grouped_gemm_esimd::simd<float, MR>        d;
+    };
+    // d and the scale bytes are a block's, kept over its k steps; the sign dword and the qh byte of
+    // a k step are a 2D load each at that step's column, so no register is picked by a runtime index
+    struct sb_t {
+        grouped_gemm_esimd::simd<float, MR>    d;
+        grouped_gemm_esimd::simd<uint32_t, MR> sc;  // the 4 scale bytes
+    };
+    // the SoA stream {d, scales} is d and scales of the block in AoS, which are not adjacent there: MS
+    // reads the scales at their SoA offset 2 in the stream, which is AoS offset 2 past 104
+    static constexpr int             META = QK_K / 4 + QK_K / 32 + QK_K / 8;
+    static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, 2 }, QH = { QK_K / 4, QK_K / 32, 2 + QK_K / 4 },
+                                               SG = { QK_K / 4 + QK_K / 32, QK_K / 8, 2 + QK_K / 4 + QK_K / 32 },
+                                               MD = { META, 2 + IQ3S_N_SCALE, 0 },
+                                               MS = { META, 2 + IQ3S_N_SCALE, META };
+    // the SoA qh and signs streams are 2D surfaces: their row pitches must be 16-byte aligned
+    static bool ok(const int K, const bool aos) {
+        return K % qk == 0 && (aos || ((K / QK_K * 8) % 16 == 0 && K / QK_K * 8 >= 64));
+    }
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+        using namespace grouped_gemm_esimd;
+        simd<uint16_t, MR> dh = s.u16(MD, ib, 0);
+        simd<uint32_t, MR> s0 = s.u16(MS, ib, 2);
+        simd<uint32_t, MR> s1 = s.u16(MS, ib, 4);
+        sb.d  = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+        sb.sc = s0 | (s1 << 16);
+    }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        const int ib = kb / 8;
+        r.q     = s.template rows<2>(QS, ib, 8 * j);
+        r.qh    = (s.template rows<1>(QH, ib, 4 * (j / 4)) >> (8 * (j % 4))) & 0xffu;
+        r.signs = s.template rows<1>(SG, ib, 4 * j);
+        r.d     = sb.d * simd<float, MR>(simd<int32_t, MR>(1 + 2 * ((sb.sc >> (4 * j)) & 0xf)));
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        // the 8 grid entries of each of the row group's rows
+        simd<uint16_t, GROUPED_GEMM_RC * 8> qb =
+            r.q.template select<GROUPED_GEMM_RC * 2, 1>(g * GROUPED_GEMM_RC * 2).template bit_cast_view<uint8_t>();
+        simd<uint32_t, GROUPED_GEMM_RC>     qr = r.qh.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+        simd<uint32_t, GROUPED_GEMM_RC * 8> qh = qr.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 8, 0>(0);
+        simd<uint32_t, GROUPED_GEMM_RC * 8> j(0, 1);
+        j &= 7;
+        simd<uint32_t, GROUPED_GEMM_RC * 8> idx = simd<uint32_t, GROUPED_GEMM_RC * 8>(qb) | (((qh >> j) & 1) << 8);
+        simd<uint32_t, GROUPED_GEMM_RC * 8> ge  = slm_gather<uint32_t, GROUPED_GEMM_RC * 8>(idx * 4);
+        return grid_signed(ge, r.signs.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+                           r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC));
+    }
+};
+
+// The K-quants and iq3_xxs: a 256-value superblock, k step j its sub-block j. Their value bytes for
+// a k step are one rectangle of a stream, in k order after a shift and a mask per dword; scales,
+// mins and d are the block's and are loaded once per block.
+
+// [qs 128][scales 12][dm]: value l of sub-block j is nibble j & 1 of byte 32 (j / 2) + l, times
+// dall * scale_j, minus dmin * min_j (6-bit scale and min, get_scale_min_k4)
+template <int MR> struct grouped_gemm_esimd_a<block_q4_K, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    static constexpr int  slm_bytes = 0;
+    static ESIMD_INLINE void init_slm() {}
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+        grouped_gemm_esimd::simd<float, MR>        d, m;
+    };
+    struct sb_t {
+        grouped_gemm_esimd::simd<uint32_t, MR> sc[3];  // the 12 scale bytes
+        grouped_gemm_esimd::simd<float, MR>    dall, dmin;
+    };
+
+    static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 16 }, SC = { QK_K / 2, K_SCALE_SIZE, 4 },
+                                               DM = { QK_K / 2 + K_SCALE_SIZE, 4, 0 };
+
+    static bool ok(const int K, const bool aos) {
+        return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / 2));
+    }
+
+    template <typename Sl>
+    static ESIMD_INLINE void load_sb(sb_t &                            sb,
+                                     const Sl &                        s,
+                                     const int                         ib,
+                                     const grouped_gemm_esimd::field & sc  = SC,
+                                     const grouped_gemm_esimd::field & dmf = DM) {
+        using namespace grouped_gemm_esimd;
+#pragma unroll
+        for (int w = 0; w < 3; ++w) {
+            sb.sc[w] = s.u32(sc, ib, 4 * w);
+        }
+        simd<uint32_t, MR> dm = s.u32(dmf, ib, 0);
+        auto                  dh = dm.template bit_cast_view<sycl::half>();
+        sb.dall                  = simd<sycl::half, MR>(dh.template select<MR, 2>(0));
+        sb.dmin                  = simd<sycl::half, MR>(dh.template select<MR, 2>(1));
+    }
+    // get_scale_min_k4 for sub-block j over the lanes
+    static ESIMD_INLINE void scale_min(sb_t &                                sb,
+                                       const int                             j,
+                                       grouped_gemm_esimd::simd<float, MR> & d,
+                                       grouped_gemm_esimd::simd<float, MR> & m) {
+        using namespace grouped_gemm_esimd;
+        simd<uint32_t, MR> sc, mb;
+        if (j < 4) {
+            sc = byte_of(sb.sc[0], j) & 63;
+            mb = byte_of(sb.sc[1], j) & 63;
+        } else {
+            sc = (byte_of(sb.sc[2], j - 4) & 0xf) | ((byte_of(sb.sc[0], j - 4) >> 6) << 4);
+            mb = (byte_of(sb.sc[2], j - 4) >> 4) | ((byte_of(sb.sc[1], j - 4) >> 6) << 4);
+        }
+        d = sb.dall * simd<float, MR>(sc);
+        m = sb.dmin * simd<float, MR>(mb);
+    }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        r.q = (s.template rows<8>(QS, kb / 8, 32 * (j / 2)) >> (4 * (j & 1))) & 0x0F0F0F0Fu;
+        scale_min(sb, j, r.d, r.m);
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        simd<float, GROUPED_GEMM_RC> d = r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+                                     m = r.m.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+        return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8), d, m, d, m);
+    }
+};
+
+// [qs 128][qh 32][scales 12][dm]: q4_K plus bit j of qh byte l as 16
+template <int MR> struct grouped_gemm_esimd_a<block_q5_K, MR> {
+    using q4                           = grouped_gemm_esimd_a<block_q4_K, MR>;
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    static constexpr int  slm_bytes = 0;
+    static ESIMD_INLINE void init_slm() {}
+    using raw  = typename q4::raw;
+    using sb_t = typename q4::sb_t;
+    static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 48 }, QH = { QK_K / 2, QK_K / 8, 16 },
+                                               SC = { QK_K / 2 + QK_K / 8, K_SCALE_SIZE, 4 },
+                                               DM = { QK_K / 2 + QK_K / 8 + K_SCALE_SIZE, 4, 0 };
+    static bool ok(const int K, const bool aos) {
+        return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 2) && grouped_gemm_esimd::surface_ok(K / 8)));
+    }
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { q4::load_sb(sb, s, ib, SC, DM); }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        const int                 ib = kb / 8;
+        simd<uint32_t, MR * 8> qh = s.template rows<8>(QH, ib, 0);
+        r.q = ((s.template rows<8>(QS, ib, 32 * (j / 2)) >> (4 * (j & 1))) & 0x0F0F0F0Fu) |
+              (((qh >> j) & 0x01010101u) << 4);
+        q4::scale_min(sb, j, r.d, r.m);
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        return q4::template decode<g>(r);
+    }
+};
+
+// [ql 128][qh 64][scales 16][d]: sub-block j is quarter r = j % 4 of half h = j / 4; value l is
+// nibble r >> 1 of ql byte 64h + 32 (r & 1) + l with bits 2r, 2r+1 of qh byte 32h + l above it,
+// minus 32, times d * the int8 scale 2j + l / 16
+template <int MR> struct grouped_gemm_esimd_a<block_q6_K, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    static constexpr int  slm_bytes = 0;
+    static ESIMD_INLINE void init_slm() {}
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+        grouped_gemm_esimd::simd<float, MR>        d0, d1;  // per k half
+    };
+    struct sb_t {
+        grouped_gemm_esimd::simd<uint32_t, MR> sc[4];  // the 16 int8 scales
+        grouped_gemm_esimd::simd<float, MR>    d;
+    };
+
+    static constexpr grouped_gemm_esimd::field QL = { 0, QK_K / 2, 0 }, QH = { QK_K / 2, QK_K / 4, QK_K / 2 },
+                                               SC = { QK_K / 2 + QK_K / 4, QK_K / 16, QK_K / 2 + QK_K / 4 },
+                                               D  = { QK_K / 2 + QK_K / 4 + QK_K / 16, 2,
+                                                      QK_K / 2 + QK_K / 4 + QK_K / 16 };
+    static bool ok(const int K, const bool aos) {
+        return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 2) && grouped_gemm_esimd::surface_ok(K / 4)));
+    }
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+        using namespace grouped_gemm_esimd;
+#pragma unroll
+        for (int w = 0; w < 4; ++w) {
+            sb.sc[w] = s.u32(SC, ib, 4 * w);
+        }
+        simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+        sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+    }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        const int ib = kb / 8, h = j / 4, q = j % 4;
+        simd<uint32_t, MR * 8> ql = s.template rows<8>(QL, ib, 64 * h + 32 * (q & 1));
+        simd<uint32_t, MR * 8> qh = s.template rows<8>(QH, ib, 32 * h);
+        r.q = ((ql >> (4 * (q >> 1))) & 0x0F0F0F0Fu) | (((qh >> (2 * q)) & 0x03030303u) << 4);
+        // scales 2j and 2j + 1: bytes 2 (j % 2) and 2 (j % 2) + 1 of dword j / 2, sign extended
+        simd<uint32_t, MR> w  = dword_of(sb.sc, j / 2);
+        simd<int32_t, MR>  s0 = simd<int32_t, MR>(w << (24 - 16 * (j % 2))) >> 24;
+        simd<int32_t, MR>  s1 = simd<int32_t, MR>(w << (16 - 16 * (j % 2))) >> 24;
+        r.d0 = sb.d * simd<float, MR>(s0);
+        r.d1 = sb.d * simd<float, MR>(s1);
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        simd<float, GROUPED_GEMM_RC> z = 0.0f;
+        return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8),
+                         r.d0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z,
+                         r.d1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z, 32.0f);
+    }
+};
+
+// [qs 64][scales 16][dm]: value l of sub-block j is bits 2 (j % 4) of byte 32 (j / 4) + l, times
+// d * (scale & 15), minus dmin * (scale >> 4), scale byte 2j + l / 16
+template <int MR> struct grouped_gemm_esimd_a<block_q2_K, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    static constexpr int  slm_bytes = 0;
+    static ESIMD_INLINE void init_slm() {}
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+        grouped_gemm_esimd::simd<float, MR>        d0, m0, d1, m1;
+    };
+    struct sb_t {
+        grouped_gemm_esimd::simd<uint32_t, MR> sc[4];
+        grouped_gemm_esimd::simd<float, MR>    d, dmin;
+    };
+
+    static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, QK_K / 16 }, SC = { QK_K / 4, QK_K / 16, 0 },
+                                               DM = { QK_K / 4 + QK_K / 16, 4, QK_K / 4 + QK_K / 16 };
+
+    static bool ok(const int K, const bool aos) {
+        return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / 4));
+    }
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+        using namespace grouped_gemm_esimd;
+#pragma unroll
+        for (int w = 0; w < 4; ++w) {
+            sb.sc[w] = s.u32(SC, ib, 4 * w);
+        }
+        simd<uint32_t, MR> dm = s.u32(DM, ib, 0);
+        auto                  dh = dm.template bit_cast_view<sycl::half>();
+        sb.d                     = simd<sycl::half, MR>(dh.template select<MR, 2>(0));
+        sb.dmin                  = simd<sycl::half, MR>(dh.template select<MR, 2>(1));
+    }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        r.q = (s.template rows<8>(QS, kb / 8, 32 * (j / 4)) >> (2 * (j % 4))) & 0x03030303u;
+        simd<uint32_t, MR> w  = dword_of(sb.sc, j / 2);
+        simd<uint32_t, MR> s0 = byte_of(w, 2 * (j % 2)), s1 = byte_of(w, 2 * (j % 2) + 1);
+        r.d0 = sb.d * simd<float, MR>(s0 & 0xf);
+        r.m0 = sb.dmin * simd<float, MR>(s0 >> 4);
+        r.d1 = sb.d * simd<float, MR>(s1 & 0xf);
+        r.m1 = sb.dmin * simd<float, MR>(s1 >> 4);
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8),
+                         r.d0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+                         r.m0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+                         r.d1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+                         r.m1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC));
+    }
+};
+
+// [qs 64][hmask 32][scales 12][d]: q2_K's 2 bits, minus 4 where bit j of hmask byte l is clear,
+// times d * (6-bit scale - 32), scale s = 2j + l / 16 split over bytes s % 8 and 8 + s % 4
+template <int MR> struct grouped_gemm_esimd_a<block_q3_K, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    static constexpr int  slm_bytes = 0;
+    static ESIMD_INLINE void init_slm() {}
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+        grouped_gemm_esimd::simd<float, MR>        d0, d1;
+    };
+    struct sb_t {
+        grouped_gemm_esimd::simd<uint32_t, MR> sc[4];  // 12 scale bytes, sc[3] unused
+        grouped_gemm_esimd::simd<float, MR>    d;
+    };
+
+    static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, QK_K / 8 }, HM = { QK_K / 4, QK_K / 8, 0 },
+                                               SC = { QK_K / 4 + QK_K / 8, 12, QK_K / 4 + QK_K / 8 },
+                                               D  = { QK_K / 4 + QK_K / 8 + 12, 2, QK_K / 4 + QK_K / 8 + 12 };
+    static bool ok(const int K, const bool aos) {
+        return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 4) && grouped_gemm_esimd::surface_ok(K / 8)));
+    }
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+        using namespace grouped_gemm_esimd;
+#pragma unroll
+        for (int w = 0; w < 3; ++w) {
+            sb.sc[w] = s.u32(SC, ib, 4 * w);
+        }
+        sb.sc[3] = 0;
+        simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+        sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+    }
+
+    static ESIMD_INLINE grouped_gemm_esimd::simd<float, MR> scale(sb_t & sb, const int sidx) {
+        using namespace grouped_gemm_esimd;
+        simd<uint32_t, MR> lo = (byte_of(dword_of(sb.sc, (sidx % 8) / 4), sidx % 4) >> (4 * (sidx / 8))) & 0xf;
+        simd<uint32_t, MR> hi = (byte_of(sb.sc[2], sidx % 4) >> (2 * (sidx / 4))) & 3;
+        return sb.d * simd<float, MR>(simd<int32_t, MR>(lo | (hi << 4)) - 32);
+    }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        const int                 ib = kb / 8;
+        simd<uint32_t, MR * 8> hm = s.template rows<8>(HM, ib, 0);
+        r.q = ((s.template rows<8>(QS, ib, 32 * (j / 4)) >> (2 * (j % 4))) & 0x03030303u) |
+              (((hm >> j) & 0x01010101u) << 2);
+        r.d0 = scale(sb, 2 * j);
+        r.d1 = scale(sb, 2 * j + 1);
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        simd<float, GROUPED_GEMM_RC> z = 0.0f;
+        return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8),
+                         r.d0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z,
+                         r.d1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z, 4.0f);
+    }
+};
+
+// [qs 96][d]: per sub-block j, 8 grid indices at 8j and an aux dword at 64 + 4j whose 4 x 7 bits
+// are sign bytes of ksigns_iq2xs and whose top 4 bits scale d * (0.5 + s) * 0.5. ksigns_iq2xs[i] is
+// i with the parity of its 7 bits as bit 7, so the sign bytes are computed, not looked up.
+template <int MR> struct grouped_gemm_esimd_a<block_iq3_xxs, MR> {
+    static constexpr bool supported = true;
+    static constexpr int  qk        = QK_K;
+    static constexpr bool whole     = false;
+    // the 256-entry grid, a read-only table in SLM
+    static constexpr int  slm_bytes = 256 * sizeof(uint32_t);
+    static ESIMD_INLINE void init_slm() {
+        using namespace grouped_gemm_esimd;
+#pragma unroll
+        for (int i = 0; i < 256; i += 64) {
+            slm_block_store<uint32_t, 64>(i * 4, block_load<uint32_t, 64>(iq3xxs_grid + i));
+        }
+    }
+    struct raw {
+        grouped_gemm_esimd::simd<uint32_t, MR * 2> q;  // [row][8 grid indices]
+        grouped_gemm_esimd::simd<uint32_t, MR>     aux;
+        grouped_gemm_esimd::simd<float, MR>        d;
+    };
+    struct sb_t {
+        grouped_gemm_esimd::simd<float, MR> d;
+    };
+
+    static constexpr grouped_gemm_esimd::field QS = { 0, 3 * QK_K / 8, 2 }, D = { 3 * QK_K / 8, 2, 0 };
+
+    static bool ok(const int K, const bool aos) {
+        return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / QK_K * 96));
+    }
+    template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+        using namespace grouped_gemm_esimd;
+        simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+        sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+    }
+    template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+        using namespace grouped_gemm_esimd;
+        const int ib = kb / 8;
+        r.q   = s.template rows<2>(QS, ib, 8 * j);
+        r.aux = s.template rows<1>(QS, ib, 64 + 4 * j);
+        r.d   = sb.d * (0.5f + simd<float, MR>(r.aux >> 28)) * 0.5f;
+    }
+
+    template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+        using namespace grouped_gemm_esimd;
+        simd<uint32_t, GROUPED_GEMM_RC * 8> qb =
+            r.q.template select<GROUPED_GEMM_RC * 2, 1>(g * GROUPED_GEMM_RC * 2).template bit_cast_view<uint8_t>();
+        simd<uint32_t, GROUPED_GEMM_RC * 8> ge = slm_gather<uint32_t, GROUPED_GEMM_RC * 8>(qb * 4);
+        simd<uint32_t, GROUPED_GEMM_RC>     ax = r.aux.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+        simd<uint32_t, GROUPED_GEMM_RC>     sg = 0;
+#pragma unroll
+        for (int il = 0; il < 4; ++il) {
+            simd<uint32_t, GROUPED_GEMM_RC> si = (ax >> (7 * il)) & 127;
+            sg |= (si | ((cbit(si) & 1) << 7)) << (8 * il);
+        }
+        return grid_signed(ge, sg, r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC));
+    }
+};
+
+// T is the dpas operand type, sycl::half or bfloat16: the type B is packed in. A is decoded in half
+// either way and converted for bf16 (see the double rounding note at the conversion). AOS: the weight
+// is in the canonical block layout rather than the reorder (SoA) one. A: the grouped_gemm_arch built for.
+template <typename block_q_t, int KS, typename T, bool AOS, typename A> struct grouped_gemm_esimd_kernel {
+    using F = grouped_gemm_esimd_a<block_q_t, A::MR>;
+
+    const char *              src0_dd;
+    size_t                    expert_stride;
+    const ggml_sycl_grouped_gemm_tile * tiles;
+    const T *                 packed;
+    float *                   dst;
+    int                       M;
+    int                       Npad;
+    int                       K;
+    ggml_sycl_grouped_gemm_rows         dst_rows;
+
+    void operator()(sycl::nd_item<1> it) const SYCL_ESIMD_KERNEL {
+        if constexpr (grouped_gemm_built<A>()) {
+            run(it);
+        }
+    }
+
+    ESIMD_INLINE void run(sycl::nd_item<1> it) const {
+        using namespace sycl::ext::intel::esimd;
+        namespace xmx = sycl::ext::intel::esimd::xmx;
+        constexpr int MR = A::MR;
+        constexpr int NW = A::NW;
+        constexpr int RC = GROUPED_GEMM_RC;
+        constexpr int RG = MR / RC; // dpas row groups of this thread
+        constexpr int BN = GGML_SYCL_GROUPED_GEMM_BN;
+        constexpr int NT = BN / NW;
+
+        if constexpr (F::slm_bytes > 0) {
+            // the format's lookup table; a work-group is one thread, so nothing else writes it
+            slm_init<F::slm_bytes>();
+            F::init_slm();
+        }
+
+        const int mblocks = M / MR;
+        const int gid     = it.get_global_id(0);
+        const int t       = gid / mblocks;
+        const int m0      = (gid % mblocks) * MR;
+
+        const ggml_sycl_grouped_gemm_tile tile = tiles[t];
+        if (tile.n1 <= tile.n0) {
+            return;
+        }
+        const int b0 = t * BN; // the tile's packed-B columns
+
+        // the slice of this expert
+        const int             bpr = K / F::qk;
+        const grouped_gemm_esimd::slice<MR, A::L2D> sl  = {
+            (const uint8_t *) src0_dd + (size_t) tile.expert * expert_stride,
+            M,
+            bpr,
+            M * bpr,
+            m0,
+            AOS,
+            (int) sizeof(block_q_t)
+        };
+        const uint32_t *      bp  = (const uint32_t *) packed;
+        const unsigned        bw  = (unsigned) Npad * 4; // bytes of one packed B row
+        const int             nstep = K / GROUPED_GEMM_QK;
+
+        simd<float, RC * NW> acc[RG][NT];
+        simd<float, RC * NW> total[KS > 1 ? RG : 1][KS > 1 ? NT : 1];
+#pragma unroll
+        for (int g = 0; g < RG; ++g) {
+#pragma unroll
+            for (int nt = 0; nt < NT; ++nt) {
+                acc[g][nt] = 0.0f;
+                if constexpr (KS > 1) {
+                    total[g][nt] = 0.0f;
+                }
+            }
+        }
+
+        // The tile's route map, gathered once up front; the K loop hides its latency, and the epilogue
+        // below reads it at compile-time indices.
+        const int          ncols = tile.n1 - tile.n0;
+        simd_mask<BN>      cm    = simd<int32_t, BN>(0, 1) < ncols;
+        simd<int32_t, BN>  i2 = 0, r1 = 0; // r1: the dst row slot, i1 % ne1
+        if (dst_rows.map) {
+            const int32_t *    mp = (const int32_t *) (dst_rows.map + tile.n0);
+            simd<uint32_t, BN> mo(0, sizeof(mmid_row_mapping));
+            simd<int32_t, BN>  i1 = gather<int32_t, BN>(mp, mo, cm);
+            i2                    = gather<int32_t, BN>(mp + 1, mo, cm);
+            // once per thread in 32 bits: a 64-bit modulo per column is emulated, and at 20 columns a
+            // tile it cost more than the whole GEMM (7.2 vs 3.1 ms)
+            r1 = i1 % simd<int32_t, BN>((int32_t) dst_rows.ne1);
+        }
+
+        // One k step at a time: the format's stored bits for the MR rows, and the 16 k pairs x NW
+        // columns of packed B per dpas column tile, one 2D block message each (without 2D loads, one
+        // block load per packed row, cut into the tiles in registers). A second stage in
+        // flight spilled 7 KB of the 256 registers and ran 1.6x slower; the other threads of the
+        // XVE hide the load latency instead.
+        // A stored block's metadata (its sub-block scales) is loaded once per block. With KS > 1 the
+        // K split of the joint_matrix kernel is reproduced for bit-identical output (a second set of
+        // accumulators); KS == 1 sums the whole K in one, which is a different f32 rounding order and
+        // frees the 4 KB of registers the second set holds.
+        constexpr int    KPB       = F::qk / GROUPED_GEMM_QK;  // k steps per stored block
+        int           split     = 0;
+        int           split_end = KS > 1 ? nstep / KS : nstep;
+        typename F::sb_t sb;
+        for (int kb = 0; kb < nstep; ++kb) {
+            const int j = kb % KPB;
+            if (j == 0) {
+                F::load_sb(sb, sl, kb / KPB);
+            }
+            typename F::raw r;
+            F::load(r, sb, sl, kb, j);
+            simd<uint32_t, 16 * NW> b[NT]; // [16 k pairs][NW columns] each
+            if constexpr (A::L2D) {
+#pragma unroll
+                for (int nt = 0; nt < NT; ++nt) {
+                    b[nt] = load_2d<uint32_t, NW, 16>(bp, bw - 1, K / 2 - 1, bw - 1, b0 + nt * NW, kb * 16);
+                }
+            } else {
+                simd<uint32_t, 16 * BN> bm;
+#pragma unroll
+                for (int r = 0; r < 16; ++r) {
+                    bm.template select<BN, 1>(r * BN) =
+                        block_load<uint32_t, BN>(bp + (size_t) (kb * 16 + r) * Npad + b0, overaligned_tag<16>{});
+                }
+                auto bv = bm.template bit_cast_view<uint32_t, 16, BN>();
+#pragma unroll
+                for (int nt = 0; nt < NT; ++nt) {
+                    b[nt] = bv.template select<16, 1, NW, 1>(0, nt * NW);
+                }
+            }
+            simd<sycl::half, MR * GROUPED_GEMM_QK> all;
+            if constexpr (F::whole) {
+                all = F::decode_all(r);
+            }
+            // one row group of the thread, g a compile-time constant: with a runtime g the larger
+            // decodes were left rolled
+            auto row_group = [&](auto gc) SYCL_ESIMD_FUNCTION {
+                constexpr int g = decltype(gc)::value;
+                grouped_gemm_esimd::a_halves av;
+                if constexpr (F::whole) {
+                    // [row][32 k] of the whole tile to this row group's [k half][row][16 k]
+                    auto am = all.template bit_cast_view<sycl::half, MR, GROUPED_GEMM_QK>();
+                    av.template select<RC * 16, 1>(0)       = am.template select<RC, 1, 16, 1>(g * RC, 0);
+                    av.template select<RC * 16, 1>(RC * 16) = am.template select<RC, 1, 16, 1>(g * RC, 16);
+                } else {
+                    av = F::template decode<g>(r);
+                }
+#pragma unroll
+                for (int h = 0; h < 2; ++h) {
+                    // A of this k16 half: rows x k [16h, 16h+16), row major
+                    simd<sycl::half, RC * 16> ah = av.template select<RC * 16, 1>(h * RC * 16);
+                    simd<T, RC * 16>          at;
+                    if constexpr (std::is_same_v<T, sycl::half>) {
+                        at = ah;
+                    } else {
+                        // rounded twice (f32 to half in the decode, half to bf16 here), so A's rounding
+                        // error is about twice the joint_matrix bf16 stage's, which rounds f32 once
+                        at = simd<float, RC * 16>(ah);
+                    }
+#pragma unroll
+                    for (int nt = 0; nt < NT; ++nt) {
+                        simd<uint32_t, 8 * NW> bh32 = b[nt].template select<8 * NW, 1>(h * 8 * NW);
+                        simd<T, 16 * NW>       bh   = bh32.template bit_cast_view<T>();
+                        acc[g][nt] = xmx::dpas<8, RC, float, float>(acc[g][nt], bh, at);
+                    }
+                }
+            };
+            grouped_gemm_esimd::static_for<0, RG>::run(row_group);
+            if constexpr (KS > 1) {
+                // the K split's range ends, in k steps; a range's partial joins the total when it closes
+                if (kb + 1 == split_end) {
+#pragma unroll
+                    for (int g = 0; g < RG; ++g) {
+#pragma unroll
+                        for (int nt = 0; nt < NT; ++nt) {
+                            total[g][nt] += acc[g][nt];
+                            acc[g][nt] = 0.0f;
+                        }
+                    }
+                    ++split;
+                    split_end = ((split + 1) * nstep) / KS;
+                }
+            }
+        }
+        (void) split;
+        (void) split_end;
+
+        // column c of the tile: MR consecutive floats of one dst row. Unrolled, so c indexes the
+        // accumulators at compile time; a runtime index sends them all to private memory.
+#pragma unroll
+        for (int c = 0; c < BN; ++c) {
+            if (c >= ncols) {
+                continue;
+            }
+            simd<float, MR> col;
+#pragma unroll
+            for (int g = 0; g < RG; ++g) {
+                if constexpr (KS > 1) {
+                    col.template select<RC, 1>(g * RC) = total[g][c / NW].template select<RC, NW>(c % NW);
+                } else {
+                    col.template select<RC, 1>(g * RC) = acc[g][c / NW].template select<RC, NW>(c % NW);
+                }
+            }
+            const int n = tile.n0 + c;
+            if (dst_rows.map) {
+                float * out = (float *) (dst_rows.base + (size_t) r1[c] * dst_rows.nb1 + i2[c] * dst_rows.nb2) + m0;
+                block_store<float, MR>(out, col, element_aligned_tag{});
+            } else {
+                block_store<float, MR>(dst + (size_t) n * M + m0, col, element_aligned_tag{});
+            }
+        }
+    }
+
+    auto get(syclexp::properties_tag) const { return syclexp::properties{ sycl::ext::intel::experimental::grf_size<256> }; }
+};
+
+// the weights' surfaces: 2D block loads (SoA with L2D) want 64-byte aligned bases, the dword gathers 4
+static bool grouped_gemm_esimd_weights_ok(const char * w, const size_t expert_stride, const bool l2d) {
+    const size_t a = l2d ? 64 : 4;
+    return (uintptr_t) w % a == 0 && expert_stride % a == 0;
+}
+
+// 0 off, 1 on (default), 2 on with the joint_matrix kernel's K-split order reproduced (bit-identical output)
+static int ggml_sycl_grouped_gemm_esimd_mode() {
+    return g_ggml_sycl_mmid_esimd;
+}
+
+bool ggml_sycl_grouped_gemm_esimd_enabled() {
+#if defined(GGML_SYCL_SUPPORT_XMX)
+    return g_ggml_sycl_enable_esimd && ggml_sycl_grouped_gemm_esimd_mode() != 0;
+#else
+    return false;
+#endif
+}
+
+template <typename block_q_t, typename T, bool AOS, typename A>
+static void grouped_gemm_esimd_launch(const char *                        src0_dd,
+                                      size_t                              expert_stride,
+                                      const ggml_sycl_grouped_gemm_tile * tiles,
+                                      const T *                           packed,
+                                      float *                             dst,
+                                      int                                 M,
+                                      int                                 Npad,
+                                      int                                 K,
+                                      const ggml_sycl_grouped_gemm_rows & dst_rows,
+                                      int64_t                             n_tiles,
+                                      dpct::queue_ptr                     stream) {
+    const sycl::nd_range<1> range((size_t) n_tiles * (M / A::MR), 1);
+    if (ggml_sycl_grouped_gemm_esimd_mode() == 2) {
+        stream->parallel_for(range, grouped_gemm_esimd_kernel<block_q_t, GGML_SYCL_GROUPED_GEMM_KSPLIT, T, AOS, A>{
+                                        src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, dst_rows });
+    } else {
+        stream->parallel_for(range, grouped_gemm_esimd_kernel<block_q_t, 1, T, AOS, A>{
+                                        src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, dst_rows });
+    }
+}
+
+template <typename block_q_t, typename T, typename A>
+static bool grouped_gemm_esimd_run(const char *                        src0_dd,
+                                   size_t                              expert_stride,
+                                   const ggml_sycl_grouped_gemm_tile * tiles,
+                                   const T *                           packed,
+                                   float *                             dst,
+                                   int                                 M,
+                                   int                                 Npad,
+                                   int                                 K,
+                                   const ggml_sycl_grouped_gemm_rows & dst_rows,
+                                   bool                                reordered,
+                                   int64_t                             n_tiles,
+                                   dpct::queue_ptr                     stream) {
+    using F = grouped_gemm_esimd_a<block_q_t, A::MR>;
+    // whole row groups and stored blocks, every K-split range non-empty (also in mode 1, so both
+    // modes take the same shapes); B is read with 2D block loads, which want 64-byte aligned
+    // surfaces and pitches, or with 16-byte aligned block loads of whole tiles
+    if (M % A::MR != 0 || !F::ok(K, !reordered) || K < GROUPED_GEMM_QK * GGML_SYCL_GROUPED_GEMM_KSPLIT ||
+        !grouped_gemm_esimd_weights_ok(src0_dd, expert_stride, reordered && A::L2D) || ((uintptr_t) packed % 64) != 0 ||
+        Npad % 16 != 0) {
+        return false;
+    }
+    if (reordered) {
+        grouped_gemm_esimd_launch<block_q_t, T, false, A>(src0_dd, expert_stride, tiles, packed, dst, M, Npad, K,
+                                                          dst_rows, n_tiles, stream);
+    } else {
+        grouped_gemm_esimd_launch<block_q_t, T, true, A>(src0_dd, expert_stride, tiles, packed, dst, M, Npad, K,
+                                                         dst_rows, n_tiles, stream);
+    }
+    return true;
+}
+
+template <typename block_q_t, typename T, int NW>
+bool ggml_sycl_grouped_gemm_esimd(const char *                        src0_dd,
+                                  size_t                              expert_stride,
+                                  const ggml_sycl_grouped_gemm_tile * tiles,
+                                  const T *                           packed,
+                                  float *                             dst,
+                                  int                                 M,
+                                  int                                 Npad,
+                                  int                                 K,
+                                  const ggml_sycl_grouped_gemm_rows & dst_rows,
+                                  bool                                reordered,
+                                  int64_t                             n_tiles,
+                                  dpct::queue_ptr                     stream) {
+    static_assert(grouped_gemm_esimd_a<block_q_t>::supported == ggml_sycl_grouped_gemm_esimd_has<block_q_t>,
+                  "the header's format list and the formats implemented here must agree");
+    static_assert(NW == 8 || NW == 16, "the dpas widths of Xe-HPG and of Xe2 / Xe-HPC");
+    if constexpr (NW == 16) {
+        return grouped_gemm_esimd_run<block_q_t, T, grouped_gemm_xe2>(src0_dd, expert_stride, tiles, packed, dst, M,
+                                                                      Npad, K, dst_rows, reordered, n_tiles, stream);
+    } else {
+        return grouped_gemm_esimd_run<block_q_t, T, grouped_gemm_xehpg>(src0_dd, expert_stride, tiles, packed, dst, M,
+                                                                        Npad, K, dst_rows, reordered, n_tiles, stream);
+    }
+}
+
+#define GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, T, NW)                                               \
+    template bool ggml_sycl_grouped_gemm_esimd<block_q_t, T, NW>(                                     \
+        const char *, size_t, const ggml_sycl_grouped_gemm_tile *, const T *, float *, int, int, int, \
+        const ggml_sycl_grouped_gemm_rows &, bool, int64_t, dpct::queue_ptr);
+#define GROUPED_GEMM_ESIMD_INSTANCE(block_q_t)                                \
+    GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::half, 16)                  \
+    GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::ext::oneapi::bfloat16, 16) \
+    GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::half, 8)                   \
+    GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::ext::oneapi::bfloat16, 8)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q8_0)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq4_nl)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq4_xs)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq3_s)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq3_xxs)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q3_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q2_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q6_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q5_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q4_K)
diff --git a/ggml/src/ggml-sycl/fused-gemm-esimd.hpp b/ggml/src/ggml-sycl/fused-gemm-esimd.hpp
new file mode 100644
index 000000000..029d19480
--- /dev/null
+++ b/ggml/src/ggml-sycl/fused-gemm-esimd.hpp
@@ -0,0 +1,33 @@
+#ifndef GGML_SYCL_FUSED_GEMM_ESIMD_HPP
+#define GGML_SYCL_FUSED_GEMM_ESIMD_HPP
+
+#include "fused-gemm.hpp"
+
+// The grouped (MUL_MAT_ID) dequant GEMM, written with ESIMD so the dequantized weights go from
+// registers straight into the dpas A operand: no SLM, no barrier. Same inputs and outputs as the
+// joint_matrix grouped kernel of the 8 x NW x 16 sub-group-NW combination of T (sycl::half or
+// bfloat16): NW = 16 on Xe2 and Xe-HPC, 8 on Xe-HPG (Arc A, ARL-H). The schedule's tiles, B packed as
+// [K/2][Npad][2] T, and dst through dst_rows. The weight is in the reorder (SoA) layout if
+// `reordered`, else in the canonical block layout. Returns false when the shape is not covered.
+template <typename block_q_t>
+constexpr bool ggml_sycl_grouped_gemm_esimd_has = std::is_same_v<block_q_t, block_q8_0> ||
+                                                  std::is_same_v<block_q_t, block_iq4_nl> ||
+                                                  std::is_same_v<block_q_t, block_iq4_xs> ||
+                                                  std::is_same_v<block_q_t, block_iq3_s> ||
+                                                  std::is_same_v<block_q_t, block_iq3_xxs> ||
+                                                  std::is_same_v<block_q_t, block_q2_K> ||
+                                                  std::is_same_v<block_q_t, block_q3_K> ||
+                                                  std::is_same_v<block_q_t, block_q4_K> ||
+                                                  std::is_same_v<block_q_t, block_q5_K> ||
+                                                  std::is_same_v<block_q_t, block_q6_K>;
+
+template <typename block_q_t, typename T, int NW>
+bool ggml_sycl_grouped_gemm_esimd(const char * src0_dd, size_t expert_stride, const ggml_sycl_grouped_gemm_tile * tiles,
+                                  const T * packed, float * dst, int M, int Npad, int K,
+                                  const ggml_sycl_grouped_gemm_rows & dst_rows, bool reordered, int64_t n_tiles,
+                                  dpct::queue_ptr stream);
+
+// GGML_SYCL_MMID_ESIMD: 1 (default) takes the ESIMD kernel above where it applies, 0 never does
+bool ggml_sycl_grouped_gemm_esimd_enabled();
+
+#endif // GGML_SYCL_FUSED_GEMM_ESIMD_HPP
diff --git a/ggml/src/ggml-sycl/fused-gemm.cpp b/ggml/src/ggml-sycl/fused-gemm.cpp
index 803a994c0..d0525e3a0 100644
--- a/ggml/src/ggml-sycl/fused-gemm.cpp
+++ b/ggml/src/ggml-sycl/fused-gemm.cpp
@@ -1,4 +1,5 @@
 #include "fused-gemm.hpp"
+#include "fused-gemm-esimd.hpp"

 #include <sycl/ext/oneapi/matrix/matrix.hpp>

@@ -17,7 +18,6 @@ namespace mx = sycl::ext::oneapi::experimental::matrix;
 // superblock into steps of this width. The sub-groups of a work-group each walk their own K range
 // and are summed at the end.
 static constexpr int FG_BK     = QK4_NL;
-static constexpr int FG_KSPLIT = 4;

 // Element traits of one joint_matrix operand type. The A stage and the B pack compute in f32 and
 // convert once, in registers, when they write the element, so any type costs the same one pass.
@@ -113,6 +113,15 @@ using fg_half = sycl::half;
 using fg_bf16 = sycl::ext::oneapi::bfloat16;
 using fg_tf32 = mx::precision::tf32;

+// The combinations the ESIMD grouped GEMM (fused-gemm-esimd.cpp) stands in for: 8 x SG x 16 with f16
+// or bf16 operands, the shape of one dpas, on sub-group 16 (Xe2, Xe-HPC) or 8 (Xe-HPG). Its B is
+// packed and tiled as theirs is.
+template <typename S> static constexpr bool fg_esimd_combo() {
+    return S::TM == 8 && S::TN == S::SG && S::TK == 16 && (S::SG == 16 || S::SG == 8) &&
+           std::is_same_v<typename S::ta, typename S::tb> &&
+           (std::is_same_v<typename S::ta, fg_half> || std::is_same_v<typename S::ta, fg_bf16>);
+}
+
 // One bit of GGML_SYCL_XMX_GATHER_SHAPES per combination. Only combinations some device lists in
 // matrix_combinations are built (appendix of sycl_ext_oneapi_matrix and the runtime's own list).
 template <typename F> static void fg_visit_combo(int idx, F && f);
@@ -152,29 +161,7 @@ template <typename F> static void fg_visit_combo(int idx, F && f) {
 // size, since IGC fails on the other ones. A JIT build keeps them all, but each combination lands in
 // its own device image (joint_matrix is an optional kernel feature) and only a combination the
 // device reports is launched, so the runtime never asks IGC for the others.
-#if defined(__SYCL_DEVICE_ONLY__)
-#    if __SYCL_TARGET_INTEL_GPU_ACM_G10__ || __SYCL_TARGET_INTEL_GPU_ACM_G11__ || __SYCL_TARGET_INTEL_GPU_ACM_G12__ || \
-        __SYCL_TARGET_INTEL_GPU_ARL_H__
-#        define FG_AOT_SG 8
-#    elif __SYCL_TARGET_INTEL_GPU_PVC__ || __SYCL_TARGET_INTEL_GPU_PVC_VG__ || __SYCL_TARGET_INTEL_GPU_BMG_G21__ || \
-        __SYCL_TARGET_INTEL_GPU_BMG_G31__ || __SYCL_TARGET_INTEL_GPU_LNL_M__ || __SYCL_TARGET_INTEL_GPU_PTL_H__ ||   \
-        __SYCL_TARGET_INTEL_GPU_PTL_U__ || __SYCL_TARGET_INTEL_GPU_WCL__ || __SYCL_TARGET_INTEL_GPU_NVL_S__ ||       \
-        __SYCL_TARGET_INTEL_GPU_NVL_U__ || __SYCL_TARGET_INTEL_GPU_NVL_P__
-#        define FG_AOT_SG 16
-#    elif __SYCL_TARGET_INTEL_GPU_TGLLP__ || __SYCL_TARGET_INTEL_GPU_RKL__ || __SYCL_TARGET_INTEL_GPU_ADL_S__ || \
-        __SYCL_TARGET_INTEL_GPU_ADL_P__ || __SYCL_TARGET_INTEL_GPU_ADL_N__ || __SYCL_TARGET_INTEL_GPU_DG1__ ||   \
-        __SYCL_TARGET_INTEL_GPU_MTL_U__ || __SYCL_TARGET_INTEL_GPU_MTL_H__
-#        define FG_AOT_SG 0 // no XMX
-#    endif
-#endif
-
-template <int SG> static constexpr bool fg_built() {
-#if defined(FG_AOT_SG)
-    return SG == FG_AOT_SG;
-#else
-    return true;
-#endif
-}
+// (FG_AOT_SG is defined in fused-gemm.hpp, which the ESIMD kernels share)

 // Upper bound on the tile count when total_rows rows are routed to n_as experts: the worst case gives
 // each expert one row and fills whole tiles with the rest. The bound depends only on the shape, not
@@ -991,6 +978,15 @@ static void grouped_dequant_gemm_launch(const char * src0_dd, const size_t exper
                                         float * dst, const int M, const int Npad, const int K, const int64_t n_tiles,
                                         const int64_t groups_m, dpct::queue_ptr stream,
                                         ggml_sycl_grouped_gemm_rows dst_rows = {}) {
+    if constexpr (fg_esimd_combo<S>() && ggml_sycl_grouped_gemm_esimd_has<block_q_t>) {
+        // the register-operand ESIMD kernel, either weight layout: no SLM A image (GGML_SYCL_MMID_ESIMD)
+        if (ggml_sycl_grouped_gemm_esimd_enabled() &&
+            ggml_sycl_grouped_gemm_esimd<block_q_t, typename S::tsb, S::SG>(src0_dd, expert_stride, tiles_ptr, packed,
+                                                                            dst, M, Npad, K, dst_rows, reordered,
+                                                                            n_tiles, stream)) {
+            return;
+        }
+    }
     stream->submit([&](sycl::handler & cgh) {
         sycl::local_accessor<typename S::tsa, 1> tile_a(FG_KSPLIT * S::SG_ROWS * FG_BK, cgh);
         sycl::local_accessor<float, 1>          tile_c(FG_KSPLIT * S::SG_ROWS * S::BN, cgh);
@@ -1087,6 +1083,45 @@ template <typename F> static bool fg_visit_type(ggml_type type, bool reordered,
     }
 }

+// The ESIMD grouped GEMM loads A straight into its dpas operands, so the joint_matrix tile shape
+// means nothing to it; it runs on the fg_esimd_combo of the picked operand type and sub-group size. A
+// grouped call whose weight it covers is moved there from any other shape of that type and sub-group
+// size, if the device lists it, whatever GGML_SYCL_XMX_GATHER_SHAPES allows: the shape bits choose
+// among joint_matrix tiles.
+// A call the ESIMD kernel then declines (a shape it does not take) runs that combination's joint_matrix
+// kernel.
+static int fg_esimd_remap(dpct::queue_ptr stream, int combo, ggml_type src0_type, bool reordered) {
+    if (combo < 0 || !ggml_sycl_grouped_gemm_esimd_enabled()) {
+        return combo;
+    }
+    bool covered = false;
+    fg_visit_type(src0_type, reordered, [&](auto tag) {
+        using T = decltype(tag);
+        covered = ggml_sycl_grouped_gemm_esimd_has<typename T::type>;
+    });
+    if (!covered) {
+        return combo;
+    }
+    int target = -1;
+    fg_visit_combo(combo, [&](auto s) {
+        using S = decltype(s);
+        if constexpr (std::is_same_v<typename S::ta, typename S::tb>) {
+            for (int idx = 0; idx < FG_N_COMBOS; ++idx) {
+                fg_visit_combo(idx, [&](auto t) {
+                    using U = decltype(t);
+                    if constexpr (fg_esimd_combo<U>() && U::SG == S::SG && std::is_same_v<typename U::ta, typename S::ta>) {
+                        target = idx;
+                    }
+                });
+            }
+        }
+    });
+    if (target < 0 || target == combo || !(fg_device_combos(stream->get_device()) & (1 << target))) {
+        return combo;
+    }
+    return target;
+}
+
 template <typename S>
 static bool fg_fused_run(ggml_type src0_type, bool src0_reordered, const void * src0, const void * src1, ggml_type src1_type,
                          float * dst, int64_t M, int64_t N, int64_t K, int64_t ldd, ggml_sycl_pool & pool,
@@ -1201,7 +1236,7 @@ bool ggml_sycl_grouped_dequant_gemm(ggml_type src0_type, bool reordered, const v
     if (!ggml_sycl_grouped_dequant_gemm_shape_ok(src0_type, M, K, total_rows, n_active)) {
         return false;
     }
-    const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec);
+    const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered);
     if (combo < 0) {
         return false;
     }
@@ -1407,7 +1442,7 @@ bool ggml_sycl_grouped_dequant_gemm_device_supported(ggml_type src0_type, bool r
     if (grouped_gemm_max_tiles(total_rows, n_as, GGML_SYCL_FG_BN) > INT32_MAX / GGML_SYCL_FG_BN) {
         return false;
     }
-    const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec);
+    const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered);
     if (combo < 0) {
         return false;
     }
@@ -1428,7 +1463,7 @@ bool ggml_sycl_grouped_dequant_gemm_device(ggml_type src0_type, bool reordered,
                                                         n_tokens, M, K, stream)) {
         return false;
     }
-    const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec);
+    const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered);
     bool launched = false;
     fg_visit_combo(combo, [&](auto s) {
         launched = fg_grouped_device_run<decltype(s)>(src0_type, reordered, src0_base, expert_stride, src1, dst,
diff --git a/ggml/src/ggml-sycl/fused-gemm.hpp b/ggml/src/ggml-sycl/fused-gemm.hpp
index 0e807a2c0..e65e98fc8 100644
--- a/ggml/src/ggml-sycl/fused-gemm.hpp
+++ b/ggml/src/ggml-sycl/fused-gemm.hpp
@@ -13,7 +13,34 @@ struct ggml_sycl_grouped_gemm_rows {
     size_t                   nb2;
 };

+// AOT: the sub-group size of the XMX combinations of the target, 0 for a target without XMX.
+// Undefined in a JIT build, which keeps every combination.
+#if defined(__SYCL_DEVICE_ONLY__)
+#    if __SYCL_TARGET_INTEL_GPU_ACM_G10__ || __SYCL_TARGET_INTEL_GPU_ACM_G11__ || __SYCL_TARGET_INTEL_GPU_ACM_G12__ || \
+        __SYCL_TARGET_INTEL_GPU_ARL_H__
+#        define FG_AOT_SG 8
+#    elif __SYCL_TARGET_INTEL_GPU_PVC__ || __SYCL_TARGET_INTEL_GPU_PVC_VG__ || __SYCL_TARGET_INTEL_GPU_BMG_G21__ || \
+        __SYCL_TARGET_INTEL_GPU_BMG_G31__ || __SYCL_TARGET_INTEL_GPU_LNL_M__ || __SYCL_TARGET_INTEL_GPU_PTL_H__ ||   \
+        __SYCL_TARGET_INTEL_GPU_PTL_U__ || __SYCL_TARGET_INTEL_GPU_WCL__ || __SYCL_TARGET_INTEL_GPU_NVL_S__ ||       \
+        __SYCL_TARGET_INTEL_GPU_NVL_U__ || __SYCL_TARGET_INTEL_GPU_NVL_P__
+#        define FG_AOT_SG 16
+#    elif __SYCL_TARGET_INTEL_GPU_TGLLP__ || __SYCL_TARGET_INTEL_GPU_RKL__ || __SYCL_TARGET_INTEL_GPU_ADL_S__ || \
+        __SYCL_TARGET_INTEL_GPU_ADL_P__ || __SYCL_TARGET_INTEL_GPU_ADL_N__ || __SYCL_TARGET_INTEL_GPU_DG1__ ||   \
+        __SYCL_TARGET_INTEL_GPU_MTL_U__ || __SYCL_TARGET_INTEL_GPU_MTL_H__
+#        define FG_AOT_SG 0 // no XMX
+#    endif
+#endif
+
+template <int SG> static constexpr bool fg_built() {
+#if defined(FG_AOT_SG)
+    return SG == FG_AOT_SG;
+#else
+    return true;
+#endif
+}
+
 static constexpr int GGML_SYCL_FG_BN = 32;
+static constexpr int FG_KSPLIT = 4;
 static constexpr int64_t GGML_SYCL_MMID_SCHED_MAX_EXPERTS = 4096;

 // Shape and type gates for the kernels below. Device capability is separate: it needs a queue to ask.
diff --git a/ggml/src/ggml-sycl/ggml-sycl.cpp b/ggml/src/ggml-sycl/ggml-sycl.cpp
index 9da436ced..5b87fa024 100644
--- a/ggml/src/ggml-sycl/ggml-sycl.cpp
+++ b/ggml/src/ggml-sycl/ggml-sycl.cpp
@@ -111,6 +111,7 @@ int g_ggml_sycl_memtrace_step = 64;
 int g_ggml_sycl_enable_vmm = 1;
 int g_ggml_sycl_enable_fusion = 1;
 int g_ggml_sycl_enable_esimd = 1;
+int g_ggml_sycl_mmid_esimd = 1;
 int g_ggml_sycl_mmvq_wide = 1;
 int g_ggml_sycl_enable_xmx_esimd = 1;
 int g_ggml_sycl_xmx_max_cols = GGML_SYCL_XMX_MAX_COLS_DEFAULT;
@@ -444,6 +445,7 @@ static void ggml_check_sycl() try {
         g_ggml_sycl_enable_vmm = ggml_sycl_get_env("GGML_SYCL_ENABLE_VMM", 1);
         g_ggml_sycl_enable_fusion = ggml_sycl_get_env("GGML_SYCL_ENABLE_FUSION", 1);
         g_ggml_sycl_enable_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_ESIMD", 1);
+        g_ggml_sycl_mmid_esimd = ggml_sycl_get_env("GGML_SYCL_MMID_ESIMD", 1);
         g_ggml_sycl_mmvq_wide = ggml_sycl_get_env("GGML_SYCL_MMVQ_WIDE", 1);
         g_ggml_sycl_enable_xmx_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_XMX_ESIMD", 1);
         g_ggml_sycl_xmx_max_cols = std::max(0, ggml_sycl_get_env("GGML_SYCL_XMX_MAX_COLS", GGML_SYCL_XMX_MAX_COLS_DEFAULT));
@@ -578,6 +580,7 @@ static void ggml_check_sycl() try {
         GGML_LOG_INFO("  GGML_SYCL_ENABLE_OPT: %d\n", g_ggml_sycl_enable_optimize);
         GGML_LOG_INFO("  GGML_SYCL_XMX_GATHER_TYPES: %d\n", g_ggml_sycl_xmx_gather_types);
         GGML_LOG_INFO("  GGML_SYCL_MMID_SCHED: %d\n", g_ggml_sycl_mmid_sched);
+        GGML_LOG_INFO("  GGML_SYCL_MMID_ESIMD: %d\n", g_ggml_sycl_mmid_esimd);
         GGML_LOG_INFO("  GGML_SYCL_XMX_GATHER_SHAPES: %d\n", g_ggml_sycl_xmx_gather_shapes);
         GGML_LOG_INFO("  GGML_SYCL_DYNAMIC_PRECISION: %s\n",
                       ggml_sycl_dynamic_precision_names[g_ggml_sycl_dynamic_precision]);