Commit 418fe97b5 for llama.cpp
commit 418fe97b5f01b11eeb89f30d8a0459433edbf1d2
Author: cwriter <silvan.niederer@bluewin.ch>
Date: Sat Nov 21 04:37:40 2026 +0100
sycl: Improve Grouped XMX GEMM by avoiding SLM staging (#31428)
diff --git a/docs/backend/SYCL.md b/docs/backend/SYCL.md
index ca0b6d7bc..4a1d16aad 100644
--- a/docs/backend/SYCL.md
+++ b/docs/backend/SYCL.md
@@ -827,6 +827,7 @@ User can use the device management in [docs/multi-gpu.md](https://github.com/ggm
| GGML_SYCL_XMX_MAX_COLS | non-negative integer, 80 (default) | Most activation columns (batch size) the `GGML_SYCL_ENABLE_XMX_ESIMD` kernels take; wider batches use the other mat-mul kernels. A larger value keeps XMX for bigger batches (as repeated launches), `0` never uses it. |
| GGML_SYCL_XMX_GLU_MAX_COLS | 0 to 16, 16 (default) | Most columns the fused gate and up XMX kernel takes; wider batches run gate and up as separate mat-vecs. 16 is the most the kernel is built for, larger values are clamped to 16 and `0` turns the fused kernel off. |
| GGML_SYCL_XMX_GATHER_TYPES | decimal bitmask, all bits set (default) | Weight formats that may use the XMX dequant-GEMM paths, which dequantize weights straight into the XMX tiles. This speeds up prompt processing of MoE models on GPUs with XMX units (Arc A- and B-series, Arc Pro, Data Center GPU Max), for example pp512 of Qwen3-30B-A3B UD-IQ3_XXS by about 50% on an Arc Pro B60. Bits:<br>* 1: IQ4_NL, 2: IQ3_S, 4: IQ4_XS, 8: IQ3_XXS, 16: IQ2_XXS, 32: IQ2_XS, 64: IQ2_S, 128: IQ1_S, 256: IQ1_M<br>* 512: Q8_0, 1024: Q4_K, 2048: Q5_K, 4096: Q6_K, 8192: Q2_K, 16384: Q3_K (MoE `MUL_MAT_ID` only)<br>Add values to combine them, for example `3` for IQ4_NL and IQ3_S; `0` disables the paths. A set bit does not force the path: batches of more than 64 tokens per expert use the library GEMM unless the device schedule width heuristic is overridden with `GGML_SYCL_MMID_SCHED=3`. Row lengths must be a multiple of 256 (32 for IQ4_NL and Q8_0). |
+| GGML_SYCL_MMID_ESIMD | 0, 1 (default), 2 | The `MUL_MAT_ID` grouped XMX dequant-GEMM as an ESIMD kernel that dequantizes the weights straight into the `dpas` register operands, with no shared local memory image of them, from either weight layout (the per-expert reorder or the canonical blocks). q8_0, iq4_nl, iq4_xs, iq3_s, iq3_xxs and q2_K to q6_K. It stands in for the 8x16x16 sub-group-16 combination of f16 or bf16 (Xe2, Xe-HPC) and for the 8x8x16 sub-group-8 one (Xe-HPG: Arc A-series, ARL-H; there it works on 16 rows a thread and reads with 1D block loads, since that architecture has half the register file and no 2D block loads); while it is on, a grouped call whose weight type it covers is moved to that combination from any other shape of its type and sub-group size, whatever `GGML_SYCL_XMX_GATHER_SHAPES` allows, because the shape bits choose among `joint_matrix` tiles and the ESIMD kernel tiles on its own. A call it declines runs that combination's `joint_matrix` kernel. 0 uses the `joint_matrix` kernel, 1 the ESIMD one, 2 the ESIMD one summing K in the `joint_matrix` kernel's order (bit-identical output for f16; slower). An AOT build gets its 256-register file through `-ze-opt-enable-auto-large-GRF-mode` (set by the CMake build). Not built with `GGML_SYCL_SUPPORT_XMX=OFF`. |
| GGML_SYCL_MMID_SCHED | decimal bitmask, 1 (default) | Build the grouped MoE routing and tile schedule on the GPU, avoiding the expert-ID host readback and CPU sort.<br>* 1: enable the device schedule.<br>* 2: override the fallback for models with few routed experts by disabling the performance heuristic that rejects batches averaging more than 64 routed rows per expert. This may reduce performance; type, shape, precision and device capability checks still apply.<br>Use `3` to enable the device schedule and override this heuristic, or `0` to use the host schedule. Bit 2 alone does not enable the device schedule. Higher bits are ignored. |
| GGML_SYCL_XMX_GATHER_SHAPES | decimal bitmask, 255 (default) | XMX `joint_matrix` combinations the paths of `GGML_SYCL_XMX_GATHER_TYPES` may use; the operand type comes from `GGML_SYCL_DYNAMIC_PRECISION` and the best supported combination is picked automatically (logged as `fg_pick_combo`). Bits:<br>* Xe2, Xe3, Xe-HPC: 1: f16 8x16x16, 2: f16 16x16x16, 4: f16 32x64x16, 8: f16 32x64x32, 32: tf32 8x16x8, 64: bf16 8x16x16<br>* Xe-HPG (Arc A770, ARL-H): 16: f16 8x8x16, 128: bf16 8x8x16<br>Clear a bit to exclude a combination, or set a single bit to force one for testing. |
| GGML_SYCL_DYNAMIC_PRECISION | `F16` (default with `GGML_SYCL_F16=ON`), `BF16`, `TF32` or `F32` (default otherwise) | Operand type of the XMX dequant-GEMM paths (`GGML_SYCL_XMX_GATHER_TYPES`); accumulation is always f32. `F16` is the fastest, but activations above 65504 overflow. `BF16` keeps the f32 range at a 7-bit mantissa, `TF32` keeps the range and the f16 mantissa but is about 30% slower and needs Xe2, Xe3 or Xe-HPC, and `F32` turns the XMX paths off. Ops that request a higher src1 precision ([TAG_GGML_PREC]) get it regardless of this setting. |
diff --git a/ggml/src/ggml-sycl/CMakeLists.txt b/ggml/src/ggml-sycl/CMakeLists.txt
index 8312a8305..0a5aede0b 100644
--- a/ggml/src/ggml-sycl/CMakeLists.txt
+++ b/ggml/src/ggml-sycl/CMakeLists.txt
@@ -218,6 +218,8 @@ endif()
if (GGML_SYCL_TARGET STREQUAL "INTEL")
add_compile_definitions(GGML_SYCL_WARP_SIZE=16)
+ # With AOT the option goes to ocloc inside -options (see GGML_SYCL_DEVICE_ARCH below): a bare
+ # -Xs hands it to ocloc as a command-line flag, which fails the device link.
if (NOT GGML_SYCL_DEVICE_ARCH)
target_link_options(ggml-sycl PRIVATE -Xs -ze-intel-greater-than-4GB-buffer-required)
else()
@@ -248,10 +250,18 @@ endif()
if (GGML_SYCL_DEVICE_ARCH)
message(STATUS "GGML_SYCL_DEVICE_ARCH=${GGML_SYCL_DEVICE_ARCH} (AOT via spir64_gen)")
+ # AOT (oneAPI 2026.2) ignores the grf_size<256> kernel property on ESIMD kernels and builds them
+ # with 128 registers, which made the ESIMD MoE GEMM 1.7-3.9x slower than under JIT; SPMD kernels
+ # keep honouring the property. A module-wide -doubleGRF also moves the ESIMD kernels that ask for
+ # 128 to 256, and a server built with it hung on its first request, so IGC picks the size per
+ # kernel from its register pressure instead (ESIMD kernels too; -ze-opt-large-grf-kernel does not
+ # reach them).
+ set(GGML_SYCL_AOT_BACKEND_OPTIONS
+ "-device ${GGML_SYCL_DEVICE_ARCH} -options '-ze-intel-greater-than-4GB-buffer-required -ze-opt-enable-auto-large-GRF-mode'")
target_compile_options(
ggml-sycl PRIVATE
-fsycl-targets=spir64_gen
- "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH} -options -ze-intel-greater-than-4GB-buffer-required\""
+ "SHELL:-Xsycl-target-backend=spir64_gen \"${GGML_SYCL_AOT_BACKEND_OPTIONS}\""
)
# Pass through parallel job (process) count for parallelising the
@@ -266,7 +276,7 @@ if (GGML_SYCL_DEVICE_ARCH)
target_link_options(
ggml-sycl PRIVATE
-fsycl-targets=spir64_gen
- "SHELL:-Xsycl-target-backend=spir64_gen \"-device ${GGML_SYCL_DEVICE_ARCH} -options -ze-intel-greater-than-4GB-buffer-required\""
+ "SHELL:-Xsycl-target-backend=spir64_gen \"${GGML_SYCL_AOT_BACKEND_OPTIONS}\""
-fsycl-max-parallel-link-jobs=${GGML_SYCL_MAX_PARALLEL_LINK_JOBS}
)
diff --git a/ggml/src/ggml-sycl/common.hpp b/ggml/src/ggml-sycl/common.hpp
index 75e783d12..ae156acca 100644
--- a/ggml/src/ggml-sycl/common.hpp
+++ b/ggml/src/ggml-sycl/common.hpp
@@ -72,6 +72,7 @@ extern int g_ggml_sycl_debug;
extern int g_ggml_sycl_enable_optimize;
extern int g_ggml_sycl_enable_fusion;
extern int g_ggml_sycl_enable_esimd;
+extern int g_ggml_sycl_mmid_esimd;
extern int g_ggml_sycl_mmvq_wide;
extern int g_ggml_sycl_enable_xmx_esimd;
extern int g_ggml_sycl_xmx_max_cols;
diff --git a/ggml/src/ggml-sycl/fused-gemm-esimd.cpp b/ggml/src/ggml-sycl/fused-gemm-esimd.cpp
new file mode 100644
index 000000000..37f87e377
--- /dev/null
+++ b/ggml/src/ggml-sycl/fused-gemm-esimd.cpp
@@ -0,0 +1,1160 @@
+#include "fused-gemm-esimd.hpp"
+
+#include <sycl/ext/intel/esimd.hpp>
+#include <sycl/ext/intel/esimd/xmx/dpas.hpp>
+#include <sycl/ext/intel/experimental/grf_size_properties.hpp>
+
+// One thread owns MR rows of one expert's weight and one BN-wide tile of the schedule, over the
+// whole K. Per k step (32 values) it loads the rows' stored bits, dequantizes them in registers to
+// the f16 A operand of dpas (RC = 8 rows x 16 k, row major) and multiplies it with the packed B,
+// which is already the dpas VNNI operand: [K/2][Npad][2] halves, NW columns a dpas.
+//
+// The joint_matrix kernel dequantizes into SLM and reads it back in the same sub-group, which is
+// staging, not sharing; dpas takes registers, so that round trip and its barriers are gone here.
+// Its K split over GGML_SYCL_GROUPED_GEMM_KSPLIT sub-groups is reproduced as sequential K ranges summed in
+// the same ascending order as its tile_c reduction, and every format below rounds A exactly as its
+// joint_matrix A stage does, so the output is the same bits.
+//
+// A weight format is a grouped_gemm_esimd_a<block, MR> with: raw, the stored bits of the MR rows for one k
+// step; load(), which fetches them from the expert's slice in either layout (grouped_gemm_esimd::slice); and
+// decode(), the f16 A of one row group.
+// It reads the same fields as the joint_matrix A stage of its format.
+//
+// The kernel is built for one grouped_gemm_arch: the dpas width NW (the columns of B one dpas takes, the
+// sub-group size of the joint_matrix combination it stands in for), the rows of one thread, and
+// whether 2D block loads exist. Xe2 and Xe-HPC: 16 wide, 32 rows, 2D loads. Xe-HPG (Arc A, ARL-H):
+// 8 wide, no 2D block messages, and 256 registers of 32 bytes, half of Xe2's file, so 16 rows: its
+// 32-row accumulators alone would take half the registers.
+static constexpr int GGML_SYCL_GROUPED_GEMM_KSPLIT = FG_KSPLIT;
+static constexpr int GGML_SYCL_GROUPED_GEMM_BN = GGML_SYCL_FG_BN;
+static constexpr int GROUPED_GEMM_MR = 32; // rows of one thread on Xe2: one 2D load covers them
+static constexpr int GROUPED_GEMM_RC = 8; // rows of one dpas
+static constexpr int GROUPED_GEMM_QK = 32; // values of one k step
+
+template <int NW_, int MR_, bool L2D_> struct grouped_gemm_arch {
+ static constexpr int NW = NW_;
+ static constexpr int MR = MR_;
+ static constexpr bool L2D = L2D_;
+};
+
+using grouped_gemm_xe2 = grouped_gemm_arch<16, GROUPED_GEMM_MR, true>;
+using grouped_gemm_xehpg = grouped_gemm_arch<8, 16, false>;
+
+// An AOT build compiles each kernel only for the targets of its dpas width: GGML_SYCL_XMX_AOT_SG for
+// spir64_gen, FG_AOT_SG (fused-gemm.hpp) for intel_gpu_* targets. A build without XMX (GGML_SYCL_SUPPORT_XMX
+// off) compiles none of them
+template <typename A> static constexpr bool grouped_gemm_built() {
+#if !defined(GGML_SYCL_SUPPORT_XMX)
+ return false;
+#elif defined(GGML_SYCL_XMX_AOT_SG)
+ return A::NW == GGML_SYCL_XMX_AOT_SG;
+#elif defined(FG_AOT_SG)
+ return A::NW == FG_AOT_SG;
+#else
+ return true;
+#endif
+}
+
+namespace grouped_gemm_esimd {
+using namespace sycl::ext::intel::esimd;
+
+// f(std::integral_constant<int, I>) for I in [0, N): register selects at a runtime offset become
+// indirect accesses, so loops whose index picks registers are spelled out at compile time
+template <int I, int N> struct static_for {
+ template <typename Fn> static ESIMD_INLINE void run(Fn & f) {
+ if constexpr (I < N) {
+ f(std::integral_constant<int, I>{});
+ static_for<I + 1, N>::run(f);
+ }
+ }
+};
+
+// One field of a format's block. In the reorder (SoA) layout it is a stream of its own, `size` bytes
+// a block, after `pre` bytes a block of the streams before it; in the canonical (AoS) layout it is
+// `aos` bytes into the block.
+struct field {
+ int pre;
+ int size;
+ int aos;
+};
+
+// Where one thread's rows sit in an expert's slice, in either layout. nblocks counts the stored blocks
+// of the slice, bpr those of one row, bs is the size of one block.
+template <int MR, bool L2D> struct slice {
+ const uint8_t * x;
+ int M;
+ int bpr;
+ int nblocks;
+ int m0;
+ bool aos;
+ int bs;
+
+ // per row, the byte offset from x of byte `inner` of field f of block ib
+ ESIMD_INLINE simd<uint32_t, MR> at(const field & f, const int ib, const int inner) const {
+ const int stride = aos ? bs : f.size;
+ simd<uint32_t, MR> o(0, (uint32_t) (bpr * stride));
+ return o + (uint32_t) ((aos ? f.aos : nblocks * f.pre) + (m0 * bpr + ib) * stride + inner);
+ }
+
+ // a field's bytes at offset `inner` start on a dword in every block (the AoS block and offset
+ // are multiples of 4); the SoA streams always do
+ ESIMD_INLINE bool dword_aligned(const field & f, const int inner) const {
+ return !aos || (bs % 4 == 0 && (f.aos + inner) % 4 == 0);
+ }
+
+ ESIMD_INLINE simd<uint16_t, MR> u16(const field & f, const int ib, const int inner) const {
+ return gather<uint16_t, MR>((const uint16_t *) x, at(f, ib, inner));
+ }
+
+ ESIMD_INLINE simd<uint32_t, MR> u32(const field & f, const int ib, const int inner) const {
+ if (dword_aligned(f, inner)) {
+ return gather<uint32_t, MR>((const uint32_t *) x, at(f, ib, inner));
+ }
+ return rows<1>(f, ib, inner);
+ }
+
+ // W dwords of each of the MR rows from byte `inner` (a multiple of 4) of field f of block ib, as
+ // [row][W]. SoA with 2D block loads (L2D): one message over the stream. Otherwise the rows' dwords
+ // are gathered, as [W][row], at offsets rounded down to a dword (every field starts on an even
+ // byte; the SoA streams are dword aligned), shifted by 16 bits where an AoS row's block starts on
+ // 2 mod 4, and transposed with strided reads.
+ template <int W> ESIMD_INLINE simd<uint32_t, MR * W> rows(const field & f, const int ib, const int inner) const {
+ if constexpr (L2D) {
+ if (!aos) {
+ const unsigned pitch = (unsigned) (bpr * f.size);
+ return load_2d<uint32_t, W, MR>((const uint32_t *) (x + (size_t) nblocks * f.pre), pitch - 1, M - 1,
+ pitch - 1, (ib * f.size + inner) / 4, m0);
+ }
+ }
+ const simd<uint32_t, MR> o = at(f, ib, inner);
+ simd<uint32_t, MR * W> g;
+ if (dword_aligned(f, inner)) {
+ g = gather<uint32_t, MR * W, W>((const uint32_t *) x, o);
+ } else {
+ // per row: sh is 16 where the run starts on 2 mod 4, else 0; hm keeps the next dword's low
+ // half only where it is needed
+ const simd<uint32_t, MR> oa = o & ~3u;
+ const simd<uint32_t, MR> sh = (o & 2u) << 3;
+ const simd<uint32_t, MR> hm = 0u - (sh >> 4);
+ simd<uint32_t, MR * W> lo = gather<uint32_t, MR * W, W>((const uint32_t *) x, oa);
+ // the dword after the run holds its last 2 bytes where a row needs it; a row that does not
+ // reads its first dword again rather than past the field
+ const simd<uint32_t, MR> tail = gather<uint32_t, MR>((const uint32_t *) x, oa + ((4u * W) & hm));
+ auto shift = [&](auto vc) SYCL_ESIMD_FUNCTION {
+ constexpr int v = decltype(vc)::value;
+ simd<uint32_t, MR> a = lo.template select<MR, 1>(v * MR);
+ simd<uint32_t, MR> b;
+ if constexpr (v + 1 < W) {
+ b = lo.template select<MR, 1>((v + 1) * MR);
+ } else {
+ b = tail;
+ }
+ g.template select<MR, 1>(v * MR) = (a >> sh) | ((b << 16) & hm);
+ };
+ static_for<0, W>::run(shift);
+ }
+ if constexpr (W == 1) {
+ return g;
+ } else {
+ simd<uint32_t, MR * W> t;
+ auto row = [&](auto rc) SYCL_ESIMD_FUNCTION {
+ constexpr int r = decltype(rc)::value;
+ t.template select<W, 1>(r * W) = g.template select<W, MR>(r);
+ };
+ static_for<0, MR>::run(row);
+ return t;
+ }
+ }
+};
+
+// A decode returns its row group's A in the order the two dpas of a k step read it, [k half][row][16
+// k]: both iq formats store k and k + 16 in the same byte or grid dword half, so that order needs
+// only reads with a stride, while [row][32 k] needed strided writes, which split into SIMD4.
+using a_halves = simd<sycl::half, GROUPED_GEMM_RC * GROUPED_GEMM_QK>;
+
+// f16 of the f32 product code * scale, each row's scale over its 16 values of a k half: the
+// expression the joint_matrix stages evaluate before E::make rounds it
+ESIMD_INLINE simd<sycl::half, GROUPED_GEMM_RC * 16> scale_half(simd<float, GROUPED_GEMM_RC * 16> v,
+ simd<float, GROUPED_GEMM_RC> d) {
+ v *= d.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+ return v;
+}
+
+// The iq4 codebook by byte, in SLM: entry b holds the values of its low and its high nibble as the
+// two 16-bit halves of one dword, as T (half, or int16 for an f32 product). A read-only table every
+// lane indexes, which is what SLM is for: register-indirect iselect() over the same lanes cost
+// 7.3 of 12.7 ms, a 16-entry SLM table per nibble about 0.4.
+template <typename T> ESIMD_INLINE void iq4_pair_table_to_slm() {
+ static_assert(sizeof(T) == 2, "two values a dword");
+ simd<uint32_t, 16> lo;
+#pragma unroll
+ for (int i = 0; i < 16; ++i) {
+ lo[i] = sycl::bit_cast<uint16_t>((T) kvalues_iq4nl[i]);
+ }
+#pragma unroll
+ for (int hi = 0; hi < 16; ++hi) {
+ slm_block_store<uint32_t, 16>(hi * 64, lo | ((uint32_t) sycl::bit_cast<uint16_t>((T) kvalues_iq4nl[hi]) << 16));
+ }
+}
+
+// 16 packed bytes a row; byte j holds k = j in its low and k = 16 + j in its high nibble (the order
+// of fg_decode_iq4_xs and of the iq4_nl stage). One gather per byte; the low halves are k half 0,
+// the high halves k half 1, each [row][16].
+template <typename T>
+ESIMD_INLINE simd<T, GROUPED_GEMM_RC * GROUPED_GEMM_QK> iq4_codes(simd<uint32_t, GROUPED_GEMM_RC * 4> q) {
+ simd<uint32_t, GROUPED_GEMM_RC * 16> off = q.template bit_cast_view<uint8_t>();
+ simd<uint32_t, GROUPED_GEMM_RC * 16> pr = slm_gather<uint32_t, GROUPED_GEMM_RC * 16>(off * 4);
+ auto pv = pr.template bit_cast_view<T>();
+ simd<T, GROUPED_GEMM_RC * GROUPED_GEMM_QK> v;
+ v.template select<GROUPED_GEMM_RC * 16, 1>(0) = pv.template select<GROUPED_GEMM_RC * 16, 2>(0);
+ v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) = pv.template select<GROUPED_GEMM_RC * 16, 2>(1);
+ return v;
+}
+// A 2D surface: rows `pitch` bytes apart, 16-byte aligned and at least 64 bytes
+constexpr bool surface_ok(const int pitch) {
+ return pitch % 16 == 0 && pitch >= 64;
+}
+
+// byte i of each lane's dword, i a runtime index: a shift by a scalar, not a register pick
+template <int MR> ESIMD_INLINE simd<uint32_t, MR> byte_of(simd<uint32_t, MR> w, const int i) {
+ return (w >> (8 * i)) & 0xffu;
+}
+
+// one of four per-lane dwords by a runtime index, as uniform branches rather than a register pick
+template <int MR> ESIMD_INLINE simd<uint32_t, MR> dword_of(simd<uint32_t, MR> * w, const int i) {
+ simd<uint32_t, MR> r = w[0];
+ if (i == 1) {
+ r = w[1];
+ } else if (i == 2) {
+ r = w[2];
+ } else if (i == 3) {
+ r = w[3];
+ }
+ return r;
+}
+
+// a row group's 32 k-ordered code bytes a row as f32, in [k half][row][16 k]
+ESIMD_INLINE simd<float, GROUPED_GEMM_RC * GROUPED_GEMM_QK> byte_codes(simd<uint32_t, GROUPED_GEMM_RC * 8> q) {
+ auto bm = q.template bit_cast_view<uint8_t, GROUPED_GEMM_RC, GROUPED_GEMM_QK>();
+ simd<float, GROUPED_GEMM_RC * GROUPED_GEMM_QK> v;
+ v.template select<GROUPED_GEMM_RC * 16, 1>(0) = bm.template select<GROUPED_GEMM_RC, 1, 16, 1>(0, 0);
+ v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) =
+ bm.template select<GROUPED_GEMM_RC, 1, 16, 1>(0, 16);
+ return v;
+}
+
+// f16 of d * code - m in f32, each row's d and m over its 16 values of a k half: the expression of
+// the K-quant stages with a min
+ESIMD_INLINE simd<sycl::half, GROUPED_GEMM_RC * 16> scale_min_half(simd<float, GROUPED_GEMM_RC * 16> v,
+ simd<float, GROUPED_GEMM_RC> d,
+ simd<float, GROUPED_GEMM_RC> m) {
+ v = v * d.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0) -
+ m.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+ return v;
+}
+
+// the two k halves of a row group, the code bytes times one (d, m) per row and half
+ESIMD_INLINE a_halves halves_dm(simd<uint32_t, GROUPED_GEMM_RC * 8> q,
+ simd<float, GROUPED_GEMM_RC> d0,
+ simd<float, GROUPED_GEMM_RC> m0,
+ simd<float, GROUPED_GEMM_RC> d1,
+ simd<float, GROUPED_GEMM_RC> m1,
+ const float bias = 0.0f) {
+ simd<float, GROUPED_GEMM_RC * GROUPED_GEMM_QK> c = byte_codes(q);
+ if (bias != 0.0f) {
+ c -= bias;
+ }
+ a_halves v;
+ v.template select<GROUPED_GEMM_RC * 16, 1>(0) =
+ scale_min_half(c.template select<GROUPED_GEMM_RC * 16, 1>(0), d0, m0);
+ v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) =
+ scale_min_half(c.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16), d1, m1);
+ return v;
+}
+
+// a row group's 32 values from grid dwords (byte b of entry e is k = 4e + b) and a sign dword per
+// row (bit k negates k), times d: the expression of the iq3 stages
+ESIMD_INLINE a_halves grid_signed(simd<uint32_t, GROUPED_GEMM_RC * 8> ge,
+ simd<uint32_t, GROUPED_GEMM_RC> signs,
+ simd<float, GROUPED_GEMM_RC> d) {
+ auto gm = ge.template bit_cast_view<uint32_t, GROUPED_GEMM_RC, 8>();
+ simd<uint32_t, GROUPED_GEMM_RC * 16> sg = signs.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+ simd<uint32_t, GROUPED_GEMM_RC * 16> k(0, 1);
+ k &= 15;
+ a_halves v;
+#pragma unroll
+ for (int h = 0; h < 2; ++h) {
+ simd<uint32_t, GROUPED_GEMM_RC * 4> eh = gm.template select<GROUPED_GEMM_RC, 1, 4, 1>(0, 4 * h);
+ simd<float, GROUPED_GEMM_RC * 16> f = eh.template bit_cast_view<uint8_t>();
+ f.merge(-f, ((sg >> (k + 16 * h)) & 1) != 0);
+ v.template select<GROUPED_GEMM_RC * 16, 1>(h * GROUPED_GEMM_RC * 16) = scale_half(f, d);
+ }
+ return v;
+}
+} // namespace grouped_gemm_esimd
+
+template <typename block_q_t, int MR = GROUPED_GEMM_MR> struct grouped_gemm_esimd_a {
+ static constexpr bool supported = false;
+};
+
+// [qs 32][d], one block a k step
+template <int MR> struct grouped_gemm_esimd_a<block_q8_0, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK8_0;
+ static constexpr bool whole = true; // decode_all() for the whole tile
+ static constexpr int slm_bytes = 0;
+ static ESIMD_INLINE void init_slm() {}
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+ grouped_gemm_esimd::simd<uint16_t, MR> d;
+ };
+
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK8_0, 2 }, D = { QK8_0, 2, 0 };
+ static bool ok(const int K, const bool aos) { return K % qk == 0 && (aos || K >= 64); }
+ struct sb_t {};
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t &, const Sl &, const int) {}
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t &, const Sl & s, const int kb, const int) {
+ r.q = s.template rows<8>(QS, kb, 0);
+ r.d = s.u16(D, kb, 0);
+ }
+ // all MR x 32 quants at once: int8 -> half, times the row's scale. q * d needs at most 19
+ // significant bits, so the f32 product the joint_matrix stage rounds to half is exact and one
+ // correctly rounded half multiply gives the same bits.
+ static ESIMD_INLINE grouped_gemm_esimd::simd<sycl::half, MR * GROUPED_GEMM_QK> decode_all(raw & r) {
+ using namespace grouped_gemm_esimd;
+ simd<sycl::half, MR * GROUPED_GEMM_QK> av = r.q.template bit_cast_view<int8_t>();
+ simd<sycl::half, MR> dh = r.d.template bit_cast_view<sycl::half>();
+ av *= dh.template replicate_vs_w_hs<MR, 1, GROUPED_GEMM_QK, 0>(0);
+ return av;
+ }
+};
+
+// [qs 16][d], one block a k step
+template <int MR> struct grouped_gemm_esimd_a<block_iq4_nl, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK4_NL;
+ static constexpr bool whole = false;
+ static constexpr int slm_bytes = 256 * sizeof(uint32_t);
+
+ static ESIMD_INLINE void init_slm() { grouped_gemm_esimd::iq4_pair_table_to_slm<sycl::half>(); }
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 4> q;
+ grouped_gemm_esimd::simd<sycl::half, MR> d;
+ };
+
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK4_NL / 2, 2 }, D = { QK4_NL / 2, 2, 0 };
+ static bool ok(const int K, const bool aos) { return K % qk == 0 && (aos || K / 2 >= 64); }
+ struct sb_t {};
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t &, const Sl &, const int) {}
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t &, const Sl & s, const int kb, const int) {
+ using namespace grouped_gemm_esimd;
+ r.q = s.template rows<4>(QS, kb, 0);
+ simd<uint16_t, MR> d = s.u16(D, kb, 0);
+ r.d = d.template bit_cast_view<sycl::half>();
+ }
+ // the codebook values are integers below 128, exact in half, and d * value has at most 18
+ // significant bits: one correctly rounded half multiply is the stage's f32 product rounded
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ a_halves v = iq4_codes<sycl::half>(r.q.template select<GROUPED_GEMM_RC * 4, 1>(g * GROUPED_GEMM_RC * 4));
+ simd<sycl::half, GROUPED_GEMM_RC> dg = r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+ simd<sycl::half, GROUPED_GEMM_RC * 16> dr = dg.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 16, 0>(0);
+ v.template select<GROUPED_GEMM_RC * 16, 1>(0) *= dr;
+ v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) *= dr;
+ return v;
+ }
+};
+
+// [qs 128][d 2][scales_h 2][scales_l 4] per 256 values; k step ib8 of a superblock is sub-block ib8
+template <int MR> struct grouped_gemm_esimd_a<block_iq4_xs, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ // int16 table and f32 products: d * (scale - 32) * value can need 24 significant bits, more
+ // than a half multiply keeps
+ static constexpr int slm_bytes = 256 * sizeof(uint32_t);
+
+ static ESIMD_INLINE void init_slm() { grouped_gemm_esimd::iq4_pair_table_to_slm<int16_t>(); }
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 4> q;
+ grouped_gemm_esimd::simd<float, MR> d;
+ };
+ struct sb_t {
+ grouped_gemm_esimd::simd<float, MR> d;
+ grouped_gemm_esimd::simd<uint32_t, MR> sh; // scales_h
+ grouped_gemm_esimd::simd<uint32_t, MR> sl; // scales_l, 4 bytes
+ };
+
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 8 }, D = { QK_K / 2, 2, 0 },
+ SH = { QK_K / 2 + 2, 2, 2 }, SL = { QK_K / 2 + 4, 4, 4 };
+ static bool ok(const int K, const bool) { return K % qk == 0; }
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+ using namespace grouped_gemm_esimd;
+ simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+ sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+ sb.sh = s.u16(SH, ib, 0);
+ sb.sl = s.u32(SL, ib, 0);
+ }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ r.q = s.template rows<4>(QS, kb / 8, 16 * j);
+ // d * (6-bit scale - 32), in f32 as the stage computes it
+ simd<int32_t, MR> sc = simd<int32_t, MR>(((sb.sl >> (4 * j)) & 0xf) | (((sb.sh >> (2 * j)) & 3) << 4)) - 32;
+ r.d = sb.d * simd<float, MR>(sc);
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ simd<int16_t, GROUPED_GEMM_RC * GROUPED_GEMM_QK> c =
+ iq4_codes<int16_t>(r.q.template select<GROUPED_GEMM_RC * 4, 1>(g * GROUPED_GEMM_RC * 4));
+ simd<float, GROUPED_GEMM_RC> dg = r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+ a_halves v;
+ v.template select<GROUPED_GEMM_RC * 16, 1>(0) = scale_half(c.template select<GROUPED_GEMM_RC * 16, 1>(0), dg);
+ v.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16) =
+ scale_half(c.template select<GROUPED_GEMM_RC * 16, 1>(GROUPED_GEMM_RC * 16), dg);
+ return v;
+ }
+};
+
+// [qs 64][qh 8][signs 32][d 2, scales 4] per 256 values. Value k of sub-block ib8 is byte k % 4 of
+// grid entry k / 4, the entry index being qs[k / 4] with bit 8 from qh bit k / 4, negated where bit
+// k of the sub-block's sign dword is set; times d * (1 + 2 * 4-bit scale)
+template <int MR> struct grouped_gemm_esimd_a<block_iq3_s, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ // the 512-entry grid in SLM, read by every lane: the global gather of it cost 2.4x the kernel
+ static constexpr int slm_bytes = 512 * sizeof(uint32_t);
+ static ESIMD_INLINE void init_slm() {
+ using namespace grouped_gemm_esimd;
+#pragma unroll
+ for (int i = 0; i < 512; i += 64) {
+ slm_block_store<uint32_t, 64>(i * 4, block_load<uint32_t, 64>(iq3s_grid + i));
+ }
+ }
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 2> q; // [row][8 qs bytes]
+ grouped_gemm_esimd::simd<uint32_t, MR> qh; // the sub-block's qh byte
+ grouped_gemm_esimd::simd<uint32_t, MR> signs; // the sub-block's 32 sign bits
+ grouped_gemm_esimd::simd<float, MR> d;
+ };
+ // d and the scale bytes are a block's, kept over its k steps; the sign dword and the qh byte of
+ // a k step are a 2D load each at that step's column, so no register is picked by a runtime index
+ struct sb_t {
+ grouped_gemm_esimd::simd<float, MR> d;
+ grouped_gemm_esimd::simd<uint32_t, MR> sc; // the 4 scale bytes
+ };
+ // the SoA stream {d, scales} is d and scales of the block in AoS, which are not adjacent there: MS
+ // reads the scales at their SoA offset 2 in the stream, which is AoS offset 2 past 104
+ static constexpr int META = QK_K / 4 + QK_K / 32 + QK_K / 8;
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, 2 }, QH = { QK_K / 4, QK_K / 32, 2 + QK_K / 4 },
+ SG = { QK_K / 4 + QK_K / 32, QK_K / 8, 2 + QK_K / 4 + QK_K / 32 },
+ MD = { META, 2 + IQ3S_N_SCALE, 0 },
+ MS = { META, 2 + IQ3S_N_SCALE, META };
+ // the SoA qh and signs streams are 2D surfaces: their row pitches must be 16-byte aligned
+ static bool ok(const int K, const bool aos) {
+ return K % qk == 0 && (aos || ((K / QK_K * 8) % 16 == 0 && K / QK_K * 8 >= 64));
+ }
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+ using namespace grouped_gemm_esimd;
+ simd<uint16_t, MR> dh = s.u16(MD, ib, 0);
+ simd<uint32_t, MR> s0 = s.u16(MS, ib, 2);
+ simd<uint32_t, MR> s1 = s.u16(MS, ib, 4);
+ sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+ sb.sc = s0 | (s1 << 16);
+ }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ const int ib = kb / 8;
+ r.q = s.template rows<2>(QS, ib, 8 * j);
+ r.qh = (s.template rows<1>(QH, ib, 4 * (j / 4)) >> (8 * (j % 4))) & 0xffu;
+ r.signs = s.template rows<1>(SG, ib, 4 * j);
+ r.d = sb.d * simd<float, MR>(simd<int32_t, MR>(1 + 2 * ((sb.sc >> (4 * j)) & 0xf)));
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ // the 8 grid entries of each of the row group's rows
+ simd<uint16_t, GROUPED_GEMM_RC * 8> qb =
+ r.q.template select<GROUPED_GEMM_RC * 2, 1>(g * GROUPED_GEMM_RC * 2).template bit_cast_view<uint8_t>();
+ simd<uint32_t, GROUPED_GEMM_RC> qr = r.qh.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+ simd<uint32_t, GROUPED_GEMM_RC * 8> qh = qr.template replicate_vs_w_hs<GROUPED_GEMM_RC, 1, 8, 0>(0);
+ simd<uint32_t, GROUPED_GEMM_RC * 8> j(0, 1);
+ j &= 7;
+ simd<uint32_t, GROUPED_GEMM_RC * 8> idx = simd<uint32_t, GROUPED_GEMM_RC * 8>(qb) | (((qh >> j) & 1) << 8);
+ simd<uint32_t, GROUPED_GEMM_RC * 8> ge = slm_gather<uint32_t, GROUPED_GEMM_RC * 8>(idx * 4);
+ return grid_signed(ge, r.signs.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+ r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC));
+ }
+};
+
+// The K-quants and iq3_xxs: a 256-value superblock, k step j its sub-block j. Their value bytes for
+// a k step are one rectangle of a stream, in k order after a shift and a mask per dword; scales,
+// mins and d are the block's and are loaded once per block.
+
+// [qs 128][scales 12][dm]: value l of sub-block j is nibble j & 1 of byte 32 (j / 2) + l, times
+// dall * scale_j, minus dmin * min_j (6-bit scale and min, get_scale_min_k4)
+template <int MR> struct grouped_gemm_esimd_a<block_q4_K, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ static constexpr int slm_bytes = 0;
+ static ESIMD_INLINE void init_slm() {}
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+ grouped_gemm_esimd::simd<float, MR> d, m;
+ };
+ struct sb_t {
+ grouped_gemm_esimd::simd<uint32_t, MR> sc[3]; // the 12 scale bytes
+ grouped_gemm_esimd::simd<float, MR> dall, dmin;
+ };
+
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 16 }, SC = { QK_K / 2, K_SCALE_SIZE, 4 },
+ DM = { QK_K / 2 + K_SCALE_SIZE, 4, 0 };
+
+ static bool ok(const int K, const bool aos) {
+ return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / 2));
+ }
+
+ template <typename Sl>
+ static ESIMD_INLINE void load_sb(sb_t & sb,
+ const Sl & s,
+ const int ib,
+ const grouped_gemm_esimd::field & sc = SC,
+ const grouped_gemm_esimd::field & dmf = DM) {
+ using namespace grouped_gemm_esimd;
+#pragma unroll
+ for (int w = 0; w < 3; ++w) {
+ sb.sc[w] = s.u32(sc, ib, 4 * w);
+ }
+ simd<uint32_t, MR> dm = s.u32(dmf, ib, 0);
+ auto dh = dm.template bit_cast_view<sycl::half>();
+ sb.dall = simd<sycl::half, MR>(dh.template select<MR, 2>(0));
+ sb.dmin = simd<sycl::half, MR>(dh.template select<MR, 2>(1));
+ }
+ // get_scale_min_k4 for sub-block j over the lanes
+ static ESIMD_INLINE void scale_min(sb_t & sb,
+ const int j,
+ grouped_gemm_esimd::simd<float, MR> & d,
+ grouped_gemm_esimd::simd<float, MR> & m) {
+ using namespace grouped_gemm_esimd;
+ simd<uint32_t, MR> sc, mb;
+ if (j < 4) {
+ sc = byte_of(sb.sc[0], j) & 63;
+ mb = byte_of(sb.sc[1], j) & 63;
+ } else {
+ sc = (byte_of(sb.sc[2], j - 4) & 0xf) | ((byte_of(sb.sc[0], j - 4) >> 6) << 4);
+ mb = (byte_of(sb.sc[2], j - 4) >> 4) | ((byte_of(sb.sc[1], j - 4) >> 6) << 4);
+ }
+ d = sb.dall * simd<float, MR>(sc);
+ m = sb.dmin * simd<float, MR>(mb);
+ }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ r.q = (s.template rows<8>(QS, kb / 8, 32 * (j / 2)) >> (4 * (j & 1))) & 0x0F0F0F0Fu;
+ scale_min(sb, j, r.d, r.m);
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ simd<float, GROUPED_GEMM_RC> d = r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+ m = r.m.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+ return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8), d, m, d, m);
+ }
+};
+
+// [qs 128][qh 32][scales 12][dm]: q4_K plus bit j of qh byte l as 16
+template <int MR> struct grouped_gemm_esimd_a<block_q5_K, MR> {
+ using q4 = grouped_gemm_esimd_a<block_q4_K, MR>;
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ static constexpr int slm_bytes = 0;
+ static ESIMD_INLINE void init_slm() {}
+ using raw = typename q4::raw;
+ using sb_t = typename q4::sb_t;
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 2, 48 }, QH = { QK_K / 2, QK_K / 8, 16 },
+ SC = { QK_K / 2 + QK_K / 8, K_SCALE_SIZE, 4 },
+ DM = { QK_K / 2 + QK_K / 8 + K_SCALE_SIZE, 4, 0 };
+ static bool ok(const int K, const bool aos) {
+ return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 2) && grouped_gemm_esimd::surface_ok(K / 8)));
+ }
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) { q4::load_sb(sb, s, ib, SC, DM); }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ const int ib = kb / 8;
+ simd<uint32_t, MR * 8> qh = s.template rows<8>(QH, ib, 0);
+ r.q = ((s.template rows<8>(QS, ib, 32 * (j / 2)) >> (4 * (j & 1))) & 0x0F0F0F0Fu) |
+ (((qh >> j) & 0x01010101u) << 4);
+ q4::scale_min(sb, j, r.d, r.m);
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ return q4::template decode<g>(r);
+ }
+};
+
+// [ql 128][qh 64][scales 16][d]: sub-block j is quarter r = j % 4 of half h = j / 4; value l is
+// nibble r >> 1 of ql byte 64h + 32 (r & 1) + l with bits 2r, 2r+1 of qh byte 32h + l above it,
+// minus 32, times d * the int8 scale 2j + l / 16
+template <int MR> struct grouped_gemm_esimd_a<block_q6_K, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ static constexpr int slm_bytes = 0;
+ static ESIMD_INLINE void init_slm() {}
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+ grouped_gemm_esimd::simd<float, MR> d0, d1; // per k half
+ };
+ struct sb_t {
+ grouped_gemm_esimd::simd<uint32_t, MR> sc[4]; // the 16 int8 scales
+ grouped_gemm_esimd::simd<float, MR> d;
+ };
+
+ static constexpr grouped_gemm_esimd::field QL = { 0, QK_K / 2, 0 }, QH = { QK_K / 2, QK_K / 4, QK_K / 2 },
+ SC = { QK_K / 2 + QK_K / 4, QK_K / 16, QK_K / 2 + QK_K / 4 },
+ D = { QK_K / 2 + QK_K / 4 + QK_K / 16, 2,
+ QK_K / 2 + QK_K / 4 + QK_K / 16 };
+ static bool ok(const int K, const bool aos) {
+ return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 2) && grouped_gemm_esimd::surface_ok(K / 4)));
+ }
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+ using namespace grouped_gemm_esimd;
+#pragma unroll
+ for (int w = 0; w < 4; ++w) {
+ sb.sc[w] = s.u32(SC, ib, 4 * w);
+ }
+ simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+ sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+ }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ const int ib = kb / 8, h = j / 4, q = j % 4;
+ simd<uint32_t, MR * 8> ql = s.template rows<8>(QL, ib, 64 * h + 32 * (q & 1));
+ simd<uint32_t, MR * 8> qh = s.template rows<8>(QH, ib, 32 * h);
+ r.q = ((ql >> (4 * (q >> 1))) & 0x0F0F0F0Fu) | (((qh >> (2 * q)) & 0x03030303u) << 4);
+ // scales 2j and 2j + 1: bytes 2 (j % 2) and 2 (j % 2) + 1 of dword j / 2, sign extended
+ simd<uint32_t, MR> w = dword_of(sb.sc, j / 2);
+ simd<int32_t, MR> s0 = simd<int32_t, MR>(w << (24 - 16 * (j % 2))) >> 24;
+ simd<int32_t, MR> s1 = simd<int32_t, MR>(w << (16 - 16 * (j % 2))) >> 24;
+ r.d0 = sb.d * simd<float, MR>(s0);
+ r.d1 = sb.d * simd<float, MR>(s1);
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ simd<float, GROUPED_GEMM_RC> z = 0.0f;
+ return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8),
+ r.d0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z,
+ r.d1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z, 32.0f);
+ }
+};
+
+// [qs 64][scales 16][dm]: value l of sub-block j is bits 2 (j % 4) of byte 32 (j / 4) + l, times
+// d * (scale & 15), minus dmin * (scale >> 4), scale byte 2j + l / 16
+template <int MR> struct grouped_gemm_esimd_a<block_q2_K, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ static constexpr int slm_bytes = 0;
+ static ESIMD_INLINE void init_slm() {}
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+ grouped_gemm_esimd::simd<float, MR> d0, m0, d1, m1;
+ };
+ struct sb_t {
+ grouped_gemm_esimd::simd<uint32_t, MR> sc[4];
+ grouped_gemm_esimd::simd<float, MR> d, dmin;
+ };
+
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, QK_K / 16 }, SC = { QK_K / 4, QK_K / 16, 0 },
+ DM = { QK_K / 4 + QK_K / 16, 4, QK_K / 4 + QK_K / 16 };
+
+ static bool ok(const int K, const bool aos) {
+ return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / 4));
+ }
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+ using namespace grouped_gemm_esimd;
+#pragma unroll
+ for (int w = 0; w < 4; ++w) {
+ sb.sc[w] = s.u32(SC, ib, 4 * w);
+ }
+ simd<uint32_t, MR> dm = s.u32(DM, ib, 0);
+ auto dh = dm.template bit_cast_view<sycl::half>();
+ sb.d = simd<sycl::half, MR>(dh.template select<MR, 2>(0));
+ sb.dmin = simd<sycl::half, MR>(dh.template select<MR, 2>(1));
+ }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ r.q = (s.template rows<8>(QS, kb / 8, 32 * (j / 4)) >> (2 * (j % 4))) & 0x03030303u;
+ simd<uint32_t, MR> w = dword_of(sb.sc, j / 2);
+ simd<uint32_t, MR> s0 = byte_of(w, 2 * (j % 2)), s1 = byte_of(w, 2 * (j % 2) + 1);
+ r.d0 = sb.d * simd<float, MR>(s0 & 0xf);
+ r.m0 = sb.dmin * simd<float, MR>(s0 >> 4);
+ r.d1 = sb.d * simd<float, MR>(s1 & 0xf);
+ r.m1 = sb.dmin * simd<float, MR>(s1 >> 4);
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8),
+ r.d0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+ r.m0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+ r.d1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC),
+ r.m1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC));
+ }
+};
+
+// [qs 64][hmask 32][scales 12][d]: q2_K's 2 bits, minus 4 where bit j of hmask byte l is clear,
+// times d * (6-bit scale - 32), scale s = 2j + l / 16 split over bytes s % 8 and 8 + s % 4
+template <int MR> struct grouped_gemm_esimd_a<block_q3_K, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ static constexpr int slm_bytes = 0;
+ static ESIMD_INLINE void init_slm() {}
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 8> q;
+ grouped_gemm_esimd::simd<float, MR> d0, d1;
+ };
+ struct sb_t {
+ grouped_gemm_esimd::simd<uint32_t, MR> sc[4]; // 12 scale bytes, sc[3] unused
+ grouped_gemm_esimd::simd<float, MR> d;
+ };
+
+ static constexpr grouped_gemm_esimd::field QS = { 0, QK_K / 4, QK_K / 8 }, HM = { QK_K / 4, QK_K / 8, 0 },
+ SC = { QK_K / 4 + QK_K / 8, 12, QK_K / 4 + QK_K / 8 },
+ D = { QK_K / 4 + QK_K / 8 + 12, 2, QK_K / 4 + QK_K / 8 + 12 };
+ static bool ok(const int K, const bool aos) {
+ return K % qk == 0 && (aos || (grouped_gemm_esimd::surface_ok(K / 4) && grouped_gemm_esimd::surface_ok(K / 8)));
+ }
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+ using namespace grouped_gemm_esimd;
+#pragma unroll
+ for (int w = 0; w < 3; ++w) {
+ sb.sc[w] = s.u32(SC, ib, 4 * w);
+ }
+ sb.sc[3] = 0;
+ simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+ sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+ }
+
+ static ESIMD_INLINE grouped_gemm_esimd::simd<float, MR> scale(sb_t & sb, const int sidx) {
+ using namespace grouped_gemm_esimd;
+ simd<uint32_t, MR> lo = (byte_of(dword_of(sb.sc, (sidx % 8) / 4), sidx % 4) >> (4 * (sidx / 8))) & 0xf;
+ simd<uint32_t, MR> hi = (byte_of(sb.sc[2], sidx % 4) >> (2 * (sidx / 4))) & 3;
+ return sb.d * simd<float, MR>(simd<int32_t, MR>(lo | (hi << 4)) - 32);
+ }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ const int ib = kb / 8;
+ simd<uint32_t, MR * 8> hm = s.template rows<8>(HM, ib, 0);
+ r.q = ((s.template rows<8>(QS, ib, 32 * (j / 4)) >> (2 * (j % 4))) & 0x03030303u) |
+ (((hm >> j) & 0x01010101u) << 2);
+ r.d0 = scale(sb, 2 * j);
+ r.d1 = scale(sb, 2 * j + 1);
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ simd<float, GROUPED_GEMM_RC> z = 0.0f;
+ return halves_dm(r.q.template select<GROUPED_GEMM_RC * 8, 1>(g * GROUPED_GEMM_RC * 8),
+ r.d0.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z,
+ r.d1.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC), z, 4.0f);
+ }
+};
+
+// [qs 96][d]: per sub-block j, 8 grid indices at 8j and an aux dword at 64 + 4j whose 4 x 7 bits
+// are sign bytes of ksigns_iq2xs and whose top 4 bits scale d * (0.5 + s) * 0.5. ksigns_iq2xs[i] is
+// i with the parity of its 7 bits as bit 7, so the sign bytes are computed, not looked up.
+template <int MR> struct grouped_gemm_esimd_a<block_iq3_xxs, MR> {
+ static constexpr bool supported = true;
+ static constexpr int qk = QK_K;
+ static constexpr bool whole = false;
+ // the 256-entry grid, a read-only table in SLM
+ static constexpr int slm_bytes = 256 * sizeof(uint32_t);
+ static ESIMD_INLINE void init_slm() {
+ using namespace grouped_gemm_esimd;
+#pragma unroll
+ for (int i = 0; i < 256; i += 64) {
+ slm_block_store<uint32_t, 64>(i * 4, block_load<uint32_t, 64>(iq3xxs_grid + i));
+ }
+ }
+ struct raw {
+ grouped_gemm_esimd::simd<uint32_t, MR * 2> q; // [row][8 grid indices]
+ grouped_gemm_esimd::simd<uint32_t, MR> aux;
+ grouped_gemm_esimd::simd<float, MR> d;
+ };
+ struct sb_t {
+ grouped_gemm_esimd::simd<float, MR> d;
+ };
+
+ static constexpr grouped_gemm_esimd::field QS = { 0, 3 * QK_K / 8, 2 }, D = { 3 * QK_K / 8, 2, 0 };
+
+ static bool ok(const int K, const bool aos) {
+ return K % qk == 0 && (aos || grouped_gemm_esimd::surface_ok(K / QK_K * 96));
+ }
+ template <typename Sl> static ESIMD_INLINE void load_sb(sb_t & sb, const Sl & s, const int ib) {
+ using namespace grouped_gemm_esimd;
+ simd<uint16_t, MR> dh = s.u16(D, ib, 0);
+ sb.d = simd<sycl::half, MR>(dh.template bit_cast_view<sycl::half>());
+ }
+ template <typename Sl> static ESIMD_INLINE void load(raw & r, sb_t & sb, const Sl & s, const int kb, const int j) {
+ using namespace grouped_gemm_esimd;
+ const int ib = kb / 8;
+ r.q = s.template rows<2>(QS, ib, 8 * j);
+ r.aux = s.template rows<1>(QS, ib, 64 + 4 * j);
+ r.d = sb.d * (0.5f + simd<float, MR>(r.aux >> 28)) * 0.5f;
+ }
+
+ template <int g> static ESIMD_INLINE grouped_gemm_esimd::a_halves decode(raw & r) {
+ using namespace grouped_gemm_esimd;
+ simd<uint32_t, GROUPED_GEMM_RC * 8> qb =
+ r.q.template select<GROUPED_GEMM_RC * 2, 1>(g * GROUPED_GEMM_RC * 2).template bit_cast_view<uint8_t>();
+ simd<uint32_t, GROUPED_GEMM_RC * 8> ge = slm_gather<uint32_t, GROUPED_GEMM_RC * 8>(qb * 4);
+ simd<uint32_t, GROUPED_GEMM_RC> ax = r.aux.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC);
+ simd<uint32_t, GROUPED_GEMM_RC> sg = 0;
+#pragma unroll
+ for (int il = 0; il < 4; ++il) {
+ simd<uint32_t, GROUPED_GEMM_RC> si = (ax >> (7 * il)) & 127;
+ sg |= (si | ((cbit(si) & 1) << 7)) << (8 * il);
+ }
+ return grid_signed(ge, sg, r.d.template select<GROUPED_GEMM_RC, 1>(g * GROUPED_GEMM_RC));
+ }
+};
+
+// T is the dpas operand type, sycl::half or bfloat16: the type B is packed in. A is decoded in half
+// either way and converted for bf16 (see the double rounding note at the conversion). AOS: the weight
+// is in the canonical block layout rather than the reorder (SoA) one. A: the grouped_gemm_arch built for.
+template <typename block_q_t, int KS, typename T, bool AOS, typename A> struct grouped_gemm_esimd_kernel {
+ using F = grouped_gemm_esimd_a<block_q_t, A::MR>;
+
+ const char * src0_dd;
+ size_t expert_stride;
+ const ggml_sycl_grouped_gemm_tile * tiles;
+ const T * packed;
+ float * dst;
+ int M;
+ int Npad;
+ int K;
+ ggml_sycl_grouped_gemm_rows dst_rows;
+
+ void operator()(sycl::nd_item<1> it) const SYCL_ESIMD_KERNEL {
+ if constexpr (grouped_gemm_built<A>()) {
+ run(it);
+ }
+ }
+
+ ESIMD_INLINE void run(sycl::nd_item<1> it) const {
+ using namespace sycl::ext::intel::esimd;
+ namespace xmx = sycl::ext::intel::esimd::xmx;
+ constexpr int MR = A::MR;
+ constexpr int NW = A::NW;
+ constexpr int RC = GROUPED_GEMM_RC;
+ constexpr int RG = MR / RC; // dpas row groups of this thread
+ constexpr int BN = GGML_SYCL_GROUPED_GEMM_BN;
+ constexpr int NT = BN / NW;
+
+ if constexpr (F::slm_bytes > 0) {
+ // the format's lookup table; a work-group is one thread, so nothing else writes it
+ slm_init<F::slm_bytes>();
+ F::init_slm();
+ }
+
+ const int mblocks = M / MR;
+ const int gid = it.get_global_id(0);
+ const int t = gid / mblocks;
+ const int m0 = (gid % mblocks) * MR;
+
+ const ggml_sycl_grouped_gemm_tile tile = tiles[t];
+ if (tile.n1 <= tile.n0) {
+ return;
+ }
+ const int b0 = t * BN; // the tile's packed-B columns
+
+ // the slice of this expert
+ const int bpr = K / F::qk;
+ const grouped_gemm_esimd::slice<MR, A::L2D> sl = {
+ (const uint8_t *) src0_dd + (size_t) tile.expert * expert_stride,
+ M,
+ bpr,
+ M * bpr,
+ m0,
+ AOS,
+ (int) sizeof(block_q_t)
+ };
+ const uint32_t * bp = (const uint32_t *) packed;
+ const unsigned bw = (unsigned) Npad * 4; // bytes of one packed B row
+ const int nstep = K / GROUPED_GEMM_QK;
+
+ simd<float, RC * NW> acc[RG][NT];
+ simd<float, RC * NW> total[KS > 1 ? RG : 1][KS > 1 ? NT : 1];
+#pragma unroll
+ for (int g = 0; g < RG; ++g) {
+#pragma unroll
+ for (int nt = 0; nt < NT; ++nt) {
+ acc[g][nt] = 0.0f;
+ if constexpr (KS > 1) {
+ total[g][nt] = 0.0f;
+ }
+ }
+ }
+
+ // The tile's route map, gathered once up front; the K loop hides its latency, and the epilogue
+ // below reads it at compile-time indices.
+ const int ncols = tile.n1 - tile.n0;
+ simd_mask<BN> cm = simd<int32_t, BN>(0, 1) < ncols;
+ simd<int32_t, BN> i2 = 0, r1 = 0; // r1: the dst row slot, i1 % ne1
+ if (dst_rows.map) {
+ const int32_t * mp = (const int32_t *) (dst_rows.map + tile.n0);
+ simd<uint32_t, BN> mo(0, sizeof(mmid_row_mapping));
+ simd<int32_t, BN> i1 = gather<int32_t, BN>(mp, mo, cm);
+ i2 = gather<int32_t, BN>(mp + 1, mo, cm);
+ // once per thread in 32 bits: a 64-bit modulo per column is emulated, and at 20 columns a
+ // tile it cost more than the whole GEMM (7.2 vs 3.1 ms)
+ r1 = i1 % simd<int32_t, BN>((int32_t) dst_rows.ne1);
+ }
+
+ // One k step at a time: the format's stored bits for the MR rows, and the 16 k pairs x NW
+ // columns of packed B per dpas column tile, one 2D block message each (without 2D loads, one
+ // block load per packed row, cut into the tiles in registers). A second stage in
+ // flight spilled 7 KB of the 256 registers and ran 1.6x slower; the other threads of the
+ // XVE hide the load latency instead.
+ // A stored block's metadata (its sub-block scales) is loaded once per block. With KS > 1 the
+ // K split of the joint_matrix kernel is reproduced for bit-identical output (a second set of
+ // accumulators); KS == 1 sums the whole K in one, which is a different f32 rounding order and
+ // frees the 4 KB of registers the second set holds.
+ constexpr int KPB = F::qk / GROUPED_GEMM_QK; // k steps per stored block
+ int split = 0;
+ int split_end = KS > 1 ? nstep / KS : nstep;
+ typename F::sb_t sb;
+ for (int kb = 0; kb < nstep; ++kb) {
+ const int j = kb % KPB;
+ if (j == 0) {
+ F::load_sb(sb, sl, kb / KPB);
+ }
+ typename F::raw r;
+ F::load(r, sb, sl, kb, j);
+ simd<uint32_t, 16 * NW> b[NT]; // [16 k pairs][NW columns] each
+ if constexpr (A::L2D) {
+#pragma unroll
+ for (int nt = 0; nt < NT; ++nt) {
+ b[nt] = load_2d<uint32_t, NW, 16>(bp, bw - 1, K / 2 - 1, bw - 1, b0 + nt * NW, kb * 16);
+ }
+ } else {
+ simd<uint32_t, 16 * BN> bm;
+#pragma unroll
+ for (int r = 0; r < 16; ++r) {
+ bm.template select<BN, 1>(r * BN) =
+ block_load<uint32_t, BN>(bp + (size_t) (kb * 16 + r) * Npad + b0, overaligned_tag<16>{});
+ }
+ auto bv = bm.template bit_cast_view<uint32_t, 16, BN>();
+#pragma unroll
+ for (int nt = 0; nt < NT; ++nt) {
+ b[nt] = bv.template select<16, 1, NW, 1>(0, nt * NW);
+ }
+ }
+ simd<sycl::half, MR * GROUPED_GEMM_QK> all;
+ if constexpr (F::whole) {
+ all = F::decode_all(r);
+ }
+ // one row group of the thread, g a compile-time constant: with a runtime g the larger
+ // decodes were left rolled
+ auto row_group = [&](auto gc) SYCL_ESIMD_FUNCTION {
+ constexpr int g = decltype(gc)::value;
+ grouped_gemm_esimd::a_halves av;
+ if constexpr (F::whole) {
+ // [row][32 k] of the whole tile to this row group's [k half][row][16 k]
+ auto am = all.template bit_cast_view<sycl::half, MR, GROUPED_GEMM_QK>();
+ av.template select<RC * 16, 1>(0) = am.template select<RC, 1, 16, 1>(g * RC, 0);
+ av.template select<RC * 16, 1>(RC * 16) = am.template select<RC, 1, 16, 1>(g * RC, 16);
+ } else {
+ av = F::template decode<g>(r);
+ }
+#pragma unroll
+ for (int h = 0; h < 2; ++h) {
+ // A of this k16 half: rows x k [16h, 16h+16), row major
+ simd<sycl::half, RC * 16> ah = av.template select<RC * 16, 1>(h * RC * 16);
+ simd<T, RC * 16> at;
+ if constexpr (std::is_same_v<T, sycl::half>) {
+ at = ah;
+ } else {
+ // rounded twice (f32 to half in the decode, half to bf16 here), so A's rounding
+ // error is about twice the joint_matrix bf16 stage's, which rounds f32 once
+ at = simd<float, RC * 16>(ah);
+ }
+#pragma unroll
+ for (int nt = 0; nt < NT; ++nt) {
+ simd<uint32_t, 8 * NW> bh32 = b[nt].template select<8 * NW, 1>(h * 8 * NW);
+ simd<T, 16 * NW> bh = bh32.template bit_cast_view<T>();
+ acc[g][nt] = xmx::dpas<8, RC, float, float>(acc[g][nt], bh, at);
+ }
+ }
+ };
+ grouped_gemm_esimd::static_for<0, RG>::run(row_group);
+ if constexpr (KS > 1) {
+ // the K split's range ends, in k steps; a range's partial joins the total when it closes
+ if (kb + 1 == split_end) {
+#pragma unroll
+ for (int g = 0; g < RG; ++g) {
+#pragma unroll
+ for (int nt = 0; nt < NT; ++nt) {
+ total[g][nt] += acc[g][nt];
+ acc[g][nt] = 0.0f;
+ }
+ }
+ ++split;
+ split_end = ((split + 1) * nstep) / KS;
+ }
+ }
+ }
+ (void) split;
+ (void) split_end;
+
+ // column c of the tile: MR consecutive floats of one dst row. Unrolled, so c indexes the
+ // accumulators at compile time; a runtime index sends them all to private memory.
+#pragma unroll
+ for (int c = 0; c < BN; ++c) {
+ if (c >= ncols) {
+ continue;
+ }
+ simd<float, MR> col;
+#pragma unroll
+ for (int g = 0; g < RG; ++g) {
+ if constexpr (KS > 1) {
+ col.template select<RC, 1>(g * RC) = total[g][c / NW].template select<RC, NW>(c % NW);
+ } else {
+ col.template select<RC, 1>(g * RC) = acc[g][c / NW].template select<RC, NW>(c % NW);
+ }
+ }
+ const int n = tile.n0 + c;
+ if (dst_rows.map) {
+ float * out = (float *) (dst_rows.base + (size_t) r1[c] * dst_rows.nb1 + i2[c] * dst_rows.nb2) + m0;
+ block_store<float, MR>(out, col, element_aligned_tag{});
+ } else {
+ block_store<float, MR>(dst + (size_t) n * M + m0, col, element_aligned_tag{});
+ }
+ }
+ }
+
+ auto get(syclexp::properties_tag) const { return syclexp::properties{ sycl::ext::intel::experimental::grf_size<256> }; }
+};
+
+// the weights' surfaces: 2D block loads (SoA with L2D) want 64-byte aligned bases, the dword gathers 4
+static bool grouped_gemm_esimd_weights_ok(const char * w, const size_t expert_stride, const bool l2d) {
+ const size_t a = l2d ? 64 : 4;
+ return (uintptr_t) w % a == 0 && expert_stride % a == 0;
+}
+
+// 0 off, 1 on (default), 2 on with the joint_matrix kernel's K-split order reproduced (bit-identical output)
+static int ggml_sycl_grouped_gemm_esimd_mode() {
+ return g_ggml_sycl_mmid_esimd;
+}
+
+bool ggml_sycl_grouped_gemm_esimd_enabled() {
+#if defined(GGML_SYCL_SUPPORT_XMX)
+ return g_ggml_sycl_enable_esimd && ggml_sycl_grouped_gemm_esimd_mode() != 0;
+#else
+ return false;
+#endif
+}
+
+template <typename block_q_t, typename T, bool AOS, typename A>
+static void grouped_gemm_esimd_launch(const char * src0_dd,
+ size_t expert_stride,
+ const ggml_sycl_grouped_gemm_tile * tiles,
+ const T * packed,
+ float * dst,
+ int M,
+ int Npad,
+ int K,
+ const ggml_sycl_grouped_gemm_rows & dst_rows,
+ int64_t n_tiles,
+ dpct::queue_ptr stream) {
+ const sycl::nd_range<1> range((size_t) n_tiles * (M / A::MR), 1);
+ if (ggml_sycl_grouped_gemm_esimd_mode() == 2) {
+ stream->parallel_for(range, grouped_gemm_esimd_kernel<block_q_t, GGML_SYCL_GROUPED_GEMM_KSPLIT, T, AOS, A>{
+ src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, dst_rows });
+ } else {
+ stream->parallel_for(range, grouped_gemm_esimd_kernel<block_q_t, 1, T, AOS, A>{
+ src0_dd, expert_stride, tiles, packed, dst, M, Npad, K, dst_rows });
+ }
+}
+
+template <typename block_q_t, typename T, typename A>
+static bool grouped_gemm_esimd_run(const char * src0_dd,
+ size_t expert_stride,
+ const ggml_sycl_grouped_gemm_tile * tiles,
+ const T * packed,
+ float * dst,
+ int M,
+ int Npad,
+ int K,
+ const ggml_sycl_grouped_gemm_rows & dst_rows,
+ bool reordered,
+ int64_t n_tiles,
+ dpct::queue_ptr stream) {
+ using F = grouped_gemm_esimd_a<block_q_t, A::MR>;
+ // whole row groups and stored blocks, every K-split range non-empty (also in mode 1, so both
+ // modes take the same shapes); B is read with 2D block loads, which want 64-byte aligned
+ // surfaces and pitches, or with 16-byte aligned block loads of whole tiles
+ if (M % A::MR != 0 || !F::ok(K, !reordered) || K < GROUPED_GEMM_QK * GGML_SYCL_GROUPED_GEMM_KSPLIT ||
+ !grouped_gemm_esimd_weights_ok(src0_dd, expert_stride, reordered && A::L2D) || ((uintptr_t) packed % 64) != 0 ||
+ Npad % 16 != 0) {
+ return false;
+ }
+ if (reordered) {
+ grouped_gemm_esimd_launch<block_q_t, T, false, A>(src0_dd, expert_stride, tiles, packed, dst, M, Npad, K,
+ dst_rows, n_tiles, stream);
+ } else {
+ grouped_gemm_esimd_launch<block_q_t, T, true, A>(src0_dd, expert_stride, tiles, packed, dst, M, Npad, K,
+ dst_rows, n_tiles, stream);
+ }
+ return true;
+}
+
+template <typename block_q_t, typename T, int NW>
+bool ggml_sycl_grouped_gemm_esimd(const char * src0_dd,
+ size_t expert_stride,
+ const ggml_sycl_grouped_gemm_tile * tiles,
+ const T * packed,
+ float * dst,
+ int M,
+ int Npad,
+ int K,
+ const ggml_sycl_grouped_gemm_rows & dst_rows,
+ bool reordered,
+ int64_t n_tiles,
+ dpct::queue_ptr stream) {
+ static_assert(grouped_gemm_esimd_a<block_q_t>::supported == ggml_sycl_grouped_gemm_esimd_has<block_q_t>,
+ "the header's format list and the formats implemented here must agree");
+ static_assert(NW == 8 || NW == 16, "the dpas widths of Xe-HPG and of Xe2 / Xe-HPC");
+ if constexpr (NW == 16) {
+ return grouped_gemm_esimd_run<block_q_t, T, grouped_gemm_xe2>(src0_dd, expert_stride, tiles, packed, dst, M,
+ Npad, K, dst_rows, reordered, n_tiles, stream);
+ } else {
+ return grouped_gemm_esimd_run<block_q_t, T, grouped_gemm_xehpg>(src0_dd, expert_stride, tiles, packed, dst, M,
+ Npad, K, dst_rows, reordered, n_tiles, stream);
+ }
+}
+
+#define GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, T, NW) \
+ template bool ggml_sycl_grouped_gemm_esimd<block_q_t, T, NW>( \
+ const char *, size_t, const ggml_sycl_grouped_gemm_tile *, const T *, float *, int, int, int, \
+ const ggml_sycl_grouped_gemm_rows &, bool, int64_t, dpct::queue_ptr);
+#define GROUPED_GEMM_ESIMD_INSTANCE(block_q_t) \
+ GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::half, 16) \
+ GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::ext::oneapi::bfloat16, 16) \
+ GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::half, 8) \
+ GROUPED_GEMM_ESIMD_INSTANCE_T(block_q_t, sycl::ext::oneapi::bfloat16, 8)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q8_0)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq4_nl)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq4_xs)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq3_s)
+GROUPED_GEMM_ESIMD_INSTANCE(block_iq3_xxs)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q3_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q2_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q6_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q5_K)
+GROUPED_GEMM_ESIMD_INSTANCE(block_q4_K)
diff --git a/ggml/src/ggml-sycl/fused-gemm-esimd.hpp b/ggml/src/ggml-sycl/fused-gemm-esimd.hpp
new file mode 100644
index 000000000..029d19480
--- /dev/null
+++ b/ggml/src/ggml-sycl/fused-gemm-esimd.hpp
@@ -0,0 +1,33 @@
+#ifndef GGML_SYCL_FUSED_GEMM_ESIMD_HPP
+#define GGML_SYCL_FUSED_GEMM_ESIMD_HPP
+
+#include "fused-gemm.hpp"
+
+// The grouped (MUL_MAT_ID) dequant GEMM, written with ESIMD so the dequantized weights go from
+// registers straight into the dpas A operand: no SLM, no barrier. Same inputs and outputs as the
+// joint_matrix grouped kernel of the 8 x NW x 16 sub-group-NW combination of T (sycl::half or
+// bfloat16): NW = 16 on Xe2 and Xe-HPC, 8 on Xe-HPG (Arc A, ARL-H). The schedule's tiles, B packed as
+// [K/2][Npad][2] T, and dst through dst_rows. The weight is in the reorder (SoA) layout if
+// `reordered`, else in the canonical block layout. Returns false when the shape is not covered.
+template <typename block_q_t>
+constexpr bool ggml_sycl_grouped_gemm_esimd_has = std::is_same_v<block_q_t, block_q8_0> ||
+ std::is_same_v<block_q_t, block_iq4_nl> ||
+ std::is_same_v<block_q_t, block_iq4_xs> ||
+ std::is_same_v<block_q_t, block_iq3_s> ||
+ std::is_same_v<block_q_t, block_iq3_xxs> ||
+ std::is_same_v<block_q_t, block_q2_K> ||
+ std::is_same_v<block_q_t, block_q3_K> ||
+ std::is_same_v<block_q_t, block_q4_K> ||
+ std::is_same_v<block_q_t, block_q5_K> ||
+ std::is_same_v<block_q_t, block_q6_K>;
+
+template <typename block_q_t, typename T, int NW>
+bool ggml_sycl_grouped_gemm_esimd(const char * src0_dd, size_t expert_stride, const ggml_sycl_grouped_gemm_tile * tiles,
+ const T * packed, float * dst, int M, int Npad, int K,
+ const ggml_sycl_grouped_gemm_rows & dst_rows, bool reordered, int64_t n_tiles,
+ dpct::queue_ptr stream);
+
+// GGML_SYCL_MMID_ESIMD: 1 (default) takes the ESIMD kernel above where it applies, 0 never does
+bool ggml_sycl_grouped_gemm_esimd_enabled();
+
+#endif // GGML_SYCL_FUSED_GEMM_ESIMD_HPP
diff --git a/ggml/src/ggml-sycl/fused-gemm.cpp b/ggml/src/ggml-sycl/fused-gemm.cpp
index 803a994c0..d0525e3a0 100644
--- a/ggml/src/ggml-sycl/fused-gemm.cpp
+++ b/ggml/src/ggml-sycl/fused-gemm.cpp
@@ -1,4 +1,5 @@
#include "fused-gemm.hpp"
+#include "fused-gemm-esimd.hpp"
#include <sycl/ext/oneapi/matrix/matrix.hpp>
@@ -17,7 +18,6 @@ namespace mx = sycl::ext::oneapi::experimental::matrix;
// superblock into steps of this width. The sub-groups of a work-group each walk their own K range
// and are summed at the end.
static constexpr int FG_BK = QK4_NL;
-static constexpr int FG_KSPLIT = 4;
// Element traits of one joint_matrix operand type. The A stage and the B pack compute in f32 and
// convert once, in registers, when they write the element, so any type costs the same one pass.
@@ -113,6 +113,15 @@ using fg_half = sycl::half;
using fg_bf16 = sycl::ext::oneapi::bfloat16;
using fg_tf32 = mx::precision::tf32;
+// The combinations the ESIMD grouped GEMM (fused-gemm-esimd.cpp) stands in for: 8 x SG x 16 with f16
+// or bf16 operands, the shape of one dpas, on sub-group 16 (Xe2, Xe-HPC) or 8 (Xe-HPG). Its B is
+// packed and tiled as theirs is.
+template <typename S> static constexpr bool fg_esimd_combo() {
+ return S::TM == 8 && S::TN == S::SG && S::TK == 16 && (S::SG == 16 || S::SG == 8) &&
+ std::is_same_v<typename S::ta, typename S::tb> &&
+ (std::is_same_v<typename S::ta, fg_half> || std::is_same_v<typename S::ta, fg_bf16>);
+}
+
// One bit of GGML_SYCL_XMX_GATHER_SHAPES per combination. Only combinations some device lists in
// matrix_combinations are built (appendix of sycl_ext_oneapi_matrix and the runtime's own list).
template <typename F> static void fg_visit_combo(int idx, F && f);
@@ -152,29 +161,7 @@ template <typename F> static void fg_visit_combo(int idx, F && f) {
// size, since IGC fails on the other ones. A JIT build keeps them all, but each combination lands in
// its own device image (joint_matrix is an optional kernel feature) and only a combination the
// device reports is launched, so the runtime never asks IGC for the others.
-#if defined(__SYCL_DEVICE_ONLY__)
-# if __SYCL_TARGET_INTEL_GPU_ACM_G10__ || __SYCL_TARGET_INTEL_GPU_ACM_G11__ || __SYCL_TARGET_INTEL_GPU_ACM_G12__ || \
- __SYCL_TARGET_INTEL_GPU_ARL_H__
-# define FG_AOT_SG 8
-# elif __SYCL_TARGET_INTEL_GPU_PVC__ || __SYCL_TARGET_INTEL_GPU_PVC_VG__ || __SYCL_TARGET_INTEL_GPU_BMG_G21__ || \
- __SYCL_TARGET_INTEL_GPU_BMG_G31__ || __SYCL_TARGET_INTEL_GPU_LNL_M__ || __SYCL_TARGET_INTEL_GPU_PTL_H__ || \
- __SYCL_TARGET_INTEL_GPU_PTL_U__ || __SYCL_TARGET_INTEL_GPU_WCL__ || __SYCL_TARGET_INTEL_GPU_NVL_S__ || \
- __SYCL_TARGET_INTEL_GPU_NVL_U__ || __SYCL_TARGET_INTEL_GPU_NVL_P__
-# define FG_AOT_SG 16
-# elif __SYCL_TARGET_INTEL_GPU_TGLLP__ || __SYCL_TARGET_INTEL_GPU_RKL__ || __SYCL_TARGET_INTEL_GPU_ADL_S__ || \
- __SYCL_TARGET_INTEL_GPU_ADL_P__ || __SYCL_TARGET_INTEL_GPU_ADL_N__ || __SYCL_TARGET_INTEL_GPU_DG1__ || \
- __SYCL_TARGET_INTEL_GPU_MTL_U__ || __SYCL_TARGET_INTEL_GPU_MTL_H__
-# define FG_AOT_SG 0 // no XMX
-# endif
-#endif
-
-template <int SG> static constexpr bool fg_built() {
-#if defined(FG_AOT_SG)
- return SG == FG_AOT_SG;
-#else
- return true;
-#endif
-}
+// (FG_AOT_SG is defined in fused-gemm.hpp, which the ESIMD kernels share)
// Upper bound on the tile count when total_rows rows are routed to n_as experts: the worst case gives
// each expert one row and fills whole tiles with the rest. The bound depends only on the shape, not
@@ -991,6 +978,15 @@ static void grouped_dequant_gemm_launch(const char * src0_dd, const size_t exper
float * dst, const int M, const int Npad, const int K, const int64_t n_tiles,
const int64_t groups_m, dpct::queue_ptr stream,
ggml_sycl_grouped_gemm_rows dst_rows = {}) {
+ if constexpr (fg_esimd_combo<S>() && ggml_sycl_grouped_gemm_esimd_has<block_q_t>) {
+ // the register-operand ESIMD kernel, either weight layout: no SLM A image (GGML_SYCL_MMID_ESIMD)
+ if (ggml_sycl_grouped_gemm_esimd_enabled() &&
+ ggml_sycl_grouped_gemm_esimd<block_q_t, typename S::tsb, S::SG>(src0_dd, expert_stride, tiles_ptr, packed,
+ dst, M, Npad, K, dst_rows, reordered,
+ n_tiles, stream)) {
+ return;
+ }
+ }
stream->submit([&](sycl::handler & cgh) {
sycl::local_accessor<typename S::tsa, 1> tile_a(FG_KSPLIT * S::SG_ROWS * FG_BK, cgh);
sycl::local_accessor<float, 1> tile_c(FG_KSPLIT * S::SG_ROWS * S::BN, cgh);
@@ -1087,6 +1083,45 @@ template <typename F> static bool fg_visit_type(ggml_type type, bool reordered,
}
}
+// The ESIMD grouped GEMM loads A straight into its dpas operands, so the joint_matrix tile shape
+// means nothing to it; it runs on the fg_esimd_combo of the picked operand type and sub-group size. A
+// grouped call whose weight it covers is moved there from any other shape of that type and sub-group
+// size, if the device lists it, whatever GGML_SYCL_XMX_GATHER_SHAPES allows: the shape bits choose
+// among joint_matrix tiles.
+// A call the ESIMD kernel then declines (a shape it does not take) runs that combination's joint_matrix
+// kernel.
+static int fg_esimd_remap(dpct::queue_ptr stream, int combo, ggml_type src0_type, bool reordered) {
+ if (combo < 0 || !ggml_sycl_grouped_gemm_esimd_enabled()) {
+ return combo;
+ }
+ bool covered = false;
+ fg_visit_type(src0_type, reordered, [&](auto tag) {
+ using T = decltype(tag);
+ covered = ggml_sycl_grouped_gemm_esimd_has<typename T::type>;
+ });
+ if (!covered) {
+ return combo;
+ }
+ int target = -1;
+ fg_visit_combo(combo, [&](auto s) {
+ using S = decltype(s);
+ if constexpr (std::is_same_v<typename S::ta, typename S::tb>) {
+ for (int idx = 0; idx < FG_N_COMBOS; ++idx) {
+ fg_visit_combo(idx, [&](auto t) {
+ using U = decltype(t);
+ if constexpr (fg_esimd_combo<U>() && U::SG == S::SG && std::is_same_v<typename U::ta, typename S::ta>) {
+ target = idx;
+ }
+ });
+ }
+ }
+ });
+ if (target < 0 || target == combo || !(fg_device_combos(stream->get_device()) & (1 << target))) {
+ return combo;
+ }
+ return target;
+}
+
template <typename S>
static bool fg_fused_run(ggml_type src0_type, bool src0_reordered, const void * src0, const void * src1, ggml_type src1_type,
float * dst, int64_t M, int64_t N, int64_t K, int64_t ldd, ggml_sycl_pool & pool,
@@ -1201,7 +1236,7 @@ bool ggml_sycl_grouped_dequant_gemm(ggml_type src0_type, bool reordered, const v
if (!ggml_sycl_grouped_dequant_gemm_shape_ok(src0_type, M, K, total_rows, n_active)) {
return false;
}
- const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec);
+ const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered);
if (combo < 0) {
return false;
}
@@ -1407,7 +1442,7 @@ bool ggml_sycl_grouped_dequant_gemm_device_supported(ggml_type src0_type, bool r
if (grouped_gemm_max_tiles(total_rows, n_as, GGML_SYCL_FG_BN) > INT32_MAX / GGML_SYCL_FG_BN) {
return false;
}
- const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec);
+ const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered);
if (combo < 0) {
return false;
}
@@ -1428,7 +1463,7 @@ bool ggml_sycl_grouped_dequant_gemm_device(ggml_type src0_type, bool reordered,
n_tokens, M, K, stream)) {
return false;
}
- const int combo = fg_pick_combo(stream, GGML_TYPE_F32, src1_prec);
+ const int combo = fg_esimd_remap(stream, fg_pick_combo(stream, GGML_TYPE_F32, src1_prec), src0_type, reordered);
bool launched = false;
fg_visit_combo(combo, [&](auto s) {
launched = fg_grouped_device_run<decltype(s)>(src0_type, reordered, src0_base, expert_stride, src1, dst,
diff --git a/ggml/src/ggml-sycl/fused-gemm.hpp b/ggml/src/ggml-sycl/fused-gemm.hpp
index 0e807a2c0..e65e98fc8 100644
--- a/ggml/src/ggml-sycl/fused-gemm.hpp
+++ b/ggml/src/ggml-sycl/fused-gemm.hpp
@@ -13,7 +13,34 @@ struct ggml_sycl_grouped_gemm_rows {
size_t nb2;
};
+// AOT: the sub-group size of the XMX combinations of the target, 0 for a target without XMX.
+// Undefined in a JIT build, which keeps every combination.
+#if defined(__SYCL_DEVICE_ONLY__)
+# if __SYCL_TARGET_INTEL_GPU_ACM_G10__ || __SYCL_TARGET_INTEL_GPU_ACM_G11__ || __SYCL_TARGET_INTEL_GPU_ACM_G12__ || \
+ __SYCL_TARGET_INTEL_GPU_ARL_H__
+# define FG_AOT_SG 8
+# elif __SYCL_TARGET_INTEL_GPU_PVC__ || __SYCL_TARGET_INTEL_GPU_PVC_VG__ || __SYCL_TARGET_INTEL_GPU_BMG_G21__ || \
+ __SYCL_TARGET_INTEL_GPU_BMG_G31__ || __SYCL_TARGET_INTEL_GPU_LNL_M__ || __SYCL_TARGET_INTEL_GPU_PTL_H__ || \
+ __SYCL_TARGET_INTEL_GPU_PTL_U__ || __SYCL_TARGET_INTEL_GPU_WCL__ || __SYCL_TARGET_INTEL_GPU_NVL_S__ || \
+ __SYCL_TARGET_INTEL_GPU_NVL_U__ || __SYCL_TARGET_INTEL_GPU_NVL_P__
+# define FG_AOT_SG 16
+# elif __SYCL_TARGET_INTEL_GPU_TGLLP__ || __SYCL_TARGET_INTEL_GPU_RKL__ || __SYCL_TARGET_INTEL_GPU_ADL_S__ || \
+ __SYCL_TARGET_INTEL_GPU_ADL_P__ || __SYCL_TARGET_INTEL_GPU_ADL_N__ || __SYCL_TARGET_INTEL_GPU_DG1__ || \
+ __SYCL_TARGET_INTEL_GPU_MTL_U__ || __SYCL_TARGET_INTEL_GPU_MTL_H__
+# define FG_AOT_SG 0 // no XMX
+# endif
+#endif
+
+template <int SG> static constexpr bool fg_built() {
+#if defined(FG_AOT_SG)
+ return SG == FG_AOT_SG;
+#else
+ return true;
+#endif
+}
+
static constexpr int GGML_SYCL_FG_BN = 32;
+static constexpr int FG_KSPLIT = 4;
static constexpr int64_t GGML_SYCL_MMID_SCHED_MAX_EXPERTS = 4096;
// Shape and type gates for the kernels below. Device capability is separate: it needs a queue to ask.
diff --git a/ggml/src/ggml-sycl/ggml-sycl.cpp b/ggml/src/ggml-sycl/ggml-sycl.cpp
index 9da436ced..5b87fa024 100644
--- a/ggml/src/ggml-sycl/ggml-sycl.cpp
+++ b/ggml/src/ggml-sycl/ggml-sycl.cpp
@@ -111,6 +111,7 @@ int g_ggml_sycl_memtrace_step = 64;
int g_ggml_sycl_enable_vmm = 1;
int g_ggml_sycl_enable_fusion = 1;
int g_ggml_sycl_enable_esimd = 1;
+int g_ggml_sycl_mmid_esimd = 1;
int g_ggml_sycl_mmvq_wide = 1;
int g_ggml_sycl_enable_xmx_esimd = 1;
int g_ggml_sycl_xmx_max_cols = GGML_SYCL_XMX_MAX_COLS_DEFAULT;
@@ -444,6 +445,7 @@ static void ggml_check_sycl() try {
g_ggml_sycl_enable_vmm = ggml_sycl_get_env("GGML_SYCL_ENABLE_VMM", 1);
g_ggml_sycl_enable_fusion = ggml_sycl_get_env("GGML_SYCL_ENABLE_FUSION", 1);
g_ggml_sycl_enable_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_ESIMD", 1);
+ g_ggml_sycl_mmid_esimd = ggml_sycl_get_env("GGML_SYCL_MMID_ESIMD", 1);
g_ggml_sycl_mmvq_wide = ggml_sycl_get_env("GGML_SYCL_MMVQ_WIDE", 1);
g_ggml_sycl_enable_xmx_esimd = ggml_sycl_get_env("GGML_SYCL_ENABLE_XMX_ESIMD", 1);
g_ggml_sycl_xmx_max_cols = std::max(0, ggml_sycl_get_env("GGML_SYCL_XMX_MAX_COLS", GGML_SYCL_XMX_MAX_COLS_DEFAULT));
@@ -578,6 +580,7 @@ static void ggml_check_sycl() try {
GGML_LOG_INFO(" GGML_SYCL_ENABLE_OPT: %d\n", g_ggml_sycl_enable_optimize);
GGML_LOG_INFO(" GGML_SYCL_XMX_GATHER_TYPES: %d\n", g_ggml_sycl_xmx_gather_types);
GGML_LOG_INFO(" GGML_SYCL_MMID_SCHED: %d\n", g_ggml_sycl_mmid_sched);
+ GGML_LOG_INFO(" GGML_SYCL_MMID_ESIMD: %d\n", g_ggml_sycl_mmid_esimd);
GGML_LOG_INFO(" GGML_SYCL_XMX_GATHER_SHAPES: %d\n", g_ggml_sycl_xmx_gather_shapes);
GGML_LOG_INFO(" GGML_SYCL_DYNAMIC_PRECISION: %s\n",
ggml_sycl_dynamic_precision_names[g_ggml_sycl_dynamic_precision]);