Commit b797b9ca8 for llama.cpp
commit b797b9ca816bbced1e71c29c8b28cb35b103b624
Author: Max Krasnyansky <maxk@qti.qualcomm.com>
Date: Thu Oct 15 20:30:11 2026 -0700
hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round
Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors
* hex-get-rows: vectorize raw k-quant ukernels
* hex-binary: refactor dma-loop thread functions (no need for macros, etc)
* hex-quant: move/cleanup flat and tiled dequant ukernels
* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights
* hex-get-rows: remove redundant cast
* hex-repack: proper filtering for BF16 that were leaking through during the tests
* hex-repack: streamling tiling rules (broken MM for now)
* hex-repack: fix mmid after tiling updates
* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars
* hex-mm: fix valid_rows calc in various kernels
* hex-mm: remove unused hvx_mm_nx_2d_flat
* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16
* hex-mm: another pass at hvx-tiled f16 matmul
* hex-get-rows: optimize small src1
* hex-get-rows: one more pass at recovering get-rows perf regression
* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py
* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul
* hex-mm: fold tiled_f16 checks into the general flow
* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op
* hex-quant: proper tail handling in new dequant ukernels
* hex-get-rows: remove unused code and cleanup support checks
* hex-mm: cleanup ne1 padding/tiling checks
* hex-mm: cleanup fusion checks for tiled tensors
* hex-repack: add a little can_repack helper to reduce duplication
* hex-repack: never repack host buffers
* hex-repack: factor out repack functions into separate .h and .cpp
* hex-tensor: cleanup tensor ifaces for tiled/padded sizes
* hex-repack: update k-quant tiling comments and cleanup #defines
* hex-dcache: factor out l2/dcache dirty tracking and flushing
* hex-tensor: update docs and remove #defs duplication
* hex-tensor: remove unused/incorrect get_data interface
* hex-get-rows: cleanup vtcm layout build
* tests: disable huge MM tests
* hex-dcache: add missing header
diff --git a/docs/backend/snapdragon/developer.md b/docs/backend/snapdragon/developer.md
index 3608250dc..e088390e8 100644
--- a/docs/backend/snapdragon/developer.md
+++ b/docs/backend/snapdragon/developer.md
@@ -134,6 +134,18 @@ Writing high-performance operators for Hexagon requires following specific guide
- **Distinction**: `octx->src[N]` pointers *can* be NULL by design and must be checked when optional.
Examples include attention masks, optional bias or weights in fused kernels, and frequency factors.
+### `supports_op` Independence from Buffer and Repack State
+
+- `supports_op` (and op-specific checkers such as `supported_mul_mat()`, `supported_get_rows()`) runs during GGML graph scheduling **before** buffers are allocated and before tensor data is populated.
+- **Never check buffer usage or repacking state in `supports_op`**:
+ - `t->buffer` may be NULL or uninitialized.
+ - `t->data` may be NULL.
+ - `buffer->usage` is unknown at scheduling time.
+ - `is_tiled` or `extra->flags & GGML_HEXAGON_TENSOR_TILED` is not set yet.
+- Operator support decisions must depend strictly on static tensor properties: operand types (`t->type`), shapes (`t->ne`), strides (`t->nb`), and operation parameters.
+- Repacking (converting weights to tiled layout) is an internal optimization applied during `set_tensor()` or rt-repack. Non-weight tensors or test harness tensors with the same type remain in flat layout.
+- Therefore, any supported type (such as BF16, F16, quantized types) must be supported both when repacked into tiles and when stored flat in memory.
+
### Multiline Macro Formatting
- Keep trailing backslashes in multiline `#define` macros cleanly aligned to a consistent column.
@@ -190,7 +202,7 @@ sessions.
Common partitioning logic is factored into reusable inline helpers in
[`htp-tensor.h`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h):
-1. [`htp_tensor_mdev_rows_per_chunk`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L71):
+1. [`htp_tensor_mdev_rows_per_chunk`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L104):
Determines the minimum number of rows per chunk so that the chunk byte size is a multiple of 128 bytes:
```
@@ -201,7 +213,7 @@ Common partitioning logic is factored into reusable inline helpers in
Returns `false` if the tensor cannot be safely row-partitioned (such as unaligned base pointer, permuted layout,
or non-128-byte aligned outer strides).
-2. [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L98):
+2. [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L131):
Calculates the per-device work range `struct htp_tensor_mdev_range { uint32_t start; uint32_t count; }` given
`total_units`, `units_per_chunk`, `mdev_idx`, `mdev_count`, and the precomputed `mdev_count_div`.
Handles chunk distribution across devices, assigns remainder units to the last device, and automatically triggers
@@ -240,11 +252,11 @@ if (nrows == 0) {
For flat element-wise operations that write directly to DDR:
- Partition total linear elements N = ne0 * ne1 * ne2 * ne3 in 128-byte cache line chunks (`elems_per_line = (elem_size == 4) ? 32 : 64`).
- Requires strict 1D contiguity:
- [`htp_tensor_is_contiguous(dst, elem_size)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L32)
+ [`htp_tensor_is_contiguous(dst, elem_size)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L65)
and 128-byte aligned destination pointer
- [`htp_tensor_mdev_data_aligned(dst)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L51).
+ [`htp_tensor_mdev_data_aligned(dst)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L84).
- If contiguous and aligned, pass `elems_per_line` to
- [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L98);
+ [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L131);
otherwise pass 0 to trigger Device 0 fallback.
### Single-Device Fallback (Device 0)
@@ -312,7 +324,7 @@ Multi-device execution synchronizes worker sessions through atomic fence slots a
(Output Slice 0) (Output Slice 1)
| |
Tensor Cache Flush Tensor Cache Flush
- (htp_tensor_flush_all) (htp_tensor_flush_all)
+ (htp_dcache_flush_dirty_tensors) (htp_dcache_flush_dirty_tensors)
| |
Post-Op/Batch Barrier ---------------------- Post-Op/Batch Barrier
(htp_mdev_group_barrier) (htp_mdev_group_barrier)
@@ -350,8 +362,8 @@ Multi-device execution synchronizes worker sessions through atomic fence slots a
### Tensor Cache Flush and Pipeline Completion
- In the kernel, ensure all pushed DMA operations have been popped in strict FIFO order to drain the queue.
-- Use [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) to flush specific dirty tensors back to DDR:
- - [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) flushes modified tensor address ranges, or the
+- Use [`htp_dcache_flush_dirty_tensors()`](../../../ggml/src/ggml-hexagon/htp/htp-dcache.h) to flush specific dirty tensors back to DDR:
+ - [`htp_dcache_flush_dirty_tensors()`](../../../ggml/src/ggml-hexagon/htp/htp-dcache.h) flushes modified tensor address ranges, or the
full D-cache when their total size exceeds the flush threshold, ensuring peer devices and the host CPU observe consistent
data in DDR.
- Never signal completion before all DMA transfers are drained and dirty tensor flushes have completed.
diff --git a/ggml/src/ggml-hexagon/CMakeLists.txt b/ggml/src/ggml-hexagon/CMakeLists.txt
index 20c720263..277a4c863 100644
--- a/ggml/src/ggml-hexagon/CMakeLists.txt
+++ b/ggml/src/ggml-hexagon/CMakeLists.txt
@@ -50,6 +50,8 @@ endif()
set(TARGET_NAME ggml-hexagon)
ggml_add_backend_library(${TARGET_NAME}
ggml-hexagon.cpp
+ htp-repack.cpp
+ htp-repack.h
htp-drv.cpp
htp-drv.h
libdl.h
diff --git a/ggml/src/ggml-hexagon/ggml-hexagon.cpp b/ggml/src/ggml-hexagon/ggml-hexagon.cpp
index 08c15ff78..54c474122 100644
--- a/ggml/src/ggml-hexagon/ggml-hexagon.cpp
+++ b/ggml/src/ggml-hexagon/ggml-hexagon.cpp
@@ -77,6 +77,7 @@
#include "htp/im2col-ops.h"
#include "htp_iface.h"
#include "htp-drv.h"
+#include "htp-repack.h"
using intvec = std::vector<int>;
using uintvec = std::vector<unsigned int>;
@@ -164,7 +165,8 @@ static int opt_opbatch = 1280; // max number of ops in a batch
static int opt_opqueue = 8; // max number of pending batches
static int opt_optrace = 0; // trace buffer size per thread (0 means default)
static int opt_oppoll = 0; // polling for batch completions
-static int opt_opfusion = 1; // enable/disable op fusion
+static int opt_opfusion = 1; // enable/disable op fusion
+static int opt_rt_repack = 1; // runtime repacking: 0 = disabled, 1 = quantized only (default), 2 = always
enum ggml_hexagon_fusion_flags {
GGML_HEXAGON_FUSE_ALLREDUCE_ADD = (1 << 1), // 2
@@ -309,42 +311,12 @@ static void ggml_hexagon_dump_trace_events(const std::string & sess_name, const
}
enum ggml_hexagon_tensor_flags {
- GGML_HEXAGON_TENSOR_REPACK = (1 << 0),
+ GGML_HEXAGON_TENSOR_TILED = (1 << 0),
GGML_HEXAGON_TENSOR_WEIGHT = (1 << 1),
GGML_HEXAGON_TENSOR_FENCE = (1 << 2),
GGML_HEXAGON_TENSOR_FUSEABLE = (1 << 3),
};
-static inline bool ggml_hexagon_is_repack_type(enum ggml_type type) {
- return type == GGML_TYPE_Q4_0 || type == GGML_TYPE_Q4_1 ||
- type == GGML_TYPE_Q8_0 || type == GGML_TYPE_IQ4_NL ||
- type == GGML_TYPE_MXFP4 || type == GGML_TYPE_Q6_K ||
- type == GGML_TYPE_Q4_K || type == GGML_TYPE_Q5_K ||
- type == GGML_TYPE_Q3_K || type == GGML_TYPE_Q2_K;
-}
-
-// Size of one repacked row in the DSP tiled layout. The Q3_K..Q6_K tiles store uncompressed scales/mins,
-// so they are larger than the ggml blocks. For the other repack types (Q2_K included) the tile has the same size.
-static inline size_t ggml_hexagon_tiled_row_size(enum ggml_type type, int64_t ne0) {
- if (type == GGML_TYPE_Q6_K) {
- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q6_K / 32);
- }
- if (type == GGML_TYPE_Q4_K) {
- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q4_1 / 32);
- }
- if (type == GGML_TYPE_Q5_K) {
- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q5_K / 32);
- }
- if (type == GGML_TYPE_Q3_K) {
- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q3_K / 32);
- }
- return ggml_row_size(type, ne0);
-}
-
-static inline bool ggml_hexagon_is_hmx_weight_type(enum ggml_type type) {
- return type == GGML_TYPE_F16 || type == GGML_TYPE_F32 || ggml_hexagon_is_repack_type(type);
-}
-
struct ggml_hexagon_session;
static void ggml_hexagon_precompute_matmul_params(
@@ -519,6 +491,32 @@ static inline bool ggml_hexagon_tensor_is_fuseable(const struct ggml_tensor * t)
return (extra->flags & GGML_HEXAGON_TENSOR_FUSEABLE) != 0;
}
+static inline bool ggml_hexagon_can_rt_repack(const struct ggml_tensor * t) {
+ if (opt_rt_repack == 0 || !htp_is_tiled_weight_tensor(t)) {
+ return false;
+ }
+ if (t->buffer && ggml_backend_buft_is_host(t->buffer->buft)) {
+ return false;
+ }
+ return opt_rt_repack > 1 || ggml_is_quantized(t->type);
+}
+
+static inline bool ggml_hexagon_tensor_is_tiled(const struct ggml_tensor * t) {
+ const auto * extra = (const ggml_hexagon_tensor_extra *) t->extra;
+ if (extra && (extra->flags & GGML_HEXAGON_TENSOR_TILED)) {
+ return true;
+ }
+ return ggml_hexagon_can_rt_repack(t);
+}
+
+static inline bool ggml_hexagon_needs_repack(const struct ggml_tensor * t) {
+ const auto * extra = (const ggml_hexagon_tensor_extra *) t->extra;
+ if (extra && (extra->flags & GGML_HEXAGON_TENSOR_TILED)) {
+ return false;
+ }
+ return ggml_hexagon_can_rt_repack(t);
+}
+
static inline bool ggml_hexagon_tensors_overlap(const struct ggml_tensor * a, const struct ggml_tensor * b) {
const uintptr_t a0 = (uintptr_t) a->data;
const uintptr_t b0 = (uintptr_t) b->data;
@@ -600,8 +598,6 @@ struct ggml_hexagon_session {
uint64_t cached_uid = 0;
std::vector<htp_opnode> cached_nodes;
- mutable std::unordered_set<const ggml_tensor *> needs_repack;
-
ggml_hexagon_mdev_group mdev;
ggml_backend_dev_t dev = nullptr;
ggml_backend_hexagon_device_context * dev_ctx = nullptr;
@@ -850,1617 +846,54 @@ struct ggml_hexagon_fence_buffer : public ggml_hexagon_shared_buffer {
if (ptr) {
memset(ptr, 0, (size_t) n_slots * GGML_HEXAGON_FENCE_SLOT_SIZE);
}
- return ptr;
- }
-
- void free_slot(void * ptr, uint32_t n_slots = 1) {
- if (!ptr) return;
- uint32_t slot = ((uint8_t *) ptr - base()) / GGML_HEXAGON_FENCE_SLOT_SIZE;
- for (uint32_t i = 0; i < n_slots; i++) {
- free_slots.push_back(slot + i);
- }
- }
-};
-
-inline uint8_t * ggml_hexagon_session::alloc_fence(uint32_t n_slots) {
- uint8_t * ptr = fence_buf->alloc_slot(n_slots);
- GGML_ASSERT(ptr);
- return ptr;
-}
-
-inline void ggml_hexagon_session::free_fence(void * ptr, uint32_t n_slots) {
- if (fence_buf) {
- fence_buf->free_slot(ptr, n_slots);
- }
-}
-
-static void ggml_backend_hexagon_buffer_free_buffer(ggml_backend_buffer_t buffer) {
- auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);
- sbuf->sess->unclone_buffer(sbuf);
- delete sbuf;
-}
-
-static void * ggml_backend_hexagon_buffer_get_base(ggml_backend_buffer_t buffer) {
- auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);
- return sbuf->base();
-}
-
-static enum ggml_status ggml_backend_hexagon_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {
- auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);
- auto sess = sbuf->sess;
-
- HEX_VERBOSE("ggml-hex: %s init-tensor %s : base %p data %p nbytes %zu\n", sess->c_name(),
- tensor->name, (void *) sbuf->base(), tensor->data, ggml_nbytes(tensor));
-
- auto extra = new ggml_hexagon_tensor_extra();
- sbuf->tensor_extra.push_back(extra);
-
- tensor->extra = extra;
- if (ggml_hexagon_is_repack_type(tensor->type)) {
- if (sess->needs_repack.count(tensor)) {
- extra->flags |= GGML_HEXAGON_TENSOR_REPACK;
- sess->needs_repack.erase(tensor);
- }
- }
-
- return GGML_STATUS_SUCCESS;
-}
-
-// ** Repack helpers for tiled quantized weights
-
-static void unpack_q4_0_quants(uint8_t * qs, const block_q4_0 * x, unsigned int bi) {
- static const int qk = QK4_0;
-
- for (unsigned int i = 0; i < qk / 2; ++i) {
- const int x0 = (x->qs[i] & 0x0F);
- const int x1 = (x->qs[i] >> 4);
- qs[bi * qk + i + 0] = x0;
- qs[bi * qk + i + qk / 2] = x1;
- }
-}
-
-static void pack_q4_0_quants(block_q4_0 * x, const uint8_t * qs, unsigned int bi) {
- static const int qk = QK4_0;
-
- for (unsigned int i = 0; i < qk / 2; ++i) {
- const uint8_t x0 = qs[bi * qk + i + 0];
- const uint8_t x1 = qs[bi * qk + i + qk / 2];
- x->qs[i] = x0 | (x1 << 4);
- }
-}
-
-static void unpack_q4_1_quants(uint8_t * qs, const block_q4_1 * x, unsigned int bi) {
- static const int qk = QK4_1;
-
- for (unsigned int i = 0; i < qk / 2; ++i) {
- const int x0 = (x->qs[i] & 0x0F);
- const int x1 = (x->qs[i] >> 4);
- qs[bi * qk + i + 0] = x0;
- qs[bi * qk + i + qk / 2] = x1;
- }
-}
-
-static void pack_q4_1_quants(block_q4_1 * x, const uint8_t * qs, unsigned int bi) {
- static const int qk = QK4_1;
-
- for (unsigned int i = 0; i < qk / 2; ++i) {
- const uint8_t x0 = qs[bi * qk + i + 0];
- const uint8_t x1 = qs[bi * qk + i + qk / 2];
- x->qs[i] = x0 | (x1 << 4);
- }
-}
-
-static void unpack_mxfp4_quants(uint8_t * qs, const block_mxfp4 * x, unsigned int bi) {
- static const int qk = QK_MXFP4;
-
- for (unsigned int i = 0; i < qk / 2; ++i) {
- const int x0 = (x->qs[i] & 0x0F);
- const int x1 = (x->qs[i] >> 4);
- qs[bi * qk + i + 0] = x0;
- qs[bi * qk + i + qk / 2] = x1;
- }
-}
-
-static void pack_mxfp4_quants(block_mxfp4 * x, const uint8_t * qs, unsigned int bi) {
- static const int qk = QK_MXFP4;
-
- for (unsigned int i = 0; i < qk / 2; ++i) {
- const uint8_t x0 = qs[bi * qk + i + 0];
- const uint8_t x1 = qs[bi * qk + i + qk / 2];
- x->qs[i] = x0 | (x1 << 4);
- }
-}
-
-// repack q4_0 data into q4_0_tiled tensor
-static void repack_q4_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- const block_q4_0 * src_matrix = (const block_q4_0 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_0;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- const block_q4_0 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));
- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = 0; ct < n_col_tiles; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;
-
- uint8_t tile_quants[32][32];
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r < ne1 && kt < ne0 / 32) {
- unpack_q4_0_quants(tile_quants[row], &src_slice[r * (ne0 / 32) + kt], 0);
- } else {
- memset(tile_quants[row], 8, 32);
- }
- }
-
- for (int cp = 0; cp < 16; cp++) {
- for (int row = 0; row < 32; row++) {
- tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];
- }
- }
-
- ggml_half * scale_dst = (ggml_half *)(tile_dst + 512);
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * (ne0 / 32) + kt].d : 0;
- }
- }
- }
- }
-}
-
-// repack q4_0_tiled tensor into q4_0 data
-static void repack_tiled_q4_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- block_q4_0 * dst_matrix = (block_q4_0 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_0;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- size_t row_size_bytes = ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);
- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);
- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;
- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;
-
- int64_t start_row = slice_offset_start / row_size_bytes;
- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;
- end_row = (std::min)(end_row, ne1);
-
- int start_ct = start_row / 32;
- int end_ct = (end_row + 31) / 32;
- end_ct = (std::min)(end_ct, n_col_tiles);
-
- block_q4_0 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_q4_0);
- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = start_ct; ct < end_ct; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;
-
- uint8_t tile_quants[32][32];
- for (int cp = 0; cp < 16; cp++) {
- for (int row = 0; row < 32; row++) {
- uint8_t val = tile_src[cp * 32 + row];
- tile_quants[row][2 * cp + 0] = val & 0x0F;
- tile_quants[row][2 * cp + 1] = val >> 4;
- }
- }
-
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- pack_q4_0_quants(&dst_slice[(r - start_row) * (ne0 / 32) + kt], tile_quants[row], 0);
- }
- }
-
- const ggml_half * scale_src = (const ggml_half *)(tile_src + 512);
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- dst_slice[(r - start_row) * (ne0 / 32) + kt].d = scale_src[row];
- }
- }
- }
- }
- }
-}
-
-// repack q4_1 data into q4_1_tiled tensor
-static void repack_q4_1_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- const block_q4_1 * src_matrix = (const block_q4_1 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- const block_q4_1 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));
- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = 0; ct < n_col_tiles; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;
-
- uint8_t tile_quants[32][32];
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r < ne1 && kt < ne0 / 32) {
- unpack_q4_1_quants(tile_quants[row], &src_slice[r * (ne0 / 32) + kt], 0);
- } else {
- memset(tile_quants[row], 0, 32);
- }
- }
-
- for (int cp = 0; cp < 16; cp++) {
- for (int row = 0; row < 32; row++) {
- tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];
- }
- }
-
- ggml_half * scale_dst = (ggml_half *)(tile_dst + 512);
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r < ne1 && kt < ne0 / 32) {
- scale_dst[2 * row + 0] = src_slice[r * (ne0 / 32) + kt].d;
- scale_dst[2 * row + 1] = src_slice[r * (ne0 / 32) + kt].m;
- } else {
- scale_dst[2 * row + 0] = 0;
- scale_dst[2 * row + 1] = 0;
- }
- }
- }
- }
- }
-}
-
-// repack q4_1_tiled tensor into q4_1 data
-static void repack_tiled_q4_1(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- block_q4_1 * dst_matrix = (block_q4_1 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- size_t row_size_bytes = ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);
- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);
- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;
- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;
-
- int64_t start_row = slice_offset_start / row_size_bytes;
- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;
- end_row = (std::min)(end_row, ne1);
-
- int start_ct = start_row / 32;
- int end_ct = (end_row + 31) / 32;
- end_ct = (std::min)(end_ct, n_col_tiles);
-
- block_q4_1 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_q4_1);
- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = start_ct; ct < end_ct; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;
-
- uint8_t tile_quants[32][32];
- for (int cp = 0; cp < 16; cp++) {
- for (int row = 0; row < 32; row++) {
- uint8_t val = tile_src[cp * 32 + row];
- tile_quants[row][2 * cp + 0] = val & 0x0F;
- tile_quants[row][2 * cp + 1] = val >> 4;
- }
- }
-
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- pack_q4_1_quants(&dst_slice[(r - start_row) * (ne0 / 32) + kt], tile_quants[row], 0);
- }
- }
-
- const ggml_half * scale_src = (const ggml_half *)(tile_src + 512);
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- dst_slice[(r - start_row) * (ne0 / 32) + kt].d = scale_src[2 * row];
- dst_slice[(r - start_row) * (ne0 / 32) + kt].m = scale_src[2 * row + 1];
- }
- }
- }
- }
- }
-}
-
-// repack q8_0 data into q8_0_tiled tensor
-static void repack_q8_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- const block_q8_0 * src_matrix = (const block_q8_0 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q8_0;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- const block_q8_0 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));
- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = 0; ct < n_col_tiles; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;
-
- for (int cp = 0; cp < 16; cp++) {
- int col0 = cp * 2;
- int col1 = col0 + 1;
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- const block_q8_0 * b = (r < ne1 && kt < ne0 / 32) ? &src_slice[r * (ne0 / 32) + kt] : NULL;
- tile_dst[cp * 64 + 2 * row + 0] = b ? b->qs[col0] : 0;
- tile_dst[cp * 64 + 2 * row + 1] = b ? b->qs[col1] : 0;
- }
- }
-
- ggml_half * scale_dst = (ggml_half *)(tile_dst + 1024);
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * (ne0 / 32) + kt].d : 0;
- }
- }
- }
- }
-}
-
-// repack q8_0_tiled tensor into q8_0 data
-static void repack_tiled_q8_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- block_q8_0 * dst_matrix = (block_q8_0 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q8_0;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- size_t row_size_bytes = ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);
- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);
- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;
- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;
-
- int64_t start_row = slice_offset_start / row_size_bytes;
- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;
- end_row = (std::min)(end_row, ne1);
-
- int start_ct = start_row / 32;
- int end_ct = (end_row + 31) / 32;
- end_ct = (std::min)(end_ct, n_col_tiles);
-
- block_q8_0 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_q8_0);
- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = start_ct; ct < end_ct; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;
-
- for (int cp = 0; cp < 16; cp++) {
- int col0 = cp * 2;
- int col1 = col0 + 1;
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- block_q8_0 & b = dst_slice[(r - start_row) * (ne0 / 32) + kt];
- b.qs[col0] = tile_src[cp * 64 + 2 * row + 0];
- b.qs[col1] = tile_src[cp * 64 + 2 * row + 1];
- }
- }
- }
-
- const ggml_half * scale_src = (const ggml_half *)(tile_src + 1024);
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- dst_slice[(r - start_row) * (ne0 / 32) + kt].d = scale_src[row];
- }
- }
- }
- }
- }
-}
-
-// repack mxfp4 data into mxfp4_tiled tensor
-static void repack_mxfp4_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- const block_mxfp4 * src_matrix = (const block_mxfp4 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_MXFP4;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- const block_mxfp4 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));
- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = 0; ct < n_col_tiles; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;
-
- uint8_t tile_quants[32][32];
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r < ne1 && kt < ne0 / 32) {
- unpack_mxfp4_quants(tile_quants[row], &src_slice[r * (ne0 / 32) + kt], 0);
- } else {
- memset(tile_quants[row], 0, 32);
- }
- }
-
- for (int cp = 0; cp < 16; cp++) {
- for (int row = 0; row < 32; row++) {
- tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];
- }
- }
-
- uint8_t * scale_dst = tile_dst + 512;
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * (ne0 / 32) + kt].e : 0;
- }
- }
- }
- }
-}
-
-// repack mxfp4_tiled tensor into mxfp4 data
-static void repack_tiled_mxfp4(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- block_mxfp4 * dst_matrix = (block_mxfp4 *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- int n_col_tiles = ne1_padded / 32;
- int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_MXFP4;
- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;
-
- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);
- size_t row_size_bytes = ggml_row_size(t->type, ne0);
- int64_t start_slice = offset / slice_size;
- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
- if (end_slice > ne2 * ne3) {
- end_slice = ne2 * ne3;
- }
-
- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);
- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);
- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;
- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;
-
- int64_t start_row = slice_offset_start / row_size_bytes;
- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;
- end_row = (std::min)(end_row, ne1);
-
- int start_ct = start_row / 32;
- int end_ct = (end_row + 31) / 32;
- end_ct = (std::min)(end_ct, n_col_tiles);
-
- block_mxfp4 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_mxfp4);
- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;
-
- for (int ct = start_ct; ct < end_ct; ct++) {
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;
-
- uint8_t tile_quants[32][32];
- for (int cp = 0; cp < 16; cp++) {
- for (int row = 0; row < 32; row++) {
- uint8_t val = tile_src[cp * 32 + row];
- tile_quants[row][2 * cp + 0] = val & 0x0F;
- tile_quants[row][2 * cp + 1] = val >> 4;
- }
- }
-
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- pack_mxfp4_quants(&dst_slice[(r - start_row) * (ne0 / 32) + kt], tile_quants[row], 0);
- }
- }
-
- const uint8_t * scale_src = tile_src + 512;
- for (int row = 0; row < 32; row++) {
- int64_t r = ct * 32 + row;
- if (r >= start_row && r < end_row && kt < ne0 / 32) {
- dst_slice[(r - start_row) * (ne0 / 32) + kt].e = scale_src[row];
- }
- }
- }
- }
- }
-}
-
-// unsigned 6-bit value (0..63) of element e of a Q6_K block, same bit layout as dequantize_row_q6_K
-static inline uint8_t q6_K_get_quant(const block_q6_K * b, int e) {
- const int c = e / 128;
- const int w = e % 128;
- const int g = w / 32;
- const int l = w % 32;
- const uint8_t * ql = b->ql + c * 64;
- const uint8_t * qh = b->qh + c * 32;
- uint8_t lo, hi;
- switch (g) {
- case 0: lo = ql[l] & 0xF; hi = (qh[l] >> 0) & 3; break;
- case 1: lo = ql[l + 32] & 0xF; hi = (qh[l] >> 2) & 3; break;
- case 2: lo = ql[l] >> 4; hi = (qh[l] >> 4) & 3; break;
- default: lo = ql[l + 32] >> 4; hi = (qh[l] >> 6) & 3; break;
- }
- return (uint8_t) (lo | (hi << 4));
-}
-
-// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q6_K in htp/matmul-ops.h
-static void repack_q6_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- const block_q6_K * src_matrix = (const block_q6_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q6_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- const block_q6_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- memset(matrix_dst, 0, matrix_size); // padding rows and the OR-ed nibbles below need zeroed tiles
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- const block_q6_K * src_row = src_slice + r * sb_per_row;
-
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const int kt_local = kt % 8; // k-tile within the super-block
- const block_q6_K * b = &src_row[kt / 8];
- const float d = GGML_FP16_TO_FP32(b->d);
-
- uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;
- uint8_t * lo_pl = tile;
- uint8_t * hi_pl = tile + 512;
- ggml_half * sc_pl = (ggml_half *) (tile + 768);
-
- for (int lk = 0; lk < 32; lk++) {
- const uint8_t q6 = q6_K_get_quant(b, kt_local * 32 + lk);
- const int g = lk >> 2;
- const int pos = row * 4 + (lk & 3);
- lo_pl[(g >> 1) * 128 + pos] |= (uint8_t) ((q6 & 0xF) << ((g & 1) * 4));
- hi_pl[(g >> 2) * 128 + pos] |= (uint8_t) ((q6 >> 4) << ((g & 3) * 2));
- }
- for (int sub = 0; sub < 2; sub++) {
- sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) b->scales[kt_local * 2 + sub]);
- }
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// Reverse of repack_q6_K_tiled. Unpacks quants losslessly and normalizes sub-block scales. Read-back only.
-static void repack_tiled_q6_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- block_q6_K * dst_matrix = (block_q6_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q6_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- block_q6_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- block_q6_K * dst_row = dst_slice + r * sb_per_row;
-
- for (int64_t sb = 0; sb < sb_per_row; sb++) {
- block_q6_K * b = &dst_row[sb];
- memset(b, 0, sizeof(block_q6_K));
-
- float sub_scales[16];
- for (int kt_local = 0; kt_local < 8; kt_local++) {
- const int kt = sb * 8 + kt_local;
- const uint8_t * tile = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;
- const uint8_t * lo_pl = tile;
- const uint8_t * hi_pl = tile + 512;
- const ggml_half * sc_pl = (const ggml_half *) (tile + 768);
-
- const int c = kt_local / 4;
- const int g = kt_local % 4;
- uint8_t * ql = b->ql + c * 64;
- uint8_t * qh = b->qh + c * 32;
-
- for (int lk = 0; lk < 32; lk++) {
- const int g_tile = lk >> 2;
- const int pos = row * 4 + (lk & 3);
- const uint8_t lo = (lo_pl[(g_tile >> 1) * 128 + pos] >> ((g_tile & 1) * 4)) & 0xF;
- const uint8_t hi = (hi_pl[(g_tile >> 2) * 128 + pos] >> ((g_tile & 3) * 2)) & 3;
-
- switch (g) {
- case 0:
- ql[lk] |= lo;
- qh[lk] |= (hi << 0);
- break;
- case 1:
- ql[lk + 32] |= lo;
- qh[lk] |= (hi << 2);
- break;
- case 2:
- ql[lk] |= (lo << 4);
- qh[lk] |= (hi << 4);
- break;
- default:
- ql[lk + 32] |= (lo << 4);
- qh[lk] |= (hi << 6);
- break;
- }
- }
-
- for (int sub = 0; sub < 2; sub++) {
- sub_scales[kt_local * 2 + sub] = GGML_FP16_TO_FP32(sc_pl[sub * 32 + row]);
- }
- }
-
- float max_abs_scale = 0.0f;
- for (int s = 0; s < 16; s++) {
- float abs_scale = fabsf(sub_scales[s]);
- if (abs_scale > max_abs_scale) {
- max_abs_scale = abs_scale;
- }
- }
-
- if (max_abs_scale == 0.0f) {
- b->d = GGML_FP32_TO_FP16(0.0f);
- memset(b->scales, 0, sizeof(b->scales));
- } else {
- float d_flt = max_abs_scale / 127.0f;
- b->d = GGML_FP32_TO_FP16(d_flt);
- float d_actual = GGML_FP16_TO_FP32(b->d);
- float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;
- for (int s = 0; s < 16; s++) {
- int sc = (int) roundf(sub_scales[s] * inv_d);
- b->scales[s] = (int8_t) (std::max)(-128, (std::min)(127, sc));
- }
- }
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// low 2 bits (0..3) of element e of a Q2_K or Q3_K block, same bit layout as dequantize_row_q2_K / q3_K
-static inline uint8_t q2_3_K_get_low2(const uint8_t * qs, int e) {
- const int c = e / 128;
- const int j = (e % 128) / 32;
- const int l = e % 32;
- return (qs[c * 32 + l] >> (2 * j)) & 3;
-}
-
-// hmask bit of element e of a Q3_K block, same bit layout as dequantize_row_q3_K
-static inline bool q3_K_get_hbit(const block_q3_K * b, int e) {
- const int c = e / 128;
- const int j = (e % 128) / 32;
- const int l = e % 32;
- return (b->hmask[l] >> (c * 4 + j)) & 1;
-}
-
-// signed 6-bit scale j (-32..31) of a Q3_K block, same packing as quantize_row_q3_K_ref
-static inline int q3_K_get_scale(const uint8_t * scales, int j) {
- const int lo = (j < 8) ? (scales[j] & 0xF) : (scales[j - 8] >> 4);
- const int hi = (scales[8 + j % 4] >> (2 * (j / 4))) & 3;
- return (lo | (hi << 4)) - 32;
-}
-
-// read-back: find fp16 d and l[j] in [lmin, lmax] with fp16(d * l[j]) == prod[j] for all j, false if none
-static bool hexagon_recover_k_scales(const ggml_half * prod, int n, int lmin, int lmax, ggml_half * d_out, int * l_out) {
- int jmax = 0;
- for (int j = 1; j < n; j++) {
- if (fabsf(GGML_FP16_TO_FP32(prod[j])) > fabsf(GGML_FP16_TO_FP32(prod[jmax]))) {
- jmax = j;
- }
- }
- const float pmax = GGML_FP16_TO_FP32(prod[jmax]);
- if (pmax == 0.0f) {
- *d_out = GGML_FP32_TO_FP16(0.0f);
- for (int j = 0; j < n; j++) {
- l_out[j] = 0;
- }
- return true;
- }
-
- // the quantizers put the largest scale at or near the range end, so try large |l| first
- const int lext = (std::max)(-lmin, lmax);
- for (int a = lext; a >= 1; a--) {
- for (int sign : { 1, -1 }) {
- const int lj = sign * a;
- if (lj < lmin || lj > lmax) {
- continue;
- }
- const ggml_half d0 = GGML_FP32_TO_FP16(pmax / (float) lj);
- for (int ulp : { 0, -1, 1 }) {
- ggml_half d = d0;
- uint16_t bits;
- memcpy(&bits, &d, sizeof(bits));
- bits = (uint16_t) (bits + ulp);
- memcpy(&d, &bits, sizeof(bits));
-
- const float df = GGML_FP16_TO_FP32(d);
- if (!std::isfinite(df) || df == 0.0f) {
- continue;
- }
- bool ok = true;
- for (int j = 0; j < n && ok; j++) {
- const int l = (int) roundf(GGML_FP16_TO_FP32(prod[j]) / df);
- const ggml_half p = GGML_FP32_TO_FP16(df * (float) l);
- ok = l >= lmin && l <= lmax && memcmp(&p, &prod[j], sizeof(p)) == 0;
- l_out[j] = l;
- }
- if (ok) {
- *d_out = d;
- return true;
- }
- }
- }
- }
- return false;
-}
-
-// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q3_K in htp/matmul-ops.h
-static void repack_q3_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- const block_q3_K * src_matrix = (const block_q3_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q3_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- const block_q3_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- memset(matrix_dst, 0, matrix_size); // padding rows and the OR-ed bits below need zeroed tiles
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- const block_q3_K * src_row = src_slice + r * sb_per_row;
-
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const int kt_local = kt % 8; // k-tile within the super-block
- const block_q3_K * b = &src_row[kt / 8];
- const float d = GGML_FP16_TO_FP32(b->d);
-
- uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;
- uint8_t * lo_pl = tile;
- uint8_t * neg_pl = tile + 256;
- ggml_half * sc_pl = (ggml_half *) (tile + 384);
-
- for (int lk = 0; lk < 32; lk++) {
- const int e = kt_local * 32 + lk;
- const int g = lk >> 2;
- const int pos = row * 4 + (lk & 3);
- lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, e) << ((g & 3) * 2));
- if (!q3_K_get_hbit(b, e)) {
- neg_pl[pos] |= (uint8_t) (1 << g);
- }
- }
- for (int sub = 0; sub < 2; sub++) {
- sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) q3_K_get_scale(b->scales, kt_local * 2 + sub));
- }
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// Reverse of repack_q3_K_tiled. Unpacks quants losslessly and normalizes sub-block scales. Read-back only.
-static void repack_tiled_q3_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- block_q3_K * dst_matrix = (block_q3_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q3_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- block_q3_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- block_q3_K * dst_row = dst_slice + r * sb_per_row;
-
- for (int64_t sb = 0; sb < sb_per_row; sb++) {
- block_q3_K * b = &dst_row[sb];
- memset(b, 0, sizeof(block_q3_K));
-
- ggml_half sub_scales[16];
- for (int kt_local = 0; kt_local < 8; kt_local++) {
- const int kt = sb * 8 + kt_local;
- const uint8_t * tile = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;
- const uint8_t * lo_pl = tile;
- const uint8_t * neg_pl = tile + 256;
- const ggml_half * sc_pl = (const ggml_half *) (tile + 384);
-
- for (int lk = 0; lk < 32; lk++) {
- const int e = kt_local * 32 + lk;
- const int g = lk >> 2;
- const int pos = row * 4 + (lk & 3);
- const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;
-
- const int c = e / 128;
- const int j = (e % 128) / 32;
- const int l = e % 32;
- b->qs[c * 32 + l] |= (uint8_t) (lo << (2 * j));
- if (!((neg_pl[pos] >> g) & 1)) {
- b->hmask[l] |= (uint8_t) (1 << (c * 4 + j));
- }
- }
-
- for (int sub = 0; sub < 2; sub++) {
- sub_scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];
- }
- }
-
- int ls[16];
- if (!hexagon_recover_k_scales(sub_scales, 16, -32, 31, &b->d, ls)) {
- // no exact match: same scale choice as quantize_row_q3_K_ref
- float max_scale = 0.0f;
- for (int s = 0; s < 16; s++) {
- if (fabsf(GGML_FP16_TO_FP32(sub_scales[s])) > fabsf(max_scale)) {
- max_scale = GGML_FP16_TO_FP32(sub_scales[s]);
- }
- }
- b->d = GGML_FP32_TO_FP16(-max_scale / 32.0f);
- const float d_actual = GGML_FP16_TO_FP32(b->d);
- const float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;
- for (int s = 0; s < 16; s++) {
- ls[s] = (std::max)(-32, (std::min)(31, (int) roundf(GGML_FP16_TO_FP32(sub_scales[s]) * inv_d)));
- }
- }
-
- for (int s = 0; s < 16; s++) {
- const int l = ls[s] + 32;
- if (s < 8) {
- b->scales[s] = l & 0xF;
- } else {
- b->scales[s - 8] |= (uint8_t) ((l & 0xF) << 4);
- }
- b->scales[s % 4 + 8] |= (uint8_t) ((l >> 4) << (2 * (s / 4)));
- }
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q2_K in htp/matmul-ops.h
-static void repack_q2_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- const block_q2_K * src_matrix = (const block_q2_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q2_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- const block_q2_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- memset(matrix_dst, 0, matrix_size); // padding rows and the OR-ed bits below need zeroed tiles
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- const block_q2_K * src_row = src_slice + r * sb_per_row;
-
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const int kt_local = kt % 8; // k-tile within the super-block
- const block_q2_K * b = &src_row[kt / 8];
-
- uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;
- uint8_t * lo_pl = tile;
- uint8_t * sc_pl = tile + 256;
- ggml_half * dd_pl = (ggml_half *) (tile + 320);
-
- for (int lk = 0; lk < 32; lk++) {
- const int g = lk >> 2;
- const int pos = row * 4 + (lk & 3);
- lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, kt_local * 32 + lk) << ((g & 3) * 2));
- }
- for (int sub = 0; sub < 2; sub++) {
- sc_pl[sub * 32 + row] = b->scales[kt_local * 2 + sub];
- }
- if (row / 8 == kt_local % 4) {
- dd_pl[row % 8] = kt_local < 4 ? b->d : (ggml_half) (b->dmin ^ 0x8000); // -dmin
- }
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// Reverse of repack_q2_K_tiled. Lossless. Read-back only.
-static void repack_tiled_q2_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- block_q2_K * dst_matrix = (block_q2_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q2_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- block_q2_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- block_q2_K * dst_row = dst_slice + r * sb_per_row;
-
- for (int64_t sb = 0; sb < sb_per_row; sb++) {
- block_q2_K * b = &dst_row[sb];
- memset(b, 0, sizeof(block_q2_K));
-
- for (int kt_local = 0; kt_local < 8; kt_local++) {
- const int kt = sb * 8 + kt_local;
- const uint8_t * tile = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;
- const uint8_t * lo_pl = tile;
- const uint8_t * sc_pl = tile + 256;
- const ggml_half * dd_pl = (const ggml_half *) (tile + 320);
-
- for (int lk = 0; lk < 32; lk++) {
- const int e = kt_local * 32 + lk;
- const int g = lk >> 2;
- const int pos = row * 4 + (lk & 3);
- const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;
- b->qs[(e / 128) * 32 + e % 32] |= (uint8_t) (lo << (2 * ((e % 128) / 32)));
- }
-
- for (int sub = 0; sub < 2; sub++) {
- b->scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];
- }
- if (row / 8 == kt_local % 4) {
- if (kt_local < 4) {
- b->d = dd_pl[row % 8];
- } else {
- b->dmin = (ggml_half) (dd_pl[row % 8] ^ 0x8000);
- }
- }
- }
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-static inline void get_scale_min_k4(int j, const uint8_t * q, uint8_t * d, uint8_t * m) {
- if (j < 4) {
- *d = q[j] & 63;
- *m = q[j + 4] & 63;
- } else {
- *d = (q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4);
- *m = (q[j + 4] >> 4) | ((q[j - 0] >> 6) << 4);
- }
-}
-
-// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q4_1 in htp/matmul-ops.h
-static void repack_q4_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- const block_q4_K * src_matrix = (const block_q4_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- const block_q4_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- memset(matrix_dst, 0, matrix_size);
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- const block_q4_K * src_row = src_slice + r * sb_per_row;
-
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const int kt_local = kt % 8;
- const block_q4_K * b = &src_row[kt / 8];
- const float d = GGML_FP16_TO_FP32(b->d);
- const float dmin = GGML_FP16_TO_FP32(b->dmin);
-
- uint8_t * tile_dst = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;
-
- uint8_t sc, m;
- get_scale_min_k4(kt_local, b->scales, &sc, &m);
-
- const float D = d * (float) sc;
- const float M = -dmin * (float) m;
-
- const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
- const int shift = (kt_local & 1) ? 4 : 0;
-
- for (int cp = 0; cp < 16; cp++) {
- const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;
- const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;
- tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);
- }
-
- ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);
- scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(D);
- scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(M);
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// Reverse of repack_q4_K_tiled. Unpacks quants and normalizes scales/mins. Read-back only.
-static void repack_tiled_q4_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- block_q4_K * dst_matrix = (block_q4_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- block_q4_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- block_q4_K * dst_row = dst_slice + r * sb_per_row;
-
- for (int64_t sb = 0; sb < sb_per_row; sb++) {
- block_q4_K * b = &dst_row[sb];
- memset(b, 0, sizeof(block_q4_K));
-
- float sub_scales[8];
- float sub_mins[8];
-
- for (int kt_local = 0; kt_local < 8; kt_local++) {
- const int kt = sb * 8 + kt_local;
- const uint8_t * tile_src = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;
- const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);
-
- uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
- const int shift = (kt_local & 1) ? 4 : 0;
-
- for (int cp = 0; cp < 16; cp++) {
- const uint8_t val = tile_src[cp * 32 + row];
- const uint8_t q0 = val & 0x0F;
- const uint8_t q1 = val >> 4;
- qs_sub[2 * cp + 0] |= (uint8_t) (q0 << shift);
- qs_sub[2 * cp + 1] |= (uint8_t) (q1 << shift);
- }
-
- const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);
- const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);
- sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;
- sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;
- }
-
- float max_scale = 0.0f;
- float max_min = 0.0f;
- for (int j = 0; j < 8; j++) {
- if (sub_scales[j] > max_scale) max_scale = sub_scales[j];
- if (sub_mins[j] > max_min) max_min = sub_mins[j];
- }
-
- float inv_scale = 0.0f;
- if (max_scale > 0.0f) {
- b->d = GGML_FP32_TO_FP16(max_scale / 63.0f);
- const float d_actual = GGML_FP16_TO_FP32(b->d);
- inv_scale = (d_actual > 0.0f) ? (1.0f / d_actual) : 0.0f;
- } else {
- b->d = GGML_FP32_TO_FP16(0.0f);
- }
-
- float inv_min = 0.0f;
- if (max_min > 0.0f) {
- b->dmin = GGML_FP32_TO_FP16(max_min / 63.0f);
- const float dmin_actual = GGML_FP16_TO_FP32(b->dmin);
- inv_min = (dmin_actual > 0.0f) ? (1.0f / dmin_actual) : 0.0f;
- } else {
- b->dmin = GGML_FP32_TO_FP16(0.0f);
- }
-
- for (int j = 0; j < 8; j++) {
- uint8_t ls = (uint8_t) roundf(inv_scale * sub_scales[j]);
- uint8_t lm = (uint8_t) roundf(inv_min * sub_mins[j]);
- ls = (std::min)((uint8_t) 63, ls);
- lm = (std::min)((uint8_t) 63, lm);
- if (j < 4) {
- b->scales[j] = ls;
- b->scales[j + 4] = lm;
- } else {
- b->scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);
- b->scales[j - 4] |= ((ls >> 4) << 6);
- b->scales[j - 0] |= ((lm >> 4) << 6);
- }
- }
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q5_K in htp/matmul-ops.h
-static void repack_q5_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- const block_q5_K * src_matrix = (const block_q5_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q5_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- const block_q5_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- memset(matrix_dst, 0, matrix_size);
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- const block_q5_K * src_row = src_slice + r * sb_per_row;
-
- for (int kt = 0; kt < n_k_tiles; kt++) {
- const int kt_local = kt % 8;
- const block_q5_K * b = &src_row[kt / 8];
- const float d = GGML_FP16_TO_FP32(b->d);
- const float dmin = GGML_FP16_TO_FP32(b->dmin);
-
- uint8_t * tile_dst = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;
- uint8_t * plane = tile_dst + 640;
-
- uint8_t sc, m;
- get_scale_min_k4(kt_local, b->scales, &sc, &m);
-
- const float D = d * (float) sc;
- const float M = -dmin * (float) m;
-
- const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
- const int shift = (kt_local & 1) ? 4 : 0;
- const uint8_t hbit = (uint8_t) (1 << kt_local);
-
- for (int cp = 0; cp < 16; cp++) {
- const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;
- const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;
- tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);
-
- const int i = cp / 4;
- const int lane = (cp % 4) * 32 + row;
- if (b->qh[2 * cp + 0] & hbit) {
- plane[lane] |= (uint8_t) (1 << (2 * i));
- }
- if (b->qh[2 * cp + 1] & hbit) {
- plane[lane] |= (uint8_t) (1 << (2 * i + 1));
- }
- }
-
- ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);
- scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(D);
- scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(M);
- }
- }
- }
- }
-
- GGML_UNUSED(size);
-}
-
-// Reverse of repack_q5_K_tiled. Unpacks quants losslessly and normalizes scales/mins. Read-back only.
-static void repack_tiled_q5_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
- GGML_ASSERT(offset == 0);
-
- block_q5_K * dst_matrix = (block_q5_K *) data;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- int64_t ne2 = t->ne[2];
- int64_t ne3 = t->ne[3];
- int64_t ne0_padded = hex_round_up(ne0, 32);
- int64_t ne1_padded = hex_round_up(ne1, 32);
-
- GGML_ASSERT(ne0 % QK_K == 0);
-
- const int n_col_tiles = ne1_padded / 32;
- const int n_k_tiles = ne0_padded / 32;
- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q5_K;
- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
-
- const int64_t sb_per_row = ne0 / QK_K;
-
- for (int i3 = 0; i3 < ne3; i3++) {
- for (int i2 = 0; i2 < ne2; i2++) {
- block_q5_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);
- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;
-
- for (int64_t r = 0; r < ne1; r++) {
- const int ct = (int) (r / 32);
- const int row = (int) (r % 32);
- block_q5_K * dst_row = dst_slice + r * sb_per_row;
-
- for (int64_t sb = 0; sb < sb_per_row; sb++) {
- block_q5_K * b = &dst_row[sb];
- memset(b, 0, sizeof(block_q5_K));
-
- float sub_scales[8];
- float sub_mins[8];
-
- for (int kt_local = 0; kt_local < 8; kt_local++) {
- const int kt = sb * 8 + kt_local;
- const uint8_t * tile_src = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;
- const uint8_t * plane = tile_src + 640;
- const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);
-
- uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
- const int shift = (kt_local & 1) ? 4 : 0;
- const uint8_t hbit = (uint8_t) (1 << kt_local);
-
- for (int cp = 0; cp < 16; cp++) {
- const uint8_t val = tile_src[cp * 32 + row];
- const uint8_t q0 = val & 0x0F;
- const uint8_t q1 = val >> 4;
- qs_sub[2 * cp + 0] |= (uint8_t) (q0 << shift);
- qs_sub[2 * cp + 1] |= (uint8_t) (q1 << shift);
-
- const int i = cp / 4;
- const int lane = (cp % 4) * 32 + row;
- if (plane[lane] & (1 << (2 * i))) {
- b->qh[2 * cp + 0] |= hbit;
- }
- if (plane[lane] & (1 << (2 * i + 1))) {
- b->qh[2 * cp + 1] |= hbit;
- }
- }
-
- const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);
- const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);
- sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;
- sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;
- }
-
- float max_scale = 0.0f;
- float max_min = 0.0f;
- for (int j = 0; j < 8; j++) {
- if (sub_scales[j] > max_scale) max_scale = sub_scales[j];
- if (sub_mins[j] > max_min) max_min = sub_mins[j];
- }
-
- float inv_scale = 0.0f;
- if (max_scale > 0.0f) {
- b->d = GGML_FP32_TO_FP16(max_scale / 63.0f);
- const float d_actual = GGML_FP16_TO_FP32(b->d);
- inv_scale = (d_actual > 0.0f) ? (1.0f / d_actual) : 0.0f;
- } else {
- b->d = GGML_FP32_TO_FP16(0.0f);
- }
-
- float inv_min = 0.0f;
- if (max_min > 0.0f) {
- b->dmin = GGML_FP32_TO_FP16(max_min / 63.0f);
- const float dmin_actual = GGML_FP16_TO_FP32(b->dmin);
- inv_min = (dmin_actual > 0.0f) ? (1.0f / dmin_actual) : 0.0f;
- } else {
- b->dmin = GGML_FP32_TO_FP16(0.0f);
- }
+ return ptr;
+ }
- for (int j = 0; j < 8; j++) {
- uint8_t ls = (uint8_t) roundf(inv_scale * sub_scales[j]);
- uint8_t lm = (uint8_t) roundf(inv_min * sub_mins[j]);
- ls = (std::min)((uint8_t) 63, ls);
- lm = (std::min)((uint8_t) 63, lm);
- if (j < 4) {
- b->scales[j] = ls;
- b->scales[j + 4] = lm;
- } else {
- b->scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);
- b->scales[j - 4] |= ((ls >> 4) << 6);
- b->scales[j - 0] |= ((lm >> 4) << 6);
- }
- }
- }
- }
+ void free_slot(void * ptr, uint32_t n_slots = 1) {
+ if (!ptr) return;
+ uint32_t slot = ((uint8_t *) ptr - base()) / GGML_HEXAGON_FENCE_SLOT_SIZE;
+ for (uint32_t i = 0; i < n_slots; i++) {
+ free_slots.push_back(slot + i);
}
}
+};
- GGML_UNUSED(size);
+inline uint8_t * ggml_hexagon_session::alloc_fence(uint32_t n_slots) {
+ uint8_t * ptr = fence_buf->alloc_slot(n_slots);
+ GGML_ASSERT(ptr);
+ return ptr;
}
-static void repack_tensor_tiled(ggml_tensor * tensor, const void * data, size_t size) {
- switch (tensor->type) {
- case GGML_TYPE_Q4_0:
- repack_q4_0_tiled(tensor, data, 0, size);
- break;
-
- case GGML_TYPE_Q4_1:
- repack_q4_1_tiled(tensor, data, 0, size);
- break;
-
- case GGML_TYPE_Q4_K:
- repack_q4_K_tiled(tensor, data, 0, size);
- break;
-
- case GGML_TYPE_Q8_0:
- repack_q8_0_tiled(tensor, data, 0, size);
- break;
+inline void ggml_hexagon_session::free_fence(void * ptr, uint32_t n_slots) {
+ if (fence_buf) {
+ fence_buf->free_slot(ptr, n_slots);
+ }
+}
- case GGML_TYPE_IQ4_NL:
- repack_q4_0_tiled(tensor, data, 0, size);
- break;
+static void ggml_backend_hexagon_buffer_free_buffer(ggml_backend_buffer_t buffer) {
+ auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);
+ sbuf->sess->unclone_buffer(sbuf);
+ delete sbuf;
+}
- case GGML_TYPE_MXFP4:
- repack_mxfp4_tiled(tensor, data, 0, size);
- break;
+static void * ggml_backend_hexagon_buffer_get_base(ggml_backend_buffer_t buffer) {
+ auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);
+ return sbuf->base();
+}
- case GGML_TYPE_Q5_K:
- repack_q5_K_tiled(tensor, data, 0, size);
- break;
+static enum ggml_status ggml_backend_hexagon_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {
+ auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);
+ auto sess = sbuf->sess;
- case GGML_TYPE_Q6_K:
- repack_q6_K_tiled(tensor, data, 0, size);
- break;
+ HEX_VERBOSE("ggml-hex: %s init-tensor %s : base %p data %p nbytes %zu\n", sess->c_name(),
+ tensor->name, (void *) sbuf->base(), tensor->data, ggml_nbytes(tensor));
- case GGML_TYPE_Q3_K:
- repack_q3_K_tiled(tensor, data, 0, size);
- break;
+ auto extra = new ggml_hexagon_tensor_extra();
+ sbuf->tensor_extra.push_back(extra);
- case GGML_TYPE_Q2_K:
- repack_q2_K_tiled(tensor, data, 0, size);
- break;
+ tensor->extra = extra;
- default:
- break;
- }
+ return GGML_STATUS_SUCCESS;
}
static void ggml_backend_hexagon_buffer_set_tensor(ggml_backend_buffer_t buffer,
@@ -2474,21 +907,21 @@ static void ggml_backend_hexagon_buffer_set_tensor(ggml_backend_buffer_t buffer,
if (ggml_backend_buffer_get_usage(buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
extra->flags |= GGML_HEXAGON_TENSOR_WEIGHT;
- if (ggml_hexagon_is_repack_type(tensor->type)) {
- extra->flags |= GGML_HEXAGON_TENSOR_REPACK;
+ if (htp_is_tiled_weight_tensor(tensor)) {
+ extra->flags |= GGML_HEXAGON_TENSOR_TILED;
}
}
HEX_VERBOSE("ggml-hex: %s set-tensor %s : data %p offset %zu size %zu usage %d flags 0x%x\n",
sess->c_name(), tensor->name, data, offset, size, (int) buffer->usage, extra->flags);
- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {
+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {
memcpy((char *) tensor->data + offset, data, size);
return;
}
if (offset == 0 && size == ggml_nbytes(tensor) && extra->shadow_buf.empty()) {
- repack_tensor_tiled(tensor, data, size);
+ htp_repack_tensor_tiled(tensor, data, size);
return;
}
@@ -2499,7 +932,7 @@ static void ggml_backend_hexagon_buffer_set_tensor(ggml_backend_buffer_t buffer,
extra->shadow_size += size;
if (extra->shadow_size >= ggml_nbytes(tensor)) {
- repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());
+ htp_repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());
extra->shadow_buf.clear();
extra->shadow_buf.shrink_to_fit();
extra->shadow_size = 0;
@@ -2518,76 +951,13 @@ static void ggml_backend_hexagon_buffer_get_tensor(ggml_backend_buffer_t buffer,
HEX_VERBOSE("ggml-hex: %s get-tensor %s : data %p offset %zu size %zu usage %d flags 0x%x\n",
sess->c_name(), tensor->name, data, offset, size, (int) buffer->usage, extra->flags);
- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {
+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {
memcpy(data, (const char *) tensor->data + offset, size);
return;
}
- switch (tensor->type) {
- case GGML_TYPE_Q4_0:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q4_0(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_Q4_1:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q4_1(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_Q4_K:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q4_K(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_Q5_K:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q5_K(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_Q8_0:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q8_0(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_IQ4_NL:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q4_0(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_MXFP4:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_mxfp4(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_Q6_K:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q6_K(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_Q3_K:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q3_K(data, tensor, offset, size);
- break;
-
- case GGML_TYPE_Q2_K:
- GGML_ASSERT(offset == 0);
- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
- repack_tiled_q2_K(data, tensor, offset, size);
- break;
-
- default:
- memcpy(data, (const char *) tensor->data + offset, size);
- break;
- }
+ GGML_ASSERT(offset == 0);
+ htp_unpack_tensor_tiled(data, tensor, offset, size);
}
static bool ggml_backend_hexagon_buffer_cpy_tensor(ggml_backend_buffer_t buffer,
@@ -2615,15 +985,15 @@ static void ggml_backend_hexagon_buffer_set_tensor_2d(ggml_backend_buffer_t buff
if (ggml_backend_buffer_get_usage(buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
extra->flags |= GGML_HEXAGON_TENSOR_WEIGHT;
- if (ggml_hexagon_is_repack_type(tensor->type)) {
- extra->flags |= GGML_HEXAGON_TENSOR_REPACK;
+ if (htp_is_tiled_weight_tensor(tensor)) {
+ extra->flags |= GGML_HEXAGON_TENSOR_TILED;
}
}
HEX_VERBOSE("ggml-hex: %s set-tensor-2d %s : data %p offset %zu size %zu n_copies %zu stride_tensor %zu stride_data %zu usage %d flags 0x%x\n",
sess->c_name(), tensor->name, data, offset, size, n_copies, stride_tensor, stride_data, (int) buffer->usage, extra->flags);
- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {
+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {
for (size_t i = 0; i < n_copies; i++) {
memcpy((uint8_t *) tensor->data + offset + i * stride_tensor, (const uint8_t *) data + i * stride_data, size);
}
@@ -2639,7 +1009,7 @@ static void ggml_backend_hexagon_buffer_set_tensor_2d(ggml_backend_buffer_t buff
extra->shadow_size += n_copies * size;
if (extra->shadow_size >= ggml_nbytes(tensor)) {
- repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());
+ htp_repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());
extra->shadow_buf.clear();
extra->shadow_buf.shrink_to_fit();
extra->shadow_size = 0;
@@ -2661,7 +1031,7 @@ static void ggml_backend_hexagon_buffer_get_tensor_2d(ggml_backend_buffer_t buff
HEX_VERBOSE("ggml-hex: %s get-tensor-2d %s : data %p offset %zu size %zu n_copies %zu stride_tensor %zu stride_data %zu usage %d\n",
sess->c_name(), tensor->name, data, offset, size, n_copies, stride_tensor, stride_data, (int) buffer->usage);
- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {
+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {
for (size_t i = 0; i < n_copies; i++) {
memcpy((uint8_t *)data + i * stride_data, (const uint8_t *)tensor->data + offset + i * stride_tensor, size);
}
@@ -2679,52 +1049,7 @@ static void ggml_backend_hexagon_buffer_get_tensor_2d(ggml_backend_buffer_t buff
GGML_ASSERT((offset + temp_size) <= ggml_nbytes(tensor));
std::vector<uint8_t> temp_buf(temp_size);
-
- switch (tensor->type) {
- case GGML_TYPE_Q4_0:
- repack_tiled_q4_0(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_Q4_1:
- repack_tiled_q4_1(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_Q4_K:
- repack_tiled_q4_K(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_Q5_K:
- repack_tiled_q5_K(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_Q8_0:
- repack_tiled_q8_0(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_IQ4_NL:
- repack_tiled_q4_0(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_MXFP4:
- repack_tiled_mxfp4(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_Q6_K:
- repack_tiled_q6_K(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_Q3_K:
- repack_tiled_q3_K(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- case GGML_TYPE_Q2_K:
- repack_tiled_q2_K(temp_buf.data(), tensor, offset, temp_size);
- break;
-
- default:
- memcpy(temp_buf.data(), (const uint8_t *) tensor->data + offset, temp_size);
- break;
- }
+ htp_unpack_tensor_tiled(temp_buf.data(), tensor, offset, temp_size);
for (size_t i = 0; i < n_copies; i++) {
memcpy((uint8_t *) data + i * stride_data, temp_buf.data() + i * stride_tensor, size);
@@ -2834,16 +1159,15 @@ static size_t ggml_backend_hexagon_buffer_type_get_alignment(ggml_backend_buffer
}
static size_t ggml_backend_hexagon_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const struct ggml_tensor * t) {
- if (ggml_hexagon_is_repack_type(t->type)) {
+ if (!ggml_backend_buft_is_host(buft) && htp_is_tiled_weight_tensor(t)) {
int64_t ne0 = hex_round_up(t->ne[0], 32);
int64_t ne1 = hex_round_up(t->ne[1], 32);
int64_t ne2 = t->ne[2];
int64_t ne3 = t->ne[3];
- return ggml_hexagon_tiled_row_size(t->type, ne0) * ne1 * ne2 * ne3;
+ size_t tiled_size = htp_tiled_row_size(t->type, ne0) * ne1 * ne2 * ne3;
+ return std::max(tiled_size, ggml_nbytes(t));
}
return ggml_nbytes(t);
-
- GGML_UNUSED(buft);
}
static size_t parse_size(const char * str, size_t default_unit = 1024 * 1024) {
@@ -3196,20 +1520,19 @@ struct ggml_hexagon_opbatch {
bool same_shape(const htp_tensor * h, const ggml_tensor * t) const {
auto extra = (ggml_hexagon_tensor_extra *) t->extra;
- int64_t ne0 = t->ne[0];
- int64_t ne1 = t->ne[1];
- const bool is_repack = (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0;
- if (is_repack) {
- ne0 = hex_round_up(ne0, 32);
- ne1 = hex_round_up(ne1, 32);
- }
- int64_t nb1 = is_repack ? (int64_t) ggml_hexagon_tiled_row_size(t->type, ne0) : t->nb[1];
- int64_t nb2 = is_repack ? nb1 * ne1 : t->nb[2];
- int64_t nb3 = is_repack ? nb2 * t->ne[2] : t->nb[3];
+ const bool is_tiled = (extra->flags & GGML_HEXAGON_TENSOR_TILED) != 0;
+ const int64_t ne0_padded = hex_round_up(t->ne[0], 32);
+ const int64_t ne1_padded = hex_round_up(t->ne[1], 32);
+
+ int64_t nb1 = is_tiled ? (int64_t) htp_tiled_row_size(t->type, ne0_padded) : t->nb[1];
+ int64_t nb2 = is_tiled ? nb1 * ne1_padded : t->nb[2];
+ int64_t nb3 = is_tiled ? nb2 * t->ne[2] : t->nb[3];
+ int64_t nb0 = (is_tiled && (t->type == GGML_TYPE_F32 || t->type == GGML_TYPE_BF16)) ? sizeof(ggml_fp16_t) : t->nb[0];
+ enum ggml_type expected_type = (is_tiled && (t->type == GGML_TYPE_F32 || t->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : t->type;
- return (h->type == t->type) &&
- (h->ne[0] == ne0) && (h->ne[1] == ne1) && (h->ne[2] == t->ne[2]) && (h->ne[3] == t->ne[3]) &&
- (h->nb[0] == t->nb[0]) && (h->nb[1] == nb1) && (h->nb[2] == nb2) && (h->nb[3] == nb3);
+ return (h->type == expected_type) &&
+ (h->ne[0] == t->ne[0]) && (h->ne[1] == t->ne[1]) && (h->ne[2] == t->ne[2]) && (h->ne[3] == t->ne[3]) &&
+ (h->nb[0] == nb0) && (h->nb[1] == nb1) && (h->nb[2] == nb2) && (h->nb[3] == nb3);
}
// add tensor and return its index
@@ -3243,23 +1566,28 @@ struct ggml_hexagon_opbatch {
h.ti = ti;
h.data = t_offset;
h.type = t->type;
-
- const bool is_repack = (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0;
- if (is_repack) {
- h.ne[0] = hex_round_up(t->ne[0], 32);
- h.ne[1] = hex_round_up(t->ne[1], 32);
- h.ne[2] = t->ne[2];
- h.ne[3] = t->ne[3];
-
- h.nb[0] = t->nb[0];
- h.nb[1] = ggml_hexagon_tiled_row_size(t->type, h.ne[0]);
- h.nb[2] = h.nb[1] * h.ne[1];
+ h.ne[0] = t->ne[0];
+ h.ne[1] = t->ne[1];
+ h.ne[2] = t->ne[2];
+ h.ne[3] = t->ne[3];
+
+ const bool is_tiled = (extra->flags & GGML_HEXAGON_TENSOR_TILED) != 0;
+ if (is_tiled) {
+ if (t->type == GGML_TYPE_F32 || t->type == GGML_TYPE_BF16) {
+ h.type = GGML_TYPE_F16;
+ h.nb[0] = sizeof(ggml_fp16_t);
+ } else {
+ h.nb[0] = t->nb[0];
+ }
+ const int64_t ne0_padded = hex_round_up(t->ne[0], 32);
+ const int64_t ne1_padded = hex_round_up(t->ne[1], 32);
+ h.nb[1] = htp_tiled_row_size(t->type, ne0_padded);
+ h.nb[2] = h.nb[1] * ne1_padded;
h.nb[3] = h.nb[2] * h.ne[2];
h.size = h.nb[3] * h.ne[3];
t_size = h.size;
} else {
h.size = t_size;
- h.ne[0] = t->ne[0]; h.ne[1] = t->ne[1]; h.ne[2] = t->ne[2]; h.ne[3] = t->ne[3];
h.nb[0] = t->nb[0]; h.nb[1] = t->nb[1]; h.nb[2] = t->nb[2]; h.nb[3] = t->nb[3];
}
@@ -3267,8 +1595,8 @@ struct ggml_hexagon_opbatch {
if ((extra->flags & GGML_HEXAGON_TENSOR_WEIGHT) != 0) {
h.flags |= HTP_TENSOR_WEIGHT;
}
- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0) {
- h.flags |= HTP_TENSOR_REPACK;
+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) != 0) {
+ h.flags |= HTP_TENSOR_TILED;
}
if ((extra->flags & GGML_HEXAGON_TENSOR_FENCE) != 0) {
h.flags |= HTP_TENSOR_FENCE;
@@ -3666,6 +1994,9 @@ struct ggml_hexagon_opbatch {
if (src2->type != GGML_TYPE_F32) return false;
+ // the fused kernel writes the shape of the MUL_MAT output: the ADD must not broadcast it
+ if (!ggml_are_same_shape(mm_out, node.dst())) return false;
+
const struct htp_mm_kernel_params * orig_kparams = (const struct htp_mm_kernel_params *) last_node.kernel_params;
struct htp_mm_kernel_params kparams;
ggml_hexagon_precompute_fused_matmul_add_params(sess, src0, src1, src2, node.dst(), &kparams);
@@ -3753,7 +2084,8 @@ struct ggml_hexagon_opbatch {
const ggml_tensor * w0 = last_node.inputs[0];
const ggml_tensor * x = last_node.inputs[curr_n];
- if (x_in != x || w_in->type != w0->type || w_in->ne[0] != w0->ne[0]) {
+ if (x_in != x || w_in->type != w0->type || w_in->ne[0] != w0->ne[0] ||
+ ggml_hexagon_tensor_is_tiled(w_in) != ggml_hexagon_tensor_is_tiled(w0)) {
return false;
}
if (!last_node.fused.empty() && (mm_is_hmx_eligible(last_node.fused[0]) != mm_is_hmx_eligible(node.node))) {
@@ -3909,7 +2241,8 @@ struct ggml_hexagon_opbatch {
const ggml_tensor * x = last_node.inputs[curr_n];
const ggml_tensor * ids = last_node.inputs[curr_n + 1];
- if (x_in != x || ids_in != ids || w_in->type != w0->type || w_in->ne[0] != w0->ne[0] || w_in->ne[2] != w0->ne[2]) {
+ if (x_in != x || ids_in != ids || w_in->type != w0->type || w_in->ne[0] != w0->ne[0] || w_in->ne[2] != w0->ne[2] ||
+ ggml_hexagon_tensor_is_tiled(w_in) != ggml_hexagon_tensor_is_tiled(w0)) {
return false;
}
if (!last_node.fused.empty() && (mm_is_hmx_eligible(last_node.fused[0]) != mm_is_hmx_eligible(node.node))) {
@@ -5433,7 +3766,6 @@ static bool ggml_hexagon_matmul_is_hmx_eligible(
const struct ggml_tensor * src0,
const struct ggml_tensor * src1,
const struct ggml_tensor * dst,
- int ne01_padded,
bool is_matmul_id,
bool is_batched
) {
@@ -5442,17 +3774,18 @@ static bool ggml_hexagon_matmul_is_hmx_eligible(
}
const int ne00 = src0->ne[0];
+ const int ne01 = src0->ne[1];
const int ne11 = src1->ne[1];
const int ne12 = src1->ne[2];
const int wtype = src0->type;
// HMX weight tiles accept non-32-aligned N for non-matmul_id.
- if (ne01_padded % 32 != 0 && is_matmul_id) {
+ if (ne01 % 32 != 0 && is_matmul_id) {
return false;
}
// HMX supports F16, F32, and repack quantized types.
- if (!ggml_hexagon_is_hmx_weight_type((ggml_type) wtype)) {
+ if (!htp_is_hmx_weight_type((ggml_type) wtype)) {
return false;
}
@@ -5461,8 +3794,12 @@ static bool ggml_hexagon_matmul_is_hmx_eligible(
return false;
}
- // Quantized HMX kernels only handle flat 2D matmul (or matmul_id wrapping flat 2D matmuls).
- if (!is_matmul_id && is_batched && wtype != GGML_TYPE_F16) {
+ // Quantized and tiled HMX kernels only handle flat 2D matmul (or matmul_id wrapping flat 2D matmuls).
+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);
+ if (ggml_is_quantized((ggml_type) wtype) && !is_tiled) {
+ return false;
+ }
+ if (!is_matmul_id && is_batched && is_tiled) {
return false;
}
@@ -5501,7 +3838,8 @@ static bool ggml_hexagon_precompute_hmx_mm_params(
size_t vtcm_budget,
struct htp_mm_kernel_params * kparams
) {
- const int aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);
+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);
+ const int aligned_tile_size = is_tiled ? htp_tensor_get_tile_padded_size(wtype) : 0;
const int n_threads = (int)sess->n_threads;
const int ne10 = act->ne[0];
@@ -5528,7 +3866,7 @@ static bool ggml_hexagon_precompute_hmx_mm_params(
bool use_grouped = false;
int act_threads_selected = 0;
- if (is_batched_val && wtype == GGML_TYPE_F16 && group_size > 1) {
+ if (is_batched_val && wtype == GGML_TYPE_F16 && !is_tiled && group_size > 1) {
// Try grouped path first
if (htp_mm_hmx_solve_batched_params(wtype, ne00_padded, ne01_padded, m_for_solver, group_size, n_threads, pipeline, src2_size, vtcm_budget, &m_chunk, &n_chunk, &act_threads_selected, &vtcm_size)) {
use_grouped = true;
@@ -5559,9 +3897,9 @@ static bool ggml_hexagon_precompute_hmx_mm_params(
kparams->n_chunk = n_chunk;
kparams->n_threads = n_threads;
kparams->n_act_threads = act_threads_selected;
- kparams->tile_size = htp_mm_get_weight_tile_size(wtype);
+ kparams->tile_size = is_tiled ? htp_tensor_get_tile_size(wtype) : 0;
kparams->aligned_tile_size = aligned_tile_size;
- kparams->act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);
+ kparams->act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);
kparams->vtcm_size = vtcm_size;
kparams->vtcm_src0_size = 0;
kparams->div_n_act_threads = init_fastdiv_values(act_threads_selected);
@@ -5571,13 +3909,11 @@ static bool ggml_hexagon_precompute_hmx_mm_params(
kparams->vtcm_dst_size = 0;
if (is_batched && !is_matmul_id) {
- kparams->kernel_type = HTP_MM_KERNEL_HMX_F16_BATCHED;
+ kparams->kernel_type = HTP_MM_KERNEL_HMX_BATCHED;
} else {
kparams->kernel_type = HTP_MM_KERNEL_HMX_2D;
}
return true;
-
- GGML_UNUSED(src0);
}
static void ggml_hexagon_precompute_hvx_mm_params(
@@ -5586,6 +3922,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(
const struct ggml_tensor * src1,
const struct ggml_tensor * dst,
int wtype,
+ bool is_tiled,
int ne02,
int ne03,
int ne10,
@@ -5608,19 +3945,19 @@ static void ggml_hexagon_precompute_hvx_mm_params(
GGML_UNUSED(ne02);
GGML_UNUSED(ne03);
- const bool is_quant = (wtype != GGML_TYPE_F16 && wtype != GGML_TYPE_F32);
const int src1_nrows = ne11 * ne12 * ne13;
- if (is_quant) {
- // Quantized HVX
- kparams->tile_size = htp_mm_get_weight_tile_size(wtype);
- kparams->aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);
+ if (is_tiled) {
+ // Tiled HVX (quantized weights or tiled F16/F32/BF16)
+ kparams->tile_size = htp_tensor_get_tile_size(wtype);
+ kparams->aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);
const bool k_align = (ne10 % 32 == 0);
if (is_matmul_id) {
- kparams->kernel_type = (src1_nrows < (int) sess->n_threads) ? HTP_MM_KERNEL_HVX_QUANT_BLOCK : HTP_MM_KERNEL_HVX_QUANT_ROW;
- kparams->act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);
+ kparams->kernel_type = (src1_nrows < (int) sess->n_threads)
+ ? HTP_MM_KERNEL_HVX_QUANT_BLOCK : HTP_MM_KERNEL_HVX_QUANT_ROW;
+ kparams->act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);
struct htp_mm_hvx_vtcm_layout L;
uint32_t max_prefetch = (src1_nrows > HTP_MM_HMX_MIN_NROWS) ? 2 : 16;
@@ -5645,13 +3982,11 @@ static void ggml_hexagon_precompute_hvx_mm_params(
kparams->vtcm_act_size = L.act_bytes;
kparams->vtcm_bias_size = 0;
kparams->vtcm_dst_size = L.dst_bytes;
- goto done_quant;
+ return;
} else {
bool try_tiled = (k_align && opt_mm_select >= 1);
if (try_tiled) {
- kparams->act_row_size = htp_mm_weight_has_offset(wtype)
- ? htp_mm_q8_1_tiled_row_size(ne10)
- : htp_mm_q8_0_tiled_row_size(ne10);
+ kparams->act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);
if (src1_nrows < (int) sess->n_threads) {
kparams->kernel_type = HTP_MM_KERNEL_HVX_QUANT_BLOCK;
} else {
@@ -5684,17 +4019,18 @@ static void ggml_hexagon_precompute_hvx_mm_params(
kparams->vtcm_act_size = L.act_bytes;
kparams->vtcm_bias_size = L.bias_bytes;
kparams->vtcm_dst_size = L.dst_bytes;
- goto done_quant;
+ return;
}
}
kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
return;
}
-
- done_quant:;
+ } else if (is_matmul_id) {
+ kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
+ return;
} else if (wtype == GGML_TYPE_F16) {
- // F16 HVX
+ // Flat F16 HVX
struct htp_mm_hvx_vtcm_layout L;
uint32_t m_chunk = 0;
if (htp_mm_hvx_solve_vtcm_params(
@@ -5715,8 +4051,8 @@ static void ggml_hexagon_precompute_hvx_mm_params(
kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
return;
- } else {
- // F32 HVX
+ } else if (wtype == GGML_TYPE_F32) {
+ // Flat F32 HVX
struct htp_mm_hvx_vtcm_layout L;
uint32_t m_chunk = 0;
if (htp_mm_hvx_solve_vtcm_params(
@@ -5735,6 +4071,31 @@ static void ggml_hexagon_precompute_hvx_mm_params(
return;
}
+ kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
+ return;
+ } else if (wtype == GGML_TYPE_BF16) {
+ // Flat BF16 HVX
+ struct htp_mm_hvx_vtcm_layout L;
+ uint32_t m_chunk = 0;
+ if (htp_mm_hvx_solve_vtcm_params(
+ HTP_MM_KERNEL_HVX_BF16_F32_VTCM, wtype, ne10, src1_nrows, sess->n_threads,
+ dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 16, vtcm_budget,
+ &L, &m_chunk)) {
+ kparams->kernel_type = HTP_MM_KERNEL_HVX_BF16_F32_VTCM;
+ kparams->m_chunk = (m_chunk < (uint32_t) src1_nrows) ? m_chunk : 0;
+ kparams->act_row_size = hex_round_up(ne10 * 4, 128);
+ kparams->vtcm_size = L.total_bytes;
+ kparams->vtcm_src0_size = L.src0_bytes;
+ kparams->vtcm_act_size = L.act_bytes;
+ kparams->vtcm_bias_size = L.bias_bytes;
+ kparams->vtcm_dst_size = L.dst_bytes;
+ kparams->n_prefetch = 16;
+ return;
+ }
+
+ kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
+ return;
+ } else {
kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
return;
}
@@ -5761,14 +4122,13 @@ static void ggml_hexagon_precompute_matmul_params_impl(
const int ne12 = src1->ne[2];
const int ne13 = src1->ne[3];
- const int wtype = src0->type;
- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);
- const int ne00_padded = is_repack ? hex_round_up(ne00, 32) : ne00;
- const int ne01_padded = is_repack ? hex_round_up(ne01, 32) : ne01;
+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);
+ const int wtype = (is_tiled && (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : src0->type;
+ const int ne00_padded = is_tiled ? hex_round_up(ne00, 32) : ne00;
const int ne11_padded = hex_round_up(ne11, 32);
// VTCM has to hold whole 32-row weight tiles, so size for the rounded-up N
// even when the tensor itself is ragged.
- const int ne01_tiled = hex_round_up(ne01_padded, 32);
+ const int ne01_tiled = hex_round_up(ne01, 32);
const bool is_matmul_id = (dst->op == GGML_OP_MUL_MAT_ID);
const bool is_batched = (ne02 * ne03 > 1 || ne12 * ne13 > 1);
@@ -5777,14 +4137,14 @@ static void ggml_hexagon_precompute_matmul_params_impl(
// Check HMX eligibility and try precomputing HMX parameters
bool hmx_enabled = (sess->n_hmx > 0) && (opt_mm_select >= 2);
- if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, src1, dst, ne01_padded, is_matmul_id, is_batched)) {
+ if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, src1, dst, is_matmul_id, is_batched)) {
if (ggml_hexagon_precompute_hmx_mm_params(sess, src0, src1, dst, wtype, ne00_padded, ne01_tiled, ne02, ne11, ne12, ne11_padded, is_matmul_id, is_batched, src2_size, vtcm_budget, kparams)) {
goto finalize;
}
}
// Fallback to HVX parameter computation
- ggml_hexagon_precompute_hvx_mm_params(sess, src0, src1, dst, wtype, ne02, ne03, ne10, ne11, ne12, ne13, is_matmul_id, src2_row_size, vtcm_budget, kparams);
+ ggml_hexagon_precompute_hvx_mm_params(sess, src0, src1, dst, wtype, is_tiled, ne02, ne03, ne10, ne11, ne12, ne13, is_matmul_id, src2_row_size, vtcm_budget, kparams);
finalize:
kparams->div_ne12_ne1 = init_fastdiv_values(ne12 * ne11);
@@ -6170,29 +4530,24 @@ static void ggml_hexagon_precompute_get_rows_params(
const uint32_t ne12 = src1->ne[2];
const uint32_t nr = ne10 * ne11 * ne12;
- const ggml_tensor * src0_base = src0->view_src ? src0->view_src : src0;
- const auto * extra = src0_base->buffer && ggml_backend_buffer_is_hexagon(src0_base->buffer) ?
- (const ggml_hexagon_tensor_extra *) src0_base->extra : nullptr;
- const bool tiled = src0->type == GGML_TYPE_Q4_0 || (extra && (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0) ||
- sess->needs_repack.count(src0_base) || sess->needs_repack.count(src0);
+ const bool tiled = ggml_hexagon_tensor_is_tiled(src0);
+ const int wtype = (tiled && (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : src0->type;
- if (src0->type == dst->type) {
- kparams->kernel_type = HTP_GET_ROWS_KERNEL_SAMETYPE;
- } else if (tiled) {
+ if (tiled) {
kparams->kernel_type = HTP_GET_ROWS_KERNEL_TILED;
+ } else if (src0->type == dst->type) {
+ kparams->kernel_type = HTP_GET_ROWS_KERNEL_SAMETYPE;
} else {
kparams->kernel_type = HTP_GET_ROWS_KERNEL_FLAT;
}
- const uint32_t chunks_per_row = 1;
- const uint32_t chunk_size = ne00;
const uint32_t total_tasks = nr;
kparams->n_threads = (std::min)((uint32_t)sess->n_threads, total_tasks);
struct htp_get_rows_vtcm_layout vtcm_layout = {};
while (kparams->n_threads > 0) {
- htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, src0->type, ne00, kparams->n_threads);
+ htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, wtype, ne00, total_tasks, src1->type, kparams->n_threads);
if (vtcm_layout.total_bytes <= sess->vtcm_size) {
break;
}
@@ -6200,19 +4555,15 @@ static void ggml_hexagon_precompute_get_rows_params(
}
if (kparams->n_threads == 0 && total_tasks > 0) {
- htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, src0->type, ne00, 1);
+ htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, wtype, ne00, total_tasks, src1->type, 1);
}
kparams->vtcm_size = (total_tasks == 0) ? 0 : vtcm_layout.total_bytes;
kparams->tasks_per_thread = kparams->n_threads > 0 ? (total_tasks + kparams->n_threads - 1) / kparams->n_threads : 0;
-
- kparams->chunks_per_row = chunks_per_row;
- kparams->chunk_size = chunk_size;
kparams->total_tasks = total_tasks;
kparams->div_ne10 = ne10 > 0 ? init_fastdiv_values(ne10) : fastdiv_values{0, 0};
kparams->div_ne10_ne11 = (ne10 * ne11) > 0 ? init_fastdiv_values(ne10 * ne11) : fastdiv_values{0, 0};
- kparams->div_chunks_per_row = chunks_per_row > 0 ? init_fastdiv_values(chunks_per_row) : fastdiv_values{0, 0};
kparams->div_ne02 = ne02 > 0 ? init_fastdiv_values(ne02) : fastdiv_values{0, 0};
kparams->div_ne03 = ne03 > 0 ? init_fastdiv_values(ne03) : fastdiv_values{0, 0};
}
@@ -6920,49 +5271,44 @@ static void ggml_hexagon_precompute_fused_mmnx_params(
const int ne12 = act->ne[2];
const int ne13 = act->ne[3];
- const int wtype = src0->type;
- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);
- const int ne00_padded = is_repack ? hex_round_up(ne00, 32) : ne00;
- const int ne01_padded = is_repack ? hex_round_up(ne01, 32) : ne01;
+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);
+ const int wtype = (is_tiled && (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : src0->type;
+ const int ne00_padded = is_tiled ? hex_round_up(ne00, 32) : ne00;
const int ne11_padded = hex_round_up(ne11, 32);
- const int ne01_tiled = hex_round_up(ne01_padded, 32);
+ const int ne01_tiled = hex_round_up(ne01, 32);
const size_t vtcm_budget = sess->vtcm_size;
const bool is_batched = (ne02 * ne03 > 1 || ne12 * ne13 > 1);
bool hmx_enabled = (sess->n_hmx > 0) && (opt_mm_select >= 2);
- if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, act, nullptr, ne01_padded, false, is_batched)) {
+ if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, act, nullptr, false, is_batched)) {
if (ggml_hexagon_precompute_hmx_mm_params(sess, src0, act, nullptr, wtype, ne00_padded, ne01_tiled, ne02, ne11, ne12, ne11_padded, false, is_batched, 0, vtcm_budget, kparams)) {
kparams->n_weights = n_weights;
goto finalize;
}
}
- if (!is_repack) {
+ if (!is_tiled || ne10 % 32 != 0) {
kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
return;
}
{
const int act_nrows = ne11 * ne12 * ne13;
- const size_t act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);
+ const size_t act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);
const size_t src0_row_size = src0->nb[1];
- uint32_t best_n_prefetch = 16;
-
- if (is_repack) {
- const uint32_t max_prefetch = (act_nrows > HTP_MM_HMX_MIN_NROWS) ? 2 : 16;
- best_n_prefetch = 2;
- for (uint32_t d = max_prefetch; d >= 2; d /= 2) {
- struct htp_mm_hvx_vtcm_layout L;
- htp_mm_hvx_vtcm_layout_build(
- &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, act_nrows, sess->n_threads,
- 0, src0_row_size, act_row_size, 0, d, false, true
- );
- if (L.total_bytes <= sess->vtcm_size) {
- best_n_prefetch = d;
- break;
- }
+ const uint32_t max_prefetch = (act_nrows > HTP_MM_HMX_MIN_NROWS) ? 2 : 16;
+ uint32_t best_n_prefetch = 2;
+ for (uint32_t d = max_prefetch; d >= 2; d /= 2) {
+ struct htp_mm_hvx_vtcm_layout L;
+ htp_mm_hvx_vtcm_layout_build(
+ &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, act_nrows, sess->n_threads,
+ 0, src0_row_size, act_row_size, 0, d, false, true
+ );
+ if (L.total_bytes <= sess->vtcm_size) {
+ best_n_prefetch = d;
+ break;
}
}
@@ -6976,15 +5322,17 @@ static void ggml_hexagon_precompute_fused_mmnx_params(
);
if (try_tiled && L.total_bytes <= sess->vtcm_size) {
- kparams->kernel_type = HTP_MM_KERNEL_HVX_QUANT_ROW;
- kparams->act_row_size = act_row_size;
- kparams->vtcm_src0_size = L.src0_bytes;
- kparams->vtcm_act_size = L.act_bytes;
- kparams->vtcm_bias_size = 0;
- kparams->vtcm_dst_size = L.dst_bytes;
- kparams->vtcm_size = L.total_bytes;
- kparams->n_prefetch = best_n_prefetch;
- kparams->n_weights = n_weights;
+ kparams->kernel_type = HTP_MM_KERNEL_HVX_QUANT_ROW;
+ kparams->tile_size = htp_tensor_get_tile_size(wtype);
+ kparams->aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);
+ kparams->act_row_size = act_row_size;
+ kparams->vtcm_src0_size = L.src0_bytes;
+ kparams->vtcm_act_size = L.act_bytes;
+ kparams->vtcm_bias_size = 0;
+ kparams->vtcm_dst_size = L.dst_bytes;
+ kparams->vtcm_size = L.total_bytes;
+ kparams->n_prefetch = best_n_prefetch;
+ kparams->n_weights = n_weights;
} else {
kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;
return;
@@ -7011,16 +5359,6 @@ static void ggml_hexagon_precompute_fused_mmidnx_params(
kparams->n_weights = n_weights;
}
-static bool ggml_hexagon_tensor_is_host(const struct ggml_hexagon_session * sess, const struct ggml_tensor * t) {
- return t && t->buffer && ggml_backend_buft_is_host(t->buffer->buft);
- GGML_UNUSED(sess);
-}
-
-static bool ggml_hexagon_tensor_is_non_host(const struct ggml_hexagon_session * sess, const struct ggml_tensor * t) {
- return t && t->buffer && !ggml_backend_buft_is_host(t->buffer->buft);
- GGML_UNUSED(sess);
-}
-
static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * sess, const struct ggml_tensor * dst) {
const struct ggml_tensor * src0 = dst->src[0];
const struct ggml_tensor * src1 = dst->src[1];
@@ -7044,7 +5382,7 @@ static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * s
case GGML_TYPE_Q6_K:
case GGML_TYPE_Q3_K:
case GGML_TYPE_Q2_K:
- if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0)) {
+ if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0) || ggml_is_view(src0)) {
return false;
}
@@ -7059,10 +5397,6 @@ static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * s
if (src1->ne[2] % src0->ne[2] != 0 || src1->ne[3] % src0->ne[3] != 0) {
return false;
}
-
- if (!src0->buffer) {
- sess->needs_repack.insert(src0);
- }
break;
case GGML_TYPE_F16:
@@ -7092,6 +5426,18 @@ static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * s
}
break;
+ case GGML_TYPE_BF16:
+ if (src0->nb[1] < src0->nb[0]) {
+ return false;
+ }
+ if (src1->ne[2] < src0->ne[2] || src1->ne[3] < src0->ne[3]) {
+ return false;
+ }
+ if (src1->ne[2] % src0->ne[2] != 0 || src1->ne[3] % src0->ne[3] != 0) {
+ return false;
+ }
+ break;
+
default:
return false;
}
@@ -7131,7 +5477,7 @@ static bool ggml_hexagon_supported_mul_mat_id(const struct ggml_hexagon_session
case GGML_TYPE_Q6_K:
case GGML_TYPE_Q3_K:
case GGML_TYPE_Q2_K:
- if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0)) {
+ if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0) || ggml_is_view(src0)) {
return false;
}
@@ -7139,9 +5485,16 @@ static bool ggml_hexagon_supported_mul_mat_id(const struct ggml_hexagon_session
src0->type == GGML_TYPE_Q3_K || src0->type == GGML_TYPE_Q2_K) ? QK_K : 32)) {
return false;
}
+ break;
- if (!src0->buffer) {
- sess->needs_repack.insert(src0);
+ case GGML_TYPE_F16:
+ case GGML_TYPE_F32:
+ case GGML_TYPE_BF16:
+ if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0)) {
+ return false;
+ }
+ if (src0->ne[0] % 32 != 0) {
+ return false;
}
break;
@@ -7564,40 +5917,12 @@ static bool ggml_hexagon_supported_get_rows(const struct ggml_hexagon_session *
const struct ggml_tensor * src1 = op->src[1]; // indices
const struct ggml_tensor * dst = op;
- if ((src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && src0->view_src) {
- return false;
- }
-
- const ggml_tensor * src0_base = src0->view_src ? src0->view_src : src0;
- bool is_repacked = false;
- if (src0_base->buffer && ggml_backend_buffer_is_hexagon(src0_base->buffer) && src0_base->extra) {
- const auto * extra = (const ggml_hexagon_tensor_extra *) src0_base->extra;
- is_repacked = (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0;
- if (is_repacked && src0->type != GGML_TYPE_Q4_0 && src0->type != GGML_TYPE_Q4_K && src0->type != GGML_TYPE_Q6_K && src0->type != GGML_TYPE_Q8_0) {
- return false;
- }
- }
- is_repacked = is_repacked || sess->needs_repack.count(src0_base) || sess->needs_repack.count(src0);
-
- // View offsets use the raw quantized layout and cannot address a tiled allocation.
- if (src0->view_src && is_repacked) {
- return false;
- }
-
- if ((src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && src0->buffer && ggml_backend_buffer_get_size(src0->buffer) != 0 && !is_repacked) {
- return false;
- }
-
- if (src0->type != dst->type && src0->ne[0] < 32) {
+ if (src0->type != GGML_TYPE_F32 && src0->type != GGML_TYPE_F16 && src0->type != GGML_TYPE_BF16 &&
+ src0->type != GGML_TYPE_Q4_0 && src0->type != GGML_TYPE_Q4_K && src0->type != GGML_TYPE_Q5_K && src0->type != GGML_TYPE_Q6_K && src0->type != GGML_TYPE_Q8_0 && src0->type != GGML_TYPE_I32) {
return false;
}
- if (src0->type != GGML_TYPE_F32 && src0->type != GGML_TYPE_F16 &&
- src0->type != GGML_TYPE_Q4_0 && src0->type != GGML_TYPE_Q4_K && src0->type != GGML_TYPE_Q6_K && src0->type != GGML_TYPE_Q8_0 && src0->type != GGML_TYPE_I32) {
- return false;
- }
-
- if ((src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && (!ggml_is_contiguous(src0) || ggml_is_permuted(src0) || src0->ne[0] % QK_K)) {
+ if (!ggml_is_contiguous_rows(src0) || !ggml_is_contiguous_rows(src1) || !ggml_is_contiguous_rows(dst)) {
return false;
}
@@ -7626,11 +5951,6 @@ static bool ggml_hexagon_supported_get_rows(const struct ggml_hexagon_session *
return false;
}
- // Tiled quantized weights have no raw fallback. Mark only accepted tensors for repacking.
- if ((src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && !src0->buffer) {
- sess->needs_repack.insert(src0);
- }
-
return true;
GGML_UNUSED(sess);
@@ -8436,20 +6756,13 @@ static bool mm_is_hmx_eligible(const ggml_tensor * t) {
if (ggml_hexagon_matmul_can_collapse(src0, src1, t)) {
const ggml_tensor src1_c = ggml_hexagon_tensor_collapse_rows(src1);
- const int wtype = src0->type;
- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);
- const int ne01_padded = is_repack ? hex_round_up(src0->ne[1], 32) : src0->ne[1];
- return ggml_hexagon_matmul_is_hmx_eligible(src0, &src1_c, t, ne01_padded, false, false);
+ return ggml_hexagon_matmul_is_hmx_eligible(src0, &src1_c, t, false, false);
}
- const int wtype = src0->type;
- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);
const bool is_matmul_id = (t->op == GGML_OP_MUL_MAT_ID);
const bool is_batched = (src0->ne[2] * src0->ne[3] > 1 || src1->ne[2] * src1->ne[3] > 1);
- const int ne01_padded = is_repack ? hex_round_up(src0->ne[1], 32) : src0->ne[1];
-
- return ggml_hexagon_matmul_is_hmx_eligible(src0, src1, t, ne01_padded, is_matmul_id, is_batched);
+ return ggml_hexagon_matmul_is_hmx_eligible(src0, src1, t, is_matmul_id, is_batched);
}
static bool is_supported_mul_mat_nx_kernel(const ggml_tensor * src0, const struct htp_mm_kernel_params * kparams) {
@@ -8457,8 +6770,8 @@ static bool is_supported_mul_mat_nx_kernel(const ggml_tensor * src0, const struc
return kparams->kernel_type == HTP_MM_KERNEL_HMX_2D;
}
- if (!ggml_hexagon_is_repack_type(src0->type) || src0->type == GGML_TYPE_Q6_K) {
- return false; // Q6_K has no fused HVX kernel
+ if (!ggml_hexagon_tensor_is_tiled(src0)) {
+ return false;
}
return kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW;
@@ -8469,7 +6782,7 @@ static bool is_supported_mul_mat_id_nx_kernel(const ggml_tensor * src0, const st
return kparams->kernel_type == HTP_MM_KERNEL_HMX_2D;
}
- if (!ggml_hexagon_is_repack_type(src0->type)) {
+ if (!ggml_hexagon_tensor_is_tiled(src0)) {
return false;
}
@@ -8484,13 +6797,13 @@ static bool is_mergeable_mul_mat(const ggml_tensor * t) {
if (src0->ne[2] != 1 || src0->ne[3] != 1) return false;
if (mm_is_hmx_eligible(t)) {
- return ggml_hexagon_is_hmx_weight_type(src0->type);
+ return htp_is_hmx_weight_type(src0->type);
}
- // HVX path requires F32 activations and repacked weights (except Q6_K)
+ // HVX path requires F32 activations and tiled weights
if (src1->type != GGML_TYPE_F32) return false;
- return ggml_hexagon_is_repack_type(src0->type) && src0->type != GGML_TYPE_Q6_K;
+ return ggml_hexagon_tensor_is_tiled(src0);
}
static bool is_mergeable_mul_mat_pair(const ggml_tensor * n1, const ggml_tensor * n2) {
@@ -8506,6 +6819,9 @@ static bool is_mergeable_mul_mat_pair(const ggml_tensor * n1, const ggml_tensor
if (n1->src[0]->type != n2->src[0]->type) {
return false;
}
+ if (ggml_hexagon_tensor_is_tiled(n1->src[0]) != ggml_hexagon_tensor_is_tiled(n2->src[0])) {
+ return false;
+ }
if (mm_is_hmx_eligible(n1) != mm_is_hmx_eligible(n2)) {
return false;
}
@@ -8516,7 +6832,11 @@ static bool is_mergeable_mul_mat_id(const ggml_tensor * t) {
if (t->op != GGML_OP_MUL_MAT_ID) return false;
const ggml_tensor * src0 = t->src[0];
- return ggml_hexagon_is_repack_type(src0->type);
+ if (mm_is_hmx_eligible(t)) {
+ return htp_is_hmx_weight_type(src0->type);
+ }
+
+ return ggml_hexagon_tensor_is_tiled(src0);
}
static bool is_mergeable_mul_mat_id_pair(const ggml_tensor * n1, const ggml_tensor * n2) {
@@ -8538,6 +6858,9 @@ static bool is_mergeable_mul_mat_id_pair(const ggml_tensor * n1, const ggml_tens
if (n1->src[0]->type != n2->src[0]->type) {
return false;
}
+ if (ggml_hexagon_tensor_is_tiled(n1->src[0]) != ggml_hexagon_tensor_is_tiled(n2->src[0])) {
+ return false;
+ }
if (mm_is_hmx_eligible(n1) != mm_is_hmx_eligible(n2)) {
return false;
}
@@ -8588,6 +6911,26 @@ static ggml_status ggml_backend_hexagon_graph_compute(ggml_backend_t backend, gg
htp_opnode node(HTP_OP_INVALID, n);
node.opcode = op_remap_to_htp(n);
+ if (node.opcode == HTP_OP_MUL_MAT || node.opcode == HTP_OP_MUL_MAT_ID || node.opcode == HTP_OP_GET_ROWS) {
+ ggml_tensor * src0 = node.node->src[0];
+ if (ggml_hexagon_needs_repack(src0)) {
+ if (opt_verbose) {
+ char dims_buf[64];
+ char strides_buf[64];
+ htp_opformat fmt;
+ fmt.format_tensor_dims(dims_buf, sizeof(dims_buf), src0);
+ fmt.format_tensor_strides(strides_buf, sizeof(strides_buf), src0);
+ GGML_LOG_DEBUG("ggml-hex: %s rt-repack %s|%s|%s|%s|size %zu\n",
+ sess->c_name(), src0->name, dims_buf, ggml_type_name(src0->type), strides_buf, ggml_nbytes(src0));
+ }
+ std::vector<uint8_t> raw(ggml_nbytes(src0));
+ memcpy(raw.data(), src0->data, raw.size());
+ htp_repack_tensor_tiled(src0, raw.data(), raw.size());
+ auto * extra = (ggml_hexagon_tensor_extra *) src0->extra;
+ extra->flags |= GGML_HEXAGON_TENSOR_TILED;
+ }
+ }
+
if (node.opcode == HTP_OP_MUL_MAT || node.opcode == HTP_OP_MUL_MAT_ID) {
ggml_hexagon_precompute_matmul_params(sess,
node.node->src[0], node.node->src[1], node.node,
@@ -9378,6 +7721,9 @@ static bool ggml_hexagon_supported_roll(const struct ggml_hexagon_session * sess
return true;
}
+// IMPORTANT: supports_op runs before buffer allocation and before tensor data is populated.
+// Never check buffer usage, buffer properties, or repacking state (is_tiled) here.
+// Support must depend only on static tensor properties (shape, type, strides).
static bool ggml_backend_hexagon_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {
auto dev_ctx = static_cast<ggml_backend_hexagon_device_context *>(dev->context);
auto sess = dev_ctx->session();
@@ -9971,12 +8317,15 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {
"please update hexagon_type to match ggml_type");
static_assert((unsigned int) HTP_TYPE_Q2_K == (unsigned int) GGML_TYPE_Q2_K,
"please update hexagon_type to match ggml_type");
+ static_assert((unsigned int) HTP_TYPE_BF16 == (unsigned int) GGML_TYPE_BF16,
+ "please update hexagon_type to match ggml_type");
const char * str_verbose = getenv("GGML_HEXAGON_VERBOSE");
const char * str_opbatch = getenv("GGML_HEXAGON_OPBATCH");
const char * str_opqueue = getenv("GGML_HEXAGON_OPQUEUE");
const char * str_oppoll = getenv("GGML_HEXAGON_OPPOLL");
const char * str_opfusion = getenv("GGML_HEXAGON_OPFUSION");
+ const char * str_rt_repack = getenv("GGML_HEXAGON_RT_REPACK");
const char * str_opfilter = getenv("GGML_HEXAGON_OPFILTER");
const char * str_profile = getenv("GGML_HEXAGON_PROFILE");
const char * str_etm = getenv("GGML_HEXAGON_ETM");
@@ -10034,6 +8383,7 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {
opt_optrace = str_optrace ? strtoul(str_optrace, NULL, 0) : (opt_opbatch * 256);
opt_oppoll = str_oppoll ? strtoul(str_oppoll, NULL, 0) : opt_oppoll;
opt_opfusion = str_opfusion ? atoi(str_opfusion) : opt_opfusion;
+ opt_rt_repack = str_rt_repack ? atoi(str_rt_repack) : opt_rt_repack;
opt_profile = str_profile ? atoi(str_profile) : 0;
opt_etm = str_etm ? atoi(str_etm) : 0;
if (str_power_mode) {
diff --git a/ggml/src/ggml-hexagon/htp-opnode.h b/ggml/src/ggml-hexagon/htp-opnode.h
index 8ff38545c..d62ce1d00 100644
--- a/ggml/src/ggml-hexagon/htp-opnode.h
+++ b/ggml/src/ggml-hexagon/htp-opnode.h
@@ -326,10 +326,12 @@ struct htp_opformat {
const auto * kparams = (const struct htp_mm_kernel_params *) node.kernel_params;
const char * path = "unknown";
int32_t type = kparams->kernel_type;
- if (type == HTP_MM_KERNEL_HMX_2D || type == HTP_MM_KERNEL_HMX_F16_BATCHED) {
- path = "hmx-tiled";
+ if (type == HTP_MM_KERNEL_HMX_2D || type == HTP_MM_KERNEL_HMX_BATCHED) {
+ path = (kparams->tile_size > 0) ? "hmx-tiled" : "hmx-flat";
} else if (type == HTP_MM_KERNEL_HVX_F16_F16_VTCM || type == HTP_MM_KERNEL_HVX_F32_F32_VTCM ||
- type == HTP_MM_KERNEL_HVX_QUANT_ROW || type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {
+ type == HTP_MM_KERNEL_HVX_BF16_F32_VTCM) {
+ path = "hvx-flat";
+ } else if (type == HTP_MM_KERNEL_HVX_QUANT_ROW || type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {
path = "hvx-tiled";
}
snprintf(str, max_size, "%s vtcm %d", path, (int) kparams->vtcm_size);
diff --git a/ggml/src/ggml-hexagon/htp-repack.cpp b/ggml/src/ggml-hexagon/htp-repack.cpp
new file mode 100644
index 000000000..38beabe82
--- /dev/null
+++ b/ggml/src/ggml-hexagon/htp-repack.cpp
@@ -0,0 +1,1000 @@
+#include "htp-repack.h"
+
+#define GGML_COMMON_IMPL_CPP
+#include "ggml-common.h"
+#include "ggml-impl.h"
+#include "ggml-quants.h"
+
+#include <algorithm>
+#include <cmath>
+#include <cstring>
+#include <initializer_list>
+
+// --- 4-bit block-32 nibble helpers (shared by Q4_0, IQ4_NL, Q4_1, MXFP4) ---
+
+static inline void unpack_nibbles_32(uint8_t * dst, const uint8_t * qs) {
+ for (unsigned int i = 0; i < 16; ++i) {
+ dst[i + 0] = qs[i] & 0x0F;
+ dst[i + 16] = qs[i] >> 4;
+ }
+}
+
+static inline void pack_nibbles_32(uint8_t * qs, const uint8_t * src) {
+ for (unsigned int i = 0; i < 16; ++i) {
+ qs[i] = src[i + 0] | (src[i + 16] << 4);
+ }
+}
+
+static inline void transpose_nibbles_to_tile(uint8_t * tile_dst, const uint8_t tile_quants[32][32]) {
+ for (int cp = 0; cp < 16; cp++) {
+ for (int row = 0; row < 32; row++) {
+ tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];
+ }
+ }
+}
+
+static inline void transpose_tile_to_nibbles(uint8_t tile_quants[32][32], const uint8_t * tile_src) {
+ for (int cp = 0; cp < 16; cp++) {
+ for (int row = 0; row < 32; row++) {
+ const uint8_t val = tile_src[cp * 32 + row];
+ tile_quants[row][2 * cp + 0] = val & 0x0F;
+ tile_quants[row][2 * cp + 1] = val >> 4;
+ }
+ }
+}
+
+// --- Generic 32x32 tile grid walkers for block-32 and float types ---
+
+template <typename SrcElemT, int ELEMS_PER_KTILE, typename PackTileFn>
+static void repack_to_tiled_grid_32(ggml_tensor * t, const void * data, size_t offset, size_t size,
+ size_t tile_size, PackTileFn && pack_tile) {
+ const SrcElemT * src_matrix = (const SrcElemT *) data;
+ const int64_t ne0 = t->ne[0];
+ const int64_t ne1 = t->ne[1];
+ const int64_t ne2 = t->ne[2];
+ const int64_t ne3 = t->ne[3];
+ const int n_col_tiles = hex_round_up(ne1, 32) / 32;
+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;
+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
+
+ const size_t slice_size = (size_t) ne1 * ggml_row_size(t->type, ne0);
+ const int64_t start_slice = offset / slice_size;
+ int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
+ if (end_slice > ne2 * ne3) {
+ end_slice = ne2 * ne3;
+ }
+
+ const int64_t row_stride = (ne0 / (32 / ELEMS_PER_KTILE));
+
+ for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
+ const SrcElemT * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * row_stride);
+ uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;
+
+ for (int ct = 0; ct < n_col_tiles; ct++) {
+ for (int kt = 0; kt < n_k_tiles; kt++) {
+ uint8_t * tile_dst = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;
+ pack_tile(tile_dst, src_slice, ne0, ne1, row_stride, ct, kt);
+ }
+ }
+ }
+}
+
+template <typename DstElemT, int ELEMS_PER_KTILE, typename UnpackTileFn>
+static void unpack_from_tiled_grid_32(void * data, const ggml_tensor * t, size_t offset, size_t size,
+ size_t tile_size, UnpackTileFn && unpack_tile) {
+ uint8_t * dst_bytes = (uint8_t *) data;
+ const int64_t ne0 = t->ne[0];
+ const int64_t ne1 = t->ne[1];
+ const int64_t ne2 = t->ne[2];
+ const int64_t ne3 = t->ne[3];
+ const int n_col_tiles = hex_round_up(ne1, 32) / 32;
+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;
+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
+
+ const size_t row_size_bytes = ggml_row_size(t->type, ne0);
+ const size_t slice_size = (size_t) ne1 * row_size_bytes;
+ const int64_t start_slice = offset / slice_size;
+ int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
+ if (end_slice > ne2 * ne3) {
+ end_slice = ne2 * ne3;
+ }
+
+ const int64_t row_stride = (ne0 / (32 / ELEMS_PER_KTILE));
+
+ for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
+ const size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);
+ const size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);
+ const size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;
+ const size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;
+
+ const int64_t start_row = slice_offset_start / row_size_bytes;
+ const int64_t end_row = (std::min)((int64_t) ((slice_offset_end + row_size_bytes - 1) / row_size_bytes), ne1);
+
+ const int start_ct = (int) (start_row / 32);
+ const int end_ct = (std::min)((int) ((end_row + 31) / 32), n_col_tiles);
+
+ DstElemT * dst_slice = (DstElemT *) (dst_bytes + (cur_start_byte - offset));
+ const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;
+
+ for (int ct = start_ct; ct < end_ct; ct++) {
+ for (int kt = 0; kt < n_k_tiles; kt++) {
+ const uint8_t * tile_src = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;
+ unpack_tile(dst_slice, tile_src, ne0, start_row, end_row, row_stride, ct, kt);
+ }
+ }
+ }
+}
+
+// --- Block-32 Quantized Types (Q4_0, IQ4_NL, Q4_1, Q8_0, MXFP4) ---
+
+static void repack_q4_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_to_tiled_grid_32<block_q4_0, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q4_0,
+ [](uint8_t * tile_dst, const block_q4_0 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {
+ uint8_t tile_quants[32][32];
+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r < ne1 && kt < ne0 / 32) {
+ const block_q4_0 & b = src_slice[r * row_stride + kt];
+ unpack_nibbles_32(tile_quants[row], b.qs);
+ scale_dst[row] = b.d;
+ } else {
+ memset(tile_quants[row], 8, 32);
+ scale_dst[row] = 0;
+ }
+ }
+ transpose_nibbles_to_tile(tile_dst, tile_quants);
+ });
+}
+
+static void repack_tiled_q4_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_from_tiled_grid_32<block_q4_0, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q4_0,
+ [](block_q4_0 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {
+ uint8_t tile_quants[32][32];
+ transpose_tile_to_nibbles(tile_quants, tile_src);
+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r >= start_row && r < end_row && kt < ne0 / 32) {
+ block_q4_0 & b = dst_slice[(r - start_row) * row_stride + kt];
+ pack_nibbles_32(b.qs, tile_quants[row]);
+ b.d = scale_src[row];
+ }
+ }
+ });
+}
+
+static void repack_q4_1_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_to_tiled_grid_32<block_q4_1, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q4_1,
+ [](uint8_t * tile_dst, const block_q4_1 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {
+ uint8_t tile_quants[32][32];
+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r < ne1 && kt < ne0 / 32) {
+ const block_q4_1 & b = src_slice[r * row_stride + kt];
+ unpack_nibbles_32(tile_quants[row], b.qs);
+ scale_dst[2 * row + 0] = b.d;
+ scale_dst[2 * row + 1] = b.m;
+ } else {
+ memset(tile_quants[row], 0, 32);
+ scale_dst[2 * row + 0] = 0;
+ scale_dst[2 * row + 1] = 0;
+ }
+ }
+ transpose_nibbles_to_tile(tile_dst, tile_quants);
+ });
+}
+
+static void repack_tiled_q4_1(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_from_tiled_grid_32<block_q4_1, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q4_1,
+ [](block_q4_1 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {
+ uint8_t tile_quants[32][32];
+ transpose_tile_to_nibbles(tile_quants, tile_src);
+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r >= start_row && r < end_row && kt < ne0 / 32) {
+ block_q4_1 & b = dst_slice[(r - start_row) * row_stride + kt];
+ pack_nibbles_32(b.qs, tile_quants[row]);
+ b.d = scale_src[2 * row + 0];
+ b.m = scale_src[2 * row + 1];
+ }
+ }
+ });
+}
+
+static void repack_mxfp4_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_to_tiled_grid_32<block_mxfp4, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_MXFP4,
+ [](uint8_t * tile_dst, const block_mxfp4 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {
+ uint8_t tile_quants[32][32];
+ uint8_t * scale_dst = tile_dst + 512;
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r < ne1 && kt < ne0 / 32) {
+ const block_mxfp4 & b = src_slice[r * row_stride + kt];
+ unpack_nibbles_32(tile_quants[row], b.qs);
+ scale_dst[row] = b.e;
+ } else {
+ memset(tile_quants[row], 0, 32);
+ scale_dst[row] = 0;
+ }
+ }
+ transpose_nibbles_to_tile(tile_dst, tile_quants);
+ });
+}
+
+static void repack_tiled_mxfp4(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_from_tiled_grid_32<block_mxfp4, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_MXFP4,
+ [](block_mxfp4 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {
+ uint8_t tile_quants[32][32];
+ transpose_tile_to_nibbles(tile_quants, tile_src);
+ const uint8_t * scale_src = tile_src + 512;
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r >= start_row && r < end_row && kt < ne0 / 32) {
+ block_mxfp4 & b = dst_slice[(r - start_row) * row_stride + kt];
+ pack_nibbles_32(b.qs, tile_quants[row]);
+ b.e = scale_src[row];
+ }
+ }
+ });
+}
+
+static void repack_q8_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_to_tiled_grid_32<block_q8_0, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q8_0,
+ [](uint8_t * tile_dst, const block_q8_0 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {
+ for (int cp = 0; cp < 16; cp++) {
+ const int col0 = cp * 2;
+ const int col1 = col0 + 1;
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ const block_q8_0 * b = (r < ne1 && kt < ne0 / 32) ? &src_slice[r * row_stride + kt] : nullptr;
+ tile_dst[cp * 64 + 2 * row + 0] = b ? b->qs[col0] : 0;
+ tile_dst[cp * 64 + 2 * row + 1] = b ? b->qs[col1] : 0;
+ }
+ }
+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 1024);
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * row_stride + kt].d : 0;
+ }
+ });
+}
+
+static void repack_tiled_q8_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_from_tiled_grid_32<block_q8_0, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q8_0,
+ [](block_q8_0 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {
+ for (int cp = 0; cp < 16; cp++) {
+ const int col0 = cp * 2;
+ const int col1 = col0 + 1;
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r >= start_row && r < end_row && kt < ne0 / 32) {
+ block_q8_0 & b = dst_slice[(r - start_row) * row_stride + kt];
+ b.qs[col0] = tile_src[cp * 64 + 2 * row + 0];
+ b.qs[col1] = tile_src[cp * 64 + 2 * row + 1];
+ }
+ }
+ }
+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 1024);
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r >= start_row && r < end_row && kt < ne0 / 32) {
+ dst_slice[(r - start_row) * row_stride + kt].d = scale_src[row];
+ }
+ }
+ });
+}
+
+// --- Float Types (F16, F32, BF16 -> f16_tiled HMX layout) ---
+
+template <typename SrcT, typename ToFp16Fn>
+static void repack_float_to_f16_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size, ToFp16Fn && to_fp16) {
+ repack_to_tiled_grid_32<SrcT, 32>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_F16,
+ [&](uint8_t * tile_bytes, const SrcT * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {
+ uint16_t * tile_dst = (uint16_t *) tile_bytes;
+ for (int cp = 0; cp < 16; cp++) {
+ const int64_t c0 = kt * 32 + cp * 2;
+ const int64_t c1 = c0 + 1;
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ const SrcT * r_ptr = (r < ne1) ? &src_slice[r * row_stride] : nullptr;
+ tile_dst[cp * 64 + row * 2 + 0] = (r_ptr && c0 < ne0) ? to_fp16(r_ptr[c0]) : 0;
+ tile_dst[cp * 64 + row * 2 + 1] = (r_ptr && c1 < ne0) ? to_fp16(r_ptr[c1]) : 0;
+ }
+ }
+ });
+}
+
+template <typename DstT, typename FromFp16Fn>
+static void unpack_f16_tiled_to_float(void * data, const ggml_tensor * t, size_t offset, size_t size, FromFp16Fn && from_fp16) {
+ unpack_from_tiled_grid_32<DstT, 32>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_F16,
+ [&](DstT * dst_slice, const uint8_t * tile_bytes, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {
+ const uint16_t * tile_src = (const uint16_t *) tile_bytes;
+ for (int cp = 0; cp < 16; cp++) {
+ const int64_t c0 = kt * 32 + cp * 2;
+ const int64_t c1 = c0 + 1;
+ for (int row = 0; row < 32; row++) {
+ const int64_t r = ct * 32 + row;
+ if (r >= start_row && r < end_row) {
+ DstT * r_ptr = dst_slice + (r - start_row) * row_stride;
+ if (c0 < ne0) r_ptr[c0] = from_fp16(tile_src[cp * 64 + row * 2 + 0]);
+ if (c1 < ne0) r_ptr[c1] = from_fp16(tile_src[cp * 64 + row * 2 + 1]);
+ }
+ }
+ }
+ });
+}
+
+// --- Generic K-Quant (super-block 256) grid walkers ---
+
+template <typename BlockT, typename PackKtFn>
+static void repack_kquant_to_tiled_grid(ggml_tensor * t, const void * data, size_t offset, size_t size,
+ size_t tile_size, PackKtFn && pack_kt) {
+ GGML_ASSERT(offset == 0);
+ GGML_UNUSED(size);
+
+ const BlockT * src_matrix = (const BlockT *) data;
+ const int64_t ne0 = t->ne[0];
+ const int64_t ne1 = t->ne[1];
+ const int64_t ne2 = t->ne[2];
+ const int64_t ne3 = t->ne[3];
+ GGML_ASSERT(ne0 % QK_K == 0);
+
+ const int n_col_tiles = hex_round_up(ne1, 32) / 32;
+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;
+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
+ const int64_t sb_per_row = ne0 / QK_K;
+
+ for (int64_t slice_idx = 0; slice_idx < ne2 * ne3; slice_idx++) {
+ const BlockT * src_slice = src_matrix + slice_idx * (ne1 * sb_per_row);
+ uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;
+
+ // Padding rows and OR-ed bit planes require zeroed tiles
+ memset(matrix_dst, 0, matrix_size);
+
+ for (int64_t r = 0; r < ne1; r++) {
+ const int ct = (int) (r / 32);
+ const int row = (int) (r % 32);
+ const BlockT * src_row = src_slice + r * sb_per_row;
+
+ for (int kt = 0; kt < n_k_tiles; kt++) {
+ uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;
+ pack_kt(tile, &src_row[kt / 8], row, kt % 8);
+ }
+ }
+ }
+}
+
+template <typename BlockT, typename UnpackSbFn>
+static void unpack_kquant_from_tiled_grid(void * data, const ggml_tensor * t, size_t offset, size_t size,
+ size_t tile_size, UnpackSbFn && unpack_sb) {
+ uint8_t * dst_bytes = (uint8_t *) data;
+ const int64_t ne0 = t->ne[0];
+ const int64_t ne1 = t->ne[1];
+ const int64_t ne2 = t->ne[2];
+ const int64_t ne3 = t->ne[3];
+ GGML_ASSERT(ne0 % QK_K == 0);
+
+ const int n_col_tiles = hex_round_up(ne1, 32) / 32;
+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;
+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;
+ const int64_t sb_per_row = ne0 / QK_K;
+
+ const size_t row_size_bytes = ggml_row_size(t->type, ne0);
+ const size_t slice_size = (size_t) ne1 * row_size_bytes;
+ const int64_t start_slice = offset / slice_size;
+ int64_t end_slice = (offset + size + slice_size - 1) / slice_size;
+ if (end_slice > ne2 * ne3) {
+ end_slice = ne2 * ne3;
+ }
+
+ for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {
+ const size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);
+ const size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);
+ const size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;
+ const size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;
+
+ const int64_t start_row = slice_offset_start / row_size_bytes;
+ const int64_t end_row = (std::min)((int64_t) ((slice_offset_end + row_size_bytes - 1) / row_size_bytes), ne1);
+
+ BlockT * dst_slice = (BlockT *) (dst_bytes + (cur_start_byte - offset));
+ const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;
+
+ for (int64_t r = start_row; r < end_row; r++) {
+ const int ct = (int) (r / 32);
+ const int row = (int) (r % 32);
+ BlockT * dst_row = dst_slice + (r - start_row) * sb_per_row;
+
+ for (int64_t sb = 0; sb < sb_per_row; sb++) {
+ BlockT * b = &dst_row[sb];
+ memset(b, 0, sizeof(BlockT));
+ const uint8_t * sb_tiles = matrix_src + ((size_t) ct * n_k_tiles + sb * 8) * tile_size;
+ unpack_sb(b, sb_tiles, tile_size, row);
+ }
+ }
+ }
+}
+
+// --- K-Quant Helpers & Per-Type Implementations (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K) ---
+
+// unsigned 6-bit value (0..63) of element e of a Q6_K block, same bit layout as dequantize_row_q6_K
+static inline uint8_t q6_K_get_quant(const block_q6_K * b, int e) {
+ const int c = e / 128;
+ const int w = e % 128;
+ const int g = w / 32;
+ const int l = w % 32;
+ const uint8_t * ql = b->ql + c * 64;
+ const uint8_t * qh = b->qh + c * 32;
+ uint8_t lo, hi;
+ switch (g) {
+ case 0: lo = ql[l] & 0xF; hi = (qh[l] >> 0) & 3; break;
+ case 1: lo = ql[l + 32] & 0xF; hi = (qh[l] >> 2) & 3; break;
+ case 2: lo = ql[l] >> 4; hi = (qh[l] >> 4) & 3; break;
+ default: lo = ql[l + 32] >> 4; hi = (qh[l] >> 6) & 3; break;
+ }
+ return (uint8_t) (lo | (hi << 4));
+}
+
+static void repack_q6_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_kquant_to_tiled_grid<block_q6_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q6_K,
+ [](uint8_t * tile, const block_q6_K * b, int row, int kt_local) {
+ const float d = GGML_FP16_TO_FP32(b->d);
+ uint8_t * lo_pl = tile;
+ uint8_t * hi_pl = tile + 512;
+ ggml_half * sc_pl = (ggml_half *) (tile + 768);
+
+ for (int lk = 0; lk < 32; lk++) {
+ const uint8_t q6 = q6_K_get_quant(b, kt_local * 32 + lk);
+ const int g = lk >> 2;
+ const int pos = row * 4 + (lk & 3);
+ lo_pl[(g >> 1) * 128 + pos] |= (uint8_t) ((q6 & 0xF) << ((g & 1) * 4));
+ hi_pl[(g >> 2) * 128 + pos] |= (uint8_t) ((q6 >> 4) << ((g & 3) * 2));
+ }
+ for (int sub = 0; sub < 2; sub++) {
+ sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) b->scales[kt_local * 2 + sub]);
+ }
+ });
+}
+
+static void repack_tiled_q6_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_kquant_from_tiled_grid<block_q6_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q6_K,
+ [](block_q6_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {
+ float sub_scales[16];
+ for (int kt_local = 0; kt_local < 8; kt_local++) {
+ const uint8_t * tile = sb_tiles + kt_local * tile_size;
+ const uint8_t * lo_pl = tile;
+ const uint8_t * hi_pl = tile + 512;
+ const ggml_half * sc_pl = (const ggml_half *) (tile + 768);
+
+ const int c = kt_local / 4;
+ const int g = kt_local % 4;
+ uint8_t * ql = b->ql + c * 64;
+ uint8_t * qh = b->qh + c * 32;
+
+ for (int lk = 0; lk < 32; lk++) {
+ const int g_tile = lk >> 2;
+ const int pos = row * 4 + (lk & 3);
+ const uint8_t lo = (lo_pl[(g_tile >> 1) * 128 + pos] >> ((g_tile & 1) * 4)) & 0xF;
+ const uint8_t hi = (hi_pl[(g_tile >> 2) * 128 + pos] >> ((g_tile & 3) * 2)) & 3;
+
+ switch (g) {
+ case 0: ql[lk] |= lo; qh[lk] |= (hi << 0); break;
+ case 1: ql[lk + 32] |= lo; qh[lk] |= (hi << 2); break;
+ case 2: ql[lk] |= (lo << 4); qh[lk] |= (hi << 4); break;
+ default: ql[lk + 32] |= (lo << 4); qh[lk] |= (hi << 6); break;
+ }
+ }
+
+ for (int sub = 0; sub < 2; sub++) {
+ sub_scales[kt_local * 2 + sub] = GGML_FP16_TO_FP32(sc_pl[sub * 32 + row]);
+ }
+ }
+
+ float max_abs_scale = 0.0f;
+ for (int s = 0; s < 16; s++) {
+ max_abs_scale = (std::max)(max_abs_scale, fabsf(sub_scales[s]));
+ }
+
+ if (max_abs_scale == 0.0f) {
+ b->d = GGML_FP32_TO_FP16(0.0f);
+ } else {
+ b->d = GGML_FP32_TO_FP16(max_abs_scale / 127.0f);
+ const float d_actual = GGML_FP16_TO_FP32(b->d);
+ const float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;
+ for (int s = 0; s < 16; s++) {
+ const int sc = (int) roundf(sub_scales[s] * inv_d);
+ b->scales[s] = (int8_t) (std::max)(-128, (std::min)(127, sc));
+ }
+ }
+ });
+}
+
+// low 2 bits (0..3) of element e of a Q2_K or Q3_K block, same bit layout as dequantize_row_q2_K / q3_K
+static inline uint8_t q2_3_K_get_low2(const uint8_t * qs, int e) {
+ const int c = e / 128;
+ const int j = (e % 128) / 32;
+ const int l = e % 32;
+ return (qs[c * 32 + l] >> (2 * j)) & 3;
+}
+
+// hmask bit of element e of a Q3_K block, same bit layout as dequantize_row_q3_K
+static inline bool q3_K_get_hbit(const block_q3_K * b, int e) {
+ const int c = e / 128;
+ const int j = (e % 128) / 32;
+ const int l = e % 32;
+ return (b->hmask[l] >> (c * 4 + j)) & 1;
+}
+
+// signed 6-bit scale j (-32..31) of a Q3_K block, same packing as quantize_row_q3_K_ref
+static inline int q3_K_get_scale(const uint8_t * scales, int j) {
+ const int lo = (j < 8) ? (scales[j] & 0xF) : (scales[j - 8] >> 4);
+ const int hi = (scales[8 + j % 4] >> (2 * (j / 4))) & 3;
+ return (lo | (hi << 4)) - 32;
+}
+
+// read-back: find fp16 d and l[j] in [lmin, lmax] with fp16(d * l[j]) == prod[j] for all j, false if none
+static bool hexagon_recover_k_scales(const ggml_half * prod, int n, int lmin, int lmax, ggml_half * d_out, int * l_out) {
+ int jmax = 0;
+ for (int j = 1; j < n; j++) {
+ if (fabsf(GGML_FP16_TO_FP32(prod[j])) > fabsf(GGML_FP16_TO_FP32(prod[jmax]))) {
+ jmax = j;
+ }
+ }
+ const float pmax = GGML_FP16_TO_FP32(prod[jmax]);
+ if (pmax == 0.0f) {
+ *d_out = GGML_FP32_TO_FP16(0.0f);
+ for (int j = 0; j < n; j++) {
+ l_out[j] = 0;
+ }
+ return true;
+ }
+
+ // the quantizers put the largest scale at or near the range end, so try large |l| first
+ const int lext = (std::max)(-lmin, lmax);
+ for (int a = lext; a >= 1; a--) {
+ for (int sign : { 1, -1 }) {
+ const int lj = sign * a;
+ if (lj < lmin || lj > lmax) {
+ continue;
+ }
+ const ggml_half d0 = GGML_FP32_TO_FP16(pmax / (float) lj);
+ for (int ulp : { 0, -1, 1 }) {
+ ggml_half d = d0;
+ uint16_t bits;
+ memcpy(&bits, &d, sizeof(bits));
+ bits = (uint16_t) (bits + ulp);
+ memcpy(&d, &bits, sizeof(bits));
+
+ const float df = GGML_FP16_TO_FP32(d);
+ if (!std::isfinite(df) || df == 0.0f) {
+ continue;
+ }
+ bool ok = true;
+ for (int j = 0; j < n && ok; j++) {
+ const int l = (int) roundf(GGML_FP16_TO_FP32(prod[j]) / df);
+ const ggml_half p = GGML_FP32_TO_FP16(df * (float) l);
+ ok = l >= lmin && l <= lmax && memcmp(&p, &prod[j], sizeof(p)) == 0;
+ l_out[j] = l;
+ }
+ if (ok) {
+ *d_out = d;
+ return true;
+ }
+ }
+ }
+ }
+ return false;
+}
+
+static void repack_q3_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_kquant_to_tiled_grid<block_q3_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q3_K,
+ [](uint8_t * tile, const block_q3_K * b, int row, int kt_local) {
+ const float d = GGML_FP16_TO_FP32(b->d);
+ uint8_t * lo_pl = tile;
+ uint8_t * neg_pl = tile + 256;
+ ggml_half * sc_pl = (ggml_half *) (tile + 384);
+
+ for (int lk = 0; lk < 32; lk++) {
+ const int e = kt_local * 32 + lk;
+ const int g = lk >> 2;
+ const int pos = row * 4 + (lk & 3);
+ lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, e) << ((g & 3) * 2));
+ if (!q3_K_get_hbit(b, e)) {
+ neg_pl[pos] |= (uint8_t) (1 << g);
+ }
+ }
+ for (int sub = 0; sub < 2; sub++) {
+ sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) q3_K_get_scale(b->scales, kt_local * 2 + sub));
+ }
+ });
+}
+
+static void repack_tiled_q3_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_kquant_from_tiled_grid<block_q3_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q3_K,
+ [](block_q3_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {
+ ggml_half sub_scales[16];
+ for (int kt_local = 0; kt_local < 8; kt_local++) {
+ const uint8_t * tile = sb_tiles + kt_local * tile_size;
+ const uint8_t * lo_pl = tile;
+ const uint8_t * neg_pl = tile + 256;
+ const ggml_half * sc_pl = (const ggml_half *) (tile + 384);
+
+ for (int lk = 0; lk < 32; lk++) {
+ const int e = kt_local * 32 + lk;
+ const int g = lk >> 2;
+ const int pos = row * 4 + (lk & 3);
+ const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;
+
+ const int c = e / 128;
+ const int j = (e % 128) / 32;
+ const int l = e % 32;
+ b->qs[c * 32 + l] |= (uint8_t) (lo << (2 * j));
+ if (!((neg_pl[pos] >> g) & 1)) {
+ b->hmask[l] |= (uint8_t) (1 << (c * 4 + j));
+ }
+ }
+
+ for (int sub = 0; sub < 2; sub++) {
+ sub_scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];
+ }
+ }
+
+ int ls[16];
+ if (!hexagon_recover_k_scales(sub_scales, 16, -32, 31, &b->d, ls)) {
+ float max_scale = 0.0f;
+ for (int s = 0; s < 16; s++) {
+ if (fabsf(GGML_FP16_TO_FP32(sub_scales[s])) > fabsf(max_scale)) {
+ max_scale = GGML_FP16_TO_FP32(sub_scales[s]);
+ }
+ }
+ b->d = GGML_FP32_TO_FP16(-max_scale / 32.0f);
+ const float d_actual = GGML_FP16_TO_FP32(b->d);
+ const float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;
+ for (int s = 0; s < 16; s++) {
+ ls[s] = (std::max)(-32, (std::min)(31, (int) roundf(GGML_FP16_TO_FP32(sub_scales[s]) * inv_d)));
+ }
+ }
+
+ for (int s = 0; s < 16; s++) {
+ const int l = ls[s] + 32;
+ if (s < 8) {
+ b->scales[s] = l & 0xF;
+ } else {
+ b->scales[s - 8] |= (uint8_t) ((l & 0xF) << 4);
+ }
+ b->scales[s % 4 + 8] |= (uint8_t) ((l >> 4) << (2 * (s / 4)));
+ }
+ });
+}
+
+static void repack_q2_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_kquant_to_tiled_grid<block_q2_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q2_K,
+ [](uint8_t * tile, const block_q2_K * b, int row, int kt_local) {
+ uint8_t * lo_pl = tile;
+ uint8_t * sc_pl = tile + 256;
+ ggml_half * dd_pl = (ggml_half *) (tile + 320);
+
+ for (int lk = 0; lk < 32; lk++) {
+ const int g = lk >> 2;
+ const int pos = row * 4 + (lk & 3);
+ lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, kt_local * 32 + lk) << ((g & 3) * 2));
+ }
+ for (int sub = 0; sub < 2; sub++) {
+ sc_pl[sub * 32 + row] = b->scales[kt_local * 2 + sub];
+ }
+ if (row / 8 == kt_local % 4) {
+ dd_pl[row % 8] = kt_local < 4 ? b->d : (ggml_half) (b->dmin ^ 0x8000);
+ }
+ });
+}
+
+static void repack_tiled_q2_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_kquant_from_tiled_grid<block_q2_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q2_K,
+ [](block_q2_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {
+ for (int kt_local = 0; kt_local < 8; kt_local++) {
+ const uint8_t * tile = sb_tiles + kt_local * tile_size;
+ const uint8_t * lo_pl = tile;
+ const uint8_t * sc_pl = tile + 256;
+ const ggml_half * dd_pl = (const ggml_half *) (tile + 320);
+
+ for (int lk = 0; lk < 32; lk++) {
+ const int e = kt_local * 32 + lk;
+ const int g = lk >> 2;
+ const int pos = row * 4 + (lk & 3);
+ const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;
+ b->qs[(e / 128) * 32 + e % 32] |= (uint8_t) (lo << (2 * ((e % 128) / 32)));
+ }
+
+ for (int sub = 0; sub < 2; sub++) {
+ b->scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];
+ }
+ if (row / 8 == kt_local % 4) {
+ if (kt_local < 4) {
+ b->d = dd_pl[row % 8];
+ } else {
+ b->dmin = (ggml_half) (dd_pl[row % 8] ^ 0x8000);
+ }
+ }
+ }
+ });
+}
+
+static inline void get_scale_min_k4(int j, const uint8_t * q, uint8_t * d, uint8_t * m) {
+ if (j < 4) {
+ *d = q[j] & 63;
+ *m = q[j + 4] & 63;
+ } else {
+ *d = (q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4);
+ *m = (q[j + 4] >> 4) | ((q[j - 0] >> 6) << 4);
+ }
+}
+
+static inline void pack_k4_scales_mins(uint8_t * scales, ggml_half * d, ggml_half * dmin,
+ const float * sub_scales, const float * sub_mins) {
+ float max_scale = 0.0f;
+ float max_min = 0.0f;
+ for (int j = 0; j < 8; j++) {
+ if (sub_scales[j] > max_scale) max_scale = sub_scales[j];
+ if (sub_mins[j] > max_min) max_min = sub_mins[j];
+ }
+
+ float inv_scale = 0.0f;
+ if (max_scale > 0.0f) {
+ *d = GGML_FP32_TO_FP16(max_scale / 63.0f);
+ const float d_actual = GGML_FP16_TO_FP32(*d);
+ inv_scale = (d_actual > 0.0f) ? (1.0f / d_actual) : 0.0f;
+ } else {
+ *d = GGML_FP32_TO_FP16(0.0f);
+ }
+
+ float inv_min = 0.0f;
+ if (max_min > 0.0f) {
+ *dmin = GGML_FP32_TO_FP16(max_min / 63.0f);
+ const float dmin_actual = GGML_FP16_TO_FP32(*dmin);
+ inv_min = (dmin_actual > 0.0f) ? (1.0f / dmin_actual) : 0.0f;
+ } else {
+ *dmin = GGML_FP32_TO_FP16(0.0f);
+ }
+
+ for (int j = 0; j < 8; j++) {
+ uint8_t ls = (std::min)((uint8_t) 63, (uint8_t) roundf(inv_scale * sub_scales[j]));
+ uint8_t lm = (std::min)((uint8_t) 63, (uint8_t) roundf(inv_min * sub_mins[j]));
+ if (j < 4) {
+ scales[j] = ls;
+ scales[j + 4] = lm;
+ } else {
+ scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);
+ scales[j - 4] |= ((ls >> 4) << 6);
+ scales[j - 0] |= ((lm >> 4) << 6);
+ }
+ }
+}
+
+static void repack_q4_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_kquant_to_tiled_grid<block_q4_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q4_K,
+ [](uint8_t * tile_dst, const block_q4_K * b, int row, int kt_local) {
+ const float d = GGML_FP16_TO_FP32(b->d);
+ const float dmin = GGML_FP16_TO_FP32(b->dmin);
+
+ uint8_t sc, m;
+ get_scale_min_k4(kt_local, b->scales, &sc, &m);
+
+ const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
+ const int shift = (kt_local & 1) ? 4 : 0;
+
+ for (int cp = 0; cp < 16; cp++) {
+ const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;
+ const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;
+ tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);
+ }
+
+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);
+ scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(d * (float) sc);
+ scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(-dmin * (float) m);
+ });
+}
+
+static void repack_tiled_q4_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_kquant_from_tiled_grid<block_q4_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q4_K,
+ [](block_q4_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {
+ float sub_scales[8];
+ float sub_mins[8];
+
+ for (int kt_local = 0; kt_local < 8; kt_local++) {
+ const uint8_t * tile_src = sb_tiles + kt_local * tile_size;
+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);
+
+ uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
+ const int shift = (kt_local & 1) ? 4 : 0;
+
+ for (int cp = 0; cp < 16; cp++) {
+ const uint8_t val = tile_src[cp * 32 + row];
+ qs_sub[2 * cp + 0] |= (uint8_t) ((val & 0x0F) << shift);
+ qs_sub[2 * cp + 1] |= (uint8_t) ((val >> 4) << shift);
+ }
+
+ const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);
+ const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);
+ sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;
+ sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;
+ }
+
+ pack_k4_scales_mins(b->scales, &b->d, &b->dmin, sub_scales, sub_mins);
+ });
+}
+
+static void repack_q5_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {
+ repack_kquant_to_tiled_grid<block_q5_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q5_K,
+ [](uint8_t * tile_dst, const block_q5_K * b, int row, int kt_local) {
+ const float d = GGML_FP16_TO_FP32(b->d);
+ const float dmin = GGML_FP16_TO_FP32(b->dmin);
+ uint8_t * plane = tile_dst + 640;
+
+ uint8_t sc, m;
+ get_scale_min_k4(kt_local, b->scales, &sc, &m);
+
+ const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
+ const int shift = (kt_local & 1) ? 4 : 0;
+ const uint8_t hbit = (uint8_t) (1 << kt_local);
+
+ for (int cp = 0; cp < 16; cp++) {
+ const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;
+ const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;
+ tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);
+
+ const int i = cp / 4;
+ const int lane = (cp % 4) * 32 + row;
+ if (b->qh[2 * cp + 0] & hbit) {
+ plane[lane] |= (uint8_t) (1 << (2 * i));
+ }
+ if (b->qh[2 * cp + 1] & hbit) {
+ plane[lane] |= (uint8_t) (1 << (2 * i + 1));
+ }
+ }
+
+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);
+ scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(d * (float) sc);
+ scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(-dmin * (float) m);
+ });
+}
+
+static void repack_tiled_q5_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {
+ unpack_kquant_from_tiled_grid<block_q5_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q5_K,
+ [](block_q5_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {
+ float sub_scales[8];
+ float sub_mins[8];
+
+ for (int kt_local = 0; kt_local < 8; kt_local++) {
+ const uint8_t * tile_src = sb_tiles + kt_local * tile_size;
+ const uint8_t * plane = tile_src + 640;
+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);
+
+ uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;
+ const int shift = (kt_local & 1) ? 4 : 0;
+ const uint8_t hbit = (uint8_t) (1 << kt_local);
+
+ for (int cp = 0; cp < 16; cp++) {
+ const uint8_t val = tile_src[cp * 32 + row];
+ qs_sub[2 * cp + 0] |= (uint8_t) ((val & 0x0F) << shift);
+ qs_sub[2 * cp + 1] |= (uint8_t) ((val >> 4) << shift);
+
+ const int i = cp / 4;
+ const int lane = (cp % 4) * 32 + row;
+ if (plane[lane] & (1 << (2 * i))) {
+ b->qh[2 * cp + 0] |= hbit;
+ }
+ if (plane[lane] & (1 << (2 * i + 1))) {
+ b->qh[2 * cp + 1] |= hbit;
+ }
+ }
+
+ const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);
+ const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);
+ sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;
+ sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;
+ }
+
+ pack_k4_scales_mins(b->scales, &b->d, &b->dmin, sub_scales, sub_mins);
+ });
+}
+
+// --- Top-level Repack / Unpack Dispatchers ---
+
+void htp_repack_tensor_tiled(ggml_tensor * tensor, const void * data, size_t size) {
+ switch (tensor->type) {
+ case GGML_TYPE_Q4_0:
+ case GGML_TYPE_IQ4_NL:
+ repack_q4_0_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_Q4_1:
+ repack_q4_1_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_Q8_0:
+ repack_q8_0_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_MXFP4:
+ repack_mxfp4_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_Q2_K:
+ repack_q2_K_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_Q3_K:
+ repack_q3_K_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_Q4_K:
+ repack_q4_K_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_Q5_K:
+ repack_q5_K_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_Q6_K:
+ repack_q6_K_tiled(tensor, data, 0, size);
+ break;
+ case GGML_TYPE_F16:
+ repack_float_to_f16_tiled<uint16_t>(tensor, data, 0, size,
+ [](uint16_t v) { return v; });
+ break;
+ case GGML_TYPE_F32:
+ repack_float_to_f16_tiled<float>(tensor, data, 0, size,
+ [](float v) { return (uint16_t) ggml_fp32_to_fp16(v); });
+ break;
+ case GGML_TYPE_BF16:
+ repack_float_to_f16_tiled<ggml_bf16_t>(tensor, data, 0, size,
+ [](ggml_bf16_t v) { return (uint16_t) ggml_fp32_to_fp16(ggml_bf16_to_fp32(v)); });
+ break;
+ default:
+ break;
+ }
+}
+
+void htp_unpack_tensor_tiled(void * data, const ggml_tensor * tensor, size_t offset, size_t size) {
+ GGML_ASSERT(offset + size <= ggml_nbytes(tensor));
+
+ switch (tensor->type) {
+ case GGML_TYPE_Q4_0:
+ case GGML_TYPE_IQ4_NL:
+ repack_tiled_q4_0(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_Q4_1:
+ repack_tiled_q4_1(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_Q8_0:
+ repack_tiled_q8_0(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_MXFP4:
+ repack_tiled_mxfp4(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_Q2_K:
+ repack_tiled_q2_K(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_Q3_K:
+ repack_tiled_q3_K(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_Q4_K:
+ repack_tiled_q4_K(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_Q5_K:
+ repack_tiled_q5_K(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_Q6_K:
+ repack_tiled_q6_K(data, tensor, offset, size);
+ break;
+ case GGML_TYPE_F16:
+ unpack_f16_tiled_to_float<uint16_t>(data, tensor, offset, size,
+ [](uint16_t v) { return v; });
+ break;
+ case GGML_TYPE_F32:
+ unpack_f16_tiled_to_float<float>(data, tensor, offset, size,
+ [](uint16_t v) { return ggml_fp16_to_fp32((ggml_fp16_t) v); });
+ break;
+ case GGML_TYPE_BF16:
+ unpack_f16_tiled_to_float<ggml_bf16_t>(data, tensor, offset, size,
+ [](uint16_t v) { return ggml_fp32_to_bf16(ggml_fp16_to_fp32((ggml_fp16_t) v)); });
+ break;
+ default:
+ memcpy(data, (const char *) tensor->data + offset, size);
+ break;
+ }
+}
diff --git a/ggml/src/ggml-hexagon/htp-repack.h b/ggml/src/ggml-hexagon/htp-repack.h
new file mode 100644
index 000000000..ae66faf61
--- /dev/null
+++ b/ggml/src/ggml-hexagon/htp-repack.h
@@ -0,0 +1,64 @@
+#ifndef HTP_REPACK_H
+#define HTP_REPACK_H
+
+#include <stddef.h>
+#include <stdint.h>
+
+#include "ggml.h"
+#include "htp/htp-tensor.h"
+
+static inline bool htp_is_tiled_weight_type(enum ggml_type type) {
+ return type == GGML_TYPE_Q4_0 || type == GGML_TYPE_Q4_1 ||
+ type == GGML_TYPE_Q8_0 || type == GGML_TYPE_IQ4_NL ||
+ type == GGML_TYPE_MXFP4 || type == GGML_TYPE_Q6_K ||
+ type == GGML_TYPE_Q4_K || type == GGML_TYPE_Q5_K ||
+ type == GGML_TYPE_Q3_K || type == GGML_TYPE_Q2_K ||
+ type == GGML_TYPE_F16 || type == GGML_TYPE_F32 ||
+ type == GGML_TYPE_BF16;
+}
+
+static inline bool htp_is_tiled_weight_tensor(const struct ggml_tensor * t) {
+ if (!ggml_is_contiguous(t) || ggml_is_view(t)) {
+ return false;
+ }
+ if (!htp_is_tiled_weight_type(t->type)) {
+ return false;
+ }
+ if (t->ne[0] < 32) {
+ return false;
+ }
+ if (!ggml_is_quantized(t->type)) {
+ return (t->ne[0] % 32 == 0) && (t->ne[1] % 32 == 0) && (t->ne[2] == 1) && (t->ne[3] == 1);
+ }
+ return t->ne[3] == 1;
+}
+
+// Size of one tiled row in the DSP tiled layout. The Q3_K..Q6_K tiles store uncompressed scales/mins,
+// so they are larger than the ggml blocks. For the other tiled types (Q2_K included) the tile has the same size.
+static inline size_t htp_tiled_row_size(enum ggml_type type, int64_t ne0) {
+ if (type == GGML_TYPE_F16 || type == GGML_TYPE_F32 || type == GGML_TYPE_BF16) {
+ return (size_t) ne0 * sizeof(ggml_fp16_t);
+ }
+ if (type == GGML_TYPE_Q6_K) {
+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q6_K / 32);
+ }
+ if (type == GGML_TYPE_Q4_K) {
+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q4_K / 32);
+ }
+ if (type == GGML_TYPE_Q5_K) {
+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q5_K / 32);
+ }
+ if (type == GGML_TYPE_Q3_K) {
+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q3_K / 32);
+ }
+ return ggml_row_size(type, ne0);
+}
+
+static inline bool htp_is_hmx_weight_type(enum ggml_type type) {
+ return htp_is_tiled_weight_type(type);
+}
+
+void htp_repack_tensor_tiled(struct ggml_tensor * t, const void * data, size_t size);
+void htp_unpack_tensor_tiled(void * data, const struct ggml_tensor * t, size_t offset, size_t size);
+
+#endif // HTP_REPACK_H
diff --git a/ggml/src/ggml-hexagon/htp/CMakeLists.txt b/ggml/src/ggml-hexagon/htp/CMakeLists.txt
index bc1474ae8..b1131317f 100644
--- a/ggml/src/ggml-hexagon/htp/CMakeLists.txt
+++ b/ggml/src/ggml-hexagon/htp/CMakeLists.txt
@@ -20,7 +20,7 @@ add_library(${HTP_LIB} SHARED
work-queue.c
dma-queue.c
hmx-queue.c
- htp-tensor.c
+ htp-dcache.c
matmul-ops.c
flash-attn-ops.c
gated-delta-net-ops.c
diff --git a/ggml/src/ggml-hexagon/htp/flash-attn-ops.c b/ggml/src/ggml-hexagon/htp/flash-attn-ops.c
index 2ee55feb9..4b2734695 100644
--- a/ggml/src/ggml-hexagon/htp/flash-attn-ops.c
+++ b/ggml/src/ggml-hexagon/htp/flash-attn-ops.c
@@ -361,7 +361,7 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_FA_K_PREP, ir);
for (uint32_t r = 0; r < current_block_size; ++r) {
__fp16 * row_k = (__fp16 *)(k_base + r * factx->size_k_row_padded);
- hvx_dequantize_row_q8_0_f16(row_k, row_k, DK);
+ hvx_dequantize_row_flat_q8_0_f16(row_k, row_k, DK);
}
htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_FA_K_PREP, ir);
}
@@ -369,7 +369,7 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_FA_V_PREP, ir);
for (uint32_t r = 0; r < current_block_size; ++r) {
__fp16 * row_v = (__fp16 *)(v_base + r * factx->size_v_row_padded);
- hvx_dequantize_row_q8_0_f16(row_v, row_v, DV);
+ hvx_dequantize_row_flat_q8_0_f16(row_v, row_v, DV);
}
htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_FA_V_PREP, ir);
}
@@ -625,7 +625,7 @@ static void fa_k_interleave_thread(unsigned int n, unsigned int i, void * data)
if (factx->k->type == HTP_TYPE_Q8_0) {
for (uint32_t r = start; r < end; ++r) {
__fp16 * row_k = (__fp16 *)((char *)args->curr_k + r * args->src_stride * sizeof(__fp16));
- hvx_dequantize_row_q8_0_f16(row_k, row_k, factx->DK);
+ hvx_dequantize_row_flat_q8_0_f16(row_k, row_k, factx->DK);
}
}
hmx_interleave_rows_to_tiles(factx->vtcm_k_tiles[args->buf_idx], (const __fp16 *) args->curr_k, total_rows, factx->DK_pad,
@@ -679,7 +679,7 @@ static void fa_v_interleave_thread(unsigned int n, unsigned int i, void * data)
if (factx->v->type == HTP_TYPE_Q8_0) {
for (uint32_t r = start; r < end; ++r) {
__fp16 * row_v = (__fp16 *)((char *)args->v_src + r * args->src_stride * sizeof(__fp16));
- hvx_dequantize_row_q8_0_f16(row_v, row_v, factx->DV);
+ hvx_dequantize_row_flat_q8_0_f16(row_v, row_v, factx->DV);
}
}
hmx_interleave_cols_to_tiles(v_tiles_dst, (const __fp16 *) args->v_src, total_rows, factx->DV_pad,
diff --git a/ggml/src/ggml-hexagon/htp/get-rows-ops.c b/ggml/src/ggml-hexagon/htp/get-rows-ops.c
index 90f38fa56..27948c37a 100644
--- a/ggml/src/ggml-hexagon/htp/get-rows-ops.c
+++ b/ggml/src/ggml-hexagon/htp/get-rows-ops.c
@@ -17,21 +17,32 @@
#include "htp-tensor.h"
#include "hvx-utils.h"
#include "hvx-quant.h"
-#include "matmul-ops.h"
#include "get-rows-ops.h"
#include "work-queue.h"
+struct get_rows_context;
+
+typedef uint32_t (*get_rows_read_index_fn_t)(const void * src1, uint32_t idx, int idx_type);
+typedef void (*get_rows_flat_compute_fn_t)(float * dst_spad, const void * src_spad, uint32_t cur_elems);
+typedef void (*get_rows_tiled_compute_fn_t)(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row);
+
struct get_rows_context {
struct htp_ops_context * octx;
const struct htp_get_rows_kernel_params * kparams;
struct htp_get_rows_vtcm_layout vtcm_layout;
- uint8_t * vtcm_base;
+ uint8_t * vtcm_base;
+ const void * src1_data;
+ get_rows_read_index_fn_t read_index;
+ get_rows_flat_compute_fn_t compute_flat;
+ get_rows_tiled_compute_fn_t compute_tiled;
uint32_t task_start;
uint32_t tasks;
uint32_t tasks_per_thread;
uint32_t tile_size;
uint32_t tile_stride;
- bool index_i32;
+ uint32_t src0_row_size;
+ uint32_t dst_row_size;
+ int idx_type;
};
#define get_rows_preamble \
@@ -64,229 +75,169 @@ struct get_rows_context {
\
const uint32_t nr = ne10 * ne11 * ne12;
-#define GET_ROWS_THREAD_ST_FN(IDX_TYPE) \
-static void get_rows_thread_st_##IDX_TYPE(unsigned int nth, unsigned int ith, void *data) { \
- struct get_rows_context * grctx = (struct get_rows_context *)data; \
- struct htp_ops_context * octx = grctx->octx; \
- const struct htp_get_rows_kernel_params * kparams = grctx->kparams; \
- get_rows_preamble; \
- const uint32_t dr = grctx->tasks_per_thread; \
- const uint32_t ir0 = grctx->task_start + dr * ith; \
- if (ir0 >= grctx->task_start + grctx->tasks) { \
- return; \
- } \
- const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks); \
- const uint32_t row_size_bytes = htp_tensor_get_row_size(octx->src[0]->type, ne00); \
- dma_queue * dma_q = octx->ctx->dma[ith]; \
- for (uint32_t i = ir0; i < ir1; ++i) { \
- const uint32_t i12 = fastdiv(i, &kparams->div_ne10_ne11); \
- const uint32_t rem = i - i12 * ne11 * ne10; \
- const uint32_t i11 = fastdiv(rem, &kparams->div_ne10); \
- const uint32_t i10 = rem - i11 * ne10; \
- const IDX_TYPE * src1_ptr = (const IDX_TYPE *)(uintptr_t)(octx->src[1]->data + i10*nb10 + i11*nb11 + i12*nb12); \
- const uint32_t i01 = (uint32_t)*src1_ptr; \
- assert(i01 < ne01); \
- const uint32_t q02 = fastdiv(i11, &kparams->div_ne02); \
- const uint32_t i02 = i11 - q02 * ne02; \
- const uint32_t q03 = fastdiv(i12, &kparams->div_ne03); \
- const uint32_t i03 = i12 - q03 * ne03; \
- const dma_addr_t src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03; \
- const dma_addr_t dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3; \
- while (!dma_queue_push(dma_q, dma_make_data(dst_data, src0_data), nb1, nb01, \
- row_size_bytes, 1)) { \
- dma_queue_pop(dma_q); \
- } \
- } \
- dma_queue_flush(dma_q); \
+static uint32_t get_rows_read_index_ddr(
+ const void * src1,
+ uint32_t idx,
+ int idx_type
+) {
+ const uint32_t off = idx * ((idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t));
+ return *(const uint32_t *) ((const uint8_t *) src1 + off);
}
-GET_ROWS_THREAD_ST_FN(int32_t)
-GET_ROWS_THREAD_ST_FN(int64_t)
-
-#define GET_ROWS_THREAD_DT_FN(TYPE_NAME, SRC0_SIZE_EXPR, IDX_TYPE, COMPUTE_EXPR) \
-static void get_rows_thread_##TYPE_NAME##_##IDX_TYPE(unsigned int nth, unsigned int ith, void *data) { \
- struct get_rows_context * grctx = (struct get_rows_context *)data; \
- struct htp_ops_context * octx = grctx->octx; \
- const struct htp_get_rows_kernel_params * kparams = grctx->kparams; \
- get_rows_preamble; \
- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \
- const uint32_t dr = grctx->tasks_per_thread; \
- const uint32_t ir0 = grctx->task_start + dr * ith; \
- if (ir0 >= grctx->task_start + grctx->tasks) { \
- return; \
- } \
- const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks); \
- const uint32_t chunks_per_row = kparams->chunks_per_row; \
- const uint32_t chunk_size = kparams->chunk_size; \
- dma_queue * dma_q = octx->ctx->dma[ith]; \
- const struct htp_get_rows_vtcm_layout * vtcm_layout = &grctx->vtcm_layout; \
- uint8_t * vtcm_src0 = grctx->vtcm_base + vtcm_layout->off_src0 + ith * vtcm_layout->src0_bytes_per_thread; \
- uint8_t * vtcm_dst = grctx->vtcm_base + vtcm_layout->off_dst + ith * vtcm_layout->dst_bytes_per_thread; \
- for (uint32_t step = 0, spad_idx = 0; step < ir1 - ir0 && spad_idx < 2; ++step, spad_idx++) { \
- const uint32_t i = ir0 + step; \
- const uint32_t row_idx = fastdiv(i, &kparams->div_chunks_per_row); \
- const uint32_t chunk_idx = i - row_idx * chunks_per_row; \
- const uint32_t i12 = fastdiv(row_idx, &kparams->div_ne10_ne11); \
- const uint32_t rem = row_idx - i12 * ne11 * ne10; \
- const uint32_t i11 = fastdiv(rem, &kparams->div_ne10); \
- const uint32_t i10 = rem - i11 * ne10; \
- const IDX_TYPE * src1_ptr = (const IDX_TYPE *)(uintptr_t)(octx->src[1]->data + i10*nb10 + i11*nb11 + i12*nb12); \
- const uint32_t i01 = (uint32_t)*src1_ptr; \
- assert(i01 < ne01); \
- const uint32_t q02 = fastdiv(i11, &kparams->div_ne02); \
- const uint32_t i02 = i11 - q02 * ne02; \
- const uint32_t q03 = fastdiv(i12, &kparams->div_ne03); \
- const uint32_t i03 = i12 - q03 * ne03; \
- const uint32_t offset = chunk_idx * chunk_size; \
- const uint32_t cur_elems = (offset < ne00) ? MIN(chunk_size, ne00 - offset) : 0; \
- const uint32_t cur_src0_bytes = SRC0_SIZE_EXPR(cur_elems); \
- const uint32_t cur_dst_bytes = cur_elems * sizeof(float); \
- const dma_addr_t src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03 + SRC0_SIZE_EXPR(offset); \
- dma_queue_push(dma_q, \
- dma_make_data(octx->dst->data, \
- vtcm_dst + spad_idx * vtcm_layout->dst_spad_half_size), \
- cur_dst_bytes, vtcm_layout->dst_spad_half_size, cur_dst_bytes, 0); \
- dma_queue_push(dma_q, \
- dma_make_data(vtcm_src0 + spad_idx * vtcm_layout->src0_spad_half_size, src0_data), \
- vtcm_layout->src0_spad_half_size, cur_src0_bytes, cur_src0_bytes, 1); \
- } \
- for (uint32_t step = 0; step < ir1 - ir0; ++step) { \
- const uint32_t i = ir0 + step; \
- void * dst_spad = (void *) dma_queue_pop(dma_q).src; \
- void * src_spad = (void *) dma_queue_pop(dma_q).dst; \
- const uint32_t row_idx = fastdiv(i, &kparams->div_chunks_per_row); \
- const uint32_t chunk_idx = i - row_idx * chunks_per_row; \
- const uint32_t i12 = fastdiv(row_idx, &kparams->div_ne10_ne11); \
- const uint32_t rem = row_idx - i12 * ne11 * ne10; \
- const uint32_t i11 = fastdiv(rem, &kparams->div_ne10); \
- const uint32_t i10 = rem - i11 * ne10; \
- const uint32_t offset = chunk_idx * chunk_size; \
- const uint32_t cur_elems = (offset < ne00) ? MIN(chunk_size, ne00 - offset) : 0; \
- const uint32_t cur_dst_bytes = cur_elems * sizeof(float); \
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, i); \
- COMPUTE_EXPR; \
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, i); \
- const dma_addr_t dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3 + offset * sizeof(float); \
- dma_queue_push(dma_q, \
- dma_make_data(dst_data, dst_spad), \
- cur_dst_bytes, vtcm_layout->dst_spad_half_size, cur_dst_bytes, 1); \
- const uint32_t next_step = step + 2; \
- if (next_step < ir1 - ir0) { \
- const uint32_t pi = ir0 + next_step; \
- const uint32_t prow_idx = fastdiv(pi, &kparams->div_chunks_per_row); \
- const uint32_t pchunk_idx = pi - prow_idx * chunks_per_row; \
- const uint32_t pi12 = fastdiv(prow_idx, &kparams->div_ne10_ne11); \
- const uint32_t prem = prow_idx - pi12 * ne11 * ne10; \
- const uint32_t pi11 = fastdiv(prem, &kparams->div_ne10); \
- const uint32_t pi10 = prem - pi11 * ne10; \
- const IDX_TYPE * psrc1_ptr = (const IDX_TYPE *)(uintptr_t)(octx->src[1]->data + pi10*nb10 + pi11*nb11 + pi12*nb12); \
- const uint32_t pi01 = (uint32_t)*psrc1_ptr; \
- assert(pi01 < ne01); \
- const uint32_t pq02 = fastdiv(pi11, &kparams->div_ne02); \
- const uint32_t pi02 = pi11 - pq02 * ne02; \
- const uint32_t pq03 = fastdiv(pi12, &kparams->div_ne03); \
- const uint32_t pi03 = pi12 - pq03 * ne03; \
- const uint32_t poffset = pchunk_idx * chunk_size; \
- const uint32_t pcur_elems = (poffset < ne00) ? MIN(chunk_size, ne00 - poffset) : 0; \
- const uint32_t pcur_src0_bytes = SRC0_SIZE_EXPR(pcur_elems); \
- const dma_addr_t psrc0_data = \
- octx->src[0]->data + pi01*nb01 + pi02*nb02 + pi03*nb03 + SRC0_SIZE_EXPR(poffset); \
- dma_queue_push(dma_q, \
- dma_make_data(src_spad, psrc0_data), \
- vtcm_layout->src0_spad_half_size, pcur_src0_bytes, pcur_src0_bytes, 1); \
- } \
- } \
- dma_queue_flush(dma_q); \
+static __attribute__((noinline)) uint32_t get_rows_read_index_vtcm(
+ const void * src1,
+ uint32_t idx,
+ int idx_type
+) {
+ const uint32_t off = idx * ((idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t));
+ const HVX_Vector v = hvx_vmem((const uint8_t *) src1 + off); // off & ~127u in HW
+ // vextract is normally slow, but here it avoids an HVX->L2 store and L1D miss
+ return (uint32_t) Q6_R_vextract_VR(v, off);
}
-#define F16_BYTES(n) ((n) * sizeof(__fp16))
-#define Q8_0_BYTES(n) (((n) / 32) * sizeof(block_q8_0))
+static void get_rows_thread_sametype(unsigned int nth, unsigned int ith, void * data) {
+ struct get_rows_context * grctx = (struct get_rows_context *) data;
+ struct htp_ops_context * octx = grctx->octx;
+ const struct htp_get_rows_kernel_params * kparams = grctx->kparams;
+ get_rows_preamble;
-static __attribute__((noinline)) void compute_get_rows_f16(float * dst_spad, const void * src_spad, uint32_t cur_elems) {
- hvx_dequantize_row_f16_f32(dst_spad, src_spad, cur_elems);
-}
+ const uint32_t dr = grctx->tasks_per_thread;
+ const uint32_t ir0 = grctx->task_start + dr * ith;
+ if (ir0 >= grctx->task_start + grctx->tasks) {
+ return;
+ }
+ const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks);
+ const uint32_t row_size_bytes = grctx->src0_row_size;
+ dma_queue * dma_q = octx->ctx->dma[ith];
-static __attribute__((noinline)) void compute_get_rows_q8_0(float * dst_spad, const void * src_spad, uint32_t cur_elems) {
- hvx_dequantize_row_q8_0_f32(dst_spad, src_spad, cur_elems);
-}
+ for (uint32_t i = ir0; i < ir1; ++i) {
+ const uint32_t i12 = fastdiv(i, &kparams->div_ne10_ne11);
+ const uint32_t rem = i - i12 * ne11 * ne10;
+ const uint32_t i11 = fastdiv(rem, &kparams->div_ne10);
+ const uint32_t i10 = rem - i11 * ne10;
-GET_ROWS_THREAD_DT_FN(f16, F16_BYTES, int32_t, { compute_get_rows_f16((float *)dst_spad, src_spad, cur_elems); })
-GET_ROWS_THREAD_DT_FN(f16, F16_BYTES, int64_t, { compute_get_rows_f16((float *)dst_spad, src_spad, cur_elems); })
-
-GET_ROWS_THREAD_DT_FN(q8_0, Q8_0_BYTES, int32_t, { compute_get_rows_q8_0((float *)dst_spad, src_spad, cur_elems); })
-GET_ROWS_THREAD_DT_FN(q8_0, Q8_0_BYTES, int64_t, { compute_get_rows_q8_0((float *)dst_spad, src_spad, cur_elems); })
-
-
-static __attribute__((noinline)) void compute_get_rows_tiled(float * dst, const uint8_t * tile, uint32_t row, bool q4, bool q4_k) {
- const HVX_VectorPred first2 = Q6_Q_vsetq_R(2);
- const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);
- HVX_Vector vq = Q6_V_vzero();
- if (q4) {
- const HVX_VectorPred first1 = Q6_Q_vsetq_R(1);
- const HVX_VectorPred first3 = Q6_Q_vsetq_R(3);
- for (int group = 3; group >= 0; --group) {
- const HVX_Vector v = Q6_V_vror_VR(hvx_vmem(tile + group * VLEN), row);
- // Four planes contribute bytes at 0, 32, 64 and 96 after rotation.
- HVX_Vector packed = Q6_V_vmux_QVV(first1, v, Q6_V_vror_VR(v, 31));
- packed = Q6_V_vmux_QVV(first2, packed, Q6_V_vror_VR(v, 62));
- packed = Q6_V_vmux_QVV(first3, packed, Q6_V_vror_VR(v, 93));
- vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));
- }
- const HVX_Vector lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));
- const HVX_Vector hi = Q6_Vub_vlsr_VubR(vq, 4);
- vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));
- if (!q4_k) {
- vq = Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(8));
- }
- } else {
- for (int group = 7; group >= 0; --group) {
- const HVX_Vector v = Q6_V_vror_VR(hvx_vmem(tile + group * VLEN), 2 * row);
- // Two planes contribute halfwords at 0 and 64 after rotation.
- const HVX_Vector packed = Q6_V_vmux_QVV(first2, v, Q6_V_vror_VR(v, 62));
- vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));
+ const uint32_t i01 = grctx->read_index(grctx->src1_data, i, grctx->idx_type);
+ assert(i01 < ne01);
+
+ const uint32_t q02 = fastdiv(i11, &kparams->div_ne02);
+ const uint32_t i02 = i11 - q02 * ne02;
+ const uint32_t q03 = fastdiv(i12, &kparams->div_ne03);
+ const uint32_t i03 = i12 - q03 * ne03;
+
+ const dma_addr_t src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03;
+ const dma_addr_t dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3;
+
+ while (!dma_queue_push(dma_q, dma_make_data(dst_data, src0_data), nb1, nb01, row_size_bytes, 1)) {
+ dma_queue_pop(dma_q);
}
}
- const HVX_Vector scales = hvx_vmem(tile + (q4 ? 512 : 1024));
- const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, (q4_k ? 4 : 2) * row));
- const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));
- const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);
- const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));
- HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);
- if (q4_k) {
- const HVX_Vector offset_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row + 2));
- const HVX_Vector offset = Q6_V_lo_W(hvx_vec_f16_to_f32(offset_hf));
- values = hvx_vec_add_f32_f32(values, offset);
+ dma_queue_flush(dma_q);
+}
+
+
+
+struct get_rows_flat_task {
+ dma_addr_t src0_data;
+ dma_addr_t dst_data;
+};
+
+static inline struct get_rows_flat_task get_rows_flat_calc_task(
+ const struct htp_ops_context * octx,
+ const struct get_rows_context * grctx,
+ uint32_t i,
+ bool need_src0
+) {
+ const struct htp_get_rows_kernel_params * kparams = grctx->kparams;
+ get_rows_preamble;
+
+ const uint32_t i12 = fastdiv(i, &kparams->div_ne10_ne11);
+ const uint32_t rem = i - i12 * ne11 * ne10;
+ const uint32_t i11 = fastdiv(rem, &kparams->div_ne10);
+ const uint32_t i10 = rem - i11 * ne10;
+
+ struct get_rows_flat_task task;
+ task.src0_data = 0;
+ task.dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3;
+
+ if (need_src0) {
+ const uint32_t i01 = grctx->read_index(grctx->src1_data, i, grctx->idx_type);
+ assert(i01 < ne01);
+
+ const uint32_t q02 = fastdiv(i11, &kparams->div_ne02);
+ const uint32_t i02 = i11 - q02 * ne02;
+ const uint32_t q03 = fastdiv(i12, &kparams->div_ne03);
+ const uint32_t i03 = i12 - q03 * ne03;
+
+ task.src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03;
}
- *(HVX_Vector *) dst = values;
+
+ return task;
}
-static __attribute__((noinline)) void compute_get_rows_q6_k(float * dst, const uint8_t * tile, uint32_t row) {
- const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);
- const HVX_VectorPred first16 = Q6_Q_vsetq_R(16 * sizeof(float));
- const HVX_Vector mask_0f = Q6_Vb_vsplat_R(0x0F);
- const HVX_Vector mask_03 = Q6_Vb_vsplat_R(0x03);
- HVX_Vector vq = Q6_V_vzero();
-
- for (int group = 7; group >= 0; --group) {
- const HVX_Vector lo_plane = Q6_V_vror_VR(hvx_vmem(tile + (group >> 1) * VLEN), 4 * row);
- const HVX_Vector hi_plane = Q6_V_vror_VR(hvx_vmem(tile + 512 + (group >> 2) * VLEN), 4 * row);
- const HVX_Vector lo = (group & 1) ? Q6_Vub_vlsr_VubR(lo_plane, 4) : Q6_V_vand_VV(lo_plane, mask_0f);
- const HVX_Vector hi = Q6_Vub_vlsr_VubR(hi_plane, 2 * (group & 3));
- const HVX_Vector packed = Q6_V_vor_VV(lo, Q6_Vw_vasl_VwR(Q6_V_vand_VV(hi, mask_03), 4));
- vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));
+static void get_rows_thread_flat(unsigned int nth, unsigned int ith, void * data) {
+ struct get_rows_context * grctx = (struct get_rows_context *) data;
+ struct htp_ops_context * octx = grctx->octx;
+ get_rows_preamble;
+
+ const uint32_t dr = grctx->tasks_per_thread;
+ const uint32_t ir0 = grctx->task_start + dr * ith;
+ if (ir0 >= grctx->task_start + grctx->tasks) {
+ return;
+ }
+ const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks);
+
+ dma_queue * dma_q = octx->ctx->dma[ith];
+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];
+ const struct htp_get_rows_vtcm_layout * vtcm_layout = &grctx->vtcm_layout;
+ uint8_t * vtcm_src0 = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_src0) + ith * vtcm_layout->src0_bytes_per_thread;
+ uint8_t * vtcm_dst = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_dst) + ith * vtcm_layout->dst_bytes_per_thread;
+ const uint32_t src0_bytes = grctx->src0_row_size;
+ const uint32_t dst_bytes = grctx->dst_row_size;
+ get_rows_flat_compute_fn_t compute = grctx->compute_flat;
+
+ for (uint32_t step = 0, spad_idx = 0; step < ir1 - ir0 && spad_idx < 2; ++step, spad_idx++) {
+ const uint32_t i = ir0 + step;
+ struct get_rows_flat_task task = get_rows_flat_calc_task(octx, grctx, i, true);
+
+ dma_queue_push(dma_q,
+ dma_make_data(octx->dst->data,
+ vtcm_dst + spad_idx * vtcm_layout->dst_spad_half_size),
+ dst_bytes, vtcm_layout->dst_spad_half_size, dst_bytes, 0);
+ dma_queue_push(dma_q,
+ dma_make_data(vtcm_src0 + spad_idx * vtcm_layout->src0_spad_half_size, task.src0_data),
+ vtcm_layout->src0_spad_half_size, src0_bytes, src0_bytes, 1);
+ }
+
+ for (uint32_t step = 0; step < ir1 - ir0; ++step) {
+ const uint32_t i = ir0 + step;
+ void * dst_spad = (void *) dma_queue_pop(dma_q).src;
+ void * src_spad = (void *) dma_queue_pop(dma_q).dst;
+
+ struct get_rows_flat_task task = get_rows_flat_calc_task(octx, grctx, i, false);
+
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);
+ compute((float *) dst_spad, src_spad, ne00);
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);
+
+ dma_queue_push(dma_q,
+ dma_make_data(task.dst_data, dst_spad),
+ dst_bytes, vtcm_layout->dst_spad_half_size, dst_bytes, 1);
+
+ const uint32_t next_step = step + 2;
+ if (next_step < ir1 - ir0) {
+ const uint32_t pi = ir0 + next_step;
+ struct get_rows_flat_task next_task = get_rows_flat_calc_task(octx, grctx, pi, true);
+ dma_queue_push(dma_q,
+ dma_make_data(src_spad, next_task.src0_data),
+ vtcm_layout->src0_spad_half_size, src0_bytes, src0_bytes, 1);
+ }
}
- const HVX_Vector scales = hvx_vmem(tile + 768);
- const HVX_Vector scale_lo_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));
- const HVX_Vector scale_hi_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 64 + 2 * row));
- const HVX_Vector scale_lo = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_lo_hf));
- const HVX_Vector scale_hi = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hi_hf));
- const HVX_Vector scale = Q6_V_vmux_QVV(first16, scale_lo, scale_hi);
- const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(32)));
- const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));
- *(HVX_Vector *) dst = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);
+ dma_queue_flush(dma_q);
}
+
+
struct get_rows_tiled_task {
dma_addr_t tile_src_base;
dma_addr_t dst_data;
@@ -307,16 +258,15 @@ static inline struct get_rows_tiled_task get_rows_tiled_calc_task(
const uint32_t rem = i - i12 * ne11 * ne10;
const uint32_t i11 = fastdiv(rem, &kparams->div_ne10);
const uint32_t i10 = rem - i11 * ne10;
- const dma_addr_t src1_data = octx->src[1]->data + i10*nb10 + i11*nb11 + i12*nb12;
- const uint32_t i01 = grctx->index_i32 ? *(const int32_t *)(uintptr_t) src1_data : (uint32_t) *(const int64_t *)(uintptr_t) src1_data;
+ const uint32_t i01 = grctx->read_index(grctx->src1_data, i, grctx->idx_type);
assert(i01 < ne01);
const uint32_t q02 = fastdiv(i11, &kparams->div_ne02);
const uint32_t i02 = i11 - q02 * ne02;
const uint32_t q03 = fastdiv(i12, &kparams->div_ne03);
const uint32_t i03 = i12 - q03 * ne03;
- const uint32_t column_tile = i01 / HTP_MM_HMX_TILE_N_ROWS;
- const uint32_t row = i01 % HTP_MM_HMX_TILE_N_ROWS;
+ const uint32_t column_tile = i01 / HTP_TENSOR_TILE_N_ROWS;
+ const uint32_t row = i01 % HTP_TENSOR_TILE_N_ROWS;
const dma_addr_t matrix = octx->src[0]->data + i02*nb02 + i03*nb03;
struct get_rows_tiled_task task;
@@ -329,7 +279,6 @@ static inline struct get_rows_tiled_task get_rows_tiled_calc_task(
static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * data) {
struct get_rows_context * grctx = (struct get_rows_context *) data;
struct htp_ops_context * octx = grctx->octx;
- const struct htp_get_rows_kernel_params * kparams = grctx->kparams;
get_rows_preamble;
const uint32_t dr = grctx->tasks_per_thread;
@@ -339,19 +288,17 @@ static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * dat
}
const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks);
- const uint32_t n_k_tiles = ne00 / HTP_MM_HMX_TILE_N_COLS;
+ const uint32_t n_k_tiles = ne00 / HTP_TENSOR_TILE_N_COLS;
const struct htp_get_rows_vtcm_layout * vtcm_layout = &grctx->vtcm_layout;
- uint8_t * src_spad_base = grctx->vtcm_base + vtcm_layout->off_src0 + ith * vtcm_layout->src0_bytes_per_thread;
- uint8_t * dst_spad_base = grctx->vtcm_base + vtcm_layout->off_dst + ith * vtcm_layout->dst_bytes_per_thread;
+ uint8_t * src_spad_base = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_src0) + ith * vtcm_layout->src0_bytes_per_thread;
+ uint8_t * dst_spad_base = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_dst) + ith * vtcm_layout->dst_bytes_per_thread;
dma_queue * dma_q = octx->ctx->dma[ith];
struct htp_thread_trace * tr = &octx->ctx->trace[ith];
const uint32_t tile_size = grctx->tile_size;
const uint32_t tile_stride = grctx->tile_stride;
- const uint32_t dst_bytes = ne00 * sizeof(float);
- const bool is_q4 = octx->src[0]->type == HTP_TYPE_Q4_0 || octx->src[0]->type == HTP_TYPE_Q4_K;
- const bool is_q4_k = octx->src[0]->type == HTP_TYPE_Q4_K;
- const bool is_q6_k = octx->src[0]->type == HTP_TYPE_Q6_K;
+ const uint32_t dst_bytes = grctx->dst_row_size;
+ get_rows_tiled_compute_fn_t compute_tiled = grctx->compute_tiled;
for (uint32_t step = 0, spad_idx = 0; step < ir1 - ir0 && spad_idx < 2; ++step, ++spad_idx) {
const uint32_t i = ir0 + step;
@@ -376,15 +323,7 @@ static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * dat
struct get_rows_tiled_task task = get_rows_tiled_calc_task(octx, grctx, i, n_k_tiles, tile_size);
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);
- for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {
- const uint8_t * tile = src_spad + k_tile * tile_stride;
- float * dst_block = dst_spad + k_tile * HTP_MM_HMX_TILE_N_COLS;
- if (is_q6_k) {
- compute_get_rows_q6_k(dst_block, tile, task.row);
- } else {
- compute_get_rows_tiled(dst_block, tile, task.row, is_q4, is_q4_k);
- }
- }
+ compute_tiled(dst_spad, src_spad, n_k_tiles, tile_stride, task.row);
htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);
// Real writeback of dst_spad
@@ -405,16 +344,40 @@ static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * dat
dma_queue_flush(dma_q);
}
+static __attribute__((noinline)) void get_rows_dma_src1(
+ struct htp_ops_context * octx,
+ uint8_t * vtcm_src1,
+ uint32_t elem_size
+) {
+ const uint32_t ne10 = octx->src[1]->ne[0];
+ const uint32_t ne11 = octx->src[1]->ne[1];
+ const uint32_t ne12 = octx->src[1]->ne[2];
+ const uint32_t ne13 = octx->src[1]->ne[3];
+ const uint32_t nb11 = octx->src[1]->nb[1];
+ const uint32_t nb12 = octx->src[1]->nb[2];
+ const uint32_t nb13 = octx->src[1]->nb[3];
+
+ dma_queue * dma_q = octx->ctx->dma[0];
+ uint8_t * cur_dst = vtcm_src1;
+ for (uint32_t i13 = 0; i13 < ne13; ++i13) {
+ for (uint32_t i12 = 0; i12 < ne12; ++i12) {
+ const dma_addr_t src_plane = octx->src[1]->data + i12 * nb12 + i13 * nb13;
+ while (!dma_queue_push(dma_q, dma_make_data(cur_dst, src_plane),
+ ne10 * elem_size, nb11, ne10 * elem_size, ne11)) {
+ dma_queue_pop(dma_q);
+ }
+ cur_dst += ne10 * ne11 * elem_size;
+ }
+ }
+ dma_queue_flush(dma_q);
+}
+
int op_get_rows(struct htp_ops_context * octx) {
const struct htp_get_rows_kernel_params * kparams = (const struct htp_get_rows_kernel_params *) octx->kernel_params;
- if (octx->src[0]->type != HTP_TYPE_F32 &&
- octx->src[0]->type != HTP_TYPE_F16 &&
- octx->src[0]->type != HTP_TYPE_Q4_0 &&
- octx->src[0]->type != HTP_TYPE_Q4_K &&
- octx->src[0]->type != HTP_TYPE_Q6_K &&
- octx->src[0]->type != HTP_TYPE_Q8_0 &&
- octx->src[0]->type != HTP_TYPE_I32) {
+ const uint32_t ne00 = octx->src[0]->ne[0];
+ const uint32_t src0_row_size = htp_tensor_get_row_size(octx->src[0]->type, ne00);
+ if (src0_row_size == 0) {
return HTP_STATUS_NO_SUPPORT;
}
@@ -432,20 +395,16 @@ int op_get_rows(struct htp_ops_context * octx) {
return HTP_STATUS_NO_SUPPORT;
}
- if (htp_tensor_is_extended(octx->src[1])) {
- return HTP_STATUS_NO_SUPPORT;
- }
-
const struct htp_tensor * dst = octx->dst;
const uint32_t total_tasks = kparams->total_tasks;
- const size_t dst_row_size = htp_tensor_get_row_size(dst->type, dst->ne[0]);
+ const uint32_t dst_row_size = (kparams->kernel_type == HTP_GET_ROWS_KERNEL_SAMETYPE) ? src0_row_size : ne00 * sizeof(float);
uint32_t task_start = 0;
uint32_t tasks = total_tasks;
if (octx->ctx->mdev.count > 1) {
uint32_t tasks_per_chunk = 1;
- htp_tensor_mdev_rows_per_chunk(dst, dst_row_size / dst->ne[0], (uint32_t) dst_row_size, &tasks_per_chunk);
+ htp_tensor_mdev_rows_per_chunk(dst, dst_row_size / dst->ne[0], dst_row_size, &tasks_per_chunk);
const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_tasks, tasks_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
task_start = range.start;
tasks = range.count;
@@ -465,15 +424,18 @@ int op_get_rows(struct htp_ops_context * octx) {
grctx.octx = octx;
grctx.kparams = kparams;
grctx.vtcm_base = (uint8_t *)octx->ctx->vtcm_base;
+ grctx.compute_flat = NULL;
+ grctx.compute_tiled = NULL;
grctx.task_start = task_start;
grctx.tasks = tasks;
grctx.tasks_per_thread = octx->ctx->mdev.count == 1 ? kparams->tasks_per_thread : fastdiv(tasks + n_threads - 1, &octx->n_threads_div);
- grctx.tile_size = htp_mm_get_weight_tile_size(octx->src[0]->type);
- grctx.tile_stride = (grctx.tile_size + 127) & ~127;
- grctx.index_i32 = octx->src[1]->type == HTP_TYPE_I32;
+ grctx.tile_size = 0;
+ grctx.tile_stride = 0;
+ grctx.src0_row_size = src0_row_size;
+ grctx.dst_row_size = dst_row_size;
+ grctx.idx_type = octx->src[1]->type;
- const uint32_t ne00 = octx->src[0]->ne[0];
- htp_get_rows_vtcm_layout_build(&grctx.vtcm_layout, kparams->kernel_type, octx->src[0]->type, ne00, n_threads);
+ htp_get_rows_vtcm_layout_build(&grctx.vtcm_layout, kparams->kernel_type, octx->src[0]->type, ne00, total_tasks, grctx.idx_type, n_threads);
if (grctx.vtcm_layout.total_bytes > octx->ctx->vtcm_size) {
FARF(ERROR, "get-rows: VTCM reservation %zu is too small, needed %zu\n",
@@ -481,27 +443,56 @@ int op_get_rows(struct htp_ops_context * octx) {
return HTP_STATUS_INVAL_PARAMS;
}
- const bool is_i32 = (octx->src[1]->type == HTP_TYPE_I32);
-
work_queue_func_t q_func = NULL;
switch (kparams->kernel_type) {
case HTP_GET_ROWS_KERNEL_SAMETYPE:
- q_func = (work_queue_func_t)(is_i32 ? get_rows_thread_st_int32_t : get_rows_thread_st_int64_t);
+ q_func = get_rows_thread_sametype;
break;
case HTP_GET_ROWS_KERNEL_TILED:
+ switch (octx->src[0]->type) {
+ case HTP_TYPE_F16: grctx.compute_tiled = hvx_dequantize_row_tiled_f16_f32; break;
+ case HTP_TYPE_Q8_0: grctx.compute_tiled = hvx_dequantize_row_tiled_q8_0_f32; break;
+ case HTP_TYPE_Q4_0: grctx.compute_tiled = hvx_dequantize_row_tiled_q4_0_f32; break;
+ case HTP_TYPE_Q4_K: grctx.compute_tiled = hvx_dequantize_row_tiled_q4_k_f32; break;
+ case HTP_TYPE_Q5_K: grctx.compute_tiled = hvx_dequantize_row_tiled_q5_k_f32; break;
+ case HTP_TYPE_Q6_K: grctx.compute_tiled = hvx_dequantize_row_tiled_q6_k_f32; break;
+ default: return HTP_STATUS_NO_SUPPORT;
+ }
+ grctx.tile_size = htp_tensor_get_tile_size(octx->src[0]->type);
+ grctx.tile_stride = htp_tensor_get_tile_padded_size(octx->src[0]->type);
q_func = get_rows_thread_tiled;
break;
case HTP_GET_ROWS_KERNEL_FLAT:
switch (octx->src[0]->type) {
- case HTP_TYPE_F16: q_func = (work_queue_func_t)(is_i32 ? get_rows_thread_f16_int32_t : get_rows_thread_f16_int64_t); break;
- case HTP_TYPE_Q8_0: q_func = (work_queue_func_t)(is_i32 ? get_rows_thread_q8_0_int32_t : get_rows_thread_q8_0_int64_t); break;
+ case HTP_TYPE_F16: grctx.compute_flat = hvx_dequantize_row_flat_f16_f32; break;
+ case HTP_TYPE_BF16: grctx.compute_flat = hvx_dequantize_row_flat_bf16_f32; break;
+ case HTP_TYPE_Q8_0: grctx.compute_flat = hvx_dequantize_row_flat_q8_0_f32; break;
+ case HTP_TYPE_Q4_0: grctx.compute_flat = hvx_dequantize_row_flat_q4_0_f32; break;
+ case HTP_TYPE_Q4_K: grctx.compute_flat = hvx_dequantize_row_flat_q4_k_f32; break;
+ case HTP_TYPE_Q5_K: grctx.compute_flat = hvx_dequantize_row_flat_q5_k_f32; break;
+ case HTP_TYPE_Q6_K: grctx.compute_flat = hvx_dequantize_row_flat_q6_k_f32; break;
default: return HTP_STATUS_NO_SUPPORT;
}
+ q_func = get_rows_thread_flat;
break;
default:
return HTP_STATUS_NO_SUPPORT;
}
+ const uint32_t elem_size = (grctx.idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t);
+ const size_t src1_bytes = (size_t) total_tasks * elem_size;
+
+ if (!htp_tensor_is_extended(octx->src[1]) && src1_bytes <= 128 &&
+ htp_tensor_is_contiguous(octx->src[1], elem_size)) {
+ grctx.src1_data = (const void *) (uintptr_t) octx->src[1]->data;
+ grctx.read_index = get_rows_read_index_ddr;
+ } else {
+ uint8_t * vtcm_src1 = VTCM_LAYOUT_PTR(uint8_t, grctx.vtcm_base, grctx.vtcm_layout.off_src1);
+ get_rows_dma_src1(octx, vtcm_src1, elem_size);
+ grctx.src1_data = vtcm_src1;
+ grctx.read_index = get_rows_read_index_vtcm;
+ }
+
FARF(HIGH, "get-rows: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : src0-vtcm-size %zu dst-vtcm-size %zu kernel-type %d n-threads %d\n",
octx->src[0]->ne[0], octx->src[0]->ne[1], octx->src[0]->ne[2], octx->src[0]->ne[3],
octx->src[1]->ne[0], octx->src[1]->ne[1], octx->src[1]->ne[2], octx->src[1]->ne[3],
diff --git a/ggml/src/ggml-hexagon/htp/get-rows-ops.h b/ggml/src/ggml-hexagon/htp/get-rows-ops.h
index 69fa54464..f44e58d16 100644
--- a/ggml/src/ggml-hexagon/htp/get-rows-ops.h
+++ b/ggml/src/ggml-hexagon/htp/get-rows-ops.h
@@ -5,7 +5,8 @@
#include <string.h>
#include "hex-fastdiv.h"
-#include "matmul-ops.h"
+#include "htp-tensor.h"
+#include "htp-vtcm.h"
enum htp_get_rows_kernel_type {
HTP_GET_ROWS_KERNEL_SAMETYPE = 0,
@@ -16,8 +17,6 @@ enum htp_get_rows_kernel_type {
struct htp_get_rows_kernel_params {
int32_t n_threads;
int32_t kernel_type;
- int32_t chunks_per_row;
- int32_t chunk_size;
int32_t total_tasks;
int32_t tasks_per_thread;
int32_t vtcm_size;
@@ -25,13 +24,14 @@ struct htp_get_rows_kernel_params {
// Fastdiv helpers
struct fastdiv_values div_ne10;
struct fastdiv_values div_ne10_ne11;
- struct fastdiv_values div_chunks_per_row;
struct fastdiv_values div_ne02;
struct fastdiv_values div_ne03;
};
struct htp_get_rows_vtcm_layout {
size_t total_bytes;
+ size_t off_src1;
+ size_t src1_bytes;
size_t off_src0;
size_t off_dst;
@@ -43,57 +43,44 @@ struct htp_get_rows_vtcm_layout {
};
static inline void htp_get_rows_vtcm_layout_build(
- struct htp_get_rows_vtcm_layout * vtcm_layout,
+ struct htp_get_rows_vtcm_layout * L,
int kernel_type,
int type,
uint32_t ne00,
+ uint32_t total_tasks,
+ int idx_type,
uint32_t n_threads) {
- if (kernel_type == HTP_GET_ROWS_KERNEL_SAMETYPE) {
- memset(vtcm_layout, 0, sizeof(*vtcm_layout));
- return;
+ memset(L, 0, sizeof(*L));
+
+ const size_t elem_size = (idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t);
+ const size_t src1_raw_bytes = total_tasks * elem_size;
+ const size_t src1_bytes = ((src1_raw_bytes + 127) & ~127) + 128;
+
+ L->src1_bytes = src1_bytes;
+ const size_t src1_alloc_bytes = (src1_bytes + 255) & ~255;
+
+ if (kernel_type != HTP_GET_ROWS_KERNEL_SAMETYPE) {
+ size_t src0_row_size = 0;
+ if (kernel_type == HTP_GET_ROWS_KERNEL_TILED) {
+ const size_t tile_stride = htp_tensor_get_tile_padded_size(type);
+ const uint32_t n_k_tiles = ne00 / HTP_TENSOR_TILE_N_COLS;
+ src0_row_size = n_k_tiles > 0 ? (n_k_tiles * tile_stride) : tile_stride;
+ } else {
+ src0_row_size = htp_tensor_get_row_size(type, ne00);
+ }
+
+ L->src0_spad_half_size = (src0_row_size + 255) & ~255;
+ L->dst_spad_half_size = (ne00 * sizeof(float) + 255) & ~255;
+ L->src0_bytes_per_thread = 2 * L->src0_spad_half_size;
+ L->dst_bytes_per_thread = 2 * L->dst_spad_half_size;
}
- if (kernel_type == HTP_GET_ROWS_KERNEL_TILED) {
- const size_t tile_size = htp_mm_get_weight_tile_size(type);
- const size_t tile_stride = (tile_size + 127) & ~127;
- const uint32_t n_k_tiles = ne00 / HTP_MM_HMX_TILE_N_COLS;
- const size_t row_tiles_size = n_k_tiles > 0 ? (n_k_tiles * tile_stride) : tile_stride;
- vtcm_layout->src0_spad_half_size = (row_tiles_size + 255) & ~255;
- vtcm_layout->dst_spad_half_size = (ne00 * sizeof(float) + 255) & ~255;
- vtcm_layout->src0_bytes_per_thread = 2 * vtcm_layout->src0_spad_half_size;
- vtcm_layout->dst_bytes_per_thread = 2 * vtcm_layout->dst_spad_half_size;
- vtcm_layout->off_src0 = 0;
- vtcm_layout->off_dst = vtcm_layout->src0_bytes_per_thread * n_threads;
- vtcm_layout->total_bytes = vtcm_layout->off_dst + vtcm_layout->dst_bytes_per_thread * n_threads;
- return;
- }
-
- uint32_t src0_row_size = 0;
- switch (type) {
- case HTP_TYPE_F16:
- src0_row_size = ne00 * 2;
- break;
- case HTP_TYPE_Q8_0:
- src0_row_size = (ne00 / 32) * 34;
- break;
- default:
- src0_row_size = 0;
- break;
- }
-
- size_t src0_row_size_aligned = (src0_row_size + 255) & ~255;
- size_t dst_row_size_aligned = (ne00 * sizeof(float) + 255) & ~255;
-
- vtcm_layout->src0_spad_half_size = src0_row_size_aligned;
- vtcm_layout->dst_spad_half_size = dst_row_size_aligned;
-
- vtcm_layout->src0_bytes_per_thread = src0_row_size_aligned * 2;
- vtcm_layout->dst_bytes_per_thread = dst_row_size_aligned * 2;
-
- vtcm_layout->off_src0 = 0;
- vtcm_layout->off_dst = vtcm_layout->off_src0 + vtcm_layout->src0_bytes_per_thread * n_threads;
- vtcm_layout->total_bytes = vtcm_layout->off_dst + vtcm_layout->dst_bytes_per_thread * n_threads;
+ size_t off = 0;
+ VTCM_LAYOUT_ALLOC(off, off_src1, src1_alloc_bytes);
+ VTCM_LAYOUT_ALLOC(off, off_src0, L->src0_bytes_per_thread * n_threads);
+ VTCM_LAYOUT_ALLOC(off, off_dst, L->dst_bytes_per_thread * n_threads);
+ L->total_bytes = off;
}
#if defined(__cplusplus)
diff --git a/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h b/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
index f0a0d9de1..309ec4877 100644
--- a/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
+++ b/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h
@@ -14,11 +14,15 @@ static const __fp16 iq4_nl_to_fp16_lut[64] __attribute__((aligned(VLEN))) = {
// --- tiled format dequantizers ---
-typedef struct {
+typedef struct tiled_dequantize_state tiled_dequantize_state_t;
+typedef void (*tiled_dequant_task_fn_t)(const tiled_dequantize_state_t * state, uint32_t start_tile, uint32_t end_tile);
+
+struct tiled_dequantize_state {
struct htp_context * ctx;
struct htp_thread_trace * traces;
__fp16 * dst;
const uint8_t * src;
+ tiled_dequant_task_fn_t task_fn;
struct fastdiv_values n_k_tiles_div;
uint32_t n_k_tiles;
@@ -31,7 +35,7 @@ typedef struct {
uint32_t k_block;
size_t row_stride;
uint32_t weight_type;
-} tiled_dequantize_state_t;
+};
// Dequantize a single tile from tiled weight data (already in VTCM) to tile-major FP16.
static void dequantize_tiled_weight_to_fp16_task_q4_0(
@@ -611,7 +615,7 @@ static void dequantize_tiled_weight_to_fp16_task_q5_k(
}
}
-// Q6_K stores 6-bit weights and one fp16 scale per 16 k, see HTP_MM_WEIGHT_TILE_SIZE_Q6_K.
+// Q6_K stores 6-bit weights and one fp16 scale per 16 k, see HTP_TENSOR_TILE_SIZE_Q6_K.
// A k-group holds 4 k per row, the HMX tile holds 2, so each group is dealt into two tiles.
static void dequantize_tiled_weight_to_fp16_task_q6_k(
const tiled_dequantize_state_t *state,
@@ -669,7 +673,7 @@ static void dequantize_tiled_weight_to_fp16_task_q6_k(
}
}
-// Q3_K stores 3-bit weights and one fp16 scale per 16 k, see HTP_MM_WEIGHT_TILE_SIZE_Q3_K.
+// Q3_K stores 3-bit weights and one fp16 scale per 16 k, see HTP_TENSOR_TILE_SIZE_Q3_K.
static void dequantize_tiled_weight_to_fp16_task_q3_k(
const tiled_dequantize_state_t *state,
uint32_t start_tile, uint32_t end_tile) {
@@ -701,7 +705,7 @@ static void dequantize_tiled_weight_to_fp16_task_q3_k(
}
}
-// Q2_K stores 2-bit weights, the ggml scale bytes and d / -dmin per super-block, see HTP_MM_WEIGHT_TILE_SIZE_Q2_K.
+// Q2_K stores 2-bit weights, the ggml scale bytes and d / -dmin per super-block, see HTP_TENSOR_TILE_SIZE_Q2_K.
static void dequantize_tiled_weight_to_fp16_task_q2_k(
const tiled_dequantize_state_t *state,
uint32_t start_tile, uint32_t end_tile) {
@@ -837,6 +841,56 @@ void quantize_f32_weight_to_fp16_tiles_task(
}
}
+static __attribute__((noinline))
+void convert_bf16_weight_to_fp16_tiles_task(
+ const tiled_dequantize_state_t *state,
+ uint32_t start_tile, uint32_t end_tile) {
+
+ const uint32_t n_k_tiles = state->n_k_tiles;
+ const struct fastdiv_values n_k_tiles_div = state->n_k_tiles_div;
+
+ const HVX_Vector v_scat_base = hvx_vmem(hmx_transpose_scatter_offsets);
+ const HVX_Vector v_scat_step = Q6_V_vsplat_R(4);
+ const HVX_VectorPred q_mask64 = Q6_Q_vsetq_R(64);
+
+ unsigned ct = fastdiv((unsigned)start_tile, &n_k_tiles_div);
+ unsigned kt = fastmodulo((unsigned)start_tile, n_k_tiles, &n_k_tiles_div);
+
+ for (unsigned t = start_tile; t < (unsigned)end_tile; ) {
+ if (kt >= (unsigned)n_k_tiles) { kt = 0; ct++; }
+
+ __fp16 *tile_base = state->dst + t * HTP_MM_HMX_TILE_N_ELMS;
+ {
+ uint32_t byte_off = kt * 32 * sizeof(uint16_t);
+
+ HVX_Vector v_off = v_scat_base;
+ for (uint32_t r = 0; r < HTP_MM_HMX_TILE_N_ROWS; r += 2) {
+ uint32_t row0 = ct * HTP_MM_HMX_TILE_N_COLS + r;
+ uint32_t row1 = row0 + 1;
+
+ const uint8_t *r0 = state->src + row0 * state->row_stride;
+ const uint8_t *r1 = state->src + row1 * state->row_stride;
+
+ HVX_Vector v0_raw = (row0 < state->n_cols) ? hvx_vmemu((const void *)(r0 + byte_off)) : Q6_V_vzero();
+ HVX_Vector v1_raw = (row1 < state->n_cols) ? hvx_vmemu((const void *)(r1 + byte_off)) : Q6_V_vzero();
+
+ HVX_Vector v0_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(Q6_Ww_vunpack_Vh(v0_raw)), 16);
+ HVX_Vector v1_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(Q6_Ww_vunpack_Vh(v1_raw)), 16);
+
+ HVX_Vector v_out = hvx_vec_f32_to_f16(v0_f32, v1_f32);
+
+ Q6_vscatter_QRMVwV(q_mask64, (size_t)tile_base, HTP_MM_HMX_TILE_SIZE - 1, v_off, v_out);
+ v_off = Q6_Vw_vadd_VwVw(v_off, v_scat_step);
+
+ HVX_Vector v_out_hi = Q6_V_vror_VR(v_out, 64);
+ Q6_vscatter_QRMVwV(q_mask64, (size_t)tile_base, HTP_MM_HMX_TILE_SIZE - 1, v_off, v_out_hi);
+ v_off = Q6_Vw_vadd_VwVw(v_off, v_scat_step);
+ }
+ }
+ ++t; ++kt;
+ }
+}
+
// --- End tiled dequantizers ---
// dot-chunk functions require external HMX lock
diff --git a/ggml/src/ggml-hexagon/htp/htp-ctx.h b/ggml/src/ggml-hexagon/htp/htp-ctx.h
index b36322ae8..22a05b917 100644
--- a/ggml/src/ggml-hexagon/htp/htp-ctx.h
+++ b/ggml/src/ggml-hexagon/htp/htp-ctx.h
@@ -3,6 +3,7 @@
#include "dma-queue.h"
#include "hmx-queue.h"
+#include "htp-dcache.h"
#include "htp-ops.h"
#include "hex-profile.h"
#include "work-queue.h"
@@ -18,8 +19,7 @@
#define HTP_MAX_NTHREADS 10
#endif
-#define HTP_MAX_MMAPS 64
-#define HTP_MAX_DIRTY_RANGES 64
+#define HTP_MAX_MMAPS 64
// Memory mapping
struct htp_mmap {
@@ -29,11 +29,6 @@ struct htp_mmap {
uint32_t flags;
};
-struct htp_dirty_range {
- uint32_t start;
- uint32_t end;
-};
-
// Scratchpad state
struct htp_spad {
const struct htp_tensor * src; // original src of the data (for reuse)
diff --git a/ggml/src/ggml-hexagon/htp/htp-tensor.c b/ggml/src/ggml-hexagon/htp/htp-dcache.c
similarity index 97%
rename from ggml/src/ggml-hexagon/htp/htp-tensor.c
rename to ggml/src/ggml-hexagon/htp/htp-dcache.c
index 03b0070d7..840b1edb7 100644
--- a/ggml/src/ggml-hexagon/htp/htp-tensor.c
+++ b/ggml/src/ggml-hexagon/htp/htp-dcache.c
@@ -1,4 +1,4 @@
-#include "htp-tensor.h"
+#include "htp-dcache.h"
#include <qurt.h>
#include <qurt_memory.h>
@@ -94,7 +94,7 @@ static void merge_dirty_ranges(struct htp_context * ctx) {
}
}
-void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {
+void htp_dcache_mark_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {
const struct htp_tensor * pending[HTP_OP_MAX_OUTPUTS];
uint32_t n_pending = 0;
@@ -301,7 +301,7 @@ static void flush_dirty_ranges(struct htp_context * ctx, const struct htp_dirty_
}
}
-void htp_flush_dirty_ranges(struct htp_context * ctx) {
+void htp_dcache_flush_dirty_ranges(struct htp_context * ctx) {
struct htp_dirty_range ranges[HTP_MAX_DIRTY_RANGES];
uint32_t n_ranges = 0;
uint64_t total_dirty = 0;
@@ -328,7 +328,7 @@ void htp_flush_dirty_ranges(struct htp_context * ctx) {
memset(ctx->dirty_ranges, 0, sizeof(ctx->dirty_ranges));
}
-void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {
+void htp_dcache_flush_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {
const struct htp_tensor * dirty_tensors[HTP_OP_MAX_INPUTS];
struct htp_dirty_range ranges[HTP_OP_MAX_INPUTS];
uint32_t n_dirty = 0;
diff --git a/ggml/src/ggml-hexagon/htp/htp-dcache.h b/ggml/src/ggml-hexagon/htp/htp-dcache.h
new file mode 100644
index 000000000..1996ab258
--- /dev/null
+++ b/ggml/src/ggml-hexagon/htp/htp-dcache.h
@@ -0,0 +1,20 @@
+#ifndef HTP_DCACHE_H
+#define HTP_DCACHE_H
+
+#include <stdint.h>
+#include "htp-ops.h"
+
+#define HTP_MAX_DIRTY_RANGES 64
+
+struct htp_dirty_range {
+ uint32_t start;
+ uint32_t end;
+};
+
+struct htp_context;
+
+void htp_dcache_flush_dirty_ranges(struct htp_context * ctx);
+void htp_dcache_flush_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);
+void htp_dcache_mark_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);
+
+#endif // HTP_DCACHE_H
diff --git a/ggml/src/ggml-hexagon/htp/htp-ops.h b/ggml/src/ggml-hexagon/htp/htp-ops.h
index 70d56e181..bb358d09a 100644
--- a/ggml/src/ggml-hexagon/htp/htp-ops.h
+++ b/ggml/src/ggml-hexagon/htp/htp-ops.h
@@ -30,21 +30,14 @@ enum htp_data_type {
HTP_TYPE_IQ4_NL = 20,
HTP_TYPE_I32 = 26,
HTP_TYPE_I64 = 27,
+ HTP_TYPE_BF16 = 30,
HTP_TYPE_MXFP4 = 39,
- // types used internally for repack, dyn.quant, etc
- HTP_TYPE_Q4_0_TILED = 200,
- HTP_TYPE_Q4_1_TILED,
- HTP_TYPE_Q8_0_TILED,
- HTP_TYPE_MXFP4_TILED,
-
HTP_TYPE_INVALID
};
-// Constats for internal types
-#define QK_Q4_0_TILED 256 // 32x32 Q4_0 tiled layout
+// Constants for internal types
#define QK_Q8_0_TILED 128 // 32x32 Q8_0 tiled layout
-#define QK_MXFP4_TILED 256 // 32x32 MXFP4 tiled layout
@@ -141,7 +134,7 @@ enum htp_op_code {
enum htp_tensor_flags {
HTP_TENSOR_WEIGHT = (1U << 0), // Tensor buffer model weight data (not compute)
- HTP_TENSOR_REPACK = (1U << 1), // Tensor is in repacked tiled format
+ HTP_TENSOR_TILED = (1U << 1), // Tensor is in tiled format
HTP_TENSOR_FENCE = (1U << 2) // Tensor is synchronization fence (explicitly managed)
};
diff --git a/ggml/src/ggml-hexagon/htp/htp-tensor.h b/ggml/src/ggml-hexagon/htp/htp-tensor.h
index f7a966836..0bce68dd0 100644
--- a/ggml/src/ggml-hexagon/htp/htp-tensor.h
+++ b/ggml/src/ggml-hexagon/htp/htp-tensor.h
@@ -8,27 +8,59 @@
#include "hex-common.h"
#include "hex-fastdiv.h"
-enum {
- HTP_TENSOR_MDEV_LINE_SIZE = 128,
-};
+#define HTP_TENSOR_MDEV_LINE_SIZE 128
+
+// Currently HMX requires 32x32 tile formats (do not change)
+#define HTP_TENSOR_TILE_N_COLS 32
+#define HTP_TENSOR_TILE_N_ROWS 32
+
+#define HTP_TENSOR_TILE_SIZE_F16 2048
+#define HTP_TENSOR_TILE_SIZE_Q4_0 576
+#define HTP_TENSOR_TILE_SIZE_Q4_1 640
+#define HTP_TENSOR_TILE_SIZE_Q8_0 1088
+#define HTP_TENSOR_TILE_SIZE_IQ4_NL 576
+#define HTP_TENSOR_TILE_SIZE_MXFP4 544
+#define HTP_TENSOR_TILE_SIZE_Q4_K 640
+#define HTP_TENSOR_TILE_SIZE_Q5_K 768
+#define HTP_TENSOR_TILE_SIZE_Q6_K 896
+#define HTP_TENSOR_TILE_SIZE_Q3_K 512
+#define HTP_TENSOR_TILE_SIZE_Q2_K 336
+
+// Q2_K, Q3_K, Q6_K use vrmpy byte order within each 128B HVX vector:
+// k = 0..31 is split into 8 groups g = k/4 of 4 k each; byte 4*r + (k%4) holds row r
+//
+// Q2_K (336B, lossless tiling with same size as block_q2_K):
+// bytes 0..255 (vec 0..1): unsigned 2-bit quants, vec m holds groups 4m..4m+3 at bit offsets 0,2,4,6
+// bytes 256..319: raw scales[] bytes (min << 4 | sc): [0..31] for k 0..15, [32..63] for k 16..31
+// bytes 320..335: 8 fp16 in tile j (0..7) of super-block: d (j < 4) or -dmin (j >= 4) for rows 8*(j%4)..8*(j%4)+7
+//
+// Q3_K (512B):
+// bytes 0..255 (vec 0..1): low 2 bits, vec m holds groups 4m..4m+3 at bit offsets 0,2,4,6
+// bytes 256..383 (vec 2): sign mask, bit g set when hmask is 0 (quant = low2 - 4)
+// bytes 384..511 (vec 3): fp16 scales d * (scales[] - 32): fp16[0..31] for k 0..15, fp16[32..63] for k 16..31
+//
+// Q4_K (640B, same layout as Q4_1 tile; each 256-k super-block splits into 8 tiles of 32 k):
+// bytes 0..511 (vec 0..3): 4-bit quants, byte 32*cp + r holds k = 2*cp (low nibble) and 2*cp + 1 (high nibble)
+// bytes 512..639 (vec 4): 32x2 fp16 per row: (d * sc, -dmin * m)
+//
+// Q5_K (768B):
+// bytes 0..639 (vec 0..4): Q4_K tile (low 4 bits + fp16 scale/min pairs)
+// bytes 640..767 (vec 5): 5th-bit plane; byte l holds the 8 high bits for lane l across vec i = 0..3 (bit 2i = lo, 2i+1 = hi)
+//
+// Q6_K (896B):
+// bytes 0..511 (vec 0..3): low 4 bits, vec i holds group 2i (low nibble) and 2i+1 (high nibble)
+// bytes 512..767 (vec 4..5): high 2 bits, vec 4+m holds groups 4m..4m+3 at bit offsets 0,2,4,6
+// bytes 768..895 (vec 6): fp16 scales d * scales[]: fp16[0..31] for k 0..15, fp16[32..63] for k 16..31
struct htp_tensor_mdev_range {
uint32_t start;
uint32_t count;
};
-static inline void * htp_tensor_data(const struct htp_tensor * t) {
- return (void *) (uintptr_t) t->data;
-}
-
static inline bool htp_tensor_is_extended(const struct htp_tensor * t) {
return t && (t->data >> 32) != 0;
}
-static inline uint32_t * htp_tensor_flags(const struct htp_tensor * t) {
- return (uint32_t *) &t->flags;
-}
-
static inline bool htp_tensor_is_contiguous(const struct htp_tensor * t, uint32_t type_size) {
uint32_t next_nb = type_size;
if (t->ne[0] != 1 && t->nb[0] != next_nb) {
@@ -125,19 +157,50 @@ static inline struct htp_tensor_mdev_range htp_tensor_mdev_partition(uint32_t to
return range;
}
-static inline uint32_t htp_tensor_get_row_size(int type, uint32_t ne00) {
+static inline uint32_t htp_tensor_get_tile_size(int type) {
switch (type) {
- case HTP_TYPE_F32: return ne00 * 4;
- case HTP_TYPE_F16: return ne00 * 2;
- case HTP_TYPE_Q8_0: return (ne00 / 32) * 34;
- case HTP_TYPE_I32: return ne00 * 4;
- default: return 0;
+ case HTP_TYPE_Q4_0: return HTP_TENSOR_TILE_SIZE_Q4_0;
+ case HTP_TYPE_IQ4_NL: return HTP_TENSOR_TILE_SIZE_IQ4_NL;
+ case HTP_TYPE_Q4_1: return HTP_TENSOR_TILE_SIZE_Q4_1;
+ case HTP_TYPE_Q4_K: return HTP_TENSOR_TILE_SIZE_Q4_K;
+ case HTP_TYPE_Q8_0: return HTP_TENSOR_TILE_SIZE_Q8_0;
+ case HTP_TYPE_Q5_K: return HTP_TENSOR_TILE_SIZE_Q5_K;
+ case HTP_TYPE_Q6_K: return HTP_TENSOR_TILE_SIZE_Q6_K;
+ case HTP_TYPE_Q3_K: return HTP_TENSOR_TILE_SIZE_Q3_K;
+ case HTP_TYPE_Q2_K: return HTP_TENSOR_TILE_SIZE_Q2_K;
+ case HTP_TYPE_MXFP4: return HTP_TENSOR_TILE_SIZE_MXFP4;
+ case HTP_TYPE_F16: return HTP_TENSOR_TILE_SIZE_F16;
+ default: return 0;
}
}
-struct htp_context;
-void htp_flush_dirty_ranges(struct htp_context * ctx);
-void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);
-void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);
+static inline uint32_t htp_tensor_get_tile_padded_size(int type) {
+ return (htp_tensor_get_tile_size(type) + 127u) & ~127u;
+}
+
+static inline bool htp_tensor_is_tiled(const struct htp_tensor * t) {
+ return t && (t->flags & HTP_TENSOR_TILED) != 0;
+}
+
+static inline uint32_t htp_tensor_get_row_size(int type, uint32_t ne00) {
+ switch (type) {
+ case HTP_TYPE_F32: return ne00 * 4;
+ case HTP_TYPE_F16:
+ case HTP_TYPE_BF16: return ne00 * 2;
+ case HTP_TYPE_Q8_0: return (ne00 / 32) * 34;
+ case HTP_TYPE_Q4_0:
+ case HTP_TYPE_IQ4_NL: return (ne00 / 32) * 18;
+ case HTP_TYPE_Q4_1: return (ne00 / 32) * 20;
+ case HTP_TYPE_MXFP4: return (ne00 / 32) * 17;
+ case HTP_TYPE_Q2_K: return (ne00 / 256) * 84;
+ case HTP_TYPE_Q3_K: return (ne00 / 256) * 110;
+ case HTP_TYPE_Q4_K: return (ne00 / 256) * 144;
+ case HTP_TYPE_Q5_K: return (ne00 / 256) * 176;
+ case HTP_TYPE_Q6_K: return (ne00 / 256) * 210;
+ case HTP_TYPE_I32: return ne00 * 4;
+ case HTP_TYPE_I64: return ne00 * 8;
+ default: return 0;
+ }
+}
#endif // HTP_TENSOR_H
diff --git a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h
index 605892aa7..d7488976b 100644
--- a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h
+++ b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h
@@ -76,7 +76,7 @@ static inline void quantize_f16_f16_kernel(
#define HVX_OP_MUL_F32(a, b) Q6_Vsf_vmpy_VsfVsf(a, b)
#endif
-static inline void vec_dot_f32_f32_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {
+static inline void flat_vec_dot_f32_f32_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {
const HVX_Vector * restrict x = (const HVX_Vector *) vx;
const HVX_Vector * restrict y = (const HVX_Vector *) vy;
@@ -104,7 +104,7 @@ static inline void vec_dot_f32_f32_aa_1x1(const uint32_t n, float * restrict s,
*s = hvx_vec_get_f32(hvx_vec_reduce_sum_f32(rsum));
}
-static inline void vec_dot_f32_f32_aa_2x1(const uint32_t n, float * restrict s0,
+static inline void flat_vec_dot_f32_f32_aa_2x1(const uint32_t n, float * restrict s0,
const void * restrict vx0, const void * restrict vx1,
const void * restrict vy0) {
const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;
@@ -143,7 +143,7 @@ static inline void vec_dot_f32_f32_aa_2x1(const uint32_t n, float * restrict s0,
hvx_vec_store_u(s0, 8, rsum);
}
-static inline void vec_dot_f32_f32_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,
+static inline void flat_vec_dot_f32_f32_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,
const void * restrict vx0, const void * restrict vx1,
const void * restrict vy0, const void * restrict vy1) {
const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;
@@ -196,10 +196,203 @@ static inline void vec_dot_f32_f32_aa_2x2(const uint32_t n, float * restrict s0,
hvx_vec_store_u(s1, 8, r0_r1_c1_sum);
}
+static inline void flat_vec_dot_bf16_f32_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {
+ const HVX_Vector * restrict x = (const HVX_Vector *) vx;
+ const HVX_Vector * restrict y = (const HVX_Vector *) vy;
+
+ uint32_t nvec = n / 64;
+ uint32_t rem = n % 64;
+
+ HVX_Vector rsum0 = Q6_V_vzero();
+ HVX_Vector rsum1 = Q6_V_vzero();
+
+ uint32_t i = 0;
+ #pragma unroll(2)
+ for (i = 0; i < nvec; i++) {
+ HVX_Vector v_bf16 = x[i];
+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);
+ HVX_Vector w0_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);
+ HVX_Vector w1_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p), 16);
+
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w0_f32, y[2 * i + 0]));
+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w1_f32, y[2 * i + 1]));
+ }
+
+ if (rem) {
+ HVX_VectorPred bmask = Q6_Q_vsetq2_R(rem * 2);
+ HVX_Vector v_bf16 = Q6_V_vand_QV(bmask, x[i]);
+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);
+ HVX_Vector w0_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);
+ HVX_Vector w1_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p), 16);
+
+ if (rem <= 32) {
+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R(rem * 4);
+ HVX_Vector y0 = Q6_V_vand_QV(a_mask, y[2 * i + 0]);
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w0_f32, y0));
+ } else {
+ HVX_Vector y0 = y[2 * i + 0];
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w0_f32, y0));
+
+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R((rem - 32) * 4);
+ HVX_Vector y1 = Q6_V_vand_QV(a_mask, y[2 * i + 1]);
+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w1_f32, y1));
+ }
+ }
+
+ HVX_Vector rsum = HVX_OP_ADD_F32(rsum0, rsum1);
+ *s = hvx_vec_get_f32(hvx_vec_reduce_sum_f32(rsum));
+}
+
+static inline void flat_vec_dot_bf16_f32_aa_2x1(const uint32_t n, float * restrict s0,
+ const void * restrict vx0, const void * restrict vx1,
+ const void * restrict vy0) {
+ const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;
+ const HVX_Vector * restrict x1 = (const HVX_Vector *) vx1;
+ const HVX_Vector * restrict y = (const HVX_Vector *) vy0;
+
+ uint32_t nvec = n / 64;
+ uint32_t rem = n % 64;
+
+ HVX_Vector rsum0 = Q6_V_vzero();
+ HVX_Vector rsum1 = Q6_V_vzero();
+
+ uint32_t i = 0;
+ #pragma unroll(2)
+ for (i = 0; i < nvec; i++) {
+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(x0[i]);
+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(x1[i]);
+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);
+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);
+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);
+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);
+
+ HVX_Vector y0 = y[2 * i + 0];
+ HVX_Vector y1 = y[2 * i + 1];
+
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w00_f32, y0));
+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w10_f32, y0));
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w01_f32, y1));
+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w11_f32, y1));
+ }
+
+ if (rem) {
+ HVX_VectorPred bmask = Q6_Q_vsetq2_R(rem * 2);
+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x0[i]));
+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x1[i]));
+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);
+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);
+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);
+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);
+
+ if (rem <= 32) {
+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R(rem * 4);
+ HVX_Vector y0 = Q6_V_vand_QV(a_mask, y[2 * i + 0]);
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w00_f32, y0));
+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w10_f32, y0));
+ } else {
+ HVX_Vector y0 = y[2 * i + 0];
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w00_f32, y0));
+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w10_f32, y0));
+
+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R((rem - 32) * 4);
+ HVX_Vector y1 = Q6_V_vand_QV(a_mask, y[2 * i + 1]);
+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w01_f32, y1));
+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w11_f32, y1));
+ }
+ }
+
+ HVX_Vector rsum = hvx_vec_reduce_sum_f32x2(rsum0, rsum1);
+ hvx_vec_store_u(s0, 8, rsum);
+}
+
+static inline void flat_vec_dot_bf16_f32_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,
+ const void * restrict vx0, const void * restrict vx1,
+ const void * restrict vy0, const void * restrict vy1) {
+ const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;
+ const HVX_Vector * restrict x1 = (const HVX_Vector *) vx1;
+ const HVX_Vector * restrict y0 = (const HVX_Vector *) vy0;
+ const HVX_Vector * restrict y1 = (const HVX_Vector *) vy1;
+
+ uint32_t nvec = n / 64;
+ uint32_t rem = n % 64;
+
+ HVX_Vector r0_c0_sum = Q6_V_vzero();
+ HVX_Vector r0_c1_sum = Q6_V_vzero();
+ HVX_Vector r1_c0_sum = Q6_V_vzero();
+ HVX_Vector r1_c1_sum = Q6_V_vzero();
+
+ uint32_t i = 0;
+ #pragma unroll(2)
+ for (i = 0; i < nvec; i++) {
+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(x0[i]);
+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(x1[i]);
+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);
+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);
+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);
+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);
+
+ HVX_Vector c0_0 = y0[2 * i + 0];
+ HVX_Vector c0_1 = y0[2 * i + 1];
+ HVX_Vector c1_0 = y1[2 * i + 0];
+ HVX_Vector c1_1 = y1[2 * i + 1];
+
+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w00_f32, c0_0));
+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w00_f32, c1_0));
+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w10_f32, c0_0));
+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w10_f32, c1_0));
+
+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w01_f32, c0_1));
+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w01_f32, c1_1));
+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w11_f32, c0_1));
+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w11_f32, c1_1));
+ }
+
+ if (rem) {
+ HVX_VectorPred bmask = Q6_Q_vsetq2_R(rem * 2);
+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x0[i]));
+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x1[i]));
+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);
+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);
+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);
+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);
+
+ if (rem <= 32) {
+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R(rem * 4);
+ HVX_Vector c0_0 = Q6_V_vand_QV(a_mask, y0[2 * i + 0]);
+ HVX_Vector c1_0 = Q6_V_vand_QV(a_mask, y1[2 * i + 0]);
+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w00_f32, c0_0));
+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w00_f32, c1_0));
+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w10_f32, c0_0));
+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w10_f32, c1_0));
+ } else {
+ HVX_Vector c0_0 = y0[2 * i + 0];
+ HVX_Vector c1_0 = y1[2 * i + 0];
+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w00_f32, c0_0));
+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w00_f32, c1_0));
+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w10_f32, c0_0));
+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w10_f32, c1_0));
+
+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R((rem - 32) * 4);
+ HVX_Vector c0_1 = Q6_V_vand_QV(a_mask, y0[2 * i + 1]);
+ HVX_Vector c1_1 = Q6_V_vand_QV(a_mask, y1[2 * i + 1]);
+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w01_f32, c0_1));
+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w01_f32, c1_1));
+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w11_f32, c0_1));
+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w11_f32, c1_1));
+ }
+ }
+
+ HVX_Vector r0_r1_c0_sum = hvx_vec_reduce_sum_f32x2(r0_c0_sum, r1_c0_sum);
+ HVX_Vector r0_r1_c1_sum = hvx_vec_reduce_sum_f32x2(r0_c1_sum, r1_c1_sum);
+
+ hvx_vec_store_u(s0, 8, r0_r1_c0_sum);
+ hvx_vec_store_u(s1, 8, r0_r1_c1_sum);
+}
+
#undef HVX_OP_ADD_F32
#undef HVX_OP_MUL_F32
-static inline void vec_dot_f16_f16_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {
+static inline void flat_vec_dot_f16_f16_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {
const HVX_Vector * restrict x = (const HVX_Vector *) vx;
const HVX_Vector * restrict y = (const HVX_Vector *) vy;
@@ -226,7 +419,7 @@ static inline void vec_dot_f16_f16_aa_1x1(const uint32_t n, float * restrict s,
hvx_vec_store_u(s, 4, hvx_vec_reduce_sum_f32(rsum));
}
-static inline void vec_dot_f16_f16_aa_2x1(const uint32_t n, float * restrict s0,
+static inline void flat_vec_dot_f16_f16_aa_2x1(const uint32_t n, float * restrict s0,
const void * restrict vx0, const void * restrict vx1,
const void * restrict vy0) {
const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;
@@ -263,7 +456,7 @@ static inline void vec_dot_f16_f16_aa_2x1(const uint32_t n, float * restrict s0,
hvx_vec_store_u(s0, 8, rsum);
}
-static inline void vec_dot_f16_f16_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,
+static inline void flat_vec_dot_f16_f16_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,
const void * restrict vx0, const void * restrict vx1,
const void * restrict vy0, const void * restrict vy1) {
const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;
diff --git a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
index 0c86e407e..91e511725 100644
--- a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
+++ b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h
@@ -274,6 +274,72 @@ static void quantize_row_f32_q8_1_s16_tiled(float * restrict x, uint8_t * restri
}
}
+static inline void quantize_half_f16_tiled_splat(HVX_Vector v_src, HVX_Vector * restrict dst) {
+ static const uint8_t __attribute__((aligned(128))) repl[128] = {
+ 0x00, 0x00, 0x00, 0x00, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ 0x20, 0x20, 0x20, 0x20, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ 0x40, 0x40, 0x40, 0x40, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ 0x20, 0x20, 0x20, 0x20, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,
+ };
+ HVX_Vector v_repl_ctrl = *(const HVX_Vector *) repl;
+
+ #pragma unroll(4)
+ for (int cp = 0; cp < 32; cp++) {
+ dst[cp] = Q6_V_vdelta_VV(cp == 0 ? v_src : Q6_V_vror_VR(v_src, cp * 4), v_repl_ctrl);
+ }
+}
+
+static inline void quantize_block_f32_f16_tiled(const float * restrict x, uint8_t * restrict y_block) {
+ assert((unsigned long) x % 128 == 0);
+ assert((unsigned long) y_block % 128 == 0);
+
+ const HVX_Vector * vx = (const HVX_Vector *) x;
+ HVX_Vector * restrict dst = (HVX_Vector *) y_block;
+
+ HVX_Vector vx01_hf = hvx_vec_f32_to_f16(vx[0], vx[1]);
+ quantize_half_f16_tiled_splat(vx01_hf, dst);
+
+ HVX_Vector vx23_hf = hvx_vec_f32_to_f16(vx[2], vx[3]);
+ quantize_half_f16_tiled_splat(vx23_hf, dst + 32);
+}
+
+static inline void quantize_block_f16_f16_tiled(const __fp16 * restrict x, uint8_t * restrict y_block) {
+ assert((unsigned long) x % 128 == 0);
+ assert((unsigned long) y_block % 128 == 0);
+
+ const HVX_Vector * vx = (const HVX_Vector *) x;
+ HVX_Vector * restrict dst = (HVX_Vector *) y_block;
+
+ quantize_half_f16_tiled_splat(vx[0], dst);
+ quantize_half_f16_tiled_splat(vx[1], dst + 32);
+}
+
+static void quantize_row_f32_f16_tiled(const float * restrict x, uint8_t * restrict y, uint32_t k) {
+ assert(k % 32 == 0);
+ const uint32_t qk = QK_Q8_0_TILED;
+ const uint32_t nb = (k + qk - 1) / qk;
+
+ for (uint32_t i = 0; i < nb; i++) {
+ uint8_t * restrict y_block = y + i * 4 * 2048;
+ quantize_block_f32_f16_tiled(x + i * qk, y_block);
+ }
+}
+
+static void quantize_row_f16_f16_tiled(const __fp16 * restrict x, uint8_t * restrict y, uint32_t k) {
+ assert(k % 32 == 0);
+ const uint32_t qk = QK_Q8_0_TILED;
+ const uint32_t nb = (k + qk - 1) / qk;
+
+ for (uint32_t i = 0; i < nb; i++) {
+ uint8_t * restrict y_block = y + i * 4 * 2048;
+ quantize_block_f16_f16_tiled(x + i * qk, y_block);
+ }
+}
+
// Dot kernels & helpers that consume tiled activations
static inline HVX_Vector hvx_vec_mul_f16_f16_to_f32_lower32(HVX_Vector v1, HVX_Vector v2) {
@@ -454,7 +520,7 @@ static inline HVX_VectorPair accum_q8_0_32x2(
return Q6_W_vcombine_VV(v_sum1, v_sum0);
}
-// Q5_K: OR 0x10 into every lane of v whose flag j is set in the plane (see HTP_MM_WEIGHT_TILE_SIZE_Q5_K)
+// Q5_K: OR 0x10 into every lane of v whose flag j is set in the plane (see HTP_TENSOR_TILE_SIZE_Q5_K)
static inline HVX_Vector hvx_q5k_or_hibit(HVX_Vector v, HVX_Vector v_plane, int j) {
HVX_VectorPred q = Q6_Q_vand_VR(v_plane, 0x01010101u << j);
return Q6_V_vandor_VQR(v, q, 0x10101010);
@@ -526,7 +592,7 @@ static inline HVX_VectorPair accum_5bit_32x2(
return Q6_W_vcombine_VV(v_sum1, v_sum0);
}
-// Q6_K weights are stored unsigned (0..63), see HTP_MM_WEIGHT_TILE_SIZE_Q6_K. Unpack k-group g of a tile to signed bytes (q - 32)
+// Q6_K weights are stored unsigned (0..63), see HTP_TENSOR_TILE_SIZE_Q6_K. Unpack k-group g of a tile to signed bytes (q - 32)
static inline HVX_Vector unpack_q6_k_group(const HVX_Vector * restrict vptr, int g, HVX_Vector mask_0f, HVX_Vector mask_03, HVX_Vector i32) {
HVX_Vector v_lo = (g & 1) ? Q6_Vub_vlsr_VubR(vptr[g >> 1], 4) : Q6_V_vand_VV(vptr[g >> 1], mask_0f);
HVX_Vector v_hi = (g & 3) ? Q6_Vub_vlsr_VubR(vptr[4 + (g >> 2)], 2 * (g & 3)) : vptr[4 + (g >> 2)];
@@ -593,7 +659,7 @@ static inline HVX_Vector scale_q6_k_32x1(HVX_VectorPair v_sums, HVX_Vector v_sca
return hvx_vec_add_f32_f32(v_lo, v_hi);
}
-// Q3_K / Q2_K: low 2 bits of k-group g, see HTP_MM_WEIGHT_TILE_SIZE_Q3_K
+// Q3_K / Q2_K: low 2 bits of k-group g, see HTP_TENSOR_TILE_SIZE_Q3_K
static inline HVX_Vector unpack_q3_k_low2(const HVX_Vector * restrict vptr, int g, HVX_Vector mask_03) {
HVX_Vector v = vptr[g >> 2];
if ((g & 3) == 3) {
@@ -1482,6 +1548,59 @@ static void tiled_vec_dot_mxfp4_32x2(const uint32_t n, float * restrict s0, floa
}
}
+static void tiled_vec_dot_f16_32x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy, uint32_t valid_rows, const float * restrict sz) {
+ const HVX_Vector * restrict vptr = (const HVX_Vector *) vx;
+ const HVX_Vector * restrict v_act = (const HVX_Vector *) vy;
+ const uint32_t n_vecs = ((n + 31) / 32) * 16;
+
+ HVX_VectorPair acc = Q6_W_vzero();
+
+ #pragma unroll(4)
+ for (uint32_t i = 0; i < n_vecs; i++) {
+ acc = hvx_vec_mpyacc_f32_f16(acc, vptr[i], v_act[i]);
+ }
+
+ HVX_Vector v_sum = hvx_vec_add_f32_f32(Q6_V_lo_W(acc), Q6_V_hi_W(acc));
+
+ if (sz) {
+ hvx_vec_store_u(s, valid_rows * sizeof(float), hvx_vec_add_f32_f32(v_sum, hvx_vmemu(sz)));
+ } else {
+ hvx_vec_store_u(s, valid_rows * sizeof(float), v_sum);
+ }
+}
+
+static void tiled_vec_dot_f16_32x2(const uint32_t n, float * restrict s0, float * restrict s1, const void * restrict vx, const void * restrict vy0, const void * restrict vy1, uint32_t valid_rows, const float * restrict sz0, const float * restrict sz1) {
+ const HVX_Vector * restrict vptr = (const HVX_Vector *) vx;
+ const HVX_Vector * restrict v_act0 = (const HVX_Vector *) vy0;
+ const HVX_Vector * restrict v_act1 = (const HVX_Vector *) vy1;
+ const uint32_t n_vecs = ((n + 31) / 32) * 16;
+
+ HVX_VectorPair acc0 = Q6_W_vzero();
+ HVX_VectorPair acc1 = Q6_W_vzero();
+
+ #pragma unroll(4)
+ for (uint32_t i = 0; i < n_vecs; i++) {
+ HVX_Vector v_w = vptr[i];
+ acc0 = hvx_vec_mpyacc_f32_f16(acc0, v_w, v_act0[i]);
+ acc1 = hvx_vec_mpyacc_f32_f16(acc1, v_w, v_act1[i]);
+ }
+
+ HVX_Vector v_sum0 = hvx_vec_add_f32_f32(Q6_V_lo_W(acc0), Q6_V_hi_W(acc0));
+ HVX_Vector v_sum1 = hvx_vec_add_f32_f32(Q6_V_lo_W(acc1), Q6_V_hi_W(acc1));
+
+ if (sz0) {
+ hvx_vec_store_u(s0, valid_rows * sizeof(float), hvx_vec_add_f32_f32(v_sum0, hvx_vmemu(sz0)));
+ } else {
+ hvx_vec_store_u(s0, valid_rows * sizeof(float), v_sum0);
+ }
+ if (sz1) {
+ hvx_vec_store_u(s1, valid_rows * sizeof(float), hvx_vec_add_f32_f32(v_sum1, hvx_vmemu(sz1)));
+ } else {
+ hvx_vec_store_u(s1, valid_rows * sizeof(float), v_sum1);
+ }
+}
+
+
static inline void quantize_f32_q8_0_tiled_kernel(
const uint8_t * restrict src_data,
uint8_t * restrict dst_data,
@@ -1533,6 +1652,40 @@ static inline void quantize_f32_q8_1_s16_tiled_kernel(
}
}
+static inline void quantize_f32_f16_tiled_kernel(
+ const uint8_t * restrict src_data,
+ uint8_t * restrict dst_data,
+ uint8_t * restrict tmp_data,
+ uint32_t ne0,
+ uint32_t nrows,
+ size_t src_row_size,
+ size_t dst_row_size
+) {
+ (void) tmp_data;
+ for (uint32_t i = 0; i < nrows; ++i) {
+ quantize_row_f32_f16_tiled((const float *) src_data, dst_data, ne0);
+ dst_data += dst_row_size;
+ src_data += src_row_size;
+ }
+}
+
+static inline void quantize_f16_f16_tiled_kernel(
+ const uint8_t * restrict src_data,
+ uint8_t * restrict dst_data,
+ uint8_t * restrict tmp_data,
+ uint32_t ne0,
+ uint32_t nrows,
+ size_t src_row_size,
+ size_t dst_row_size
+) {
+ (void) tmp_data;
+ for (uint32_t i = 0; i < nrows; ++i) {
+ quantize_row_f16_f16_tiled((const __fp16 *) src_data, dst_data, ne0);
+ dst_data += dst_row_size;
+ src_data += src_row_size;
+ }
+}
+
static inline void quantize_f32_q8_0_tiled_block_kernel(
const float * restrict src,
uint8_t * restrict dst,
@@ -1622,3 +1775,63 @@ static inline void quantize_f32_q8_1_s16_tiled_block_kernel(
}
}
}
+
+static inline void quantize_f32_f16_tiled_block_kernel(
+ const float * restrict src,
+ uint8_t * restrict dst,
+ uint8_t * restrict tmp_data,
+ uint32_t ne0,
+ uint32_t ib_first,
+ uint32_t ib_last,
+ size_t src_row_size,
+ size_t dst_row_size,
+ uint32_t r,
+ uint32_t c
+) {
+ (void) tmp_data;
+ const uint32_t qk = QK_Q8_0_TILED;
+ const uint32_t nb = (ne0 + qk - 1) / qk;
+
+ for (uint32_t ib = ib_first; ib < ib_last; ++ib) {
+ const float * restrict src_ptr = (const float *) ((const uint8_t *) src + r * src_row_size + c * qk * sizeof(float));
+ uint8_t * restrict dst_ptr = dst + r * dst_row_size + c * 4 * 2048;
+
+ quantize_block_f32_f16_tiled(src_ptr, dst_ptr);
+
+ c++;
+ if (c == nb) {
+ c = 0;
+ r++;
+ }
+ }
+}
+
+static inline void quantize_f16_f16_tiled_block_kernel(
+ const __fp16 * restrict src,
+ uint8_t * restrict dst,
+ uint8_t * restrict tmp_data,
+ uint32_t ne0,
+ uint32_t ib_first,
+ uint32_t ib_last,
+ size_t src_row_size,
+ size_t dst_row_size,
+ uint32_t r,
+ uint32_t c
+) {
+ (void) tmp_data;
+ const uint32_t qk = QK_Q8_0_TILED;
+ const uint32_t nb = (ne0 + qk - 1) / qk;
+
+ for (uint32_t ib = ib_first; ib < ib_last; ++ib) {
+ const __fp16 * restrict src_ptr = (const __fp16 *) ((const uint8_t *) src + r * src_row_size + c * qk * sizeof(__fp16));
+ uint8_t * restrict dst_ptr = dst + r * dst_row_size + c * 4 * 2048;
+
+ quantize_block_f16_f16_tiled(src_ptr, dst_ptr);
+
+ c++;
+ if (c == nb) {
+ c = 0;
+ r++;
+ }
+ }
+}
diff --git a/ggml/src/ggml-hexagon/htp/hvx-quant.h b/ggml/src/ggml-hexagon/htp/hvx-quant.h
index 6b172cd63..b7837967a 100644
--- a/ggml/src/ggml-hexagon/htp/hvx-quant.h
+++ b/ggml/src/ggml-hexagon/htp/hvx-quant.h
@@ -5,6 +5,7 @@
#include <stdint.h>
#include <string.h>
+#include "htp-tensor.h"
#include "hvx-arith.h"
#include "hvx-base.h"
#include "hvx-reduce.h"
@@ -96,7 +97,7 @@ static inline void hvx_quantize_row_q8_0_f32(void * restrict dst_ptr, const floa
}
}
-static inline void hvx_dequantize_row_q8_0_f32(float * restrict dst_ptr, const void * restrict src_ptr, int n) {
+static inline void hvx_dequantize_row_flat_q8_0_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
const int nb = n / QK8_0;
const block_q8_0 * src = (const block_q8_0 *) src_ptr;
@@ -120,7 +121,7 @@ static inline void hvx_dequantize_row_q8_0_f32(float * restrict dst_ptr, const v
}
}
-static inline void hvx_dequantize_row_q8_0_f16(__fp16 * restrict dst_ptr, const void * restrict src_ptr, int n) {
+static inline void hvx_dequantize_row_flat_q8_0_f16(__fp16 * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
const int nb = n / QK8_0;
const block_q8_0 * src = (const block_q8_0 *) src_ptr;
@@ -146,20 +147,460 @@ static inline void hvx_dequantize_row_q8_0_f16(__fp16 * restrict dst_ptr, const
}
}
-static inline void hvx_dequantize_row_f16_f32(float * restrict dst_ptr, const void * restrict src_ptr, int n) {
- const int nb = n / 32;
+static inline void hvx_dequantize_row_flat_f16_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
+ const uint32_t nvec = n / VLEN_FP32;
+ const uint32_t nloe = n % VLEN_FP32;
const _Float16 * src = (const _Float16 *) src_ptr;
- for (int i = 0; i < nb; i++) {
- HVX_Vector v_f16 = *(const HVX_UVector *) (src + i * 32);
+ for (uint32_t i = 0; i < nvec; i++) {
+ HVX_Vector v_f16 = *(const HVX_UVector *) (src + i * VLEN_FP32);
HVX_VectorPair vp_f32 = hvx_vec_f16_to_f32(v_f16);
HVX_Vector res = Q6_V_lo_W(vp_f32);
- float * block_dst = dst_ptr + i * 32;
- hvx_vmem(block_dst) = res;
+ hvx_vmem(dst_ptr + i * VLEN_FP32) = res;
+ }
+
+ if (nloe) {
+ HVX_Vector v_f16 = *(const HVX_UVector *) (src + nvec * VLEN_FP32);
+ HVX_VectorPair vp_f32 = hvx_vec_f16_to_f32(v_f16);
+ HVX_Vector res = Q6_V_lo_W(vp_f32);
+
+ hvx_vec_store_a(dst_ptr + nvec * VLEN_FP32, nloe * sizeof(float), res);
+ }
+}
+
+static inline void hvx_dequantize_row_flat_bf16_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
+ const uint32_t nvec = n / VLEN_FP32;
+ const uint32_t nloe = n % VLEN_FP32;
+ const uint16_t * src = (const uint16_t *) src_ptr;
+
+ for (uint32_t i = 0; i < nvec; i++) {
+ HVX_Vector v_bf16 = *(const HVX_UVector *) (src + i * VLEN_FP32);
+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);
+ HVX_Vector res = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);
+
+ hvx_vmem(dst_ptr + i * VLEN_FP32) = res;
+ }
+
+ if (nloe) {
+ HVX_Vector v_bf16 = *(const HVX_UVector *) (src + nvec * VLEN_FP32);
+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);
+ HVX_Vector res = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);
+
+ hvx_vec_store_a(dst_ptr + nvec * VLEN_FP32, nloe * sizeof(float), res);
+ }
+}
+
+static inline void hvx_dequantize_row_flat_q4_0_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
+ const int nb = n / QK4_0;
+ const block_q4_0 * src = (const block_q4_0 *) src_ptr;
+
+ const HVX_VectorPred q16 = Q6_Q_vsetq_R(16);
+ const HVX_Vector v_eight = Q6_Vb_vsplat_R(8);
+ const HVX_Vector v_zero = Q6_V_vzero();
+
+ for (int i = 0; i < nb; i++) {
+ HVX_Vector vd_f16 = Q6_Vh_vsplat_R(*(const int16_t *) &src[i].d);
+ HVX_VectorPair vp_f32 = hvx_vec_f16_to_f32(vd_f16);
+ HVX_Vector vd = Q6_V_lo_W(vp_f32);
+
+ HVX_Vector vq = *(const HVX_UVector *) src[i].qs;
+ HVX_Vector v_lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));
+ HVX_Vector v_hi = Q6_Vub_vlsr_VubR(vq, 4);
+
+ HVX_Vector v_lo_s8 = Q6_Vb_vsub_VbVb(v_lo, v_eight);
+ HVX_Vector v_hi_s8 = Q6_Vb_vsub_VbVb(v_hi, v_eight);
+
+ HVX_Vector v32 = Q6_V_vmux_QVV(q16, v_lo_s8, Q6_V_vlalign_VVR(v_hi_s8, v_zero, 16));
+
+ HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(v32);
+ HVX_Vector v_i16 = Q6_V_lo_W(p16);
+ HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(v_i16);
+ HVX_Vector v_i32 = Q6_V_lo_W(p32);
+
+ HVX_Vector v_f32 = Q6_Vsf_equals_Vw(v_i32);
+ HVX_Vector res = hvx_vec_mul_f32_f32(v_f32, vd);
+
+ hvx_vmem(dst_ptr + i * QK4_0) = res;
+ }
+}
+
+static inline void hvx_dequantize_row_flat_q4_k_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
+ const int nb = n / QK_K;
+ const block_q4_K * src = (const block_q4_K *) src_ptr;
+
+ for (int i = 0; i < nb; i++) {
+ const float d = GGML_FP16_TO_FP32(src[i].d);
+ const float min = GGML_FP16_TO_FP32(src[i].dmin);
+ const uint8_t * sc = src[i].scales;
+
+ float * y = dst_ptr + i * QK_K;
+
+ for (int step = 0; step < 4; ++step) {
+ uint8_t sc1, m1_u, sc2, m2_u;
+ if (step < 2) {
+ sc1 = sc[2 * step + 0] & 63;
+ m1_u = sc[2 * step + 4] & 63;
+ sc2 = sc[2 * step + 1] & 63;
+ m2_u = sc[2 * step + 5] & 63;
+ } else {
+ const int s = step - 2;
+ sc1 = (sc[8 + 2 * s + 0] & 0xF) | ((sc[2 * s + 0] >> 6) << 4);
+ m1_u = (sc[8 + 2 * s + 0] >> 4) | ((sc[4 + 2 * s + 0] >> 6) << 4);
+ sc2 = (sc[8 + 2 * s + 1] & 0xF) | ((sc[2 * s + 1] >> 6) << 4);
+ m2_u = (sc[8 + 2 * s + 1] >> 4) | ((sc[4 + 2 * s + 1] >> 6) << 4);
+ }
+
+ HVX_Vector vq = *(const HVX_UVector *) (src[i].qs + step * 32);
+ HVX_Vector v_lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));
+ HVX_Vector v_hi = Q6_Vub_vlsr_VubR(vq, 4);
+
+ HVX_VectorPair p16_lo = Q6_Wuh_vunpack_Vub(v_lo);
+ HVX_VectorPair p32_lo = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_lo));
+ HVX_Vector v_i32_lo = Q6_V_lo_W(p32_lo);
+ HVX_Vector v_f32_lo = Q6_Vsf_equals_Vw(v_i32_lo);
+
+ HVX_Vector vd1 = hvx_vec_splat_f32(d * sc1);
+ HVX_Vector vm1 = hvx_vec_splat_f32(min * m1_u);
+ HVX_Vector v_out1 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_lo, vd1), vm1);
+ hvx_vmem(y + 0) = v_out1;
+
+ HVX_VectorPair p16_hi = Q6_Wuh_vunpack_Vub(v_hi);
+ HVX_VectorPair p32_hi = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_hi));
+ HVX_Vector v_i32_hi = Q6_V_lo_W(p32_hi);
+ HVX_Vector v_f32_hi = Q6_Vsf_equals_Vw(v_i32_hi);
+
+ HVX_Vector vd2 = hvx_vec_splat_f32(d * sc2);
+ HVX_Vector vm2 = hvx_vec_splat_f32(min * m2_u);
+ HVX_Vector v_out2 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_hi, vd2), vm2);
+ hvx_vmem(y + 32) = v_out2;
+
+ y += 64;
+ }
}
}
+static inline void hvx_dequantize_row_flat_q5_k_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
+ const int nb = n / QK_K;
+ const block_q5_K * src = (const block_q5_K *) src_ptr;
+ for (int i = 0; i < nb; i++) {
+ const float d = GGML_FP16_TO_FP32(src[i].d);
+ const float min = GGML_FP16_TO_FP32(src[i].dmin);
+ const uint8_t * sc = src[i].scales;
+
+ HVX_Vector v_qh = *(const HVX_UVector *) src[i].qh;
+ float * y = dst_ptr + i * QK_K;
+
+ for (int step = 0; step < 4; ++step) {
+ uint8_t sc1, m1_u, sc2, m2_u;
+ if (step < 2) {
+ sc1 = sc[2 * step + 0] & 63;
+ m1_u = sc[2 * step + 4] & 63;
+ sc2 = sc[2 * step + 1] & 63;
+ m2_u = sc[2 * step + 5] & 63;
+ } else {
+ const int s = step - 2;
+ sc1 = (sc[8 + 2 * s + 0] & 0xF) | ((sc[2 * s + 0] >> 6) << 4);
+ m1_u = (sc[8 + 2 * s + 0] >> 4) | ((sc[4 + 2 * s + 0] >> 6) << 4);
+ sc2 = (sc[8 + 2 * s + 1] & 0xF) | ((sc[2 * s + 1] >> 6) << 4);
+ m2_u = (sc[8 + 2 * s + 1] >> 4) | ((sc[4 + 2 * s + 1] >> 6) << 4);
+ }
+
+ HVX_Vector v_ql = *(const HVX_UVector *) (src[i].qs + step * 32);
+ HVX_Vector v_lo = Q6_V_vand_VV(v_ql, Q6_Vb_vsplat_R(0x0F));
+ HVX_Vector v_hi = Q6_Vub_vlsr_VubR(v_ql, 4);
+
+ HVX_Vector v_h1 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 2 * step), Q6_Vb_vsplat_R(1)), 4);
+ HVX_Vector v_val1 = Q6_V_vor_VV(v_lo, v_h1);
+
+ HVX_VectorPair p16_1 = Q6_Wuh_vunpack_Vub(v_val1);
+ HVX_VectorPair p32_1 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_1));
+ HVX_Vector v_i32_1 = Q6_V_lo_W(p32_1);
+ HVX_Vector v_f32_1 = Q6_Vsf_equals_Vw(v_i32_1);
+
+ HVX_Vector vd1 = hvx_vec_splat_f32(d * sc1);
+ HVX_Vector vm1 = hvx_vec_splat_f32(min * m1_u);
+ HVX_Vector v_out1 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_1, vd1), vm1);
+ hvx_vmem(y + 0) = v_out1;
+
+ HVX_Vector v_h2 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 2 * step + 1), Q6_Vb_vsplat_R(1)), 4);
+ HVX_Vector v_val2 = Q6_V_vor_VV(v_hi, v_h2);
+
+ HVX_VectorPair p16_2 = Q6_Wuh_vunpack_Vub(v_val2);
+ HVX_VectorPair p32_2 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_2));
+ HVX_Vector v_i32_2 = Q6_V_lo_W(p32_2);
+ HVX_Vector v_f32_2 = Q6_Vsf_equals_Vw(v_i32_2);
+
+ HVX_Vector vd2 = hvx_vec_splat_f32(d * sc2);
+ HVX_Vector vm2 = hvx_vec_splat_f32(min * m2_u);
+ HVX_Vector v_out2 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_2, vd2), vm2);
+ hvx_vmem(y + 32) = v_out2;
+
+ y += 64;
+ }
+ }
+}
+
+static inline void hvx_dequantize_row_flat_q6_k_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {
+ const int nb = n / QK_K;
+ const block_q6_K * src = (const block_q6_K *) src_ptr;
+
+ const HVX_VectorPred q64 = Q6_Q_vsetq_R(64);
+ const HVX_Vector v32_b = Q6_Vb_vsplat_R(32);
+ const HVX_Vector mask_0f = Q6_Vb_vsplat_R(0x0F);
+ const HVX_Vector mask_03 = Q6_Vb_vsplat_R(0x03);
+
+ for (int i = 0; i < nb; i++) {
+ const float d = GGML_FP16_TO_FP32(src[i].d);
+
+ const uint8_t * restrict ql = src[i].ql;
+ const uint8_t * restrict qh = src[i].qh;
+ const int8_t * restrict sc = src[i].scales;
+
+ float * y = dst_ptr + i * QK_K;
+
+ for (int half = 0; half < 2; ++half) {
+ float s[8];
+ for (int j = 0; j < 8; ++j) {
+ s[j] = d * sc[j];
+ }
+
+ HVX_Vector vscale1 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[0]), hvx_vec_splat_f32(s[1]));
+ HVX_Vector vscale2 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[2]), hvx_vec_splat_f32(s[3]));
+ HVX_Vector vscale3 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[4]), hvx_vec_splat_f32(s[5]));
+ HVX_Vector vscale4 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[6]), hvx_vec_splat_f32(s[7]));
+
+ HVX_Vector v_ql0 = *(const HVX_UVector *) (ql);
+ HVX_Vector v_ql1 = *(const HVX_UVector *) (ql + 32);
+ HVX_Vector v_qh = *(const HVX_UVector *) (qh);
+
+ HVX_Vector ql0_lo = Q6_V_vand_VV(v_ql0, mask_0f);
+ HVX_Vector qh_0 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(v_qh, mask_03), 4);
+ HVX_Vector v_q1 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql0_lo, qh_0), v32_b);
+
+ HVX_Vector ql1_lo = Q6_V_vand_VV(v_ql1, mask_0f);
+ HVX_Vector qh_1 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 2), mask_03), 4);
+ HVX_Vector v_q2 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql1_lo, qh_1), v32_b);
+
+ HVX_Vector ql0_hi = Q6_Vub_vlsr_VubR(v_ql0, 4);
+ HVX_Vector qh_2 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 4), mask_03), 4);
+ HVX_Vector v_q3 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql0_hi, qh_2), v32_b);
+
+ HVX_Vector ql1_hi = Q6_Vub_vlsr_VubR(v_ql1, 4);
+ HVX_Vector qh_3 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 6), mask_03), 4);
+ HVX_Vector v_q4 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql1_hi, qh_3), v32_b);
+
+ HVX_VectorPair p16_1 = Q6_Wh_vunpack_Vb(v_q1);
+ HVX_VectorPair p32_1 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_1));
+ HVX_Vector v_f32_1 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_1));
+ hvx_vmem(y + 0) = hvx_vec_mul_f32_f32(v_f32_1, vscale1);
+
+ HVX_VectorPair p16_2 = Q6_Wh_vunpack_Vb(v_q2);
+ HVX_VectorPair p32_2 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_2));
+ HVX_Vector v_f32_2 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_2));
+ hvx_vmem(y + 32) = hvx_vec_mul_f32_f32(v_f32_2, vscale2);
+
+ HVX_VectorPair p16_3 = Q6_Wh_vunpack_Vb(v_q3);
+ HVX_VectorPair p32_3 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_3));
+ HVX_Vector v_f32_3 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_3));
+ hvx_vmem(y + 64) = hvx_vec_mul_f32_f32(v_f32_3, vscale3);
+
+ HVX_VectorPair p16_4 = Q6_Wh_vunpack_Vb(v_q4);
+ HVX_VectorPair p32_4 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_4));
+ HVX_Vector v_f32_4 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_4));
+ hvx_vmem(y + 96) = hvx_vec_mul_f32_f32(v_f32_4, vscale4);
+
+ y += 128;
+ ql += 64;
+ qh += 32;
+ sc += 8;
+ }
+ }
+}
+
+// bytes row, row + 32, row + 64 and row + 96 of v, to bytes 0..3
+static inline HVX_Vector hvx_dequantize_gather4(HVX_Vector v, uint32_t row) {
+ v = Q6_V_vror_VR(v, row);
+ HVX_Vector packed = Q6_V_vmux_QVV(Q6_Q_vsetq_R(1), v, Q6_V_vror_VR(v, 31));
+ packed = Q6_V_vmux_QVV(Q6_Q_vsetq_R(2), packed, Q6_V_vror_VR(v, 62));
+ return Q6_V_vmux_QVV(Q6_Q_vsetq_R(3), packed, Q6_V_vror_VR(v, 93));
+}
+
+static inline void hvx_dequantize_row_tiled_q8_0_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {
+ const HVX_VectorPred first2 = Q6_Q_vsetq_R(2);
+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);
+
+ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {
+ const uint8_t * tile = src_spad + k_tile * tile_stride;
+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;
+
+ HVX_Vector vq = Q6_V_vzero();
+ for (int group = 7; group >= 0; --group) {
+ const HVX_Vector v = Q6_V_vror_VR(hvx_vmem(tile + group * VLEN), 2 * row);
+ const HVX_Vector packed = Q6_V_vmux_QVV(first2, v, Q6_V_vror_VR(v, 62));
+ vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));
+ }
+
+ const HVX_Vector scales = hvx_vmem(tile + 1024);
+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));
+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));
+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);
+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));
+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);
+ *(HVX_Vector *) dst = values;
+ }
+}
+
+static inline void hvx_dequantize_row_tiled_q4_0_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {
+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);
+
+ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {
+ const uint8_t * tile = src_spad + k_tile * tile_stride;
+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;
+
+ HVX_Vector vq = Q6_V_vzero();
+ for (int group = 3; group >= 0; --group) {
+ vq = Q6_V_vmux_QVV(first4, hvx_dequantize_gather4(hvx_vmem(tile + group * VLEN), row), Q6_V_vror_VR(vq, VLEN - 4));
+ }
+
+ const HVX_Vector lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));
+ const HVX_Vector hi = Q6_Vub_vlsr_VubR(vq, 4);
+ vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));
+ vq = Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(8));
+
+ const HVX_Vector scales = hvx_vmem(tile + 512);
+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));
+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));
+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);
+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));
+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);
+ *(HVX_Vector *) dst = values;
+ }
+}
+
+static inline void hvx_dequantize_row_tiled_q4_k_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {
+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);
+
+ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {
+ const uint8_t * tile = src_spad + k_tile * tile_stride;
+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;
+
+ HVX_Vector vq = Q6_V_vzero();
+ for (int group = 3; group >= 0; --group) {
+ vq = Q6_V_vmux_QVV(first4, hvx_dequantize_gather4(hvx_vmem(tile + group * VLEN), row), Q6_V_vror_VR(vq, VLEN - 4));
+ }
+
+ const HVX_Vector lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));
+ const HVX_Vector hi = Q6_Vub_vlsr_VubR(vq, 4);
+ vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));
+
+ const HVX_Vector scales = hvx_vmem(tile + 512);
+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row));
+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));
+ const HVX_Vector offset_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row + 2));
+ const HVX_Vector offset = Q6_V_lo_W(hvx_vec_f16_to_f32(offset_hf));
+
+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);
+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));
+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);
+ *(HVX_Vector *) dst = hvx_vec_add_f32_f32(values, offset);
+ }
+}
+
+static inline void hvx_dequantize_row_tiled_q5_k_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {
+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);
+ const HVX_Vector one = Q6_Vb_vsplat_R(1);
+
+ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {
+ const uint8_t * tile = src_spad + k_tile * tile_stride;
+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;
+
+ const HVX_Vector p5 = hvx_dequantize_gather4(hvx_vmem(tile + 640), row);
+ HVX_Vector vq = Q6_V_vzero();
+ HVX_Vector vh = Q6_V_vzero();
+ for (int group = 3; group >= 0; --group) {
+ vq = Q6_V_vmux_QVV(first4, hvx_dequantize_gather4(hvx_vmem(tile + group * VLEN), row), Q6_V_vror_VR(vq, VLEN - 4));
+ vh = Q6_V_vmux_QVV(first4, Q6_Vub_vlsr_VubR(p5, 2 * group), Q6_V_vror_VR(vh, VLEN - 4));
+ }
+
+ const HVX_Vector lo = Q6_V_vor_VV(Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F)), Q6_Vw_vasl_VwR(Q6_V_vand_VV(vh, one), 4));
+ const HVX_Vector hi = Q6_V_vor_VV(Q6_Vub_vlsr_VubR(vq, 4), Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(vh, 1), one), 4));
+ vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));
+
+ const HVX_Vector scales = hvx_vmem(tile + 512);
+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row));
+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));
+ const HVX_Vector offset_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row + 2));
+ const HVX_Vector offset = Q6_V_lo_W(hvx_vec_f16_to_f32(offset_hf));
+
+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);
+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));
+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);
+ *(HVX_Vector *) dst = hvx_vec_add_f32_f32(values, offset);
+ }
+}
+
+static inline void hvx_dequantize_row_tiled_q6_k_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {
+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);
+ const HVX_VectorPred first16 = Q6_Q_vsetq_R(16 * sizeof(float));
+ const HVX_Vector mask_0f = Q6_Vb_vsplat_R(0x0F);
+ const HVX_Vector mask_03 = Q6_Vb_vsplat_R(0x03);
+
+ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {
+ const uint8_t * tile = src_spad + k_tile * tile_stride;
+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;
+
+ HVX_Vector vq = Q6_V_vzero();
+ for (int group = 7; group >= 0; --group) {
+ const HVX_Vector lo_plane = Q6_V_vror_VR(hvx_vmem(tile + (group >> 1) * VLEN), 4 * row);
+ const HVX_Vector hi_plane = Q6_V_vror_VR(hvx_vmem(tile + 512 + (group >> 2) * VLEN), 4 * row);
+ const HVX_Vector lo = (group & 1) ? Q6_Vub_vlsr_VubR(lo_plane, 4) : Q6_V_vand_VV(lo_plane, mask_0f);
+ const HVX_Vector hi = Q6_Vub_vlsr_VubR(hi_plane, 2 * (group & 3));
+ const HVX_Vector packed = Q6_V_vor_VV(lo, Q6_Vw_vasl_VwR(Q6_V_vand_VV(hi, mask_03), 4));
+ vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));
+ }
+
+ const HVX_Vector scales = hvx_vmem(tile + 768);
+ const HVX_Vector scale_lo_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));
+ const HVX_Vector scale_hi_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 64 + 2 * row));
+ const HVX_Vector scale_lo = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_lo_hf));
+ const HVX_Vector scale_hi = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hi_hf));
+ const HVX_Vector scale = Q6_V_vmux_QVV(first16, scale_lo, scale_hi);
+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(32)));
+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));
+ *(HVX_Vector *) dst = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);
+ }
+}
+
+static inline HVX_Vector hvx_dequantize_tiled_f16_gather4(const uint8_t * p, uint32_t r_byte) {
+ HVX_Vector w0 = Q6_V_vror_VR(hvx_vmem(p + 0 * VLEN), r_byte);
+ HVX_Vector w1 = Q6_V_vror_VR(hvx_vmem(p + 1 * VLEN), r_byte);
+ HVX_Vector w01 = Q6_V_lo_W(Q6_W_vshuff_VVR(w1, w0, 4));
+ HVX_Vector w2 = Q6_V_vror_VR(hvx_vmem(p + 2 * VLEN), r_byte);
+ HVX_Vector w3 = Q6_V_vror_VR(hvx_vmem(p + 3 * VLEN), r_byte);
+ HVX_Vector w23 = Q6_V_lo_W(Q6_W_vshuff_VVR(w3, w2, 4));
+ return Q6_V_lo_W(Q6_W_vshuff_VVR(w23, w01, 8));
+}
+
+static inline void hvx_dequantize_row_tiled_f16_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {
+ const uint32_t r_byte = 4 * row;
+
+ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {
+ const uint8_t * tile = src_spad + k_tile * tile_stride;
+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;
+
+ HVX_Vector g0 = hvx_dequantize_tiled_f16_gather4(tile + 0 * VLEN, r_byte);
+ HVX_Vector g1 = hvx_dequantize_tiled_f16_gather4(tile + 4 * VLEN, r_byte);
+ HVX_Vector g01 = Q6_V_lo_W(Q6_W_vshuff_VVR(g1, g0, 16));
+
+ HVX_Vector g2 = hvx_dequantize_tiled_f16_gather4(tile + 8 * VLEN, r_byte);
+ HVX_Vector g3 = hvx_dequantize_tiled_f16_gather4(tile + 12 * VLEN, r_byte);
+ HVX_Vector g23 = Q6_V_lo_W(Q6_W_vshuff_VVR(g3, g2, 16));
+
+ HVX_Vector g = Q6_V_lo_W(Q6_W_vshuff_VVR(g23, g01, 32));
+ *(HVX_Vector *) dst = Q6_V_lo_W(hvx_vec_f16_to_f32(g));
+ }
+}
#endif // HVX_QUANT_H
diff --git a/ggml/src/ggml-hexagon/htp/main.c b/ggml/src/ggml-hexagon/htp/main.c
index df7c4a5bd..2374add3f 100644
--- a/ggml/src/ggml-hexagon/htp/main.c
+++ b/ggml/src/ggml-hexagon/htp/main.c
@@ -28,8 +28,8 @@
#include "ggml-common.h"
#include "hex-bitmap.h"
#include "htp-ctx.h"
+#include "htp-dcache.h"
#include "htp-ops.h"
-#include "htp-tensor.h"
#include "htp_iface.h"
#include "work-queue.h"
#include "hex-profile.h"
@@ -835,7 +835,7 @@ static int op_fence(struct htp_ops_context * octx) {
atomic_uint * sync_fence = (atomic_uint *) (uintptr_t) sync->data;
if (mode == 1) {
- htp_flush_dirty_ranges(ctx);
+ htp_dcache_flush_dirty_ranges(ctx);
htp_mdev_group_barrier(octx);
@@ -1211,7 +1211,7 @@ static int proc_op_req(struct htp_ops_context * octx, struct htp_buf_desc * bufs
src->ne[0], src->ne[1], src->ne[2], src->ne[3]);
}
- htp_tensor_flush_all(octx->ctx, octx->src, HTP_OP_MAX_INPUTS);
+ htp_dcache_flush_dirty_tensors(octx->ctx, octx->src, HTP_OP_MAX_INPUTS);
// Prep output tensors
for (uint32_t i = 0; i < HTP_OP_MAX_OUTPUTS; i++) {
@@ -1227,7 +1227,7 @@ static int proc_op_req(struct htp_ops_context * octx, struct htp_buf_desc * bufs
dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);
}
- htp_tensor_dirty_all(octx->ctx, octx->dsts, HTP_OP_MAX_OUTPUTS);
+ htp_dcache_mark_dirty_tensors(octx->ctx, octx->dsts, HTP_OP_MAX_OUTPUTS);
htp_mdev_group_barrier(octx);
diff --git a/ggml/src/ggml-hexagon/htp/matmul-ops.c b/ggml/src/ggml-hexagon/htp/matmul-ops.c
index 28aed4680..3837c290d 100644
--- a/ggml/src/ggml-hexagon/htp/matmul-ops.c
+++ b/ggml/src/ggml-hexagon/htp/matmul-ops.c
@@ -38,8 +38,12 @@ typedef struct {
int act_stride;
uint32_t act_elem_size; // 4=F32 src1, 2=F16 src1
int weight_stride;
+ int weight_type;
int dst_stride;
+ int dst_cols;
uint32_t bias_stride;
+ int tile_size;
+ int aligned_tile_size;
int ne02;
int ne03;
int ne12;
@@ -56,7 +60,7 @@ typedef struct {
int r3;
struct fastdiv_values div_r2;
struct fastdiv_values div_r3;
-} hmx_mm_f16_f32_batched_params_t;
+} hmx_mm_batched_params_t;
static bool htp_matmul_has_extended_weight(const struct htp_ops_context * octx, uint32_t n_weights) {
for (uint32_t i = 0; i < n_weights; ++i) {
@@ -89,6 +93,11 @@ struct htp_mm_context {
const void * restrict vy, uint32_t valid_rows,
const float * restrict sz);
+ void (*vec_dot_32x2)(const uint32_t n, float * restrict s0, float * restrict s1,
+ const void * restrict vx,
+ const void * restrict vy0, const void * restrict vy1, uint32_t valid_rows,
+ const float * restrict sz0, const float * restrict sz1);
+
// Precomputed values
uint32_t src0_nrows_per_thread;
uint32_t src0_row_start;
@@ -238,314 +247,301 @@ static const uint8_t __attribute__((aligned(VLEN))) kvalues_mxfp4_lut[] = {
#include "hvx-mm-kernels-float.h"
#include "hmx-mm-kernels-tiled.h"
-// Specialized repacked matmul macros
-#define MATMUL_2D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X2, DOT_2X1) \
-static void hvx_mm_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \
- htp_matmul_preamble; \
- \
- const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \
- const uint32_t act_nrows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13); \
- const uint32_t cur_m_start = mmctx->cur_m_start; \
- \
- const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \
- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \
- \
- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \
- \
- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \
- const uint32_t n_prefetch = kparams->n_prefetch; \
- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \
- \
- const size_t dst_row_size = nb1; \
- const size_t act_row_size = nb11; \
- const size_t act_stride = mmctx->vtcm_act_stride; \
- const size_t src2_stride = src2 ? ((src2->ne[1] == 1) ? 0 : src2->nb[1]) : 0; \
- \
- uint8_t * restrict vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith; \
- uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \
- uint8_t * restrict act_data = mmctx->vtcm_act; \
- \
- const dma_addr_t src0_row = src0->data; \
- \
- const uint32_t tile_size = TILE_SIZE; \
- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \
- \
- uint32_t n_k_tiles_w = ne00 / 32; \
- uint32_t n_k_tiles_a = ne10 / 32; \
- uint32_t tile_row_stride = n_k_tiles_w * tile_size; \
- uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \
- \
- uint32_t ct_start = src0_start_row / 32; \
- uint32_t ct_end = (src0_end_row + 31) / 32; \
- \
- uint32_t push_ct = ct_start; \
- if (src0_start_row < src0_end_row) { \
- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \
- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned, \
- src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- } \
- } \
- \
- if (src0_start_row >= src0_end_row) { \
- return; \
- } \
- \
- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \
- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \
- \
- int valid_rows = (int)ne0 - (int)(ct * 32); \
- valid_rows = MIN(32, MAX(0, valid_rows)); \
- \
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- uint32_t ir1 = 0; \
- for (; ir1 + 1 < act_nrows; ir1 += 2) { \
- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride); \
- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride); \
- float * restrict dst_row0 = (float *) (dst->data + ((cur_m_start + ir1+0) * dst_row_size)); \
- float * restrict dst_row1 = (float *) (dst->data + ((cur_m_start + ir1+1) * dst_row_size)); \
- \
- float * dst_ptr0 = &dst_row0[ct * 32]; \
- float * dst_ptr1 = &dst_row1[ct * 32]; \
- \
- const float * src2_ptr0 = NULL; \
- const float * src2_ptr1 = NULL; \
- if (src2) { \
- const float * restrict src2_row0 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+0) * src2_stride)); \
- const float * restrict src2_row1 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+1) * src2_stride)); \
- src2_ptr0 = &src2_row0[ct * 32]; \
- src2_ptr1 = &src2_row1[ct * 32]; \
- } \
- DOT_2X2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, src2_ptr0, src2_ptr1); \
- } \
- \
- for (; ir1 < act_nrows; ++ir1) { \
- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride); \
- float * restrict dst_row = (float *) (dst->data + ((cur_m_start + ir1) * dst_row_size)); \
- float * dst_ptr = &dst_row[ct * 32]; \
- \
- const float * src2_ptr = NULL; \
- if (src2) { \
- const float * restrict src2_row = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1) * src2_stride)); \
- src2_ptr = &src2_row[ct * 32]; \
- } \
- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, src2_ptr); \
- } \
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- \
- if (push_ct < ct_end) { \
- dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride), \
- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- push_ct++; \
- } \
- } \
-}
+static void hvx_mm_2d_tiled(unsigned int nth, unsigned int ith, void * data) {
+ htp_matmul_preamble;
+
+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;
+ const uint32_t act_nrows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13);
+ const uint32_t cur_m_start = mmctx->cur_m_start;
+
+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;
+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);
+
+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];
+
+ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
+ const uint32_t n_prefetch = kparams->n_prefetch;
+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);
+
+ const size_t dst_row_size = nb1;
+ const size_t act_row_size = nb11;
+ const size_t act_stride = mmctx->vtcm_act_stride;
+ const size_t src2_stride = src2 ? ((src2->ne[1] == 1) ? 0 : src2->nb[1]) : 0;
+
+ uint8_t * restrict vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith;
+ uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;
+ uint8_t * restrict act_data = mmctx->vtcm_act;
+
+ const dma_addr_t src0_row = src0->data;
+
+ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);
+
+ uint32_t n_k_tiles_w = ne00 / 32;
+ uint32_t n_k_tiles_a = ne10 / 32;
+ uint32_t tile_row_stride = n_k_tiles_w * tile_size;
+ uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;
+
+ uint32_t ct_start = src0_start_row / 32;
+ uint32_t ct_end = (src0_end_row + 31) / 32;
+
+ uint32_t push_ct = ct_start;
+ if (src0_start_row < src0_end_row) {
+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {
+ dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned,
+ src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ }
+ }
+
+ if (src0_start_row >= src0_end_row) {
+ return;
+ }
+
+ for (uint32_t ct = ct_start; ct < ct_end; ct++) {
+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
+
+ int valid_rows = (int)ne0 - (int)(ct * 32);
+ valid_rows = MIN(32, MAX(0, valid_rows));
+
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+ uint32_t ir1 = 0;
+ for (; ir1 + 1 < act_nrows; ir1 += 2) {
+ const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride);
+ const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride);
+ float * restrict dst_row0 = (float *) (dst->data + ((cur_m_start + ir1+0) * dst_row_size));
+ float * restrict dst_row1 = (float *) (dst->data + ((cur_m_start + ir1+1) * dst_row_size));
+
+ float * dst_ptr0 = &dst_row0[ct * 32];
+ float * dst_ptr1 = &dst_row1[ct * 32];
-#define MATVEC_2D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X1) \
-static void hvx_mv_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \
- htp_matmul_preamble; \
- \
- const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \
- \
- const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \
- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \
- \
- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \
- \
- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \
- const uint32_t n_prefetch = kparams->n_prefetch; \
- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \
- \
- const size_t dst_row_size = nb1; \
- const size_t act_row_size = nb11; \
- const size_t act_stride = mmctx->vtcm_act_stride; \
- \
- uint8_t * vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith; \
- uint8_t * vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \
- uint8_t * act_data = mmctx->vtcm_act; \
- \
- float * tmp = (float *) vtcm_dst_ptr; \
- \
- const dma_addr_t src0_row = src0->data; \
- \
- const uint8_t * restrict act_col = (const uint8_t *) act_data; \
- float * restrict dst_col = (float *) dst->data; \
- \
- const uint32_t tile_size = TILE_SIZE; \
- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \
- \
- uint32_t n_k_tiles_w = ne00 / 32; \
- uint32_t n_k_tiles_a = ne10 / 32; \
- uint32_t tile_row_stride = n_k_tiles_w * tile_size; \
- uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \
- \
- uint32_t ct_start = src0_start_row / 32; \
- uint32_t ct_end = (src0_end_row + 31) / 32; \
- \
- uint32_t push_ct = ct_start; \
- if (src0_start_row < src0_end_row) { \
- if (src2) { \
- float * vtcm_bias_ptr = (float *) mmctx->vtcm_bias + src0_start_row; \
- const dma_addr_t src2_addr = src2->data + src0_start_row * sizeof(float); \
- int slice_size = (int)MIN(src0_end_row, ne0) - (int)src0_start_row; \
- if (slice_size > 0) { \
- dma_queue_push(dma_q, dma_make_data(vtcm_bias_ptr, src2_addr), \
- slice_size * sizeof(float), slice_size * sizeof(float), slice_size * sizeof(float), 1); \
- dma_queue_pop_nowait(dma_q); \
- } \
- } \
- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \
- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned, \
- src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- } \
- } \
- \
- if (src0_start_row >= src0_end_row) { \
- return; \
- } \
- \
- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \
- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \
- \
- float * dst_ptr = &tmp[ct * 32 - src0_start_row]; \
- int valid_rows = (int)ne0 - (int)(ct * 32); \
- valid_rows = MIN(32, MAX(0, valid_rows)); \
- \
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL); \
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- \
- if (push_ct < ct_end) { \
- dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride), \
- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- push_ct++; \
- } \
- } \
- \
- int copy_cnt = (int)MIN(src0_end_row, ne0) - (int)src0_start_row; \
- if (copy_cnt > 0) { \
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct_end); \
- if (src2) { \
- hvx_add_f32_uuu((uint8_t *) &dst_col[src0_start_row], \
- (const uint8_t *) tmp, \
- (const uint8_t *) ((const float *) mmctx->vtcm_bias + src0_start_row), \
- copy_cnt); \
- } else { \
- hvx_copy_f32_ua((uint8_t *) &dst_col[src0_start_row], (uint8_t *) tmp, copy_cnt); \
- } \
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct_end); \
- } \
+ const float * src2_ptr0 = NULL;
+ const float * src2_ptr1 = NULL;
+ if (src2) {
+ const float * restrict src2_row0 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+0) * src2_stride));
+ const float * restrict src2_row1 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+1) * src2_stride));
+ src2_ptr0 = &src2_row0[ct * 32];
+ src2_ptr1 = &src2_row1[ct * 32];
+ }
+ mmctx->vec_dot_32x2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, src2_ptr0, src2_ptr1);
+ }
+
+ for (; ir1 < act_nrows; ++ir1) {
+ const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);
+ float * restrict dst_row = (float *) (dst->data + ((cur_m_start + ir1) * dst_row_size));
+ float * dst_ptr = &dst_row[ct * 32];
+
+ const float * src2_ptr = NULL;
+ if (src2) {
+ const float * restrict src2_row = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1) * src2_stride));
+ src2_ptr = &src2_row[ct * 32];
+ }
+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, src2_ptr);
+ }
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+
+ if (push_ct < ct_end) {
+ dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride),
+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ push_ct++;
+ }
+ }
}
-#define MATMUL_NX_2D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X2, DOT_2X1) \
-static void hvx_mm_nx_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \
- struct htp_mm_context * mmctx = data; \
- struct htp_ops_context * octx = mmctx->octx; \
- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \
- const uint32_t n_weights = kparams->n_weights; \
- \
- const struct htp_tensor * restrict act = octx->src[n_weights]; /* x */ \
- const uint32_t ne10 = act->ne[0]; \
- const uint32_t act_nrows = act->ne[1] * act->ne[2] * act->ne[3]; \
- const size_t act_stride = mmctx->vtcm_act_stride; \
- \
- uint8_t * restrict vtcm_weight_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \
- uint8_t * restrict act_data = mmctx->vtcm_act; \
- \
- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \
- const uint32_t n_prefetch = kparams->n_prefetch; \
- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \
- \
- const uint32_t tile_size = TILE_SIZE; \
- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \
- uint32_t n_k_tiles_a = ne10 / 32; \
- uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \
- \
- for (uint32_t widx = 0; widx < n_weights; widx++) { \
- const struct htp_tensor * restrict src_w = octx->src[widx]; \
- const struct htp_tensor * restrict dst = octx->dsts[widx]; \
- if (!src_w || !dst) continue; \
- dma_queue * dma_q = octx->ctx->dma[ith]; \
- \
- const uint32_t ne00 = src_w->ne[0]; \
- const uint32_t ne01 = src_w->ne[1]; \
- const size_t dst_row_size = dst->nb[1]; \
- const dma_addr_t src_w_row = src_w->data; \
- \
- uint32_t n_k_tiles_w = ne00 / 32; \
- uint32_t tile_row_stride = n_k_tiles_w * tile_size; \
- \
- uint32_t src0_start_row = 0; \
- uint32_t src0_end_row = ne01; \
- if (octx->ctx->mdev.count > 1) { \
- const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float)); \
- const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, \
- octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div); \
- src0_start_row = range.start; \
- src0_end_row = range.start + range.count; \
- } \
- \
- const uint32_t nrows = src0_end_row - src0_start_row; \
- uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div); \
- src0_nrows_per_thread = hex_round_up(src0_nrows_per_thread, 32); \
- \
- const uint32_t start_row = src0_start_row + src0_nrows_per_thread * ith; \
- const uint32_t end_row = MIN(start_row + src0_nrows_per_thread, src0_end_row); \
- if (start_row >= end_row) continue; \
- \
- uint32_t ct_start = start_row / 32; \
- uint32_t ct_end = (end_row + 31) / 32; \
- \
- uint32_t push_ct = ct_start; \
- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \
- dma_queue_push(dma_q, dma_make_data(vtcm_weight_ptr + d * tile_row_transfer_size_aligned, \
- src_w_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- } \
- \
- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \
- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \
- int valid_rows = (int)ne01 - (int)(ct * 32); \
- valid_rows = MIN(32, MAX(0, valid_rows)); \
- \
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- uint32_t ir1 = 0; \
- for (; ir1 + 1 < act_nrows; ir1 += 2) { \
- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride); \
- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride); \
- \
- float * restrict dst_row0 = (float *) (dst->data + ((ir1+0) * dst_row_size)); \
- float * restrict dst_row1 = (float *) (dst->data + ((ir1+1) * dst_row_size)); \
- float * dst_ptr0 = &dst_row0[ct * 32]; \
- float * dst_ptr1 = &dst_row1[ct * 32]; \
- \
- DOT_2X2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL); \
- } \
- \
- for (; ir1 < act_nrows; ++ir1) { \
- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride); \
- float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size)); \
- float * dst_ptr = &dst_row[ct * 32]; \
- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL); \
- } \
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- \
- if (push_ct < ct_end) { \
- dma_queue_push(dma_q, dma_make_data(w_tile, src_w_row + push_ct * tile_row_stride), \
- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- push_ct++; \
- } \
- } \
- } \
+static void hvx_mv_2d_tiled(unsigned int nth, unsigned int ith, void * data) {
+ htp_matmul_preamble;
+
+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;
+
+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;
+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);
+
+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];
+
+ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
+ const uint32_t n_prefetch = kparams->n_prefetch;
+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);
+
+ const size_t dst_row_size = nb1;
+ const size_t act_row_size = nb11;
+ const size_t act_stride = mmctx->vtcm_act_stride;
+
+ uint8_t * vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith;
+ uint8_t * vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;
+ uint8_t * act_data = mmctx->vtcm_act;
+
+ float * tmp = (float *) vtcm_dst_ptr;
+
+ const dma_addr_t src0_row = src0->data;
+
+ const uint8_t * restrict act_col = (const uint8_t *) act_data;
+ float * restrict dst_col = (float *) dst->data;
+
+ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);
+
+ uint32_t n_k_tiles_w = ne00 / 32;
+ uint32_t n_k_tiles_a = ne10 / 32;
+ uint32_t tile_row_stride = n_k_tiles_w * tile_size;
+ uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;
+
+ uint32_t ct_start = src0_start_row / 32;
+ uint32_t ct_end = (src0_end_row + 31) / 32;
+
+ uint32_t push_ct = ct_start;
+ if (src0_start_row < src0_end_row) {
+ if (src2) {
+ float * vtcm_bias_ptr = (float *) mmctx->vtcm_bias + src0_start_row;
+ const dma_addr_t src2_addr = src2->data + src0_start_row * sizeof(float);
+ int slice_size = (int)MIN(src0_end_row, ne0) - (int)src0_start_row;
+ if (slice_size > 0) {
+ dma_queue_push(dma_q, dma_make_data(vtcm_bias_ptr, src2_addr),
+ slice_size * sizeof(float), slice_size * sizeof(float), slice_size * sizeof(float), 1);
+ dma_queue_pop_nowait(dma_q);
+ }
+ }
+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {
+ dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned,
+ src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ }
+ }
+
+ if (src0_start_row >= src0_end_row) {
+ return;
+ }
+
+ for (uint32_t ct = ct_start; ct < ct_end; ct++) {
+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
+
+ float * dst_ptr = &tmp[ct * 32 - src0_start_row];
+ int valid_rows = (int)ne0 - (int)(ct * 32);
+ valid_rows = MIN(32, MAX(0, valid_rows));
+
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL);
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+
+ if (push_ct < ct_end) {
+ dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride),
+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ push_ct++;
+ }
+ }
+
+ int copy_cnt = (int)MIN(src0_end_row, ne0) - (int)src0_start_row;
+ if (copy_cnt > 0) {
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct_end);
+ if (src2) {
+ hvx_add_f32_uuu((uint8_t *) &dst_col[src0_start_row],
+ (const uint8_t *) tmp,
+ (const uint8_t *) ((const float *) mmctx->vtcm_bias + src0_start_row),
+ copy_cnt);
+ } else {
+ hvx_copy_f32_ua((uint8_t *) &dst_col[src0_start_row], (uint8_t *) tmp, copy_cnt);
+ }
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct_end);
+ }
}
-MATMUL_2D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x2, tiled_vec_dot_q4_0_32x1)
-MATMUL_2D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x2, tiled_vec_dot_q4_1_32x1)
-MATMUL_2D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x2, tiled_vec_dot_q8_0_32x1)
-MATMUL_2D_REPACKED_IMPL(q6_k, 896, tiled_vec_dot_q6_k_32x2, tiled_vec_dot_q6_k_32x1)
-MATMUL_2D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x2, tiled_vec_dot_q5_k_32x1)
-MATMUL_2D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x2, tiled_vec_dot_q3_k_32x1)
-MATMUL_2D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x2, tiled_vec_dot_q2_k_32x1)
-MATMUL_2D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x2, tiled_vec_dot_iq4nl_32x1)
-MATMUL_2D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x2, tiled_vec_dot_mxfp4_32x1)
+static void hvx_mm_nx_2d_tiled(unsigned int nth, unsigned int ith, void * data) {
+ struct htp_mm_context * mmctx = data;
+ struct htp_ops_context * octx = mmctx->octx;
+ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
+ const uint32_t n_weights = kparams->n_weights;
+
+ const struct htp_tensor * restrict act = octx->src[n_weights]; /* x */
+ const uint32_t ne10 = act->ne[0];
+ const uint32_t act_nrows = act->ne[1] * act->ne[2] * act->ne[3];
+ const size_t act_stride = mmctx->vtcm_act_stride;
+
+ uint8_t * restrict vtcm_weight_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;
+ uint8_t * restrict act_data = mmctx->vtcm_act;
+
+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];
+ const uint32_t n_prefetch = kparams->n_prefetch;
+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);
+
+ uint32_t n_k_tiles_a = ne10 / 32;
+
+ for (uint32_t widx = 0; widx < n_weights; widx++) {
+ const struct htp_tensor * restrict src_w = octx->src[widx];
+ const struct htp_tensor * restrict dst = octx->dsts[widx];
+ if (!src_w || !dst) continue;
+ dma_queue * dma_q = octx->ctx->dma[ith];
+
+ const uint32_t tile_size = htp_tensor_get_tile_size(src_w->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src_w->type);
+ const uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;
+
+ const uint32_t ne00 = src_w->ne[0];
+ const uint32_t ne01 = src_w->ne[1];
+ const size_t dst_row_size = dst->nb[1];
+ const dma_addr_t src_w_row = src_w->data;
+
+ uint32_t n_k_tiles_w = ne00 / 32;
+ uint32_t tile_row_stride = n_k_tiles_w * tile_size;
+
+ uint32_t src0_start_row = 0;
+ uint32_t src0_end_row = ne01;
+ if (octx->ctx->mdev.count > 1) {
+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));
+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0,
+ octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
+ src0_start_row = range.start;
+ src0_end_row = range.start + range.count;
+ }
+
+ const uint32_t nrows = src0_end_row - src0_start_row;
+ uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div);
+ src0_nrows_per_thread = hex_round_up(src0_nrows_per_thread, 32);
+
+ const uint32_t start_row = src0_start_row + src0_nrows_per_thread * ith;
+ const uint32_t end_row = MIN(start_row + src0_nrows_per_thread, src0_end_row);
+ if (start_row >= end_row) continue;
+
+ uint32_t ct_start = start_row / 32;
+ uint32_t ct_end = (end_row + 31) / 32;
+
+ uint32_t push_ct = ct_start;
+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {
+ dma_queue_push(dma_q, dma_make_data(vtcm_weight_ptr + d * tile_row_transfer_size_aligned,
+ src_w_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ }
+
+ for (uint32_t ct = ct_start; ct < ct_end; ct++) {
+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
+ int valid_rows = (int)dst->ne[0] - (int)(ct * 32);
+ valid_rows = MIN(32, MAX(0, valid_rows));
+
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+ uint32_t ir1 = 0;
+ for (; ir1 + 1 < act_nrows; ir1 += 2) {
+ const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride);
+ const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride);
+
+ float * restrict dst_row0 = (float *) (dst->data + ((ir1+0) * dst_row_size));
+ float * restrict dst_row1 = (float *) (dst->data + ((ir1+1) * dst_row_size));
+ float * dst_ptr0 = &dst_row0[ct * 32];
+ float * dst_ptr1 = &dst_row1[ct * 32];
+
+ mmctx->vec_dot_32x2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL);
+ }
+
+ for (; ir1 < act_nrows; ++ir1) {
+ const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);
+ float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size));
+ float * dst_ptr = &dst_row[ct * 32];
+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL);
+ }
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+
+ if (push_ct < ct_end) {
+ dma_queue_push(dma_q, dma_make_data(w_tile, src_w_row + push_ct * tile_row_stride),
+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ push_ct++;
+ }
+ }
+ }
+}
static void hvx_mm_transfer_act_dma(
struct htp_ops_context * octx,
@@ -649,12 +645,14 @@ static void name(unsigned int nth, unsigned int ith, void * data) {
htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, ir_first); \
}
-QUANTIZE_IMPL(quantize_f32_q8_0_tiled, "quantize-f32-q8_0_tiled", quantize_f32_q8_0_tiled_kernel, htp_mm_q8_0_tiled_row_size(ne0))
-QUANTIZE_IMPL(quantize_f32_q8_1_tiled, "quantize-f32-q8_1_tiled", quantize_f32_q8_1_tiled_kernel, htp_mm_q8_1_tiled_row_size(ne0))
+QUANTIZE_IMPL(quantize_f32_q8_0_tiled, "quantize-f32-q8_0_tiled", quantize_f32_q8_0_tiled_kernel, htp_mm_q8_0_tiled_row_size(ne0))
+QUANTIZE_IMPL(quantize_f32_q8_1_tiled, "quantize-f32-q8_1_tiled", quantize_f32_q8_1_tiled_kernel, htp_mm_q8_1_tiled_row_size(ne0))
QUANTIZE_IMPL(quantize_f32_q8_1_s16_tiled, "quantize-f32-q8_1_s16_tiled", quantize_f32_q8_1_s16_tiled_kernel, htp_mm_q8_1_tiled_row_size(ne0))
-QUANTIZE_IMPL(quantize_f32_f32, "quantize-f32-f32", quantize_f32_f32_kernel, mmctx->vtcm_act_stride)
-QUANTIZE_IMPL(quantize_f32_f16, "quantize-f32-f16", quantize_f32_f16_kernel, mmctx->vtcm_act_stride)
-QUANTIZE_IMPL(quantize_f16_f16, "quantize-f16-f16", quantize_f16_f16_kernel, mmctx->vtcm_act_stride)
+QUANTIZE_IMPL(quantize_f32_f16_tiled, "quantize-f32-f16_tiled", quantize_f32_f16_tiled_kernel, htp_mm_f16_tiled_row_size(ne0))
+QUANTIZE_IMPL(quantize_f16_f16_tiled, "quantize-f16-f16_tiled", quantize_f16_f16_tiled_kernel, htp_mm_f16_tiled_row_size(ne0))
+QUANTIZE_IMPL(quantize_f32_f32, "quantize-f32-f32", quantize_f32_f32_kernel, mmctx->vtcm_act_stride)
+QUANTIZE_IMPL(quantize_f32_f16, "quantize-f32-f16", quantize_f32_f16_kernel, mmctx->vtcm_act_stride)
+QUANTIZE_IMPL(quantize_f16_f16, "quantize-f16-f16", quantize_f16_f16_kernel, mmctx->vtcm_act_stride)
static void quantize_f32_q8_0_tiled_block(unsigned int nth, unsigned int ith, void * data) {
(void) nth;
@@ -740,178 +738,209 @@ static void quantize_f32_q8_1_s16_tiled_block(unsigned int nth, unsigned int ith
htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);
}
-// q8_1 for weight types with offsets (q8_1_s16 for Q2_K), otherwise q8_0
-static inline worker_callback_t htp_mm_act_quant_row_func(int weight_type) {
+static void quantize_f32_f16_tiled_block(unsigned int nth, unsigned int ith, void * data) {
+ (void) nth;
+ struct htp_mm_context * mmctx = data;
+ if (mmctx->quant_ib_first[ith] >= mmctx->quant_ib_last[ith]) {
+ return;
+ }
+ struct htp_ops_context * octx = mmctx->octx;
+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);
+
+ const struct htp_tensor * src = mmctx->act;
+
+ quantize_f32_f16_tiled_block_kernel(
+ (const float *) mmctx->vtcm_act_raw,
+ mmctx->vtcm_act,
+ NULL,
+ src->ne[0],
+ mmctx->quant_ib_first[ith],
+ mmctx->quant_ib_last[ith],
+ mmctx->vtcm_act_raw_stride,
+ htp_mm_f16_tiled_row_size(src->ne[0]),
+ mmctx->quant_r[ith],
+ mmctx->quant_c[ith]
+ );
+
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);
+}
+
+static void quantize_f16_f16_tiled_block(unsigned int nth, unsigned int ith, void * data) {
+ (void) nth;
+ struct htp_mm_context * mmctx = data;
+ if (mmctx->quant_ib_first[ith] >= mmctx->quant_ib_last[ith]) {
+ return;
+ }
+ struct htp_ops_context * octx = mmctx->octx;
+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);
+
+ const struct htp_tensor * src = mmctx->act;
+
+ quantize_f16_f16_tiled_block_kernel(
+ (const __fp16 *) mmctx->vtcm_act_raw,
+ mmctx->vtcm_act,
+ NULL,
+ src->ne[0],
+ mmctx->quant_ib_first[ith],
+ mmctx->quant_ib_last[ith],
+ mmctx->vtcm_act_raw_stride,
+ htp_mm_f16_tiled_row_size(src->ne[0]),
+ mmctx->quant_r[ith],
+ mmctx->quant_c[ith]
+ );
+
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);
+}
+
+// q8_1 for weight types with offsets (q8_1_s16 for Q2_K), f16_tiled for F16, otherwise q8_0
+static inline worker_callback_t htp_mm_act_quant_row_func(int weight_type, int act_type) {
+ if (weight_type == HTP_TYPE_F16) {
+ return (act_type == HTP_TYPE_F16) ? quantize_f16_f16_tiled : quantize_f32_f16_tiled;
+ }
if (weight_type == HTP_TYPE_Q2_K) {
return quantize_f32_q8_1_s16_tiled;
}
return htp_mm_weight_has_offset(weight_type) ? quantize_f32_q8_1_tiled : quantize_f32_q8_0_tiled;
}
-static inline worker_callback_t htp_mm_act_quant_block_func(int weight_type) {
+static inline worker_callback_t htp_mm_act_quant_block_func(int weight_type, int act_type) {
+ if (weight_type == HTP_TYPE_F16) {
+ return (act_type == HTP_TYPE_F16) ? quantize_f16_f16_tiled_block : quantize_f32_f16_tiled_block;
+ }
if (weight_type == HTP_TYPE_Q2_K) {
return quantize_f32_q8_1_s16_tiled_block;
}
return htp_mm_weight_has_offset(weight_type) ? quantize_f32_q8_1_tiled_block : quantize_f32_q8_0_tiled_block;
}
-MATVEC_2D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x1)
-MATVEC_2D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x1)
-MATVEC_2D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x1)
-MATVEC_2D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x1)
-MATVEC_2D_REPACKED_IMPL(q6_k, 896, tiled_vec_dot_q6_k_32x1)
-MATVEC_2D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x1)
-MATVEC_2D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x1)
-MATVEC_2D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x1)
-MATVEC_2D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x1)
-
-MATMUL_NX_2D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x2, tiled_vec_dot_q4_0_32x1)
-MATMUL_NX_2D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x2, tiled_vec_dot_q4_1_32x1)
-MATMUL_NX_2D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x2, tiled_vec_dot_q8_0_32x1)
-MATMUL_NX_2D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x2, tiled_vec_dot_iq4nl_32x1)
-MATMUL_NX_2D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x2, tiled_vec_dot_mxfp4_32x1)
-MATMUL_NX_2D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x2, tiled_vec_dot_q5_k_32x1)
-MATMUL_NX_2D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x2, tiled_vec_dot_q3_k_32x1)
-MATMUL_NX_2D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x2, tiled_vec_dot_q2_k_32x1)
-
-#define MATMUL_4D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X2, DOT_2X1) \
-static void hvx_mm_4d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \
- htp_matmul_preamble; \
- \
- const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \
- const uint32_t cur_m_rows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13); \
- const uint32_t cur_m_start = mmctx->cur_m_start; \
- \
- const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \
- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \
- \
- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \
- \
- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \
- const uint32_t n_prefetch = kparams->n_prefetch; \
- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \
- \
- const size_t dst_row_size = nb1; \
- const size_t act_stride = mmctx->vtcm_act_stride; \
- \
- uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \
- uint8_t * restrict act_data = mmctx->vtcm_act; \
- \
- const uint32_t tile_size = TILE_SIZE; \
- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \
- \
- const uint32_t n_k_tiles_w = ne00 / 32; \
- const uint32_t n_k_tiles_a = ne10 / 32; \
- const uint32_t tile_row_stride = n_k_tiles_w * tile_size; \
- const uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \
- const uint32_t src0_slice_stride = ((ne01 + 31) / 32) * tile_row_stride; \
- \
- const uint32_t ct_start = src0_start_row / 32; \
- const uint32_t ct_end = (src0_end_row + 31) / 32; \
- \
- \
- if (src0_start_row >= src0_end_row || cur_m_rows == 0) { \
- return; \
- } \
- \
- const uint32_t total_batches = ne12 * ne13; \
- const uint32_t b_start = fastdiv(cur_m_start, &kparams->div_ne1); \
- uint32_t b_end = fastdiv(cur_m_start + cur_m_rows + ne11 - 1, &kparams->div_ne1); \
- b_end = MIN(b_end, total_batches); \
- \
- uint32_t b_grp_start = b_start; \
- while (b_grp_start < b_end) { \
- const uint32_t b3 = fastdiv(b_grp_start, &kparams->div_ne12); \
- const uint32_t b2 = b_grp_start - b3 * ne12; \
- const uint32_t i02 = fastdiv(b2, &kparams->div_r2); \
- const uint32_t i03 = fastdiv(b3, &kparams->div_r3); \
- \
- uint32_t b_grp_end = b_grp_start + 1; \
- while (b_grp_end < b_end) { \
- const uint32_t cur_b3 = fastdiv(b_grp_end, &kparams->div_ne12); \
- const uint32_t cur_b2 = b_grp_end - cur_b3 * ne12; \
- const uint32_t cur_i02 = fastdiv(cur_b2, &kparams->div_r2); \
- const uint32_t cur_i03 = fastdiv(cur_b3, &kparams->div_r3); \
- if (cur_i02 != i02 || cur_i03 != i03) { \
- break; \
- } \
- b_grp_end++; \
- } \
- \
- const uint32_t slice_idx = i03 * ne02 + i02; \
- const dma_addr_t src0_slice = src0->data + (size_t) slice_idx * src0_slice_stride; \
- \
- uint32_t push_ct = ct_start; \
- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \
- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned, \
- src0_slice + (size_t) push_ct * tile_row_stride), \
- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- } \
- \
- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \
- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \
- \
- int valid_rows = (int)ne0 - (int)(ct * 32); \
- valid_rows = MIN(32, MAX(0, valid_rows)); \
- \
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- for (uint32_t b = b_grp_start; b < b_grp_end; b++) { \
- const uint32_t b_m_start = b * ne11; \
- const uint32_t m_first = MAX(cur_m_start, b_m_start); \
- const uint32_t m_last = MIN(cur_m_start + cur_m_rows, b_m_start + ne11); \
- if (m_first >= m_last) continue; \
- \
- const uint32_t cur_b3 = fastdiv(b, &kparams->div_ne12); \
- const uint32_t cur_b2 = b - cur_b3 * ne12; \
- uint8_t * dst_batch_base = (uint8_t *) dst->data + (size_t) cur_b2 * nb2 + (size_t) cur_b3 * nb3; \
- \
- const uint32_t chunk_m_offset = m_first - cur_m_start; \
- const uint32_t dst_m_offset = m_first - b_m_start; \
- const uint32_t batch_nrows = m_last - m_first; \
- \
- uint32_t ir1 = 0; \
- for (; ir1 + 1 < batch_nrows; ir1 += 2) { \
- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 0) * act_stride); \
- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 1) * act_stride); \
- float * restrict dst_row0 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 0) * dst_row_size); \
- float * restrict dst_row1 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 1) * dst_row_size); \
- float * dst_ptr0 = &dst_row0[ct * 32]; \
- float * dst_ptr1 = &dst_row1[ct * 32]; \
- DOT_2X2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL); \
- } \
- for (; ir1 < batch_nrows; ++ir1) { \
- const uint8_t * restrict act_col = (const uint8_t *) (act_data + (chunk_m_offset + ir1) * act_stride); \
- float * restrict dst_row = (float *) (dst_batch_base + (dst_m_offset + ir1) * dst_row_size); \
- float * dst_ptr = &dst_row[ct * 32]; \
- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL); \
- } \
- } \
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \
- \
- if (push_ct < ct_end) { \
- dma_queue_push(dma_q, dma_make_data(w_tile, src0_slice + (size_t) push_ct * tile_row_stride), \
- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \
- push_ct++; \
- } \
- } \
- b_grp_start = b_grp_end; \
- } \
- if (src2) { \
- hvx_tensor_add_f32_grid(dst, src2, cur_m_start, cur_m_start + cur_m_rows, src0_start_row, src0_end_row, &kparams->div_ne12_ne1, &kparams->div_ne1); \
- } \
+static void hvx_mm_4d_tiled(unsigned int nth, unsigned int ith, void * data) {
+ htp_matmul_preamble;
+
+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;
+ const uint32_t cur_m_rows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13);
+ const uint32_t cur_m_start = mmctx->cur_m_start;
+
+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;
+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);
+
+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];
+
+ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
+ const uint32_t n_prefetch = kparams->n_prefetch;
+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);
+
+ const size_t dst_row_size = nb1;
+ const size_t act_stride = mmctx->vtcm_act_stride;
+
+ uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;
+ uint8_t * restrict act_data = mmctx->vtcm_act;
+
+ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);
+
+ const uint32_t n_k_tiles_w = ne00 / 32;
+ const uint32_t n_k_tiles_a = ne10 / 32;
+ const uint32_t tile_row_stride = n_k_tiles_w * tile_size;
+ const uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;
+ const uint32_t src0_slice_stride = ((ne01 + 31) / 32) * tile_row_stride;
+
+ const uint32_t ct_start = src0_start_row / 32;
+ const uint32_t ct_end = (src0_end_row + 31) / 32;
+
+ if (src0_start_row >= src0_end_row || cur_m_rows == 0) {
+ return;
+ }
+
+ const uint32_t total_batches = ne12 * ne13;
+ const uint32_t b_start = fastdiv(cur_m_start, &kparams->div_ne1);
+ uint32_t b_end = fastdiv(cur_m_start + cur_m_rows + ne11 - 1, &kparams->div_ne1);
+ b_end = MIN(b_end, total_batches);
+
+ uint32_t b_grp_start = b_start;
+ while (b_grp_start < b_end) {
+ const uint32_t b3 = fastdiv(b_grp_start, &kparams->div_ne12);
+ const uint32_t b2 = b_grp_start - b3 * ne12;
+ const uint32_t i02 = fastdiv(b2, &kparams->div_r2);
+ const uint32_t i03 = fastdiv(b3, &kparams->div_r3);
+
+ uint32_t b_grp_end = b_grp_start + 1;
+ while (b_grp_end < b_end) {
+ const uint32_t cur_b3 = fastdiv(b_grp_end, &kparams->div_ne12);
+ const uint32_t cur_b2 = b_grp_end - cur_b3 * ne12;
+ const uint32_t cur_i02 = fastdiv(cur_b2, &kparams->div_r2);
+ const uint32_t cur_i03 = fastdiv(cur_b3, &kparams->div_r3);
+ if (cur_i02 != i02 || cur_i03 != i03) {
+ break;
+ }
+ b_grp_end++;
+ }
+
+ const uint32_t slice_idx = i03 * ne02 + i02;
+ const dma_addr_t src0_slice = src0->data + (size_t) slice_idx * src0_slice_stride;
+
+ uint32_t push_ct = ct_start;
+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {
+ dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned,
+ src0_slice + (size_t) push_ct * tile_row_stride),
+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ }
+
+ for (uint32_t ct = ct_start; ct < ct_end; ct++) {
+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
+
+ int valid_rows = (int)ne0 - (int)(ct * 32);
+ valid_rows = MIN(32, MAX(0, valid_rows));
+
+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+ for (uint32_t b = b_grp_start; b < b_grp_end; b++) {
+ const uint32_t b_m_start = b * ne11;
+ const uint32_t m_first = MAX(cur_m_start, b_m_start);
+ const uint32_t m_last = MIN(cur_m_start + cur_m_rows, b_m_start + ne11);
+ if (m_first >= m_last) continue;
+
+ const uint32_t cur_b3 = fastdiv(b, &kparams->div_ne12);
+ const uint32_t cur_b2 = b - cur_b3 * ne12;
+ uint8_t * dst_batch_base = (uint8_t *) dst->data + (size_t) cur_b2 * nb2 + (size_t) cur_b3 * nb3;
+
+ const uint32_t chunk_m_offset = m_first - cur_m_start;
+ const uint32_t dst_m_offset = m_first - b_m_start;
+ const uint32_t batch_nrows = m_last - m_first;
+
+ uint32_t ir1 = 0;
+ for (; ir1 + 1 < batch_nrows; ir1 += 2) {
+ const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 0) * act_stride);
+ const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 1) * act_stride);
+ float * restrict dst_row0 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 0) * dst_row_size);
+ float * restrict dst_row1 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 1) * dst_row_size);
+ float * dst_ptr0 = &dst_row0[ct * 32];
+ float * dst_ptr1 = &dst_row1[ct * 32];
+ mmctx->vec_dot_32x2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL);
+ }
+ for (; ir1 < batch_nrows; ++ir1) {
+ const uint8_t * restrict act_col = (const uint8_t *) (act_data + (chunk_m_offset + ir1) * act_stride);
+ float * restrict dst_row = (float *) (dst_batch_base + (dst_m_offset + ir1) * dst_row_size);
+ float * dst_ptr = &dst_row[ct * 32];
+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL);
+ }
+ }
+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);
+
+ if (push_ct < ct_end) {
+ dma_queue_push(dma_q, dma_make_data(w_tile, src0_slice + (size_t) push_ct * tile_row_stride),
+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);
+ push_ct++;
+ }
+ }
+ b_grp_start = b_grp_end;
+ }
+ if (src2) {
+ hvx_tensor_add_f32_grid(dst, src2, cur_m_start, cur_m_start + cur_m_rows, src0_start_row, src0_end_row, &kparams->div_ne12_ne1, &kparams->div_ne1);
+ }
}
-MATMUL_4D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x2, tiled_vec_dot_q4_0_32x1)
-MATMUL_4D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x2, tiled_vec_dot_q4_1_32x1)
-MATMUL_4D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x2, tiled_vec_dot_q8_0_32x1)
-MATMUL_4D_REPACKED_IMPL(q6_k, 896, tiled_vec_dot_q6_k_32x2, tiled_vec_dot_q6_k_32x1)
-MATMUL_4D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x2, tiled_vec_dot_q5_k_32x1)
-MATMUL_4D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x2, tiled_vec_dot_q3_k_32x1)
-MATMUL_4D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x2, tiled_vec_dot_q2_k_32x1)
-MATMUL_4D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x2, tiled_vec_dot_iq4nl_32x1)
-MATMUL_4D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x2, tiled_vec_dot_mxfp4_32x1)
-
-static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {
+static void hvx_mm_2d_flat(unsigned int nth, unsigned int ith, void * data) {
htp_matmul_preamble;
const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
@@ -1014,7 +1043,7 @@ static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {
}
}
-static void hvx_mv_2d(unsigned int nth, unsigned int ith, void * data) {
+static void hvx_mv_2d_flat(unsigned int nth, unsigned int ith, void * data) {
htp_matmul_preamble;
const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;
@@ -1119,7 +1148,7 @@ static void hvx_mv_2d(unsigned int nth, unsigned int ith, void * data) {
}
}
-static void hvx_mm_4d(unsigned int nth, unsigned int ith, void * data) {
+static void hvx_mm_4d_flat(unsigned int nth, unsigned int ith, void * data) {
htp_matmul_preamble;
const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
@@ -1268,7 +1297,7 @@ static void hvx_mm_4d(unsigned int nth, unsigned int ith, void * data) {
#define MMID_MATRIX_ROW(row_id, i1) matrix_rows[(row_id) * mmctx->mapping_stride + (i1)]
-static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {
+static void hvx_mm_id_tiled(unsigned int nth, unsigned int ith, void * data) {
htp_matmul_preamble;
const struct htp_tensor * restrict ids = octx->src[2];
@@ -1312,8 +1341,8 @@ static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {
const dma_addr_t src0_row = src0->data + cur_a * nb02;
- const uint32_t tile_size = htp_mm_get_weight_tile_size(src0->type);
- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src0->type);
+ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);
const uint32_t n_k_tiles_w = ne00 / 32;
const uint32_t n_k_tiles_a = ne10 / 32;
const uint32_t tile_row_stride = n_k_tiles_w * tile_size;
@@ -1331,7 +1360,7 @@ static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {
for (uint32_t ct = ct_start; ct < ct_end; ct++) {
const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
- int valid_rows = (int)ne01 - (int)(ct * 32);
+ int valid_rows = (int)ne0 - (int)(ct * 32);
valid_rows = MIN(32, MAX(0, valid_rows));
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
@@ -1357,7 +1386,7 @@ static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {
}
}
-static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {
+static void hvx_mv_id_tiled(unsigned int nth, unsigned int ith, void * data) {
htp_matmul_preamble;
const struct htp_tensor * restrict ids = octx->src[2];
@@ -1400,8 +1429,8 @@ static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {
const uint8_t * restrict act_col = (const uint8_t *) act_data;
float * restrict dst_row = (float *) (dst->data + ie1 * nb1);
- const uint32_t tile_size = htp_mm_get_weight_tile_size(src0->type);
- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src0->type);
+ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);
const uint32_t n_k_tiles_w = ne00 / 32;
const uint32_t n_k_tiles_a = ne10 / 32;
const uint32_t tile_row_stride = n_k_tiles_w * tile_size;
@@ -1419,7 +1448,7 @@ static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {
for (uint32_t ct = ct_start; ct < ct_end; ct++) {
const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
- int valid_rows = (int)ne01 - (int)(ct * 32);
+ int valid_rows = (int)ne0 - (int)(ct * 32);
valid_rows = MIN(32, MAX(0, valid_rows));
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
@@ -1435,7 +1464,7 @@ static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {
}
}
-static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {
+static void hvx_mv_id_nx_tiled(unsigned int nth, unsigned int ith, void * data) {
struct htp_mm_context * mmctx = (struct htp_mm_context *) data;
struct htp_ops_context * octx = mmctx->octx;
const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
@@ -1489,8 +1518,8 @@ static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {
const uint8_t * restrict act_col = (const uint8_t *) act_data;
float * restrict dst_row = (float *) (dst->data + ie1 * dst->nb[1]);
- const uint32_t tile_size = htp_mm_get_weight_tile_size(src_w->type);
- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src_w->type);
+ const uint32_t tile_size = htp_tensor_get_tile_size(src_w->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src_w->type);
const uint32_t n_k_tiles_w = src_w->ne[0] / 32;
const uint32_t n_k_tiles_a = act->ne[0] / 32;
const uint32_t tile_row_stride = n_k_tiles_w * tile_size;
@@ -1508,7 +1537,7 @@ static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {
for (uint32_t ct = ct_start; ct < ct_end; ct++) {
const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
- int valid_rows = (int)src_w->ne[1] - (int)(ct * 32);
+ int valid_rows = (int)dst->ne[0] - (int)(ct * 32);
valid_rows = MIN(32, MAX(0, valid_rows));
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
@@ -1525,7 +1554,7 @@ static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {
}
}
-static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {
+static void hvx_mm_id_nx_tiled(unsigned int nth, unsigned int ith, void * data) {
struct htp_mm_context * mmctx = (struct htp_mm_context *) data;
struct htp_ops_context * octx = mmctx->octx;
const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
@@ -1580,8 +1609,8 @@ static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {
const dma_addr_t src0_row = src_w->data + cur_a * src_w->nb[2];
- const uint32_t tile_size = htp_mm_get_weight_tile_size(src_w->type);
- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src_w->type);
+ const uint32_t tile_size = htp_tensor_get_tile_size(src_w->type);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src_w->type);
const uint32_t n_k_tiles_w = src_w->ne[0] / 32;
const uint32_t n_k_tiles_a = act->ne[0] / 32;
const uint32_t tile_row_stride = n_k_tiles_w * tile_size;
@@ -1599,7 +1628,7 @@ static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {
for (uint32_t ct = ct_start; ct < ct_end; ct++) {
const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;
- int valid_rows = (int)src_w->ne[1] - (int)(ct * 32);
+ int valid_rows = (int)dst->ne[0] - (int)(ct * 32);
valid_rows = MIN(32, MAX(0, valid_rows));
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);
@@ -1626,44 +1655,58 @@ static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {
}
}
-static int hvx_mm_init_vec_dot(struct htp_mm_context * mmctx, enum htp_data_type type) {
+static int hvx_mm_init_tiled_vec_dot(struct htp_mm_context * mmctx, enum htp_data_type type) {
switch (type) {
case HTP_TYPE_Q4_0:
mmctx->type = "q4_0_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_q4_0_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_q4_0_32x2;
return 0;
case HTP_TYPE_Q4_1:
case HTP_TYPE_Q4_K:
mmctx->type = "q4_1_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_q4_1_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_q4_1_32x2;
return 0;
case HTP_TYPE_Q8_0:
mmctx->type = "q8_0_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_q8_0_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_q8_0_32x2;
return 0;
case HTP_TYPE_Q5_K:
mmctx->type = "q5_k_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_q5_k_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_q5_k_32x2;
return 0;
case HTP_TYPE_Q6_K:
mmctx->type = "q6_k_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_q6_k_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_q6_k_32x2;
return 0;
case HTP_TYPE_Q3_K:
mmctx->type = "q3_k_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_q3_k_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_q3_k_32x2;
return 0;
case HTP_TYPE_Q2_K:
mmctx->type = "q2_k_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_q2_k_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_q2_k_32x2;
return 0;
case HTP_TYPE_IQ4_NL:
mmctx->type = "iq4nl_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_iq4nl_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_iq4nl_32x2;
return 0;
case HTP_TYPE_MXFP4:
mmctx->type = "mxfp4_tiled-f32";
mmctx->vec_dot_32x1 = tiled_vec_dot_mxfp4_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_mxfp4_32x2;
+ return 0;
+ case HTP_TYPE_F16:
+ mmctx->type = "f16_tiled-f16";
+ mmctx->vec_dot_32x1 = tiled_vec_dot_f16_32x1;
+ mmctx->vec_dot_32x2 = tiled_vec_dot_f16_32x2;
return 0;
default:
return -1;
@@ -1705,15 +1748,11 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
mmctx->src0_row_start = src0_row_start;
mmctx->src0_row_end = src0_row_end;
- bool is_repacked = (src0->type == HTP_TYPE_Q4_0 || src0->type == HTP_TYPE_Q4_1 ||
- src0->type == HTP_TYPE_Q8_0 || src0->type == HTP_TYPE_IQ4_NL ||
- src0->type == HTP_TYPE_MXFP4 || src0->type == HTP_TYPE_Q6_K ||
- src0->type == HTP_TYPE_Q4_K || src0->type == HTP_TYPE_Q5_K ||
- src0->type == HTP_TYPE_Q3_K || src0->type == HTP_TYPE_Q2_K);
+ const bool is_tiled = htp_tensor_is_tiled(src0);
// Compute src0_nrows_per_thread
mmctx->src0_nrows_per_thread = fastdiv(nrows + octx->n_threads - 1, &octx->n_threads_div);
- if (is_repacked) {
+ if (is_tiled) {
mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);
} else {
mmctx->src0_nrows_per_thread += (mmctx->src0_nrows_per_thread & 1); // round up to even
@@ -1730,59 +1769,11 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
uint32_t n_quant_tasks = 1;
const bool is_batched = (ne12 > 1 || ne13 > 1 || ne02 > 1 || ne03 > 1);
if (is_batched) {
- if (is_repacked) {
- switch (src0->type) {
- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mm_4d_repacked_q4_0; break;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mm_4d_repacked_q4_1; break;
- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mm_4d_repacked_q8_0; break;
- case HTP_TYPE_Q6_K: matmul_job_func = hvx_mm_4d_repacked_q6_k; break;
- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mm_4d_repacked_q5_k; break;
- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mm_4d_repacked_q3_k; break;
- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mm_4d_repacked_q2_k; break;
- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mm_4d_repacked_iq4nl; break;
- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mm_4d_repacked_mxfp4; break;
- default: return HTP_STATUS_NO_SUPPORT;
- }
- } else {
- matmul_job_func = hvx_mm_4d;
- }
+ matmul_job_func = is_tiled ? hvx_mm_4d_tiled : hvx_mm_4d_flat;
} else if (act_nrows > 1) {
- if (is_repacked) {
- switch (src0->type) {
- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mm_2d_repacked_q4_0; break;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mm_2d_repacked_q4_1; break;
- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mm_2d_repacked_q8_0; break;
- case HTP_TYPE_Q6_K: matmul_job_func = hvx_mm_2d_repacked_q6_k; break;
- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mm_2d_repacked_q5_k; break;
- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mm_2d_repacked_q3_k; break;
- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mm_2d_repacked_q2_k; break;
- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mm_2d_repacked_iq4nl; break;
- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mm_2d_repacked_mxfp4; break;
- default: return HTP_STATUS_NO_SUPPORT;
- }
- } else {
- matmul_job_func = hvx_mm_2d;
- }
+ matmul_job_func = is_tiled ? hvx_mm_2d_tiled : hvx_mm_2d_flat;
} else {
- if (is_repacked) {
- switch (src0->type) {
- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mv_2d_repacked_q4_0; break;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mv_2d_repacked_q4_1; break;
- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mv_2d_repacked_q8_0; break;
- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mv_2d_repacked_q5_k; break;
- case HTP_TYPE_Q6_K: matmul_job_func = hvx_mv_2d_repacked_q6_k; break;
- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mv_2d_repacked_q3_k; break;
- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mv_2d_repacked_q2_k; break;
- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mv_2d_repacked_iq4nl; break;
- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mv_2d_repacked_mxfp4; break;
- default: return HTP_STATUS_NO_SUPPORT;
- }
- } else {
- matmul_job_func = hvx_mv_2d;
- }
+ matmul_job_func = is_tiled ? hvx_mv_2d_tiled : hvx_mv_2d_flat;
}
bool need_quant = true;
@@ -1792,9 +1783,9 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
quant_task_func = (act->type == HTP_TYPE_F32) ? quantize_f32_f16 : quantize_f16_f16;
need_quant = (act->type == HTP_TYPE_F32);
mmctx->type = (act->type == HTP_TYPE_F32) ? "f32-f16" : "f16-f16";
- mmctx->vec_dot_1x1 = vec_dot_f16_f16_aa_1x1;
- mmctx->vec_dot_2x1 = vec_dot_f16_f16_aa_2x1;
- mmctx->vec_dot_2x2 = vec_dot_f16_f16_aa_2x2;
+ mmctx->vec_dot_1x1 = flat_vec_dot_f16_f16_aa_1x1;
+ mmctx->vec_dot_2x1 = flat_vec_dot_f16_f16_aa_2x1;
+ mmctx->vec_dot_2x2 = flat_vec_dot_f16_f16_aa_2x2;
act_row_size = kparams->act_row_size;
break;
@@ -1802,16 +1793,26 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
quant_task_func = NULL;
need_quant = false;
mmctx->type = "f32-f32";
- mmctx->vec_dot_1x1 = vec_dot_f32_f32_aa_1x1;
- mmctx->vec_dot_2x1 = vec_dot_f32_f32_aa_2x1;
- mmctx->vec_dot_2x2 = vec_dot_f32_f32_aa_2x2;
+ mmctx->vec_dot_1x1 = flat_vec_dot_f32_f32_aa_1x1;
+ mmctx->vec_dot_2x1 = flat_vec_dot_f32_f32_aa_2x1;
+ mmctx->vec_dot_2x2 = flat_vec_dot_f32_f32_aa_2x2;
+ act_row_size = kparams->act_row_size;
+ break;
+
+ case HTP_MM_KERNEL_HVX_BF16_F32_VTCM:
+ quant_task_func = NULL;
+ need_quant = false;
+ mmctx->type = "bf16-f32";
+ mmctx->vec_dot_1x1 = flat_vec_dot_bf16_f32_aa_1x1;
+ mmctx->vec_dot_2x1 = flat_vec_dot_bf16_f32_aa_2x1;
+ mmctx->vec_dot_2x2 = flat_vec_dot_bf16_f32_aa_2x2;
act_row_size = kparams->act_row_size;
break;
case HTP_MM_KERNEL_HVX_QUANT_BLOCK:
case HTP_MM_KERNEL_HVX_QUANT_ROW:
- default:
- if (hvx_mm_init_vec_dot(mmctx, src0->type) != 0) {
+ default: {
+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) != 0) {
return HTP_STATUS_NO_SUPPORT;
}
@@ -1821,7 +1822,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
if (act_nrows < octx->n_threads && !is_batched) {
n_quant_tasks = MIN(total_nb, octx->n_threads);
- quant_task_func = htp_mm_act_quant_block_func(src0->type);
+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);
for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {
uint32_t ib_first = (total_nb * ith) / n_quant_tasks;
uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;
@@ -1832,10 +1833,11 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
}
} else {
n_quant_tasks = MIN(act_nrows, octx->n_threads);
- quant_task_func = htp_mm_act_quant_row_func(src0->type);
+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);
}
act_row_size = kparams->act_row_size;
break;
+ }
}
const uint32_t m_chunk = (kparams->m_chunk > 0 && (uint32_t) kparams->m_chunk < act_nrows)
@@ -1848,6 +1850,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
if (kparams->kernel_type == HTP_MM_KERNEL_HVX_F16_F16_VTCM ||
kparams->kernel_type == HTP_MM_KERNEL_HVX_F32_F32_VTCM ||
+ kparams->kernel_type == HTP_MM_KERNEL_HVX_BF16_F32_VTCM ||
kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW ||
kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {
mmctx->vtcm_act_size_per_thread = L.act_bytes;
@@ -1908,7 +1911,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
work_queue_func_t q_func;
if (cur_m_rows < octx->n_threads && (kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_BLOCK || kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW)) {
quant_tasks = MIN(total_nb, octx->n_threads);
- q_func = htp_mm_act_quant_block_func(src0->type);
+ q_func = htp_mm_act_quant_block_func(src0->type, act->type);
for (uint32_t ith = 0; ith < quant_tasks; ++ith) {
uint32_t ib_first = (total_nb * ith) / quant_tasks;
uint32_t ib_last = (total_nb * (ith + 1)) / quant_tasks;
@@ -1949,158 +1952,18 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {
return HTP_STATUS_OK;
}
-static void hvx_mm_nx_2d(unsigned int nth, unsigned int ith, void * data) {
- struct htp_mm_context * mmctx = data;
- struct htp_ops_context * octx = mmctx->octx;
- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;
- const uint32_t n_weights = kparams->n_weights;
-
- const struct htp_tensor * restrict act = octx->src[n_weights];
- const uint32_t act_nrows = act->ne[1] * act->ne[2] * act->ne[3];
- const size_t act_stride = mmctx->vtcm_act_stride;
-
- uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;
- uint8_t * restrict act_data = mmctx->vtcm_act;
-
- const uint32_t n_prefetch = kparams->n_prefetch;
- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);
- const uint32_t prefetch_mask = n_prefetch - 1;
-
- struct htp_thread_trace * tr = &octx->ctx->trace[ith];
-
-
- for (uint32_t widx = 0; widx < n_weights; widx++) {
- const struct htp_tensor * restrict src_w = octx->src[widx];
- const struct htp_tensor * restrict dst = octx->dsts[widx];
- if (!src_w || !dst) continue;
- dma_queue * dma_q = octx->ctx->dma[ith];
-
- const uint32_t ne00 = src_w->ne[0];
- const uint32_t ne01 = src_w->ne[1];
- uint32_t start_row = 0;
- uint32_t end_row = ne01;
- if (octx->ctx->mdev.count > 1) {
- const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));
- const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
- start_row = range.start;
- end_row = range.start + range.count;
- }
-
- const uint32_t nrows = end_row - start_row;
- uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div);
- src0_nrows_per_thread += (src0_nrows_per_thread & 1);
-
- const uint32_t src0_start_row = start_row + src0_nrows_per_thread * ith;
- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, end_row);
- const uint32_t src0_end_row_x2 = src0_start_row + ((src0_end_row - src0_start_row) & ~1U);
- if (src0_start_row >= src0_end_row) continue;
-
- const size_t dst_row_size = dst->nb[1];
- const size_t src0_row_size = src_w->nb[1];
- const size_t src0_stride = hex_round_up(src0_row_size, 128);
-
- const dma_addr_t src0_row = src_w->data;
-
- for (uint32_t ir0 = src0_start_row; ir0 < src0_end_row_x2; ir0 += 2) {
- const int is0 = (ir0 - src0_start_row);
- if (is0 >= (int)n_prefetch) break;
- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + is0 * src0_stride, src0_row + ir0 * src0_row_size),
- src0_stride, src0_row_size, src0_row_size, 2);
- }
-
- for (uint32_t ir0 = src0_start_row; ir0 < src0_end_row_x2; ir0 += 2) {
- const uint8_t * ss0 = (void *) dma_queue_pop(dma_q).dst;
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ir0);
- uint32_t ir1 = 0;
- for (; ir1 + 1 < act_nrows; ir1 += 2) {
- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride);
- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride);
- float * restrict dst_row0 = (float *) (dst->data + ((ir1+0) * dst_row_size));
- float * restrict dst_row1 = (float *) (dst->data + ((ir1+1) * dst_row_size));
- mmctx->vec_dot_2x2(ne00, &dst_row0[ir0], &dst_row1[ir0], ss0, ss0 + src0_stride, act_col0, act_col1);
- }
- for (; ir1 < act_nrows; ++ir1) {
- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);
- float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size));
- mmctx->vec_dot_2x1(ne00, &dst_row[ir0], ss0, ss0 + src0_stride, act_col);
- }
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ir0);
-
- const int pr0 = (ir0 + n_prefetch);
- const int is0 = (pr0 - src0_start_row) & prefetch_mask;
- if (pr0 < src0_end_row_x2) {
- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + is0 * src0_stride, src0_row + pr0 * src0_row_size),
- src0_stride, src0_row_size, src0_row_size, 2);
- }
- }
-
- if (src0_end_row != src0_end_row_x2) {
- uint32_t ir0 = src0_end_row_x2;
- const int is0 = (ir0 - src0_start_row) & prefetch_mask;
- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + is0 * src0_stride, src0_row + ir0 * src0_row_size),
- src0_stride, src0_row_size, src0_row_size, 1);
- const uint8_t * ss0 = (void *) dma_queue_pop(dma_q).dst;
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ir0);
- for (uint32_t ir1 = 0; ir1 < act_nrows; ++ir1) {
- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);
- float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size));
- mmctx->vec_dot_1x1(ne00, &dst_row[ir0], ss0, act_col);
- }
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ir0);
- }
- }
-}
-
-#define DEQUANTIZE_WORKER_LOOP_IMPL(SUFFIX) \
-static void dequantize_tiled_worker_loop_##SUFFIX(unsigned int n, unsigned int i, void *data) { \
- tiled_dequantize_state_t *state = (tiled_dequantize_state_t *)data; \
- struct htp_thread_trace * tr = &state->traces[i]; \
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i); \
- for (unsigned int task_id = i; task_id < (unsigned int)state->n_tasks; task_id += n) { \
- int start = task_id * state->n_tiles_per_task; \
- int end = hex_smin(start + state->n_tiles_per_task, state->n_tot_tiles); \
- dequantize_tiled_weight_to_fp16_task_##SUFFIX(state, start, end); \
- } \
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i); \
-}
-
-DEQUANTIZE_WORKER_LOOP_IMPL(q4_0)
-DEQUANTIZE_WORKER_LOOP_IMPL(q4_1)
-DEQUANTIZE_WORKER_LOOP_IMPL(iq4_nl)
-DEQUANTIZE_WORKER_LOOP_IMPL(mxfp4)
-DEQUANTIZE_WORKER_LOOP_IMPL(q8_0)
-DEQUANTIZE_WORKER_LOOP_IMPL(q6_k)
-DEQUANTIZE_WORKER_LOOP_IMPL(q5_k)
-DEQUANTIZE_WORKER_LOOP_IMPL(q3_k)
-DEQUANTIZE_WORKER_LOOP_IMPL(q2_k)
-
-static void convert_f16_worker_loop(unsigned int n, unsigned int i, void *data) {
- tiled_dequantize_state_t *state = (tiled_dequantize_state_t *)data;
+static void dequantize_tiled_worker_loop(unsigned int n, unsigned int i, void * data) {
+ tiled_dequantize_state_t * state = (tiled_dequantize_state_t *) data;
struct htp_thread_trace * tr = &state->traces[i];
htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i);
- for (unsigned int task_id = i; task_id < (unsigned int)state->n_tasks; task_id += n) {
+ for (unsigned int task_id = i; task_id < (unsigned int) state->n_tasks; task_id += n) {
int start = task_id * state->n_tiles_per_task;
int end = hex_smin(start + state->n_tiles_per_task, state->n_tot_tiles);
- convert_f16_weight_to_fp16_tiles_task(state, start, end);
+ state->task_fn(state, start, end);
}
htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i);
}
-static void quantize_f32_worker_loop(unsigned int n, unsigned int i, void *data) {
- tiled_dequantize_state_t *state = (tiled_dequantize_state_t *)data;
-
- struct htp_thread_trace * tr = &state->traces[i];
- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_A_QUANT, i);
-
- for (unsigned int task_id = i; task_id < (unsigned int)state->n_tasks; task_id += n) {
- int start = task_id * state->n_tiles_per_task;
- int end = hex_smin(start + state->n_tiles_per_task, state->n_tot_tiles);
- quantize_f32_weight_to_fp16_tiles_task(state, start, end);
- }
-
- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, i);
-}
-
static void transfer_output_chunk_worker_fn(unsigned int n, unsigned int i, void *data) {
output_transfer_task_state_t *st = (output_transfer_task_state_t *) data;
@@ -2450,13 +2313,36 @@ static void transfer_output_chunk_scattered_worker_fn(unsigned int n, unsigned i
// --- HMX Dispatchers & Entry Points ---
+static inline tiled_dequant_task_fn_t hmx_get_dequant_worker_fn(int weight_type, bool is_tiled) {
+ switch (weight_type) {
+ case HTP_TYPE_Q4_0: return dequantize_tiled_weight_to_fp16_task_q4_0;
+ case HTP_TYPE_IQ4_NL: return dequantize_tiled_weight_to_fp16_task_iq4_nl;
+ case HTP_TYPE_Q4_1:
+ case HTP_TYPE_Q4_K: return dequantize_tiled_weight_to_fp16_task_q4_1;
+ case HTP_TYPE_MXFP4: return dequantize_tiled_weight_to_fp16_task_mxfp4;
+ case HTP_TYPE_Q8_0: return dequantize_tiled_weight_to_fp16_task_q8_0;
+ case HTP_TYPE_Q5_K: return dequantize_tiled_weight_to_fp16_task_q5_k;
+ case HTP_TYPE_Q6_K: return dequantize_tiled_weight_to_fp16_task_q6_k;
+ case HTP_TYPE_Q3_K: return dequantize_tiled_weight_to_fp16_task_q3_k;
+ case HTP_TYPE_Q2_K: return dequantize_tiled_weight_to_fp16_task_q2_k;
+ case HTP_TYPE_F16: return is_tiled ? NULL : convert_f16_weight_to_fp16_tiles_task;
+ case HTP_TYPE_BF16: return convert_bf16_weight_to_fp16_tiles_task;
+ case HTP_TYPE_F32: return quantize_f32_weight_to_fp16_tiles_task;
+ default: return NULL;
+ }
+}
+
static void dequantize_tiled_weight_chunk_to_fp16_tiles(
struct htp_context *ctx, __fp16 *vtcm_dst,
const void *weight_src_ddr,
int n_cols, int k_block,
size_t row_stride, int weight_type,
int n_k_tiles, struct fastdiv_values n_k_tiles_div,
- worker_callback_t dequant_worker_fn, int n_threads) {
+ tiled_dequant_task_fn_t dequant_worker_fn, int n_threads) {
+
+ if (!dequant_worker_fn) {
+ return;
+ }
assert(k_block % HTP_MM_HMX_TILE_N_COLS == 0);
@@ -2471,6 +2357,7 @@ static void dequantize_tiled_weight_chunk_to_fp16_tiles(
state.n_tiles_per_task = n_tiles_per_task;
state.dst = vtcm_dst;
state.src = (const uint8_t *)weight_src_ddr;
+ state.task_fn = dequant_worker_fn;
state.n_cols = n_cols;
state.k_block = k_block;
state.row_stride = row_stride;
@@ -2480,14 +2367,14 @@ static void dequantize_tiled_weight_chunk_to_fp16_tiles(
state.traces = ctx->trace;
state.ctx = ctx;
- state.tile_size = htp_mm_get_weight_tile_size(weight_type);
- state.aligned_tile_size = htp_mm_get_weight_aligned_tile_size(weight_type);
+ state.tile_size = htp_tensor_get_tile_size(weight_type);
+ state.aligned_tile_size = htp_tensor_get_tile_padded_size(weight_type);
if (state.n_tasks == 1 || n_threads == 1) {
- dequant_worker_fn(1, 0, &state);
+ dequantize_tiled_worker_loop(1, 0, &state);
} else {
int n_tasks = hex_smin((int) state.n_tasks, n_threads);
- worker_pool_run_func(ctx->worker_pool, dequant_worker_fn, &state, n_tasks);
+ worker_pool_run_func(ctx->worker_pool, dequantize_tiled_worker_loop, &state, n_tasks);
}
}
@@ -2764,53 +2651,35 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
struct htp_thread_trace * tr = &ctx->trace[0];
htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);
- // Quantized weights are repacked and padded to 32, so we it has to be 32-aligned.
- // Only F16/F32 weights can be non-32-aligned, they will be padded in the following kernel.
- const bool wtype_is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);
- if (k % 32 != 0 || (wtype_is_quant && n % 32 != 0)) {
+ const bool is_tiled = (tile_size > 0);
+
+ if (k % 32 != 0) {
return -1;
}
- if (!hex_is_aligned(dst, VLEN) || (act_dma_addr & (VLEN - 1)) != 0) { return -1; }
+ if (!hex_is_aligned(dst, sizeof(float))) { return -1; }
- size_t row_stride = htp_mm_get_tiled_row_stride(weight_type, k);
+ size_t row_stride = htp_tensor_get_row_size(weight_type, k);
if (row_stride == 0) {
return -1;
}
- worker_callback_t dequant_worker_fn = NULL;
- switch (weight_type) {
- case HTP_TYPE_Q4_0: dequant_worker_fn = dequantize_tiled_worker_loop_q4_0; break;
- case HTP_TYPE_IQ4_NL: dequant_worker_fn = dequantize_tiled_worker_loop_iq4_nl; break;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K: dequant_worker_fn = dequantize_tiled_worker_loop_q4_1; break;
- case HTP_TYPE_MXFP4: dequant_worker_fn = dequantize_tiled_worker_loop_mxfp4; break;
- case HTP_TYPE_Q8_0: dequant_worker_fn = dequantize_tiled_worker_loop_q8_0; break;
- case HTP_TYPE_Q5_K: dequant_worker_fn = dequantize_tiled_worker_loop_q5_k; break;
- case HTP_TYPE_Q6_K: dequant_worker_fn = dequantize_tiled_worker_loop_q6_k; break;
- case HTP_TYPE_Q3_K: dequant_worker_fn = dequantize_tiled_worker_loop_q3_k; break;
- case HTP_TYPE_Q2_K: dequant_worker_fn = dequantize_tiled_worker_loop_q2_k; break;
- case HTP_TYPE_F16: dequant_worker_fn = convert_f16_worker_loop; break;
- case HTP_TYPE_F32: dequant_worker_fn = quantize_f32_worker_loop; break;
- default:
- return -1;
- }
+ tiled_dequant_task_fn_t dequant_worker_fn = hmx_get_dequant_worker_fn(weight_type, is_tiled);
const int n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;
const struct fastdiv_values n_k_tiles_div = init_fastdiv_values(n_k_tiles);
- const bool is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);
const size_t vec_dot_size = k * sizeof(__fp16);
const size_t vtcm_budget = ctx->vtcm_size;
- const uint32_t dma_dst_stride = is_quant ? aligned_tile_size : row_stride;
- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;
- const uint32_t dma_width_bytes = is_quant ? tile_size : row_stride;
+ const uint32_t dma_dst_stride = is_tiled ? aligned_tile_size : row_stride;
+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;
+ const uint32_t dma_width_bytes = is_tiled ? tile_size : row_stride;
size_t m_chunk_n_rows = m_chunk;
size_t n_chunk_n_cols = n_chunk;
size_t vtcm_used = vtcm_size;
- const size_t qweight_row_stride = is_quant ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;
+ const size_t qweight_row_stride = is_tiled ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;
struct htp_mm_hmx_vtcm_layout L;
htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_2D, weight_type, k, m_chunk_n_rows, n_chunk_n_cols, 1, pipeline, act_threads, aligned_tile_size, src2_bytes);
@@ -2827,7 +2696,6 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
VTCM_LAYOUT_PTR(__fp16, base, L.off_weight[0]),
VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_weight[1], pipeline)
};
-
__fp16 *vtcm_f16_act = VTCM_LAYOUT_PTR(__fp16, base, L.off_act);
float *vtcm_f32_act = VTCM_LAYOUT_PTR(float, base, L.off_act_f32);
__fp16 *vtcm_output = VTCM_LAYOUT_PTR(__fp16, base, L.off_dst[0]);
@@ -2855,13 +2723,23 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
if (pipeline) {
// --- Asynchronous Pipelined Loop ---
hmx_matmul_job_t job_slots[2]; // persistent double-buffered job descriptors
+ void *vtcm_raw_bufs[4] = {
+ vtcm_weight_raw[0],
+ vtcm_weight_raw[1],
+ dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0,
+ dequant_worker_fn ? vtcm_weight_raw[1] : vtcm_scratch1,
+ };
+ void *vtcm_weight_bufs[4] = {
+ dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0],
+ dequant_worker_fn ? vtcm_scratch1 : vtcm_weight_raw[1],
+ vtcm_scratch0,
+ vtcm_scratch1,
+ };
+ void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };
for (size_t mr = 0; mr < m; mr += m_chunk_n_rows) {
const size_t n_rows = hex_smin(m - mr, m_chunk_n_rows);
- void *vtcm_weight_bufs[2] = { vtcm_scratch0, vtcm_scratch1 };
- void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };
-
struct activation_transfer_params act_params = {
.ctx = ctx,
.dst = vtcm_f16_act,
@@ -2881,14 +2759,14 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
// Prologue: push A0 and optionally A1 (if n_chunk_cnt > 1)
const size_t n_cols_A0 = hex_smin(n - 0 * n_chunk_n_cols, n_chunk_n_cols);
- const uint32_t height_A0 = is_quant ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;
- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight),
+ const uint32_t height_A0 = is_tiled ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight),
dma_dst_stride, dma_src_stride, dma_width_bytes, height_A0);
if (1 < n_chunk_cnt) {
const size_t n_cols_A1 = hex_smin(n - 1 * n_chunk_n_cols, n_chunk_n_cols);
- const uint32_t height_A1 = is_quant ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;
- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[1], weight + n_chunk_n_cols * weight_stride),
+ const uint32_t height_A1 = is_tiled ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[1], weight + n_chunk_n_cols * weight_stride),
dma_dst_stride, dma_src_stride, dma_width_bytes, height_A1);
}
@@ -2905,20 +2783,20 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
// 2. dequantize A_i
dequantize_tiled_weight_chunk_to_fp16_tiles(
- ctx, vtcm_weight_bufs[i % 2], curr_raw,
+ ctx, vtcm_weight_bufs[i % 4], curr_raw,
n_cols, k, row_stride, weight_type,
n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);
// 3. push A_{i+2} (if i+2 < n_chunk_cnt)
if (i + 2 < n_chunk_cnt) {
- const uint32_t height_p2 = is_quant ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;
- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_p2 * weight_stride),
+ const uint32_t height_p2 = is_tiled ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 2) % 4], weight + nc_p2 * weight_stride),
dma_dst_stride, dma_src_stride, dma_width_bytes, height_p2);
}
// 4. submit C_i
hmx_matmul_job_init(&job_slots[i % 2], (__fp16 *) vtcm_output_bufs[i % 2],
- (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2],
+ (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 4],
vtcm_scales, hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS),
hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS), k / HTP_MM_HMX_TILE_N_ROWS);
hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job_slots[i % 2]));
@@ -2951,6 +2829,9 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
} else {
// --- Synchronous loop (m <= 32 or fallback) ---
hmx_matmul_job_t job;
+ void *vtcm_raw_bufs[2] = { vtcm_weight_raw[0], dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0 };
+ void *vtcm_weight_bufs[2] = { dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0], vtcm_scratch0 };
+
for (size_t mr = 0; mr < m; mr += m_chunk_n_rows) {
const size_t n_rows = hex_smin(m - mr, m_chunk_n_rows);
@@ -2974,11 +2855,12 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
// A0: Pre-fetch the first weight chunk (nc = 0)
if (n > 0) {
const size_t n_cols = hex_smin(n, n_chunk_n_cols);
- const uint32_t height = is_quant ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;
- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);
+ const uint32_t height = is_tiled ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);
}
- for (size_t nc = 0; nc < n; nc += n_chunk_n_cols) {
+ for (int i = 0; i < n_chunk_cnt; ++i) {
+ const size_t nc = (size_t) i * n_chunk_n_cols;
const size_t n_cols = hex_smin(n - nc, n_chunk_n_cols);
const size_t n_row_tiles = hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS);
const size_t n_col_tiles = hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS);
@@ -2988,7 +2870,7 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
// B: Weight Dequantize (Threaded)
dequantize_tiled_weight_chunk_to_fp16_tiles(
- ctx, vtcm_scratch0, curr_raw,
+ ctx, vtcm_weight_bufs[i % 2], curr_raw,
n_cols, k, row_stride, weight_type,
n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);
@@ -2996,12 +2878,12 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,
const size_t nc_next = nc + n_chunk_n_cols;
if (nc_next < n) {
const size_t n_cols_next = hex_smin(n - nc_next, n_chunk_n_cols);
- const uint32_t height_next = is_quant ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;
- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);
+ const uint32_t height_next = is_tiled ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 1) % 2], weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);
}
// C: HMX Compute (Queue-based)
- hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, vtcm_scratch0, vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);
+ hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2], vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);
hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job));
hmx_queue_pop(ctx->hmx_queue);
@@ -3041,35 +2923,20 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
const dma_addr_t act_dma_addr = act->data;
if (k % 32 != 0) { return HTP_STATUS_NO_SUPPORT; }
- if ((act_dma_addr & (VLEN - 1)) != 0) { return HTP_STATUS_NO_SUPPORT; }
- size_t row_stride = htp_mm_get_tiled_row_stride(weight_type, k);
+ size_t row_stride = htp_tensor_get_row_size(weight_type, k);
if (row_stride == 0) {
return HTP_STATUS_NO_SUPPORT;
}
- worker_callback_t dequant_worker_fn = NULL;
- switch (weight_type) {
- case HTP_TYPE_Q4_0: dequant_worker_fn = dequantize_tiled_worker_loop_q4_0; break;
- case HTP_TYPE_IQ4_NL: dequant_worker_fn = dequantize_tiled_worker_loop_iq4_nl; break;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K: dequant_worker_fn = dequantize_tiled_worker_loop_q4_1; break;
- case HTP_TYPE_MXFP4: dequant_worker_fn = dequantize_tiled_worker_loop_mxfp4; break;
- case HTP_TYPE_Q8_0: dequant_worker_fn = dequantize_tiled_worker_loop_q8_0; break;
- case HTP_TYPE_Q5_K: dequant_worker_fn = dequantize_tiled_worker_loop_q5_k; break;
- case HTP_TYPE_Q6_K: dequant_worker_fn = dequantize_tiled_worker_loop_q6_k; break;
- case HTP_TYPE_Q3_K: dequant_worker_fn = dequantize_tiled_worker_loop_q3_k; break;
- case HTP_TYPE_Q2_K: dequant_worker_fn = dequantize_tiled_worker_loop_q2_k; break;
- case HTP_TYPE_F16: dequant_worker_fn = convert_f16_worker_loop; break;
- case HTP_TYPE_F32: dequant_worker_fn = quantize_f32_worker_loop; break;
- default:
- return HTP_STATUS_NO_SUPPORT;
- }
-
const int n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;
const struct fastdiv_values n_k_tiles_div = init_fastdiv_values(n_k_tiles);
- const bool is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);
+ const int tile_size = kparams->tile_size;
+ const int aligned_tile_size = kparams->aligned_tile_size;
+
+ const bool is_tiled = (tile_size > 0);
+ tiled_dequant_task_fn_t dequant_worker_fn = hmx_get_dequant_worker_fn(weight_type, is_tiled);
const size_t vtcm_budget = ctx->vtcm_size;
const int m_chunk_n_rows = kparams->m_chunk;
@@ -3079,11 +2946,9 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
const int act_threads = kparams->n_act_threads;
const struct fastdiv_values * act_threads_div = &kparams->div_n_act_threads;
const struct fastdiv_values * k_div = &kparams->div_ne00_padded;
- const int tile_size = kparams->tile_size;
- const int aligned_tile_size = kparams->aligned_tile_size;
- const uint32_t dma_dst_stride = is_quant ? aligned_tile_size : row_stride;
- const uint32_t dma_width_bytes = is_quant ? tile_size : row_stride;
+ const uint32_t dma_dst_stride = is_tiled ? aligned_tile_size : row_stride;
+ const uint32_t dma_width_bytes = is_tiled ? tile_size : row_stride;
struct htp_mm_hmx_vtcm_layout L;
htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_2D, weight_type, k, m_chunk_n_rows, n_chunk_n_cols, 1, pipeline, act_threads, aligned_tile_size, 0);
@@ -3099,7 +2964,6 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
VTCM_LAYOUT_PTR(__fp16, base, L.off_weight[0]),
VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_weight[1], pipeline)
};
-
__fp16 *vtcm_f16_act = VTCM_LAYOUT_PTR(__fp16, base, L.off_act);
float *vtcm_f32_act = VTCM_LAYOUT_PTR(float, base, L.off_act_f32);
__fp16 *vtcm_output = VTCM_LAYOUT_PTR(__fp16, base, L.off_dst[0]);
@@ -3141,13 +3005,23 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
if (pipeline) {
hmx_matmul_job_t job_slots[2];
+ void *vtcm_raw_bufs[4] = {
+ vtcm_weight_raw[0],
+ vtcm_weight_raw[1],
+ dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0,
+ dequant_worker_fn ? vtcm_weight_raw[1] : vtcm_scratch1,
+ };
+ void *vtcm_weight_bufs[4] = {
+ dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0],
+ dequant_worker_fn ? vtcm_scratch1 : vtcm_weight_raw[1],
+ vtcm_scratch0,
+ vtcm_scratch1,
+ };
+ void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };
for (size_t mr = (size_t) m_start; mr < mr_end; mr += m_chunk_n_rows) {
const size_t n_rows = hex_smin(mr_end - mr, m_chunk_n_rows);
- void *vtcm_weight_bufs[2] = { vtcm_scratch0, vtcm_scratch1 };
- void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };
-
struct activation_transfer_params act_params = {
.ctx = ctx,
.dst = vtcm_f16_act,
@@ -3180,17 +3054,17 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
const int dst_cols = (int) dst->ne[0];
const int n_chunk_cnt = hmx_ceil_div(n, n_chunk_n_cols);
- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;
+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;
const size_t n_cols_A0 = hex_smin(n - 0 * n_chunk_n_cols, n_chunk_n_cols);
- const uint32_t height_A0 = is_quant ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;
- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight),
+ const uint32_t height_A0 = is_tiled ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight),
dma_dst_stride, dma_src_stride, dma_width_bytes, height_A0);
if (1 < n_chunk_cnt) {
const size_t n_cols_A1 = hex_smin(n - 1 * n_chunk_n_cols, n_chunk_n_cols);
- const uint32_t height_A1 = is_quant ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;
- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[1], weight + n_chunk_n_cols * weight_stride),
+ const uint32_t height_A1 = is_tiled ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[1], weight + n_chunk_n_cols * weight_stride),
dma_dst_stride, dma_src_stride, dma_width_bytes, height_A1);
}
@@ -3204,18 +3078,18 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
void * curr_raw = (void *) dma_queue_pop(weight_dma).dst;
dequantize_tiled_weight_chunk_to_fp16_tiles(
- ctx, vtcm_weight_bufs[i % 2], curr_raw,
+ ctx, vtcm_weight_bufs[i % 4], curr_raw,
n_cols, k, row_stride, weight_type,
n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);
if (i + 2 < n_chunk_cnt) {
- const uint32_t height_p2 = is_quant ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;
- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_p2 * weight_stride),
+ const uint32_t height_p2 = is_tiled ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 2) % 4], weight + nc_p2 * weight_stride),
dma_dst_stride, dma_src_stride, dma_width_bytes, height_p2);
}
hmx_matmul_job_init(&job_slots[i % 2], (__fp16 *) vtcm_output_bufs[i % 2],
- (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2],
+ (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 4],
vtcm_scales, hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS),
hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS), k / HTP_MM_HMX_TILE_N_ROWS);
hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job_slots[i % 2]));
@@ -3244,6 +3118,9 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
}
} else {
hmx_matmul_job_t job;
+ void *vtcm_raw_bufs[2] = { vtcm_weight_raw[0], dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0 };
+ void *vtcm_weight_bufs[2] = { dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0], vtcm_scratch0 };
+
for (size_t mr = (size_t) m_start; mr < mr_end; mr += m_chunk_n_rows) {
const size_t n_rows = hex_smin(mr_end - mr, m_chunk_n_rows);
@@ -3277,16 +3154,18 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
const size_t weight_stride = src_w->nb[1];
const size_t dst_stride = dst->nb[1] / sizeof(float);
const int dst_cols = (int) dst->ne[0];
+ const int n_chunk_cnt = hmx_ceil_div(n, n_chunk_n_cols);
- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;
+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;
if (n > 0) {
const size_t n_cols = hex_smin(n, n_chunk_n_cols);
- const uint32_t height = is_quant ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;
- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);
+ const uint32_t height = is_tiled ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);
}
- for (size_t nc = 0; nc < n; nc += n_chunk_n_cols) {
+ for (int i = 0; i < n_chunk_cnt; ++i) {
+ const size_t nc = (size_t) i * n_chunk_n_cols;
const size_t n_cols = hex_smin(n - nc, n_chunk_n_cols);
const size_t n_row_tiles = hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS);
const size_t n_col_tiles = hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS);
@@ -3294,18 +3173,18 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
void * curr_raw = (void *) dma_queue_pop(weight_dma).dst;
dequantize_tiled_weight_chunk_to_fp16_tiles(
- ctx, vtcm_scratch0, curr_raw,
+ ctx, vtcm_weight_bufs[i % 2], curr_raw,
n_cols, k, row_stride, weight_type,
n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);
const size_t nc_next = nc + n_chunk_n_cols;
if (nc_next < n) {
const size_t n_cols_next = hex_smin(n - nc_next, n_chunk_n_cols);
- const uint32_t height_next = is_quant ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;
- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);
+ const uint32_t height_next = is_tiled ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 1) % 2], weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);
}
- hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, vtcm_scratch0, vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);
+ hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2], vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);
hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job));
hmx_queue_pop(ctx->hmx_queue);
@@ -3322,30 +3201,30 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
return HTP_STATUS_OK;
}
-static inline dma_addr_t hmx_mm_weight_batch_data(const hmx_mm_f16_f32_batched_params_t *params,
+static inline dma_addr_t hmx_mm_weight_batch_data(const hmx_mm_batched_params_t *params,
int dst_b2, int dst_b3) {
const size_t b2_idx = (params->r2 <= 1) ? (size_t) dst_b2 : (size_t) fastdiv((uint32_t) dst_b2, ¶ms->div_r2);
const size_t b3_idx = (params->r3 <= 1) ? (size_t) dst_b3 : (size_t) fastdiv((uint32_t) dst_b3, ¶ms->div_r3);
return params->weight + b2_idx * params->src0_nb2 + b3_idx * params->src0_nb3;
}
-static inline dma_addr_t hmx_mm_act_batch_addr(const hmx_mm_f16_f32_batched_params_t *params,
+static inline dma_addr_t hmx_mm_act_batch_addr(const hmx_mm_batched_params_t *params,
int dst_b2, int dst_b3) {
return params->act_dma_addr + dst_b2 * params->act_nb2 +
dst_b3 * params->act_nb3;
}
-static inline float *hmx_mm_dst_batch_ptr(const hmx_mm_f16_f32_batched_params_t *params,
+static inline float *hmx_mm_dst_batch_ptr(const hmx_mm_batched_params_t *params,
int dst_b2, int dst_b3) {
return (float *) ((uint8_t *) params->dst +
(size_t) dst_b2 * params->dst_nb2 +
(size_t) dst_b3 * params->dst_nb3);
}
-static int hmx_mm_f16_f32_batched_simple(struct htp_context *ctx,
- const hmx_mm_f16_f32_batched_params_t *params,
- int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads, int vtcm_size,
- const struct fastdiv_values * act_threads_div, const struct fastdiv_values * k_div) {
+static int hmx_mm_batched_simple(struct htp_context *ctx,
+ const hmx_mm_batched_params_t *params,
+ int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads, int vtcm_size,
+ const struct fastdiv_values * act_threads_div, const struct fastdiv_values * k_div) {
int ret = 0;
for (int b3 = 0; b3 < params->ne13 && ret == 0; ++b3) {
for (int b2 = 0; b2 < params->ne12 && ret == 0; ++b2) {
@@ -3357,54 +3236,53 @@ static int hmx_mm_f16_f32_batched_simple(struct htp_context *ctx,
hmx_mm_act_batch_addr(params, b2, b3),
hmx_mm_weight_batch_data(params, b2, b3),
params->m, params->k, params->n,
- params->act_stride, params->act_elem_size, params->weight_stride * (int)sizeof(__fp16),
- HTP_TYPE_F16, params->k, params->dst_stride, params->bias_stride, params->n,
+ params->act_stride, params->act_elem_size, params->weight_stride,
+ params->weight_type, params->k, params->dst_stride, params->bias_stride, params->dst_cols,
m_chunk, n_chunk, pipeline, n_threads, act_threads,
- act_threads_div, k_div, 0, 0, vtcm_size);
+ act_threads_div, k_div, params->tile_size, params->aligned_tile_size, vtcm_size);
}
}
return ret;
}
-static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_batched_params_t *params,
- int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads,
- const struct fastdiv_values * act_threads_div,
- const struct fastdiv_values * k_div,
- int vtcm_size) {
- if (params->act_stride < params->k || params->weight_stride < params->k || params->dst_stride < params->n) { return -1; }
+static int hmx_mm_batched(struct htp_context *ctx, const hmx_mm_batched_params_t *params,
+ int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads,
+ const struct fastdiv_values * act_threads_div,
+ const struct fastdiv_values * k_div,
+ int vtcm_size) {
+ const size_t row_stride = htp_tensor_get_row_size(params->weight_type, params->k);
+ if (row_stride == 0) { return -1; }
+ if (params->act_stride < params->k || (params->tile_size == 0 && (size_t) params->weight_stride < row_stride) || params->dst_stride < params->dst_cols) { return -1; }
if (params->ne02 <= 0 || params->ne03 <= 0 || params->ne12 <= 0 || params->ne13 <= 0) { return -1; }
if (params->ne12 % params->ne02 != 0 || params->ne13 % params->ne03 != 0) { return -1; }
// N (the weight row count) does not have to be 32-aligned:
if (params->k % 32 != 0) { return -1; }
- if (!hex_is_aligned(params->dst, VLEN) || (params->act_dma_addr & (VLEN - 1)) != 0) { return -1; }
+ if (!hex_is_aligned(params->dst, sizeof(float))) { return -1; }
const int group_size = params->r2;
const size_t vtcm_budget = ctx->vtcm_size;
// Check if the precomputed parameters are grouped or simple.
// If simple, or if group_size <= 1, we use simple fallback loop.
- // Grouped path is only valid if group_size > 1 and it fits within VTCM budget.
- bool run_grouped = (group_size > 1 && (size_t)vtcm_size <= vtcm_budget);
+ // Grouped path is only valid for flat F16 if group_size > 1 and it fits within VTCM budget.
+ bool run_grouped = (params->weight_type == HTP_TYPE_F16 && params->tile_size == 0 && group_size > 1 && (size_t)vtcm_size <= vtcm_budget);
if (!run_grouped) {
- return hmx_mm_f16_f32_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);
+ return hmx_mm_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);
}
struct htp_thread_trace * tr = &ctx->trace[0];
htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);
- const size_t vec_dot_size = params->k * sizeof(__fp16);
-
size_t m_chunk_n_rows = m_chunk;
size_t n_chunk_n_cols = n_chunk;
- size_t vtcm_used = vtcm_size;
struct htp_mm_hmx_vtcm_layout L;
- htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_F16_BATCHED, HTP_TYPE_F16, params->k, m_chunk_n_rows, n_chunk_n_cols, group_size, false, act_threads, 0, params->bias_bytes);
+ htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_BATCHED, HTP_TYPE_F16, params->k, m_chunk_n_rows, n_chunk_n_cols, group_size, false, act_threads, 0, params->bias_bytes);
if (L.total_bytes > vtcm_budget) {
FARF(HIGH, "%s: grouped layout overflowed VTCM, falling back to simple batched loop", __func__);
htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);
- return hmx_mm_f16_f32_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);
+ return hmx_mm_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);
}
uint8_t * const base = (uint8_t *) ctx->vtcm_base;
@@ -3431,7 +3309,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_
L.total_bytes, vtcm_budget);
const size_t fp16_row_bytes = (size_t) params->k * sizeof(__fp16);
- const size_t weight_row_bytes = (size_t) params->weight_stride * sizeof(__fp16);
+ const size_t weight_row_bytes = (size_t) params->weight_stride;
htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);
@@ -3481,7 +3359,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_
}
if (n_chunk_n_cols < (size_t) params->n) {
const size_t n_cols_second = hex_smin((size_t) params->n - n_chunk_n_cols, n_chunk_n_cols);
- const dma_addr_t second_weight_chunk = weight_group + n_chunk_n_cols * params->weight_stride * sizeof(__fp16);
+ const dma_addr_t second_weight_chunk = weight_group + n_chunk_n_cols * weight_row_bytes;
dma_queue_push(weight_dma, dma_make_data(vtcm_scratch1, second_weight_chunk),
fp16_row_bytes, weight_row_bytes, fp16_row_bytes, n_cols_second);
}
@@ -3500,7 +3378,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_
const size_t nc_next = nc + n_chunk_n_cols * 2;
if (nc_next < (size_t) params->n) {
const size_t n_cols_next = hex_smin((size_t) params->n - nc_next, n_chunk_n_cols);
- const dma_addr_t next_weight_chunk = weight_group + nc_next * params->weight_stride * sizeof(__fp16);
+ const dma_addr_t next_weight_chunk = weight_group + nc_next * weight_row_bytes;
dma_queue_push(weight_dma, dma_make_data(curr_raw, next_weight_chunk),
fp16_row_bytes, weight_row_bytes, fp16_row_bytes, n_cols_next);
@@ -3519,7 +3397,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_
{
float *output = hmx_mm_dst_batch_ptr(params, b2_base + g, b3) + mr * params->dst_stride + nc;
const float *bias_chunk = has_bias ? (vtcm_bias + mr * params->bias_stride + nc) : NULL;
- int chunk_dst_cols = params->n - (int)nc;
+ int chunk_dst_cols = params->dst_cols - (int)nc;
if (chunk_dst_cols > 0) {
transfer_output_chunk_threaded(ctx, output, bias_chunk, vtcm_output, (int) n_rows, (int) n_cols,
params->dst_stride, params->bias_stride, chunk_dst_cols, n_threads);
@@ -3642,6 +3520,8 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,
size_t dst_nb1, size_t dst_nb2,
int weight_stride,
int weight_type,
+ int tile_size,
+ int aligned_tile_size,
const struct mmid_row_mapping *matrix_rows,
int cur_a,
int mapping_stride,
@@ -3658,47 +3538,27 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,
if (k % 32 != 0 || n % 32 != 0) { return -1; }
if (!hex_is_aligned(dst, VLEN) || !hex_is_aligned(activation, VLEN)) { return -1; }
- size_t row_stride = htp_mm_get_tiled_row_stride(weight_type, k);
+ size_t row_stride = htp_tensor_get_row_size(weight_type, k);
if (row_stride == 0) {
return -1;
}
- worker_callback_t dequant_worker_fn = NULL;
- switch (weight_type) {
- case HTP_TYPE_Q4_0: dequant_worker_fn = dequantize_tiled_worker_loop_q4_0; break;
- case HTP_TYPE_IQ4_NL: dequant_worker_fn = dequantize_tiled_worker_loop_iq4_nl; break;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K: dequant_worker_fn = dequantize_tiled_worker_loop_q4_1; break;
- case HTP_TYPE_MXFP4: dequant_worker_fn = dequantize_tiled_worker_loop_mxfp4; break;
- case HTP_TYPE_Q8_0: dequant_worker_fn = dequantize_tiled_worker_loop_q8_0; break;
- case HTP_TYPE_Q5_K: dequant_worker_fn = dequantize_tiled_worker_loop_q5_k; break;
- case HTP_TYPE_Q6_K: dequant_worker_fn = dequantize_tiled_worker_loop_q6_k; break;
- case HTP_TYPE_Q3_K: dequant_worker_fn = dequantize_tiled_worker_loop_q3_k; break;
- case HTP_TYPE_Q2_K: dequant_worker_fn = dequantize_tiled_worker_loop_q2_k; break;
- case HTP_TYPE_F16: dequant_worker_fn = convert_f16_worker_loop; break;
- case HTP_TYPE_F32: dequant_worker_fn = quantize_f32_worker_loop; break;
- default:
- return -1;
- }
-
const int n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;
const struct fastdiv_values n_k_tiles_div = init_fastdiv_values(n_k_tiles);
- const bool is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);
-
const size_t vec_dot_size = k * sizeof(__fp16);
const size_t vtcm_budget = ctx->vtcm_size;
size_t vtcm_used = 0;
- int tile_size = htp_mm_get_weight_tile_size(weight_type);
- int aligned_tile_size = htp_mm_get_weight_aligned_tile_size(weight_type);
+ const bool is_tiled = (tile_size > 0);
+ tiled_dequant_task_fn_t dequant_worker_fn = hmx_get_dequant_worker_fn(weight_type, is_tiled);
- const uint32_t dma_dst_stride = is_quant ? aligned_tile_size : row_stride;
- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;
- const uint32_t dma_width_bytes = is_quant ? tile_size : row_stride;
+ const uint32_t dma_dst_stride = is_tiled ? aligned_tile_size : row_stride;
+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;
+ const uint32_t dma_width_bytes = is_tiled ? tile_size : row_stride;
- const size_t qweight_row_stride = is_quant ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;
- const size_t weight_row_stride = is_quant ? qweight_row_stride : row_stride;
+ const size_t qweight_row_stride = is_tiled ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;
+ const size_t weight_row_stride = is_tiled ? qweight_row_stride : row_stride;
size_t size_per_n = 0, size_per_m = 0, size_per_mn = 0;
htp_mm_hmx_get_2d_chunk_costs(weight_type, k, /*pipeline=*/false, aligned_tile_size,
@@ -3738,6 +3598,9 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,
htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);
hmx_matmul_job_t job;
+ void *vtcm_raw_bufs[2] = { vtcm_weight, dequant_worker_fn ? vtcm_weight : vtcm_scratch0 };
+ void *vtcm_weight_bufs[2] = { dequant_worker_fn ? vtcm_scratch0 : vtcm_weight, vtcm_scratch0 };
+ const int n_chunk_cnt = hmx_ceil_div(n, n_chunk_n_cols);
for (size_t mr = (size_t) m_start; mr < (size_t) m_end; mr += m_chunk_n_rows) {
const size_t n_rows = hex_smin((size_t) m_end - mr, m_chunk_n_rows);
@@ -3750,12 +3613,13 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,
// A0: Pre-fetch the first weight chunk (nc = 0)
if (n > 0) {
const size_t n_cols = hex_smin((size_t) n, n_chunk_n_cols);
- const uint32_t height = is_quant ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;
- dma_queue_push(weight_dma, dma_make_data(vtcm_weight, weight),
+ const uint32_t height = is_tiled ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight),
dma_dst_stride, dma_src_stride, dma_width_bytes, height);
}
- for (size_t nc = 0; nc < (size_t) n; nc += n_chunk_n_cols) {
+ for (int i = 0; i < n_chunk_cnt; ++i) {
+ const size_t nc = (size_t) i * n_chunk_n_cols;
const size_t n_cols = hex_smin((size_t) n - nc, n_chunk_n_cols);
const size_t n_col_tiles = hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS);
@@ -3764,7 +3628,7 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,
// B: Weight Dequantize (Threaded)
dequantize_tiled_weight_chunk_to_fp16_tiles(
- ctx, vtcm_scratch0, curr_raw,
+ ctx, vtcm_weight_bufs[i % 2], curr_raw,
n_cols, k, row_stride, weight_type,
n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads
);
@@ -3773,13 +3637,13 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,
const size_t nc_next = nc + n_chunk_n_cols;
if (nc_next < (size_t) n) {
const size_t n_cols_next = hex_smin((size_t) n - nc_next, n_chunk_n_cols);
- const uint32_t height_next = is_quant ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;
- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_next * weight_stride),
+ const uint32_t height_next = is_tiled ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;
+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 1) % 2], weight + nc_next * weight_stride),
dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);
}
// C: HMX Compute (Queue-based)
- hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, vtcm_scratch0, vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);
+ hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2], vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);
hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job));
hmx_queue_pop(ctx->hmx_queue);
@@ -3803,7 +3667,6 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_k
const int m_total = (int) act->ne[1];
const uint32_t act_elem_size = (act->type == HTP_TYPE_F16) ? sizeof(__fp16) : sizeof(float);
const int act_stride = (int) (act->nb[1] / act_elem_size);
- const int wgt_stride = (int)(src0->nb[1] / sizeof(__fp16));
int m_start = 0;
int m_rows = m_total;
@@ -3842,46 +3705,50 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_k
int ret = -1;
const int n_threads = kparams->n_threads;
- if (kparams->kernel_type == HTP_MM_KERNEL_HMX_F16_BATCHED) {
- hmx_mm_f16_f32_batched_params_t batch_params = {
- .dst = dst_ptr,
- .bias_addr = src2_addr,
- .bias_bytes = src2_bytes,
- .act_dma_addr = act_addr,
- .weight = src0->data,
- .weight_dma = octx->ctx->dma[0],
- .m = m_rows,
- .k = k,
- .n = n,
- .act_stride = act_stride,
- .act_elem_size = act_elem_size,
- .weight_stride = wgt_stride,
- .dst_stride = dst_stride,
- .bias_stride = src2_stride,
- .ne02 = ne02,
- .ne03 = ne03,
- .ne12 = ne12,
- .ne13 = ne13,
- .src0_nb2 = src0->nb[2],
- .src0_nb3 = src0->nb[3],
- .act_nb2 = act->nb[2],
- .act_nb3 = act->nb[3],
- .dst_nb2 = dst->nb[2],
- .dst_nb3 = dst->nb[3],
- .bias_nb2 = src2_nb2,
- .bias_nb3 = src2_nb3,
- .r2 = (ne02 > 0) ? (ne12 / ne02) : 1,
- .r3 = (ne03 > 0) ? (ne13 / ne03) : 1,
- .div_r2 = kparams->div_r2,
- .div_r3 = kparams->div_r3,
+ if (kparams->kernel_type == HTP_MM_KERNEL_HMX_BATCHED) {
+ hmx_mm_batched_params_t batch_params = {
+ .dst = dst_ptr,
+ .bias_addr = src2_addr,
+ .bias_bytes = src2_bytes,
+ .act_dma_addr = act_addr,
+ .weight = src0->data,
+ .weight_dma = octx->ctx->dma[0],
+ .m = m_rows,
+ .k = k,
+ .n = n,
+ .act_stride = act_stride,
+ .act_elem_size = act_elem_size,
+ .weight_stride = (int) src0->nb[1],
+ .weight_type = (int) src0->type,
+ .dst_stride = dst_stride,
+ .dst_cols = (int) dst->ne[0],
+ .bias_stride = src2_stride,
+ .tile_size = kparams->tile_size,
+ .aligned_tile_size = kparams->aligned_tile_size,
+ .ne02 = ne02,
+ .ne03 = ne03,
+ .ne12 = ne12,
+ .ne13 = ne13,
+ .src0_nb2 = src0->nb[2],
+ .src0_nb3 = src0->nb[3],
+ .act_nb2 = act->nb[2],
+ .act_nb3 = act->nb[3],
+ .dst_nb2 = dst->nb[2],
+ .dst_nb3 = dst->nb[3],
+ .bias_nb2 = src2_nb2,
+ .bias_nb3 = src2_nb3,
+ .r2 = (ne02 > 0) ? (ne12 / ne02) : 1,
+ .r3 = (ne03 > 0) ? (ne13 / ne03) : 1,
+ .div_r2 = kparams->div_r2,
+ .div_r3 = kparams->div_r3,
};
- ret = hmx_mm_f16_f32_batched(octx->ctx, &batch_params,
- kparams->m_chunk, kparams->n_chunk,
- kparams->pipeline, n_threads,
- kparams->n_act_threads,
- &kparams->div_n_act_threads,
- &kparams->div_ne00_padded,
- kparams->vtcm_size);
+ ret = hmx_mm_batched(octx->ctx, &batch_params,
+ kparams->m_chunk, kparams->n_chunk,
+ kparams->pipeline, n_threads,
+ kparams->n_act_threads,
+ &kparams->div_n_act_threads,
+ &kparams->div_ne00_padded,
+ kparams->vtcm_size);
} else {
ret = hmx_mm_2d_f32(
octx->ctx, octx->ctx->dma[0], dst_ptr, src2_addr, src2_bytes,
@@ -4014,6 +3881,7 @@ static int hmx_mm_op_matmul_id(
nb11, nb12,
nb1, nb2,
(int) src0->nb[1], (int) src0->type,
+ kparams->tile_size, kparams->aligned_tile_size,
matrix_rows, cur_a, mmctx->mapping_stride,
m_start, m_end, (int) octx->n_threads);
if (ret != 0) {
@@ -4049,7 +3917,7 @@ static int hvx_mm_matmul_id(
uint32_t n_quant_tasks = 1;
if (act_nrows < octx->n_threads) {
n_quant_tasks = MIN(total_nb, octx->n_threads);
- quant_task_func = htp_mm_act_quant_block_func(src0->type);
+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);
for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {
uint32_t ib_first = (total_nb * ith) / n_quant_tasks;
uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;
@@ -4060,9 +3928,9 @@ static int hvx_mm_matmul_id(
}
} else {
n_quant_tasks = MIN(act_nrows, octx->n_threads);
- quant_task_func = htp_mm_act_quant_row_func(src0->type);
+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);
}
- size_t act_row_size = htp_mm_weight_has_offset(src0->type) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);
+ size_t act_row_size = htp_mm_tiled_act_row_size(src0->type, ne10);
struct htp_mm_hvx_vtcm_layout L;
htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, ne10, act_nrows, octx->n_threads,
@@ -4177,6 +4045,7 @@ static int hmx_mm_op_matmul_id_nx(
act->nb[1], act->nb[2],
dst->nb[1], dst->nb[2],
(int) src_w->nb[1], (int) src_w->type,
+ kparams->tile_size, kparams->aligned_tile_size,
matrix_rows, cur_a, mmctx->mapping_stride,
m_start, m_end, (int) octx->n_threads);
if (ret != 0) {
@@ -4215,7 +4084,7 @@ static int hvx_mm_matmul_id_nx(
uint32_t n_quant_tasks = 1;
if (act_nrows < octx->n_threads) {
n_quant_tasks = MIN(total_nb, octx->n_threads);
- quant_task_func = htp_mm_act_quant_block_func(src0->type);
+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);
for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {
uint32_t ib_first = (total_nb * ith) / n_quant_tasks;
uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;
@@ -4226,9 +4095,9 @@ static int hvx_mm_matmul_id_nx(
}
} else {
n_quant_tasks = MIN(act_nrows, octx->n_threads);
- quant_task_func = htp_mm_act_quant_row_func(src0->type);
+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);
}
- size_t act_row_size = htp_mm_weight_has_offset(src0->type) ? htp_mm_q8_1_tiled_row_size(act->ne[0]) : htp_mm_q8_0_tiled_row_size(act->ne[0]);
+ size_t act_row_size = htp_mm_tiled_act_row_size(src0->type, act->ne[0]);
struct htp_mm_hvx_vtcm_layout L;
htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, act->ne[0], act_nrows, octx->n_threads,
@@ -4451,8 +4320,8 @@ int op_matmul_id(struct htp_ops_context * octx) {
mmctx->src0_nrows_per_thread = fastdiv(nrows + octx->n_threads - 1, &octx->n_threads_div);
mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);
- if (hvx_mm_init_vec_dot(mmctx, src0->type) == 0) {
- s = hvx_mm_matmul_id(octx, mmctx, act_nrows > 1 ? hvx_mm_id : hvx_mv_id);
+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) == 0) {
+ s = hvx_mm_matmul_id(octx, mmctx, act_nrows > 1 ? hvx_mm_id_tiled : hvx_mv_id_tiled);
} else {
s = HTP_STATUS_NO_SUPPORT;
}
@@ -4551,8 +4420,8 @@ int op_matmul_id_nx(struct htp_ops_context * octx) {
if (kparams->n_hmx) {
s = hmx_mm_op_matmul_id_nx(octx, mmctx);
} else {
- if (hvx_mm_init_vec_dot(mmctx, src0->type) == 0) {
- s = hvx_mm_matmul_id_nx(octx, mmctx, act_nrows > 1 ? hvx_mm_id_nx : hvx_mv_id_nx);
+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) == 0) {
+ s = hvx_mm_matmul_id_nx(octx, mmctx, act_nrows > 1 ? hvx_mm_id_nx_tiled : hvx_mv_id_nx_tiled);
} else {
s = HTP_STATUS_NO_SUPPORT;
}
@@ -4584,12 +4453,6 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {
const struct htp_tensor * restrict src0 = octx->src[0]; // first weight
const struct htp_tensor * restrict act = octx->src[n_weights]; // activation x
- bool is_repacked = (src0->type == HTP_TYPE_Q4_0 || src0->type == HTP_TYPE_Q4_1 ||
- src0->type == HTP_TYPE_Q8_0 || src0->type == HTP_TYPE_IQ4_NL ||
- src0->type == HTP_TYPE_MXFP4 || src0->type == HTP_TYPE_Q4_K ||
- src0->type == HTP_TYPE_Q5_K || src0->type == HTP_TYPE_Q3_K ||
- src0->type == HTP_TYPE_Q2_K);
-
struct htp_mm_context mmctx_struct = {0};
struct htp_mm_context * mmctx = &mmctx_struct;
mmctx->octx = octx;
@@ -4601,9 +4464,8 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {
mmctx->cur_m_rows = act_nrows;
const size_t src0_row_size = src0->nb[1];
- const size_t src0_row_size_padded = hex_round_up(src0_row_size, 128);
- if (hvx_mm_init_vec_dot(mmctx, src0->type) != 0) {
+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) != 0) {
return HTP_STATUS_NO_SUPPORT;
}
@@ -4615,7 +4477,7 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {
uint32_t n_quant_tasks = 1;
if (act_nrows < octx->n_threads) {
n_quant_tasks = MIN(total_nb, octx->n_threads);
- quant_task_func = htp_mm_act_quant_block_func(src0->type);
+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);
for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {
uint32_t ib_first = (total_nb * ith) / n_quant_tasks;
uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;
@@ -4626,12 +4488,10 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {
}
} else {
n_quant_tasks = MIN(act_nrows, octx->n_threads);
- quant_task_func = htp_mm_act_quant_row_func(src0->type);
+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);
}
- const size_t act_row_size = htp_mm_weight_has_offset(src0->type)
- ? htp_mm_q8_1_tiled_row_size(act->ne[0])
- : htp_mm_q8_0_tiled_row_size(act->ne[0]);
+ const size_t act_row_size = htp_mm_tiled_act_row_size(src0->type, act->ne[0]);
struct htp_mm_hvx_vtcm_layout L;
htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, act->ne[0], act_nrows, octx->n_threads,
@@ -4651,7 +4511,7 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {
mmctx->vtcm_dst = VTCM_LAYOUT_PTR(uint8_t, base, L.off_dst);
mmctx->vtcm_act_raw = VTCM_LAYOUT_PTR(uint8_t, base, L.off_act_raw);
- mmctx->vtcm_src0_stride = is_repacked ? 0 : src0_row_size_padded;
+ mmctx->vtcm_src0_stride = 0;
mmctx->vtcm_act_stride = act_row_size;
mmctx->vtcm_act_raw_stride = hex_round_up(act->ne[0] * sizeof(float), QK_Q8_0_TILED * sizeof(float));
@@ -4659,28 +4519,6 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {
mmctx->vtcm_act_size_per_thread = L.act_bytes;
mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->n_threads_div);
- // Run fused matmul
- const uint32_t n_matmul_jobs = octx->n_threads;
- worker_callback_t matmul_job_func;
- if (is_repacked) {
- switch (src0->type) {
- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mm_nx_2d_repacked_q4_0; break;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mm_nx_2d_repacked_q4_1; break;
- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mm_nx_2d_repacked_q5_k; break;
- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mm_nx_2d_repacked_q3_k; break;
- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mm_nx_2d_repacked_q2_k; break;
- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mm_nx_2d_repacked_q8_0; break;
- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mm_nx_2d_repacked_iq4nl; break;
- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mm_nx_2d_repacked_mxfp4; break;
- default:
- htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);
- return HTP_STATUS_NO_SUPPORT;
- }
- } else {
- matmul_job_func = hvx_mm_nx_2d;
- }
-
htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);
hvx_mm_transfer_act_dma(octx, kparams, act, mmctx->vtcm_act_raw, mmctx->vtcm_act_raw_stride, 0, act_nrows);
@@ -4689,7 +4527,7 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {
mmctx->n_quant_tasks = n_quant_tasks;
work_queue_run(octx->ctx->work_queue, quant_task_func, mmctx, n_quant_tasks);
- work_queue_run(octx->ctx->work_queue, matmul_job_func, mmctx, n_matmul_jobs);
+ work_queue_run(octx->ctx->work_queue, hvx_mm_nx_2d_tiled, mmctx, octx->n_threads);
return HTP_STATUS_OK;
}
diff --git a/ggml/src/ggml-hexagon/htp/matmul-ops.h b/ggml/src/ggml-hexagon/htp/matmul-ops.h
index 522d8779f..339eea4a7 100644
--- a/ggml/src/ggml-hexagon/htp/matmul-ops.h
+++ b/ggml/src/ggml-hexagon/htp/matmul-ops.h
@@ -4,6 +4,7 @@
#include <stdint.h>
#include <stddef.h>
#include "htp-ops.h"
+#include "htp-tensor.h"
#include "hex-fastdiv.h"
#include "hex-common.h"
#include "htp-vtcm.h"
@@ -19,45 +20,10 @@ extern "C" {
#define HTP_MM_HMX_TILE_N_ELMS 1024
#define HTP_MM_HMX_MIN_NROWS 4
-// --- Weight Repacked Tile Sizes ---
-#define HTP_MM_WEIGHT_TILE_SIZE_Q4_0 576
-#define HTP_MM_WEIGHT_TILE_SIZE_Q4_1 640
-#define HTP_MM_WEIGHT_TILE_SIZE_Q8_0 1088
-#define HTP_MM_WEIGHT_TILE_SIZE_IQ4_NL 576
-#define HTP_MM_WEIGHT_TILE_SIZE_MXFP4 544
-// Q5_K: the Q4_1 tile (640) followed by a 128-byte plane with the 5th bit of every quant, transposed so that
-// plane byte l holds the eight flags of lane l: bit 2i = low nibble of nibble vector i, bit 2i+1 = high nibble
-#define HTP_MM_WEIGHT_TILE_SIZE_Q5_K 768
-// Q6_K native 6-bit tile (32 rows x 32 k), vrmpy-ready: byte 4*row+b of a vector holds k = 4*group+b
-// vectors 0..3: low nibbles, vector i holds group 2i (low nibble) and group 2i+1 (high nibble)
-// vectors 4..5: high 2 bits, vector m holds groups 4m..4m+3 at bit offsets 0,2,4,6
-// vector 6: fp16 scales per row, d * scales[]: k 0..15 in lanes 0..31, k 16..31 in lanes 32..63
-#define HTP_MM_WEIGHT_TILE_SIZE_Q6_K 896
-// Q3_K native 3-bit tile, vrmpy-ready like Q6_K
-// vectors 0..1: low 2 bits, vector m holds groups 4m..4m+3 at bit offsets 0,2,4,6
-// vector 2: bit g set where the hmask bit of group g is clear (quant = low 2 bits - 4)
-// vector 3: fp16 scales per row, d * (scales[] - 32): k 0..15 in lanes 0..31, k 16..31 in lanes 32..63
-#define HTP_MM_WEIGHT_TILE_SIZE_Q3_K 512
-// Q2_K native 2-bit tile with the ggml scale bytes, vrmpy-ready like Q6_K (same bits per weight as block_q2_K)
-// bytes 0..255: unsigned 2-bit quants, vector m holds groups 4m..4m+3 at bit offsets 0,2,4,6
-// bytes 256..319: scales[] bytes (min << 4 | scale), sub-block s of row r at 32 * s + r
-// bytes 320..335: 8 fp16, tile j of a super-block: d (j < 4) or -dmin (j >= 4) of rows 8 * (j % 4) .. 8 * (j % 4) + 7
-#define HTP_MM_WEIGHT_TILE_SIZE_Q2_K 336
-
-// --- Weight Repacked Aligned Tile Sizes ---
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_0 640
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_1 640
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q8_0 1152
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_IQ4_NL 640
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_MXFP4 640
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q5_K 768
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q6_K 896
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q3_K 512
-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q2_K 384
-
// --- Activation Tiled Block Sizes (including padding) ---
#define HTP_MM_ACT_TILE_SIZE_Q8_0 1152
#define HTP_MM_ACT_TILE_SIZE_Q8_1 1280
+#define HTP_MM_ACT_TILE_SIZE_F16 2048
#define HTP_MM_MAX_PREFETCH 16
@@ -74,11 +40,12 @@ enum htp_mm_kernel_type {
// HMX paths
HTP_MM_KERNEL_HMX_2D,
- HTP_MM_KERNEL_HMX_F16_BATCHED,
+ HTP_MM_KERNEL_HMX_BATCHED,
// HVX floating-point paths
HTP_MM_KERNEL_HVX_F16_F16_VTCM,
HTP_MM_KERNEL_HVX_F32_F32_VTCM,
+ HTP_MM_KERNEL_HVX_BF16_F32_VTCM,
// HVX quantized paths
HTP_MM_KERNEL_HVX_QUANT_ROW, // standard row-wise parallel quantization
@@ -212,57 +179,6 @@ next_nc:
return 0;
}
-// --- Tile Size Helpers ---
-static inline uint32_t htp_mm_get_weight_tile_size(int weight_type) {
- switch (weight_type) {
- case HTP_TYPE_Q4_0:
- case HTP_TYPE_IQ4_NL:
- return HTP_MM_WEIGHT_TILE_SIZE_Q4_0;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K:
- return HTP_MM_WEIGHT_TILE_SIZE_Q4_1;
- case HTP_TYPE_Q8_0:
- return HTP_MM_WEIGHT_TILE_SIZE_Q8_0;
- case HTP_TYPE_Q5_K:
- return HTP_MM_WEIGHT_TILE_SIZE_Q5_K;
- case HTP_TYPE_Q6_K:
- return HTP_MM_WEIGHT_TILE_SIZE_Q6_K;
- case HTP_TYPE_Q3_K:
- return HTP_MM_WEIGHT_TILE_SIZE_Q3_K;
- case HTP_TYPE_Q2_K:
- return HTP_MM_WEIGHT_TILE_SIZE_Q2_K;
- case HTP_TYPE_MXFP4:
- return HTP_MM_WEIGHT_TILE_SIZE_MXFP4;
- default:
- return 0;
- }
-}
-
-static inline uint32_t htp_mm_get_weight_aligned_tile_size(int weight_type) {
- switch (weight_type) {
- case HTP_TYPE_Q4_0:
- case HTP_TYPE_IQ4_NL:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_0;
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_1;
- case HTP_TYPE_Q8_0:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q8_0;
- case HTP_TYPE_Q5_K:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q5_K;
- case HTP_TYPE_Q6_K:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q6_K;
- case HTP_TYPE_Q3_K:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q3_K;
- case HTP_TYPE_Q2_K:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q2_K;
- case HTP_TYPE_MXFP4:
- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_MXFP4;
- default:
- return 0;
- }
-}
-
// weight types whose tiles carry a per-block offset (x = d * q + m): the activations need block sums (q8_1)
// (Q2_K: per-16 k sums, q8_1_s16)
static inline bool htp_mm_weight_has_offset(int weight_type) {
@@ -283,27 +199,17 @@ static inline size_t htp_mm_q8_1_tiled_row_size(uint32_t ne) {
return nb_32 * HTP_MM_ACT_TILE_SIZE_Q8_1;
}
-static inline size_t htp_mm_get_tiled_row_stride(int weight_type, uint32_t k) {
- uint32_t nb = (k + QK_Q4_0_TILED - 1) / QK_Q4_0_TILED;
- switch (weight_type) {
- case HTP_TYPE_Q4_0:
- case HTP_TYPE_IQ4_NL:
- case HTP_TYPE_Q4_1:
- case HTP_TYPE_Q4_K:
- case HTP_TYPE_Q8_0:
- case HTP_TYPE_Q5_K:
- case HTP_TYPE_Q6_K:
- case HTP_TYPE_Q3_K:
- case HTP_TYPE_Q2_K:
- case HTP_TYPE_MXFP4:
- return (size_t) nb * htp_mm_get_weight_tile_size(weight_type);
- case HTP_TYPE_F16:
- return (size_t) k * sizeof(__fp16);
- case HTP_TYPE_F32:
- return (size_t) k * sizeof(float);
- default:
- return 0;
+static inline size_t htp_mm_f16_tiled_row_size(uint32_t ne) {
+ const uint32_t ne_padded = ((ne + 127) / 128) * 128;
+ const uint32_t nb_32 = ne_padded / 32;
+ return nb_32 * HTP_MM_ACT_TILE_SIZE_F16;
+}
+
+static inline size_t htp_mm_tiled_act_row_size(int weight_type, uint32_t ne) {
+ if (weight_type == HTP_TYPE_F16) {
+ return htp_mm_f16_tiled_row_size(ne);
}
+ return htp_mm_weight_has_offset(weight_type) ? htp_mm_q8_1_tiled_row_size(ne) : htp_mm_q8_0_tiled_row_size(ne);
}
static inline size_t htp_mm_round_up(size_t n, size_t m) {
@@ -318,8 +224,8 @@ static inline void htp_mm_hmx_get_2d_chunk_costs(
int wtype, uint32_t k, bool pipeline, uint32_t aligned_tile_size,
size_t * size_per_n_out, size_t * size_per_m_out, size_t * size_per_mn_out
) {
- const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32);
- const size_t row_stride = htp_mm_get_tiled_row_stride(wtype, k);
+ const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32 && wtype != HTP_TYPE_BF16);
+ const size_t row_stride = htp_tensor_get_row_size(wtype, k);
const size_t vec_dot_size = k * sizeof(uint16_t);
const uint32_t n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;
const size_t qweight_row_stride = is_quant ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;
@@ -404,7 +310,7 @@ static inline void htp_mm_hmx_vtcm_layout_build(
) {
size_t off = 0;
- if (kernel_type == HTP_MM_KERNEL_HMX_F16_BATCHED) {
+ if (kernel_type == HTP_MM_KERNEL_HMX_BATCHED) {
const size_t vec_dot_size = k * sizeof(uint16_t);
const size_t act_head_stride = mc * k;
const size_t weight_area_size = hex_align_up(nc * vec_dot_size, HTP_MM_HMX_TILE_SIZE);
@@ -450,8 +356,8 @@ static inline void htp_mm_hmx_vtcm_layout_build(
off = off_group_a + hex_smax(group_b_size, group_c_size);
} else {
// HTP_MM_KERNEL_HMX_2D
- const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32);
- const size_t row_stride = htp_mm_get_tiled_row_stride(wtype, k);
+ const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32 && wtype != HTP_TYPE_BF16);
+ const size_t row_stride = htp_tensor_get_row_size(wtype, k);
const size_t vec_dot_size = k * sizeof(uint16_t);
const uint32_t n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;
@@ -527,28 +433,13 @@ static inline void htp_mm_hvx_vtcm_layout_build(
size_t dst_sz = 0;
size_t act_raw_sz = 0;
- const bool is_repack = (wtype == HTP_TYPE_Q4_0 || wtype == HTP_TYPE_Q4_1 ||
- wtype == HTP_TYPE_Q8_0 || wtype == HTP_TYPE_IQ4_NL ||
- wtype == HTP_TYPE_MXFP4 || wtype == HTP_TYPE_Q6_K ||
- wtype == HTP_TYPE_Q4_K || wtype == HTP_TYPE_Q5_K ||
- wtype == HTP_TYPE_Q3_K || wtype == HTP_TYPE_Q2_K);
-
if (is_fused_nx) {
- const size_t src0_row_size_padded = hex_round_up(src0_row_size, 128);
+ uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);
+ uint32_t n_k_tiles = hex_round_up(ne10, 32) / 32;
+ uint32_t tile_row_size = n_k_tiles * aligned_tile_size;
+ size_t weight_sz_per_thread = hex_round_up(n_prefetch * tile_row_size, 128);
- size_t weight_sz_per_thread = 0;
-
- if (is_repack) {
- uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);
- uint32_t n_k_tiles = hex_round_up(ne10, 32) / 32;
- uint32_t tile_row_size = n_k_tiles * aligned_tile_size;
-
- weight_sz_per_thread = hex_round_up(n_prefetch * tile_row_size, 128);
- } else {
- weight_sz_per_thread = hex_round_up(n_prefetch * src0_row_size_padded, 128);
- }
-
- size_t tiled_act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);
+ size_t tiled_act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);
size_t q_act_sz = hex_round_up(tiled_act_row_size * act_nrows, 128);
size_t raw_row_size = hex_round_up(ne10 * sizeof(float), QK_Q8_0_TILED * sizeof(float));
@@ -558,20 +449,13 @@ static inline void htp_mm_hvx_vtcm_layout_build(
dst_sz = 0;
act_raw_sz = hex_round_up(raw_row_size * act_nrows, 128);
} else if (is_matmul_id) {
- const size_t src0_row_size_padded = htp_mm_round_up(src0_row_size, 128);
- const size_t act_row_size_tiled = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10)
- : htp_mm_q8_0_tiled_row_size(ne10);
-
- size_t src0_sz_per_thread = htp_mm_round_up(n_prefetch * src0_row_size_padded, 256);
- act_sz = htp_mm_round_up(act_row_size_tiled * act_nrows, 256);
-
- if (is_repack) {
- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);
- const uint32_t n_k_tiles = ne10 / 32;
- const uint32_t tile_row_size = n_k_tiles * aligned_tile_size;
- size_t repacked_vtcm_size = htp_mm_round_up(n_prefetch * tile_row_size, 256);
- src0_sz_per_thread = repacked_vtcm_size;
- }
+ const size_t act_row_size_tiled = htp_mm_tiled_act_row_size(wtype, ne10);
+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);
+ const uint32_t n_k_tiles = ne10 / 32;
+ const uint32_t tile_row_size = n_k_tiles * aligned_tile_size;
+ size_t src0_sz_per_thread = htp_mm_round_up(n_prefetch * tile_row_size, 256);
+
+ act_sz = htp_mm_round_up(act_row_size_tiled * act_nrows, 256);
size_t raw_row_size = hex_round_up(ne10 * sizeof(float), QK_Q8_0_TILED * sizeof(float));
@@ -592,7 +476,8 @@ static inline void htp_mm_hvx_vtcm_layout_build(
act_raw_sz = hex_round_up(hex_round_up(ne10 * sizeof(float), 128) * act_nrows, 128);
break;
}
- case HTP_MM_KERNEL_HVX_F32_F32_VTCM: {
+ case HTP_MM_KERNEL_HVX_F32_F32_VTCM:
+ case HTP_MM_KERNEL_HVX_BF16_F32_VTCM: {
size_t f32_act_row_size = htp_mm_round_up(ne10 * 4, 128);
act_sz = htp_mm_round_up(f32_act_row_size * act_nrows, 256);
src0_sz = htp_mm_round_up(n_prefetch * src0_row_size_padded, 256) * n_threads;
@@ -602,21 +487,15 @@ static inline void htp_mm_hvx_vtcm_layout_build(
}
case HTP_MM_KERNEL_HVX_QUANT_BLOCK:
case HTP_MM_KERNEL_HVX_QUANT_ROW: {
- size_t q_act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);
+ size_t q_act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);
+ uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);
+ uint32_t n_k_tiles = (ne10 + 31) / 32;
+ uint32_t tile_row_size = n_k_tiles * aligned_tile_size;
+ size_t tiled_vtcm_size = htp_mm_round_up(n_prefetch * tile_row_size, 256);
- src0_sz = htp_mm_round_up(n_prefetch * src0_row_size_padded, 256);
+ src0_sz = tiled_vtcm_size * n_threads;
act_sz = htp_mm_round_up(q_act_row_size * act_nrows, 256);
- src0_sz = src0_sz * n_threads;
-
- if (is_repack) {
- uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);
- uint32_t n_k_tiles = ne10 / 32;
- uint32_t tile_row_size = n_k_tiles * aligned_tile_size;
- size_t repacked_vtcm_size = htp_mm_round_up(n_prefetch * tile_row_size, 256);
- src0_sz = repacked_vtcm_size * n_threads;
- }
-
size_t dst_slice_per_thread = (dst_nrows > 0 && act_nrows == 1) ? htp_mm_round_up((dst_row_size + n_threads - 1) / n_threads, 128) : 0;
dst_sz = dst_slice_per_thread * n_threads;
size_t raw_row_size = hex_round_up(ne10 * sizeof(float), QK_Q8_0_TILED * sizeof(float));
@@ -687,9 +566,7 @@ static inline bool htp_mm_hvx_solve_vtcm_params(
const size_t avail_act = vtcm_budget - fixed_bytes;
size_t row_size = 0;
if (kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW || kernel_type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {
- row_size = htp_mm_weight_has_offset(wtype)
- ? htp_mm_q8_1_tiled_row_size(ne10)
- : htp_mm_q8_0_tiled_row_size(ne10);
+ row_size = htp_mm_tiled_act_row_size(wtype, ne10);
} else if (kernel_type == HTP_MM_KERNEL_HVX_F16_F16_VTCM) {
row_size = hex_round_up(ne10 * 2, 128);
} else {
@@ -749,7 +626,7 @@ static inline size_t htp_mm_hmx_get_batched_vtcm_size(
int wtype, uint32_t k, size_t mc, size_t nc, uint32_t group_size, bool pipeline, uint32_t act_threads, size_t bias_size) {
(void)pipeline;
struct htp_mm_hmx_vtcm_layout L;
- htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_F16_BATCHED, wtype, k, mc, nc, group_size, false, act_threads, 0, bias_size);
+ htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_BATCHED, wtype, k, mc, nc, group_size, false, act_threads, 0, bias_size);
return L.total_bytes;
}
diff --git a/scripts/snapdragon/run.py b/scripts/snapdragon/run.py
index dec4b996f..7e11d90de 100755
--- a/scripts/snapdragon/run.py
+++ b/scripts/snapdragon/run.py
@@ -27,6 +27,7 @@ MANAGED_ENV_NAMES = (
"GGML_HEXAGON_OPPOLL",
"GGML_HEXAGON_OPFILTER",
"GGML_HEXAGON_OPFUSION",
+ "GGML_HEXAGON_RT_REPACK",
"GGML_HEXAGON_VMEM",
"GGML_HEXAGON_MBUF",
"GGML_HEXAGON_MM_SELECT",
@@ -167,6 +168,7 @@ def main():
parser.add_argument("--hex-oppoll", default="1", help="Enable (1) or Disable (0) polling for NPU opbatch completion (GGML_HEXAGON_OPPOLL) (default: 1)")
parser.add_argument("--hex-opfilter", help="Regex pattern to filter/select which operators are offloaded to NPU (GGML_HEXAGON_OPFILTER)")
parser.add_argument("--hex-opfusion", help="NPU graph node fusion optimization level (0: disabled, 1: enabled) (GGML_HEXAGON_OPFUSION)")
+ parser.add_argument("--hex-rt-repack", help="Runtime weight repacking mode (0: disabled, 1: quantized only, 2: always) (GGML_HEXAGON_RT_REPACK)")
parser.add_argument("--hex-vmem", help="Maximum NPU VMEM size limit in MB to allocate (GGML_HEXAGON_VMEM)")
parser.add_argument("--hex-mbuf", help="Host buffer size limits in MB (supports K/M/G suffix): <dyn>[,<static>[,<total>]] (default: 512,1024,0) (GGML_HEXAGON_MBUF)")
parser.add_argument("--hex-mm-select", help="Select MUL_MAT and MUL_MAT_ID kernel (GGML_HEXAGON_MM_SELECT) 2:HMX,1:HVX,0:disable")
@@ -311,6 +313,7 @@ def main():
set_env("GGML_HEXAGON_OPPOLL", args.hex_oppoll)
set_env("GGML_HEXAGON_OPFILTER", args.hex_opfilter)
set_env("GGML_HEXAGON_OPFUSION", args.hex_opfusion)
+ set_env("GGML_HEXAGON_RT_REPACK", args.hex_rt_repack)
set_env("GGML_HEXAGON_VMEM", args.hex_vmem)
set_env("GGML_HEXAGON_MBUF", args.hex_mbuf)
set_env("GGML_HEXAGON_MM_SELECT", args.hex_mm_select)
diff --git a/tests/test-backend-ops.cpp b/tests/test-backend-ops.cpp
index ba7f7eb4e..b50364660 100644
--- a/tests/test-backend-ops.cpp
+++ b/tests/test-backend-ops.cpp
@@ -10741,11 +10741,14 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
// testing very large N
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F32, 16, 2105344, 32, {1, 1}, {1, 1}));
+
+#if 0
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 16, 8400000, 32, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_0, GGML_TYPE_F32, 16, 8400000, 32, {1, 1}, {1, 1}));
// same shapes with a repeat in dim 3
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 16, 8400000, 32, {1, 1}, {2, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_0, GGML_TYPE_F32, 16, 8400000, 32, {1, 1}, {2, 1}));
+#endif
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 31, 509, 2051, {1, 1}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 32, 509, 2112, {1, 1}, {1, 1}));