Commit 4c4a0f0a4 for llama.cpp
commit 4c4a0f0a4246bc36a3724d12238d5624714dcb2f
Author: Aman Gupta <amangupta052@gmail.com>
Date: Fri Nov 20 16:13:00 2026 +0800
llama : reserve graph nodes for the MoE cache (#32636)
The MoE cache adds a slot lookup to every MoE layer, and with
--moe-cache-cpu-share a second expert path, but graph_max_nodes() did
not account for it. Models on the default 8*n_tensors budget with few
tensors per layer (cohere2moe: 443 tensors, 49 layers) overflow the
graph context when the cache is combined with a quantized KV cache
(K/V rotation), aborting in ggml_new_tensor with GGML_ASSERT(obj_new).
Fixes #32584
Assisted-by: Claude Opus 5.5
diff --git a/src/llama-context.cpp b/src/llama-context.cpp
index 6252147a7..7f1cc4c44 100644
--- a/src/llama-context.cpp
+++ b/src/llama-context.cpp
@@ -2949,6 +2949,12 @@ uint32_t llama_context::graph_max_nodes() const {
}
}
+ // the MoE cache adds a lookup to each MoE layer, and a second expert path with a CPU share (up to ~16 tensors per layer),
+ // which the per-tensor estimates above do not cover for models with few tensors per layer
+ if (cparams.moe_cache_size > 0) {
+ res += 32u*model.hparams.n_layer_all;
+ }
+
// every sampler builds n_outputs_max_per_seq chains, see llm_graph_context::build_sampling
for (const auto & [seq_id, sampler] : sampling.samplers) {
res += llama_sampler_backend_n_nodes(sampler) * cparams.n_outputs_max_per_seq;