Combining --enable-prefix-caching with the DFlash drafter crashed engine
init with "block_size must be divisible by hash_block_size" in
HybridKVCacheCoordinator. The drafter's attention KV page is ~2x the
target's, so vLLM's page-size unification scales the target's mamba+attn
block 2240->4480 to match; the (align-mode) mamba block then differs from
cache_config.block_size and resolve_kv_cache_block_sizes backs off to
hash_block_size = LCM (4480), which the drafter group (still 2240) is not
divisible by. But the GCD (2240) divides every group and is the correct
finer hash granularity; the back-off's `block_size != cache_block` test is
a buggy proxy for "non-align mamba".
patch_prefix_align.py makes the back-off align-aware (only back off when
mamba_cache_mode != "align"), so resolve uses the GCD. Prefix caching is on
by default (a win for multi-turn / long-context, neutral for single-turn
c=1); set PREFIX_CACHE=0 to disable.
Validated on GB10: READY, DFlash accept ~7.7 tok/step on code, ~13x
warm-prefix TTFT (2.30s->0.18s), KV pool ~422k tokens (no regression).
See docs/FINDINGS.md section 1a.
A forum user hit HTTP 400 "'auto' tool choice requires --enable-auto-tool-choice
and --tool-call-parser to be set" — the server shipped without tool-calling
enabled, which is exactly what agent/tool workloads (the whole point) need.
- serve.sh + mtp_serve.sh now pass --enable-auto-tool-choice --tool-call-parser
qwen3_xml --reasoning-parser qwen3 by default (overridable/disable via
TOOL_PARSER= / REASONING_PARSER=).
- Parser matters: Qwen3.5 emits the XML tool format
(<function=name><parameter=k>v</parameter></function>), so hermes returns 200
but with EMPTY tool_calls (the call lands in content). qwen3_xml parses it.
Verified on the box: tool_choice=auto -> tool_calls=[get_weather {city:Paris}];
plain chat unaffected; <think> -> reasoning_content.
- README: document tool-calling defaults + that the server binds 0.0.0.0
(LAN-reachable at http://<spark-ip>:8000; no auth).
Reclaims over-reserved memory rather than spending headroom, so it honors
the conservative 0.82 / 14 GB-OS-reserve choice (still ~15 GiB free under
peak load) while closing most of the gap to the dflash pool (456k):
- int8 lm-head: build int8 on first warmup forward, then free the dead bf16
copy (~1.4 GiB) + empty_cache() so the block lands in the KV pool. Safe:
the DFlash drafter shares the one int8 lm_head module (verified one int8
build; drafter ckpt has no lm_head/embed) and tie_word_embeddings=False,
so bf16 has no reader (bias fallback = int8-GEMV + add-bias).
SPARK_KEEP_BF16_LMHEAD=1 restores keep-bf16.
- VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 in serve.sh/mtp_serve.sh:
returns the CUDA-graph over-estimate (~0.6 GiB; actual capture ~0.14 GiB
from the wide 0.82 headroom). Overridable with =1.
- scripts/stress_mem.sh: drive the concurrency bank while sampling host RAM
to prove a gpu-mem setting survives peak load without swapping.
Validated: pool 426,610 tokens, coherent output, drafter acceptance 4-12,
no new swap, throughput unchanged. max-batched-tokens 8192->4096 tested as
a KV lever and rejected (only +2.5k tokens, costs prefill speed).
Entrpi