TBNilles/qwen3.5-122B-A10B-on-spark
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

kv: recover dense pool 376k->427k (+13%) at the same 0.82 headroom

Browse Source 
Reclaims over-reserved memory rather than spending headroom, so it honors
the conservative 0.82 / 14 GB-OS-reserve choice (still ~15 GiB free under
peak load) while closing most of the gap to the dflash pool (456k):

- int8 lm-head: build int8 on first warmup forward, then free the dead bf16
  copy (~1.4 GiB) + empty_cache() so the block lands in the KV pool. Safe:
  the DFlash drafter shares the one int8 lm_head module (verified one int8
  build; drafter ckpt has no lm_head/embed) and tie_word_embeddings=False,
  so bf16 has no reader (bias fallback = int8-GEMV + add-bias).
  SPARK_KEEP_BF16_LMHEAD=1 restores keep-bf16.
- VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 in serve.sh/mtp_serve.sh:
  returns the CUDA-graph over-estimate (~0.6 GiB; actual capture ~0.14 GiB
  from the wide 0.82 headroom). Overridable with =1.
- scripts/stress_mem.sh: drive the concurrency bank while sampling host RAM
  to prove a gpu-mem setting survives peak load without swapping.

Validated: pool 426,610 tokens, coherent output, drafter acceptance 4-12,
no new swap, throughput unchanged. max-batched-tokens 8192->4096 tested as
a KV lever and rejected (only +2.5k tokens, costs prefill speed).
This commit is contained in:
Entrpi
2026-06-24 17:41:07 +10:00
parent 1d0f2e1af3
commit 2cb5033fb8
6 changed files with 105 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+16 -8
View File
@@ -98,9 +98,17 @@ target hardware at the shipped defaults (`gpu-mem 0.82`, `ctx 262144`, `seqs 3`)
| Measurement (default `dense` profile) | Value |
|---|---|
| Free memory at READY | **~18 GiB** (responsive, no swap) |
| GPU KV cache pool | **376,518 tokens** (`dflash` profile: 456,664) |
| Max concurrency at full 262 144 | **1.44×** (`dflash`: 1.74×) |
| Free memory at READY | **~16 GiB** (responsive, no swap; ~15 GiB under peak 3-stream load) |
| GPU KV cache pool | **426,610 tokens** (`dflash` profile: 456,664) |
| Max concurrency at full 262 144 | **1.63×** (`dflash`: 1.74×) |
The `dense` pool was lifted from 376,518 → **426,610 tokens** (+13 %) at the *same*
`0.82` headroom by reclaiming over-reserved memory rather than spending headroom:
the int8 lm-head frees its now-dead bf16 copy (~1.4 GiB — the DFlash drafter shares
the same int8 lm_head, and `tie_word_embeddings=False`, so it is genuinely unused
after quantization), and `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` returns the
CUDA-graph over-estimate (~0.6 GiB; actual capture is ~0.14 GiB, drawn from the wide
0.82 headroom). Both are validated coherent with full drafter acceptance (4–12).
Decode-only throughput (streaming, excludes prefill), by workload and concurrency:
@@ -116,16 +124,16 @@ exceeds the ~81 headline, which is the median over 10 varied real turns includin
longer, slower-prefilling contexts.)
A typical load — three streams under ~100 k each (≈ <300 k tokens) — fits the
376 k pool with margin, and a single stream can still reach the full 262 144
426 k pool with margin, and a single stream can still reach the full 262 144
context. At `gpu-mem` 0.88–0.89 the static footprint leaves only ~5 GiB free; the
host then swaps and requests stall. `0.82` is the validated value (~18 GiB free
on `dense`). Defaults (override via flags or environment variables):
host then swaps and requests stall. `0.82` is the validated value (~16 GiB free
on `dense`, ~15 GiB under peak load). Defaults (override via flags or environment variables):
| Flag / env | Default | Note |
|---|---|---|
| `--gpu-mem` / `GPU_MEM` | **0.82** | ~14 GiB free (validated); 0.88+ over-subscribes and swaps |
| `--ctx` / `CTX` (`MAX_MODEL_LEN`) | **262144** | model native max; a single stream can reach any length up to this. Costs only KV-pool sizing — the CUDA-graph compile range tracks `max-batched-tokens`, not `ctx` |
| `--max-num-seqs` / `MAX_NUM_SEQS` | **3** | concurrent-stream cap; the pool holds ~1.4× (`dense`) / ~1.7× (`dflash`) a full-262 k context, ample for <100 k streams |
| `--max-num-seqs` / `MAX_NUM_SEQS` | **3** | concurrent-stream cap; the pool holds ~1.6× (`dense`) / ~1.7× (`dflash`) a full-262 k context, ample for <100 k streams |
| `--max-batched-tokens` / `MAX_BATCHED_TOKENS` | **8192** | chunked-prefill chunk, kept **below** `ctx` so a long prefill does not batch all at once |
The default operating point is a single stream (no contention; ~81 tok/s on agent
@@ -159,7 +167,7 @@ Selected with `--profile`:
| Profile | Stack | Best for | Measured |
|---|---|---|---|
| **`dense`** *(default)* | hybrid INT4+FP8 + int8 lm-head + DFlash n=12 | general — downloads the prebuilt hybrid; ≈ dflash on agents, +28% on base | 36.0 base (+28%) · 59.0 albond-bench · ~81 Hermes |
| `dflash` | INT4 + DFlash n=12 | agent path; largest KV pool (456k vs 376k) | **~81 tok/s** Hermes · 53.7 albond-bench |
| `dflash` | INT4 + DFlash n=12 | agent path; largest KV pool (456k vs 426k) | **~81 tok/s** Hermes · 53.7 albond-bench |
| `base` | plain INT4, no speculative decode | airtight baseline | 28.2 tok/s c=1 |
| `mtp` | INT4 + native MTP-2 head | comparison | ~40 tok/s Hermes |