diff --git a/README.md b/README.md index 3c8a8ca..3bf4055 100644 --- a/README.md +++ b/README.md @@ -70,41 +70,43 @@ anything else needs `--force`. ### Memory & context (defaults tuned for: 1 orchestrator + up to 3 subagents) -Target use case — **one user**, one main Hermes orchestrator thread (~100 k -context average, safe up to the model's 262 144 max) that can spin up **up to 3 -subagents** (<100 k each). Default operation is a single decode stream; we want -all **4 concurrent** streams to be smoothly additive (no preemption). +Target use case — **one user**, one main orchestrator thread (~100 k context +average, safe up to the model's 262 144 max) that dispatches **up to 3 subagents** +(<100 k each). With prefix-caching off (DFlash requires it) the orchestrator is +*not* a persistent stream — when it dispatches it has ended its turn and is idle +until the subagents return — so the concurrent peak is the **3 subagents**, not 4. -That fits easily, because the model is **36/48 linear-attention (GDN) + 12 -full-attention** layers with GQA `num_key_value_heads=2`, `head_dim=256` → **only -~24 KiB/token of KV**. The GDN layers hold a *fixed* per-sequence state (~0.1–0.2 -GiB) that does **not** grow with context. On the 128 GB (119 GiB) GB10, reserving -~15 GB for the OS leaves ~105 GiB for vLLM: +These numbers are **measured on the box**, not estimated (an earlier naive +"24 KiB/token → ~1.3 M pool" projection was wrong — the per-sequence GDN/mamba +state and the activation reserve cost far more than the attention KV alone). At +the shipped defaults (`gpu-mem 0.82`, `ctx 262144`, `seqs 3`): -| | | +| | measured | |---|---| -| weights (INT4) + DFlash drafter | ~64 GiB | -| CUDA graphs + activations | ~10 GiB | -| GDN state × 4 seq slots | ~0.5–1 GiB | -| **attention-KV pool** | **~30 GiB ≈ ~1.3 M tokens** | +| free memory at READY | **~14 GiB** (responsive, no swap) | +| GPU KV cache pool | **456,664 tokens** | +| max concurrency @ full 262 144 | **1.74×** | +| concurrency 1 → 2 → 3 (prose) | per-request **26.5 → 18.8 → 17.4** tok/s · aggregate **26.5 → 36.8 → 51.6** | -A full 262 144 context is only ~6 GiB of KV, so the pool holds **~5 full-context -sequences**. The intended load — orchestrator @ up to 262 144 + 3 subagents @ -~100 k ≈ 560 k tokens (~13 GiB) — uses well under half the pool, so the 4 streams -**never preempt**. Defaults (override via flags/env): +Your realistic peak — 3 subagents <100 k (≈ <300 k tokens) — fits the 456 k pool +with margin, and the orchestrator can still run to the full 262 k when it's the +active stream. **Why not higher:** `gpu-mem 0.88–0.89` over-subscribes this box — +the static footprint left only ~5 GiB free, it swapped, and requests hung. `0.82` +is the validated sweet spot (~14 GiB headroom). Defaults (override via flags/env): | Flag / env | Default | Note | |---|---|---| -| `--gpu-mem` / `GPU_MEM` | **0.88** | reserves ~15 GB for OS/other apps; drop to 0.86 if the OOM-guard fires on first load | -| `--ctx` / `CTX` (`MAX_MODEL_LEN`) | **262144** | model native max — keeps the orchestrator safe at any length | -| `--max-num-seqs` / `MAX_NUM_SEQS` | **4** | 1 orchestrator + 3 subagents; all four fit the full context (pool ÷ ctx ≈ 5) | +| `--gpu-mem` / `GPU_MEM` | **0.82** | ~14 GiB free (validated). 0.88+ over-subscribes → swap → hangs. | +| `--ctx` / `CTX` (`MAX_MODEL_LEN`) | **262144** | model native max — orchestrator safe at any length (costs only KV-pool sizing; graph range is tied to `max-batched-tokens`) | +| `--max-num-seqs` / `MAX_NUM_SEQS` | **3** | the concurrent-subagent peak; pool ÷ ctx ≈ 1.74× full-context, ample for <100 k subagents | | `--max-batched-tokens` / `MAX_BATCHED_TOKENS` | **8192** | chunked-prefill chunk — kept **below** ctx so a long prefill doesn't batch all at once | **Single-stream is the default operating point** (DFlash speculative decode is a -single-stream lever; at 1 active stream there's no contention and you get the full -~81 tok/s on agent turns). Concurrency 2–4 is memory-safe and additive for these -contexts; per-request tok/s eases down as the decode batch grows (more routed-expert -traffic per step) — the cost is throughput-vs-latency, not safety. +single-stream lever; at 1 active stream there's no contention — ~81 tok/s on agent +turns). 2–3 concurrent is additive: per-request tok/s eases down as the decode +batch grows (more routed-expert traffic/step) while aggregate rises — the cost is +throughput-vs-latency, not safety. For a 4th simultaneous stream raise +`--max-num-seqs 4` (it'll queue, not crash) or trim `--ctx`. > Unified-memory OOM **hard-freezes** the box, and vLLM's profiler can undershoot > peak by a couple GB — always bring the server up under diff --git a/install.sh b/install.sh index b5d1fa5..dd33494 100755 --- a/install.sh +++ b/install.sh @@ -56,8 +56,8 @@ PROFILE="dflash" # dflash | dense | base | mtp NSPEC="" # override num_speculative_tokens (default per profile) PORT="${PORT:-8000}" CTX="${CTX:-262144}" # max-model-len: model native max (KV is ~24 KiB/token) -GPU_MEM="${GPU_MEM:-0.88}" # reserves ~15 GB on a 128 GB (119 GiB) GB10 -MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}" # ~5 fit full native ctx each; raise for short-ctx concurrency +GPU_MEM="${GPU_MEM:-0.82}" # VALIDATED: ~14 GiB free (0.88+ over-subscribes -> swap) +MAX_NUM_SEQS="${MAX_NUM_SEQS:-3}" # 3 concurrent streams (e.g. 3 subagents); pool ~457k tok, 1.74x @ 262k MAX_BATCHED_TOKENS="${MAX_BATCHED_TOKENS:-8192}" # chunked-prefill chunk (decoupled from ctx) BACKEND="${BACKEND:-flash_attn}" @@ -94,8 +94,8 @@ Flags: --nspec N num_speculative_tokens (default 12 dflash/dense, 2 mtp, 0 base). --port N Server port (default: $PORT). --ctx N max-model-len (default: $CTX = model native max). - --gpu-mem F gpu-memory-utilization (default: $GPU_MEM reserves ~15 GB on 119 GiB). - --max-num-seqs N concurrent sequences (default: $MAX_NUM_SEQS; ~5 fit full ctx each). + --gpu-mem F gpu-memory-utilization (default: $GPU_MEM = ~14 GiB free; 0.88+ over-subscribes/swaps). + --max-num-seqs N concurrent sequences (default: $MAX_NUM_SEQS; KV pool ~457k tokens, 1.74x at full 262k). --max-batched-tokens N chunked-prefill chunk (default: $MAX_BATCHED_TOKENS; keep < ctx). --force Skip the GB10/SM121 host check. --no-smoke Start the server but skip the Paris smoke test. diff --git a/runtime/mtp_serve.sh b/runtime/mtp_serve.sh index 0488932..6f76985 100755 --- a/runtime/mtp_serve.sh +++ b/runtime/mtp_serve.sh @@ -8,8 +8,8 @@ NSPEC="${1:-2}" BACKEND="${2:-flash_attn}" MODEL="${MODEL:-Intel/Qwen3.5-122B-A10B-int4-AutoRound}" MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}" -GPU_MEM="${GPU_MEM:-0.88}" -MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}" +GPU_MEM="${GPU_MEM:-0.82}" +MAX_NUM_SEQS="${MAX_NUM_SEQS:-3}" MAX_BATCHED_TOKENS="${MAX_BATCHED_TOKENS:-8192}" PORT="${PORT:-8000}" echo "[mtp] qwen3_5_mtp — backend=$BACKEND, num_speculative_tokens=$NSPEC, model=$MODEL" diff --git a/runtime/serve.sh b/runtime/serve.sh index f2a9507..1305c60 100755 --- a/runtime/serve.sh +++ b/runtime/serve.sh @@ -20,8 +20,8 @@ BACKEND="${2:-flash_attn}" MODEL="${MODEL:-Intel/Qwen3.5-122B-A10B-int4-AutoRound}" DRAFT="${DRAFT:-z-lab/Qwen3.5-122B-A10B-DFlash}" MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}" # model native max; KV is ~24 KiB/token so it fits -GPU_MEM="${GPU_MEM:-0.88}" # ~15 GB reserved on a 128 GB (119 GiB) GB10 -MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}" # ~5 seqs fit the full native ctx each (KV pool ~1.38M / 262144) +GPU_MEM="${GPU_MEM:-0.82}" # VALIDATED: ~14 GiB free on 128 GB (119 GiB) GB10 (0.88+ over-subscribes -> swap) +MAX_NUM_SEQS="${MAX_NUM_SEQS:-3}" # 3 concurrent streams; KV pool ~457k tokens, 1.74x at full 262144 MAX_BATCHED_TOKENS="${MAX_BATCHED_TOKENS:-8192}" # chunked-prefill chunk (NOT = max-model-len) PORT="${PORT:-8000}"