From dabc8665071c2fdb2a885db968f62764a9ddb0b5 Mon Sep 17 00:00:00 2001 From: ent Date: Wed, 24 Jun 2026 13:31:36 +1000 Subject: [PATCH] tune for 1 orchestrator + up to 3 subagents: gpu-mem 0.88 (~15GB reserve), seqs=4, ctx 262144; document use case + tradeoffs --- README.md | 39 ++++++++++++++++++++++++++------------- install.sh | 4 ++-- runtime/mtp_serve.sh | 2 +- runtime/serve.sh | 2 +- 4 files changed, 30 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 9f69ee9..3c8a8ca 100644 --- a/README.md +++ b/README.md @@ -68,31 +68,44 @@ Preview without running: `... | bash -s -- --help`. GB10 is detected via `nvidia-smi --query-gpu=compute_cap` returning `12.1`; anything else needs `--force`. -### Memory & context (defaults are tuned for max context + KV depth) +### Memory & context (defaults tuned for: 1 orchestrator + up to 3 subagents) -This model is **36/48 linear-attention (GDN) + 12 full-attention** layers, and the -attention layers use GQA `num_key_value_heads=2`, `head_dim=256` → **only -~24 KiB/token of KV**. A full **262 144** context (the model's native max) is just -~6 GiB of KV; the GDN layers hold a *fixed* per-sequence state (~0.2 GiB) that -does **not** grow with context. So on the 128 GB (119 GiB) GB10, reserving 14 GB -for the OS leaves ~106 GiB for vLLM: +Target use case — **one user**, one main Hermes orchestrator thread (~100 k +context average, safe up to the model's 262 144 max) that can spin up **up to 3 +subagents** (<100 k each). Default operation is a single decode stream; we want +all **4 concurrent** streams to be smoothly additive (no preemption). + +That fits easily, because the model is **36/48 linear-attention (GDN) + 12 +full-attention** layers with GQA `num_key_value_heads=2`, `head_dim=256` → **only +~24 KiB/token of KV**. The GDN layers hold a *fixed* per-sequence state (~0.1–0.2 +GiB) that does **not** grow with context. On the 128 GB (119 GiB) GB10, reserving +~15 GB for the OS leaves ~105 GiB for vLLM: | | | |---|---| | weights (INT4) + DFlash drafter | ~64 GiB | | CUDA graphs + activations | ~10 GiB | -| **KV pool** | **~32 GiB ≈ 1.38 M tokens** | +| GDN state × 4 seq slots | ~0.5–1 GiB | +| **attention-KV pool** | **~30 GiB ≈ ~1.3 M tokens** | -The KV pool (~1.38 M tokens) dwarfs a single 262 144 context, so single context is -capped by the *model*, not memory. Defaults (override via flags/env): +A full 262 144 context is only ~6 GiB of KV, so the pool holds **~5 full-context +sequences**. The intended load — orchestrator @ up to 262 144 + 3 subagents @ +~100 k ≈ 560 k tokens (~13 GiB) — uses well under half the pool, so the 4 streams +**never preempt**. Defaults (override via flags/env): | Flag / env | Default | Note | |---|---|---| -| `--gpu-mem` / `GPU_MEM` | **0.89** | reserves ~14 GB; drop to 0.87 if the OOM-guard fires on first load | -| `--ctx` / `CTX` (`MAX_MODEL_LEN`) | **262144** | model native max | -| `--max-num-seqs` / `MAX_NUM_SEQS` | **4** | ~5 sequences fit the full 262 144 context each (pool ÷ ctx); set higher for short-context concurrency (preempts if many grow long) | +| `--gpu-mem` / `GPU_MEM` | **0.88** | reserves ~15 GB for OS/other apps; drop to 0.86 if the OOM-guard fires on first load | +| `--ctx` / `CTX` (`MAX_MODEL_LEN`) | **262144** | model native max — keeps the orchestrator safe at any length | +| `--max-num-seqs` / `MAX_NUM_SEQS` | **4** | 1 orchestrator + 3 subagents; all four fit the full context (pool ÷ ctx ≈ 5) | | `--max-batched-tokens` / `MAX_BATCHED_TOKENS` | **8192** | chunked-prefill chunk — kept **below** ctx so a long prefill doesn't batch all at once | +**Single-stream is the default operating point** (DFlash speculative decode is a +single-stream lever; at 1 active stream there's no contention and you get the full +~81 tok/s on agent turns). Concurrency 2–4 is memory-safe and additive for these +contexts; per-request tok/s eases down as the decode batch grows (more routed-expert +traffic per step) — the cost is throughput-vs-latency, not safety. + > Unified-memory OOM **hard-freezes** the box, and vLLM's profiler can undershoot > peak by a couple GB — always bring the server up under > [`scripts/monitor.sh`](scripts/monitor.sh) (OOM auto-kill guard) the first time diff --git a/install.sh b/install.sh index de5fdc2..b5d1fa5 100755 --- a/install.sh +++ b/install.sh @@ -56,7 +56,7 @@ PROFILE="dflash" # dflash | dense | base | mtp NSPEC="" # override num_speculative_tokens (default per profile) PORT="${PORT:-8000}" CTX="${CTX:-262144}" # max-model-len: model native max (KV is ~24 KiB/token) -GPU_MEM="${GPU_MEM:-0.89}" # reserves ~14 GB on a 128 GB (119 GiB) GB10 +GPU_MEM="${GPU_MEM:-0.88}" # reserves ~15 GB on a 128 GB (119 GiB) GB10 MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}" # ~5 fit full native ctx each; raise for short-ctx concurrency MAX_BATCHED_TOKENS="${MAX_BATCHED_TOKENS:-8192}" # chunked-prefill chunk (decoupled from ctx) BACKEND="${BACKEND:-flash_attn}" @@ -94,7 +94,7 @@ Flags: --nspec N num_speculative_tokens (default 12 dflash/dense, 2 mtp, 0 base). --port N Server port (default: $PORT). --ctx N max-model-len (default: $CTX = model native max). - --gpu-mem F gpu-memory-utilization (default: $GPU_MEM reserves ~14 GB on 119 GiB). + --gpu-mem F gpu-memory-utilization (default: $GPU_MEM reserves ~15 GB on 119 GiB). --max-num-seqs N concurrent sequences (default: $MAX_NUM_SEQS; ~5 fit full ctx each). --max-batched-tokens N chunked-prefill chunk (default: $MAX_BATCHED_TOKENS; keep < ctx). --force Skip the GB10/SM121 host check. diff --git a/runtime/mtp_serve.sh b/runtime/mtp_serve.sh index 284e6ba..0488932 100755 --- a/runtime/mtp_serve.sh +++ b/runtime/mtp_serve.sh @@ -8,7 +8,7 @@ NSPEC="${1:-2}" BACKEND="${2:-flash_attn}" MODEL="${MODEL:-Intel/Qwen3.5-122B-A10B-int4-AutoRound}" MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}" -GPU_MEM="${GPU_MEM:-0.89}" +GPU_MEM="${GPU_MEM:-0.88}" MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}" MAX_BATCHED_TOKENS="${MAX_BATCHED_TOKENS:-8192}" PORT="${PORT:-8000}" diff --git a/runtime/serve.sh b/runtime/serve.sh index dfea383..f2a9507 100755 --- a/runtime/serve.sh +++ b/runtime/serve.sh @@ -20,7 +20,7 @@ BACKEND="${2:-flash_attn}" MODEL="${MODEL:-Intel/Qwen3.5-122B-A10B-int4-AutoRound}" DRAFT="${DRAFT:-z-lab/Qwen3.5-122B-A10B-DFlash}" MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}" # model native max; KV is ~24 KiB/token so it fits -GPU_MEM="${GPU_MEM:-0.89}" # ~14 GB reserved on a 128 GB (119 GiB) GB10 +GPU_MEM="${GPU_MEM:-0.88}" # ~15 GB reserved on a 128 GB (119 GiB) GB10 MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}" # ~5 seqs fit the full native ctx each (KV pool ~1.38M / 262144) MAX_BATCHED_TOKENS="${MAX_BATCHED_TOKENS:-8192}" # chunked-prefill chunk (NOT = max-model-len) PORT="${PORT:-8000}"