tune for 1 orchestrator + up to 3 subagents: gpu-mem 0.88 (~15GB reserve), seqs=4, ctx 262144; document use case + tradeoffs

README.md

@@ -68,31 +68,44 @@ Preview without running: `... | bash -s -- --help`.
 GB10 is detected via `nvidia-smi --query-gpu=compute_cap` returning `12.1`;
 anything else needs `--force`.
 
-### Memory & context (defaults are tuned for max context + KV depth)
+### Memory & context (defaults tuned for: 1 orchestrator + up to 3 subagents)
 
-This model is **36/48 linear-attention (GDN) + 12 full-attention** layers, and the
-attention layers use GQA `num_key_value_heads=2`, `head_dim=256` → **only
-~24 KiB/token of KV**. A full **262 144** context (the model's native max) is just
-~6 GiB of KV; the GDN layers hold a *fixed* per-sequence state (~0.2 GiB) that
-does **not** grow with context. So on the 128 GB (119 GiB) GB10, reserving 14 GB
-for the OS leaves ~106 GiB for vLLM:
+Target use case — **one user**, one main Hermes orchestrator thread (~100 k
+context average, safe up to the model's 262 144 max) that can spin up **up to 3
+subagents** (<100 k each). Default operation is a single decode stream; we want
+all **4 concurrent** streams to be smoothly additive (no preemption).
+
+That fits easily, because the model is **36/48 linear-attention (GDN) + 12
+full-attention** layers with GQA `num_key_value_heads=2`, `head_dim=256` → **only
+~24 KiB/token of KV**. The GDN layers hold a *fixed* per-sequence state (~0.1–0.2
+GiB) that does **not** grow with context. On the 128 GB (119 GiB) GB10, reserving
+~15 GB for the OS leaves ~105 GiB for vLLM:
 
 | | |
 |---|---|
 | weights (INT4) + DFlash drafter | ~64 GiB |
 | CUDA graphs + activations | ~10 GiB |
-| **KV pool** | **~32 GiB ≈ 1.38 M tokens** |
+| GDN state × 4 seq slots | ~0.5–1 GiB |
+| **attention-KV pool** | **~30 GiB ≈ ~1.3 M tokens** |
 
-The KV pool (~1.38 M tokens) dwarfs a single 262 144 context, so single context is
-capped by the *model*, not memory. Defaults (override via flags/env):
+A full 262 144 context is only ~6 GiB of KV, so the pool holds **~5 full-context
+sequences**. The intended load — orchestrator @ up to 262 144 + 3 subagents @
+~100 k ≈ 560 k tokens (~13 GiB) — uses well under half the pool, so the 4 streams
+**never preempt**. Defaults (override via flags/env):
 
 | Flag / env | Default | Note |
 |---|---|---|
-| `--gpu-mem` / `GPU_MEM` | **0.89** | reserves ~14 GB; drop to 0.87 if the OOM-guard fires on first load |
-| `--ctx` / `CTX` (`MAX_MODEL_LEN`) | **262144** | model native max |
-| `--max-num-seqs` / `MAX_NUM_SEQS` | **4** | ~5 sequences fit the full 262 144 context each (pool ÷ ctx); set higher for short-context concurrency (preempts if many grow long) |
+| `--gpu-mem` / `GPU_MEM` | **0.88** | reserves ~15 GB for OS/other apps; drop to 0.86 if the OOM-guard fires on first load |
+| `--ctx` / `CTX` (`MAX_MODEL_LEN`) | **262144** | model native max — keeps the orchestrator safe at any length |
+| `--max-num-seqs` / `MAX_NUM_SEQS` | **4** | 1 orchestrator + 3 subagents; all four fit the full context (pool ÷ ctx ≈ 5) |
 | `--max-batched-tokens` / `MAX_BATCHED_TOKENS` | **8192** | chunked-prefill chunk — kept **below** ctx so a long prefill doesn't batch all at once |
 
+**Single-stream is the default operating point** (DFlash speculative decode is a
+single-stream lever; at 1 active stream there's no contention and you get the full
+~81 tok/s on agent turns). Concurrency 2–4 is memory-safe and additive for these
+contexts; per-request tok/s eases down as the decode batch grows (more routed-expert
+traffic per step) — the cost is throughput-vs-latency, not safety.
+
 > Unified-memory OOM **hard-freezes** the box, and vLLM's profiler can undershoot
 > peak by a couple GB — always bring the server up under
 > [`scripts/monitor.sh`](scripts/monitor.sh) (OOM auto-kill guard) the first time