From dc9d40a1c4462862b5c650a4e0c4948f3177cad5 Mon Sep 17 00:00:00 2001 From: ent Date: Wed, 24 Jun 2026 13:08:52 +1000 Subject: [PATCH] docs: credit albond's recipe as the foundation + head-to-head comparison (59.0 vs 51.58 e2e, same method) --- README.md | 67 +++++++++++++++++++++++++++++++++++++++++++----- docs/FINDINGS.md | 7 +++++ 2 files changed, 67 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 2809c11..7846449 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,13 @@ with a per-token bandwidth model that explains every number. **Status:** Working end-to-end, one-shot install. On real Hermes-agent tool-call turns, **DFlash decode reaches a median ~81 tok/s on GB10** — -**~2× the native MTP-2 head (~40 tok/s)** on the same workload, and above the -best previously published number for this model on Spark (albond's fully-patched -MTP stack, 51.58 tok/s end-to-end). DFlash's acceptance is task-dependent (it -block-drafts 12 tokens in one parallel forward), so the win is largest on -structured/tool-call/code traffic and collapses to parity on open-ended prose. +**~2× the native MTP-2 head (~40 tok/s)** on the same workload, and above +[**albond's**](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) +fully-patched MTP recipe (51.58 tok/s end-to-end) — **the recipe this repo +gratefully builds on** (see [Building on the albond recipe](#building-on-the-albond-recipe)). +DFlash's acceptance is task-dependent (it block-drafts 12 tokens in one parallel +forward), so the win is largest on structured/tool-call/code traffic and +collapses to parity on open-ended prose. A separate **dense-bandwidth stack** (hybrid INT4+FP8 shared experts + int8 lm-head) adds **+28 % to no-spec / base decode** (28.2 → 36.0 tok/s) but, by the @@ -116,6 +118,56 @@ hybrid INT4+FP8 (BF16 shared experts → calibrated FP8) and int8 lm-head (the | + int8 lm-head | 32.7 (+16%) | — | — | | **+ both** | **36.0 (+28%)** | **59.0 (+10%)** | ~80–87 *(noise)* | +## Building on the albond recipe + +This repo stands on **[albond's DGX-Spark Qwen3.5-122B recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)** — +the first working high-throughput recipe for this model on Spark, and the +reference we measured everything against. On eugr's vLLM 0.19.1 fork, albond +established: + +- the **rebuilt hybrid INT4+FP8 checkpoint** (BF16 dense → calibrated FP8), +- the **INT8 lm-head** patch (the single biggest dense-bandwidth lever), +- **MTP-2** native speculative decode, and +- the **end-to-end benchmark methodology** (completion_tokens / total wallclock, + incl. prefill) we report against — reproduced verbatim in + [`scripts/bench_albond.py`](scripts/bench_albond.py). + +We gratefully build on all of it. What this repo adds is **carrying that recipe +forward to the latest vLLM and a stronger drafter**, and getting the community +patches to stack together there: + +1. **Forward-ported the dense levers to vLLM 0.23.** albond's patches target the + 0.19 fork and don't drop in cleanly. The hybrid-FP8 dispatch had to be + re-expressed against 0.23's `maybe_update_config(model_name, hf_config=…)` + quant-config hook; the INT8 lm-head needed a from-scratch integration (the + prior port zeroed the lm-head weight — which corrupts the **DFlash-shared** + head → garbage — and looped per-row for batch>4, which is *slower* under spec; + both fixed in [`patch_int8_lmhead_v3.py`](runtime/patch_int8_lmhead_v3.py)). +2. **Swapped MTP-2 for the DFlash block-diffusion drafter** and got it running on + the hybrid 122B in vLLM (the [KV-unify fix](runtime/patch_unify2.py)). DFlash + block-drafts 12 tokens in one parallel forward vs MTP's sequential head + (acceptance-capped at ~3), so it pulls ~2× ahead on agent/code traffic. +3. **Composed the dense levers *with* DFlash** instead of MTP. + +### Where we land — albond's own end-to-end method, same hardware class + +| Stack | Spec | Dense patches | e2e tok/s | +|---|---|---|---| +| **albond** (published) | MTP-2 | hybrid-FP8 + INT8 lm-head + PR#38325 | 51.58 | +| this repo — `dflash` | DFlash n=12 | *none* | **53.7** (+4%) | +| this repo — `dense` | DFlash n=12 | hybrid-FP8 + INT8 lm-head | **59.0** (+14%) | + +On the **real agent workload** (decode-only, regenerating live tool-call turns), +DFlash's parallel block-drafting pulls further ahead of MTP's sequential head: +**~81 vs ~40 tok/s**. + +> **Stated plainly:** albond's 51.58 is his published figure on his stack +> (vLLM 0.19 + MTP); our figures are on this stack (vLLM 0.23 + DFlash). Both use +> the **same e2e harness** on the **same hardware class** (DGX Spark / GB10) — a +> fair best-on-each-stack comparison, not a single-variable controlled run. Note +> that **unpatched DFlash (53.7) already clears albond's fully-patched MTP**, so +> the dense stack is upside on top of the drafter swap, not the source of the win. + ## The amortization law The dense levers cut **always-on** weight reads (shared experts + lm-head, read @@ -208,10 +260,11 @@ python3 scripts/hermes_bench.py --base-url http://127.0.0.1:8000 # real agent ## Acknowledgements -- [`z-lab`](https://huggingface.co/z-lab) / [Modal](https://modal.com/blog/spec-is-all-u-need) — the DFlash drafter and block-diffusion speculative decode. +- **[`albond`](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) — the foundation this builds on.** The hybrid INT4+FP8 checkpoint, the INT8 lm-head, MTP-2, and the end-to-end benchmark methodology. This repo is a grateful forward-port and recomposition of that recipe onto vLLM 0.23 + DFlash. +- [`z-lab`](https://huggingface.co/z-lab) / [Modal](https://modal.com/blog/spec-is-all-u-need) — the DFlash drafter and the block-diffusion speculative-decode work. - [`Intel/AutoRound`](https://huggingface.co/Intel) — the INT4 target quantization. - [`vLLM`](https://github.com/vllm-project/vllm) and the AEON sm121 image maintainers — the engine and the DFlash-enabled GB10 build. -- [`albond`](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) — the MTP/hybrid-FP8/int8-lmhead recipe and the end-to-end benchmark methodology. +- [`eugr/spark-vllm-docker`](https://github.com/eugr/spark-vllm-docker) — the vLLM-on-Spark base that albond's recipe (and much of this ecosystem) started from. ## License diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index 5b09b0f..856fed3 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -4,6 +4,13 @@ Single-stream (c=1) decode of `Qwen3.5-122B-A10B` (hybrid GDN + mamba + 128-expe MoE, ~10B active) on GB10 / SM121, 128 GiB unified, ~273 GB/s. The agent this backs (Hermes) is ~73 % tool-calls. All numbers temperature 0. +> **Credit.** This builds on [albond's recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) +> (hybrid INT4+FP8 + INT8 lm-head + MTP-2 on vLLM 0.19, and the e2e benchmark +> method). The new work here is forward-porting the dense levers to vLLM 0.23, +> swapping MTP for DFlash, and composing them — landing at **59.0 e2e tok/s vs +> albond's 51.58** on his own harness (+14%), and ~2× on real agent traffic. +> See README → *Building on the albond recipe* for the side-by-side. + ## 1. Getting DFlash to run on the hybrid 122B in vLLM The DFlash drafter is **non-causal** (it block-drafts 16 tokens in one parallel