docs: credit albond's recipe as the foundation + head-to-head comparison (59.0 vs 51.58 e2e, same method)
This commit is contained in:
@@ -9,11 +9,13 @@ with a per-token bandwidth model that explains every number.
|
||||
|
||||
**Status:** Working end-to-end, one-shot install. On real Hermes-agent
|
||||
tool-call turns, **DFlash decode reaches a median ~81 tok/s on GB10** —
|
||||
**~2× the native MTP-2 head (~40 tok/s)** on the same workload, and above the
|
||||
best previously published number for this model on Spark (albond's fully-patched
|
||||
MTP stack, 51.58 tok/s end-to-end). DFlash's acceptance is task-dependent (it
|
||||
block-drafts 12 tokens in one parallel forward), so the win is largest on
|
||||
structured/tool-call/code traffic and collapses to parity on open-ended prose.
|
||||
**~2× the native MTP-2 head (~40 tok/s)** on the same workload, and above
|
||||
[**albond's**](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)
|
||||
fully-patched MTP recipe (51.58 tok/s end-to-end) — **the recipe this repo
|
||||
gratefully builds on** (see [Building on the albond recipe](#building-on-the-albond-recipe)).
|
||||
DFlash's acceptance is task-dependent (it block-drafts 12 tokens in one parallel
|
||||
forward), so the win is largest on structured/tool-call/code traffic and
|
||||
collapses to parity on open-ended prose.
|
||||
|
||||
A separate **dense-bandwidth stack** (hybrid INT4+FP8 shared experts + int8
|
||||
lm-head) adds **+28 % to no-spec / base decode** (28.2 → 36.0 tok/s) but, by the
|
||||
@@ -116,6 +118,56 @@ hybrid INT4+FP8 (BF16 shared experts → calibrated FP8) and int8 lm-head (the
|
||||
| + int8 lm-head | 32.7 (+16%) | — | — |
|
||||
| **+ both** | **36.0 (+28%)** | **59.0 (+10%)** | ~80–87 *(noise)* |
|
||||
|
||||
## Building on the albond recipe
|
||||
|
||||
This repo stands on **[albond's DGX-Spark Qwen3.5-122B recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)** —
|
||||
the first working high-throughput recipe for this model on Spark, and the
|
||||
reference we measured everything against. On eugr's vLLM 0.19.1 fork, albond
|
||||
established:
|
||||
|
||||
- the **rebuilt hybrid INT4+FP8 checkpoint** (BF16 dense → calibrated FP8),
|
||||
- the **INT8 lm-head** patch (the single biggest dense-bandwidth lever),
|
||||
- **MTP-2** native speculative decode, and
|
||||
- the **end-to-end benchmark methodology** (completion_tokens / total wallclock,
|
||||
incl. prefill) we report against — reproduced verbatim in
|
||||
[`scripts/bench_albond.py`](scripts/bench_albond.py).
|
||||
|
||||
We gratefully build on all of it. What this repo adds is **carrying that recipe
|
||||
forward to the latest vLLM and a stronger drafter**, and getting the community
|
||||
patches to stack together there:
|
||||
|
||||
1. **Forward-ported the dense levers to vLLM 0.23.** albond's patches target the
|
||||
0.19 fork and don't drop in cleanly. The hybrid-FP8 dispatch had to be
|
||||
re-expressed against 0.23's `maybe_update_config(model_name, hf_config=…)`
|
||||
quant-config hook; the INT8 lm-head needed a from-scratch integration (the
|
||||
prior port zeroed the lm-head weight — which corrupts the **DFlash-shared**
|
||||
head → garbage — and looped per-row for batch>4, which is *slower* under spec;
|
||||
both fixed in [`patch_int8_lmhead_v3.py`](runtime/patch_int8_lmhead_v3.py)).
|
||||
2. **Swapped MTP-2 for the DFlash block-diffusion drafter** and got it running on
|
||||
the hybrid 122B in vLLM (the [KV-unify fix](runtime/patch_unify2.py)). DFlash
|
||||
block-drafts 12 tokens in one parallel forward vs MTP's sequential head
|
||||
(acceptance-capped at ~3), so it pulls ~2× ahead on agent/code traffic.
|
||||
3. **Composed the dense levers *with* DFlash** instead of MTP.
|
||||
|
||||
### Where we land — albond's own end-to-end method, same hardware class
|
||||
|
||||
| Stack | Spec | Dense patches | e2e tok/s |
|
||||
|---|---|---|---|
|
||||
| **albond** (published) | MTP-2 | hybrid-FP8 + INT8 lm-head + PR#38325 | 51.58 |
|
||||
| this repo — `dflash` | DFlash n=12 | *none* | **53.7** (+4%) |
|
||||
| this repo — `dense` | DFlash n=12 | hybrid-FP8 + INT8 lm-head | **59.0** (+14%) |
|
||||
|
||||
On the **real agent workload** (decode-only, regenerating live tool-call turns),
|
||||
DFlash's parallel block-drafting pulls further ahead of MTP's sequential head:
|
||||
**~81 vs ~40 tok/s**.
|
||||
|
||||
> **Stated plainly:** albond's 51.58 is his published figure on his stack
|
||||
> (vLLM 0.19 + MTP); our figures are on this stack (vLLM 0.23 + DFlash). Both use
|
||||
> the **same e2e harness** on the **same hardware class** (DGX Spark / GB10) — a
|
||||
> fair best-on-each-stack comparison, not a single-variable controlled run. Note
|
||||
> that **unpatched DFlash (53.7) already clears albond's fully-patched MTP**, so
|
||||
> the dense stack is upside on top of the drafter swap, not the source of the win.
|
||||
|
||||
## The amortization law
|
||||
|
||||
The dense levers cut **always-on** weight reads (shared experts + lm-head, read
|
||||
@@ -208,10 +260,11 @@ python3 scripts/hermes_bench.py --base-url http://127.0.0.1:8000 # real agent
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- [`z-lab`](https://huggingface.co/z-lab) / [Modal](https://modal.com/blog/spec-is-all-u-need) — the DFlash drafter and block-diffusion speculative decode.
|
||||
- **[`albond`](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) — the foundation this builds on.** The hybrid INT4+FP8 checkpoint, the INT8 lm-head, MTP-2, and the end-to-end benchmark methodology. This repo is a grateful forward-port and recomposition of that recipe onto vLLM 0.23 + DFlash.
|
||||
- [`z-lab`](https://huggingface.co/z-lab) / [Modal](https://modal.com/blog/spec-is-all-u-need) — the DFlash drafter and the block-diffusion speculative-decode work.
|
||||
- [`Intel/AutoRound`](https://huggingface.co/Intel) — the INT4 target quantization.
|
||||
- [`vLLM`](https://github.com/vllm-project/vllm) and the AEON sm121 image maintainers — the engine and the DFlash-enabled GB10 build.
|
||||
- [`albond`](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) — the MTP/hybrid-FP8/int8-lmhead recipe and the end-to-end benchmark methodology.
|
||||
- [`eugr/spark-vllm-docker`](https://github.com/eugr/spark-vllm-docker) — the vLLM-on-Spark base that albond's recipe (and much of this ecosystem) started from.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -4,6 +4,13 @@ Single-stream (c=1) decode of `Qwen3.5-122B-A10B` (hybrid GDN + mamba + 128-expe
|
||||
MoE, ~10B active) on GB10 / SM121, 128 GiB unified, ~273 GB/s. The agent this
|
||||
backs (Hermes) is ~73 % tool-calls. All numbers temperature 0.
|
||||
|
||||
> **Credit.** This builds on [albond's recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)
|
||||
> (hybrid INT4+FP8 + INT8 lm-head + MTP-2 on vLLM 0.19, and the e2e benchmark
|
||||
> method). The new work here is forward-porting the dense levers to vLLM 0.23,
|
||||
> swapping MTP for DFlash, and composing them — landing at **59.0 e2e tok/s vs
|
||||
> albond's 51.58** on his own harness (+14%), and ~2× on real agent traffic.
|
||||
> See README → *Building on the albond recipe* for the side-by-side.
|
||||
|
||||
## 1. Getting DFlash to run on the hybrid 122B in vLLM
|
||||
|
||||
The DFlash drafter is **non-causal** (it block-drafts 16 tokens in one parallel
|
||||
|
||||
Reference in New Issue
Block a user