TBNilles/qwen3.5-122B-A10B-on-spark
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refactor README so dense is unambiguously the default and the hosted hybrid (bleysg/...int4-fp8-hybrid) is the primary/chosen checkpoint (title link, model bullet, quick start, related-work)

Browse Source
This commit is contained in:
ent
2026-06-24 16:45:57 +10:00
parent d3688cc833
commit 4e5c9552fe
1 changed files with 26 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+26 -14
View File
@@ -1,11 +1,15 @@
# qwen3.5-122B-A10B-on-spark # qwen3.5-122B-A10B-on-spark
[`Qwen3.5-122B-A10B`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) [`Qwen3.5-122B-A10B`](https://huggingface.co/bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid)
(hybrid GDN + mamba + 128-expert MoE, ~10B active) on a single (hybrid GDN + mamba + 128-expert MoE, ~10B active) on a single
**NVIDIA DGX Spark** (GB10 / SM121, 128 GB / 119 GiB unified) under **vLLM**, with **NVIDIA DGX Spark** (GB10 / SM121, 128 GB / 119 GiB unified) under **vLLM**, with
**[DFlash](https://modal.com/blog/spec-is-all-u-need) block-diffusion speculative **[DFlash](https://modal.com/blog/spec-is-all-u-need) block-diffusion speculative
decode** and an optional **dense-bandwidth patch stack** — measured end-to-end, decode** and a **dense-bandwidth patch stack** — measured end-to-end, with a
with a per-token bandwidth model behind the numbers. per-token bandwidth model behind the numbers.
**The default `dense` profile** serves a purpose-built **hybrid INT4+FP8
checkpoint**, [`bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid`](https://huggingface.co/bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid),
downloaded ready-to-run — no local checkpoint build.
**Status:** working end-to-end, one-shot install. On real agent tool-call turns, **Status:** working end-to-end, one-shot install. On real agent tool-call turns,
**DFlash decode reaches a median ~81 tok/s on GB10** — about **2× the native **DFlash decode reaches a median ~81 tok/s on GB10** — about **2× the native
@@ -25,9 +29,9 @@ plain DFlash on high-acceptance agent traffic (per the
The **`dflash`** profile drops the dense patches for the largest KV pool and the The **`dflash`** profile drops the dense patches for the largest KV pool and the
same ~81 tok/s on agents. same ~81 tok/s on agents.
- **Engine:** [`vLLM`](https://github.com/vllm-project/vllm) 0.23, sm121 build with the DFlash PRs, via the prebuilt image `ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix`. No host build — the four runtime patches in [`runtime/`](runtime/) are applied at serve time. - **Model (default):** [`bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid`](https://huggingface.co/bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid) — the prebuilt hybrid the `dense` profile downloads and serves: INT4 (AutoRound/GPTQ) routed experts + attention, **calibrated FP8** shared experts, BF16 embeddings / head; ~67 GiB safetensors (*not* GGUF — vLLM serves it directly). Built from [`Intel/Qwen3.5-122B-A10B-int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) (INT4 base) + [`Qwen/Qwen3.5-122B-A10B-FP8`](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8) (FP8 shared experts). The `dflash` / `base` / `mtp` profiles serve the plain `Intel/…int4-AutoRound` instead.
- **Target:** [`Intel/Qwen3.5-122B-A10B-int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) — INT4 (AutoRound/GPTQ) routed experts + attention, BF16 shared experts / embeddings / head, ~62 GiB. Safetensors, *not* GGUF — vLLM serves the HF checkpoint directly. - **Drafter:** [`z-lab/Qwen3.5-122B-A10B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) — 0.8B / 6-layer non-causal block-diffusion drafter (block 16), sharing the model's `embed_tokens` + `lm_head`, ~1.6 GiB.
- **Drafter:** [`z-lab/Qwen3.5-122B-A10B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) — 0.8B / 6-layer non-causal block-diffusion drafter (block 16), sharing the target's `embed_tokens` + `lm_head`, ~1.6 GiB. - **Engine:** [`vLLM`](https://github.com/vllm-project/vllm) 0.23, sm121 build with the DFlash PRs, via the prebuilt image `ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix`. No host build — the runtime patches in [`runtime/`](runtime/) are applied at serve time.
- **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GB LPDDR5X unified (~119 GiB usable), ~273 GB/s. - **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GB LPDDR5X unified (~119 GiB usable), ~273 GB/s.
## Quick start ## Quick start
@@ -42,16 +46,23 @@ This command:
1. Verifies the host (aarch64, GB10 / SM121, Docker GPU access, free disk). 1. Verifies the host (aarch64, GB10 / SM121, Docker GPU access, free disk).
2. Pulls the sm121 vLLM image (~40 GiB, one-time). 2. Pulls the sm121 vLLM image (~40 GiB, one-time).
3. Downloads the INT4 target (~62 GiB) and the DFlash drafter (~1.6 GiB) into the HF cache. 3. Downloads the hybrid checkpoint (~67 GiB) and the DFlash drafter (~1.6 GiB) into the HF cache.
4. Starts the `dflash` profile on `:8000`, waits until READY, and runs the 4. Starts the **`dense`** profile on `:8000`, waits until READY (~3 min), and runs
"capital of France" smoke test (asserts "Paris"). the "capital of France" smoke test (asserts "Paris").
To reuse a checkpoint that is already present and skip the ~62 GiB download: For the plain-DFlash agent profile (largest KV pool, no dense patches, serves the
Intel INT4 checkpoint):
```bash ```bash
# point at an existing checkpoint directory (mounted read-only at /model): ./install.sh --start --profile dflash
./install.sh --start --model-dir /path/to/Qwen3.5-122B-A10B-int4-AutoRound ```
# or reuse an existing HF cache (the download becomes a no-op if already present):
To reuse a checkpoint already on disk and skip the download:
```bash
# an existing checkpoint directory (mounted read-only at /model):
./install.sh --start --model-dir /path/to/checkpoint
# or reuse an existing HF cache (download becomes a no-op if already present):
./install.sh --start --hf-home /mnt/big/hf ./install.sh --start --hf-home /mnt/big/hf
``` ```
@@ -329,8 +340,9 @@ python3 scripts/conc_workloads.py --base-url http://127.0.0.1:8000 # concurrency
| Project | Role | | Project | Role |
|---|---| |---|---|
| [`bleysg/...int4-fp8-hybrid`](https://huggingface.co/bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid) | **the default served checkpoint** — this project's prebuilt hybrid INT4+FP8 |
| [`vLLM`](https://github.com/vllm-project/vllm) | the inference engine; served unmodified-on-disk | | [`vLLM`](https://github.com/vllm-project/vllm) | the inference engine; served unmodified-on-disk |
| [`Intel/...int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) · [`z-lab/...DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) | the target and drafter weights | | [`Intel/...int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) (INT4 base) · [`Qwen/...FP8`](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8) (FP8 donor) · [`z-lab/...DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) (drafter) | upstream weights the hybrid + `dflash` profiles build on |
| [`albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4`](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) | the MTP + hybrid-FP8 + int8-lm-head recipe; benchmark reference and source of the ported dense levers | | [`albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4`](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) | the MTP + hybrid-FP8 + int8-lm-head recipe; benchmark reference and source of the ported dense levers |
| [`Entrpi/ds4-on-spark`](https://github.com/Entrpi/ds4-on-spark) | sibling repo, same hardware, different model (DeepSeek-V4-Flash via ds4) | | [`Entrpi/ds4-on-spark`](https://github.com/Entrpi/ds4-on-spark) | sibling repo, same hardware, different model (DeepSeek-V4-Flash via ds4) |
| [Modal: *Speculative decoding is all you need*](https://modal.com/blog/spec-is-all-u-need) | the DFlash block-diffusion drafter and the task-dependent-acceptance framing | | [Modal: *Speculative decoding is all you need*](https://modal.com/blog/spec-is-all-u-need) | the DFlash block-diffusion drafter and the task-dependent-acceptance framing |