docs: GB10 memory is 128 GB (119 GiB), not 128 GiB

This commit is contained in:
ent
2026-06-24 13:10:36 +10:00
parent dc9d40a1c4
commit 70c91f94a7
3 changed files with 7 additions and 7 deletions
+4 -4
View File
@@ -2,7 +2,7 @@
[`Qwen3.5-122B-A10B`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound)
(hybrid GDN + mamba + 128-expert MoE, ~10B active) running on a single
**NVIDIA DGX Spark** (GB10 / SM121, 128 GiB unified) under **vLLM**, with
**NVIDIA DGX Spark** (GB10 / SM121, 128 GB / 119 GiB unified) under **vLLM**, with
**[DFlash](https://modal.com/blog/spec-is-all-u-need) block-diffusion speculative
decode** and an optional **dense-bandwidth patch stack** — measured end-to-end,
with a per-token bandwidth model that explains every number.
@@ -25,7 +25,7 @@ so it's a lever for *base / low-acceptance* serving, not for the agent path.
- **Engine:** [`vLLM`](https://github.com/vllm-project/vllm) 0.23, sm121 build with the DFlash PRs, via the prebuilt image `ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix`. No host build — the four runtime patches in [`runtime/`](runtime/) are applied at serve time.
- **Target:** [`Intel/Qwen3.5-122B-A10B-int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) — INT4 (AutoRound/GPTQ) routed experts + attention, BF16 shared experts/embeddings/head, ~62 GiB. (Safetensors, *not* GGUF — vLLM serves HF checkpoints directly.)
- **Drafter:** [`z-lab/Qwen3.5-122B-A10B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) — 0.8B / 6-layer non-causal block-diffusion drafter (block 16), shares the target's `embed_tokens` + `lm_head`, ~1.6 GiB.
- **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GiB LPDDR5X unified, ~273 GB/s.
- **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GB LPDDR5X unified (~119 GiB usable), ~273 GB/s.
## Quick start
@@ -58,12 +58,12 @@ Preview without running: `... | bash -s -- --help`.
| | |
|---|---|
| Validated on | NVIDIA DGX Spark (GB10, SM121, 128 GiB unified) |
| Validated on | NVIDIA DGX Spark (GB10, SM121, 128 GB / 119 GiB unified) |
| Likely to work | other Blackwell with `--force` (untested) |
| Runtime | Docker + NVIDIA container runtime (`docker run --gpus all`) |
| Disk | ≥ 75 GiB free (image + weights); ≥ 150 GiB if `--build-hybrid` |
| OS | aarch64 Linux (Grace) |
| Memory | 128 GiB unified is enough for the model + DFlash drafter + KV @ 16k |
| Memory | 128 GB / 119 GiB unified is enough for the model + DFlash drafter + KV @ 16k |
GB10 is detected via `nvidia-smi --query-gpu=compute_cap` returning `12.1`;
anything else needs `--force`.
+1 -1
View File
@@ -1,7 +1,7 @@
# FINDINGS — DFlash + dense levers for Qwen3.5-122B-A10B on DGX Spark
Single-stream (c=1) decode of `Qwen3.5-122B-A10B` (hybrid GDN + mamba + 128-expert
MoE, ~10B active) on GB10 / SM121, 128 GiB unified, ~273 GB/s. The agent this
MoE, ~10B active) on GB10 / SM121, 128 GB / 119 GiB unified, ~273 GB/s. The agent this
backs (Hermes) is ~73 % tool-calls. All numbers temperature 0.
> **Credit.** This builds on [albond's recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)
+2 -2
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# install.sh — Qwen3.5-122B-A10B + DFlash speculative decode on NVIDIA DGX Spark
# (GB10 / SM121, 128 GiB unified), via vLLM in Docker.
# (GB10 / SM121, 128 GB / 119 GiB unified), via vLLM in Docker.
#
# curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash
# curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash -s -- --help
@@ -92,7 +92,7 @@ Flags:
--nspec N num_speculative_tokens (default 12 dflash/dense, 2 mtp, 0 base).
--port N Server port (default: $PORT).
--ctx N max-model-len (default: $CTX).
--gpu-mem F gpu-memory-utilization, keep <=0.84 on 128GiB (default: $GPU_MEM).
--gpu-mem F gpu-memory-utilization, keep <=0.84 on 128GB (default: $GPU_MEM).
--force Skip the GB10/SM121 host check.
--no-smoke Start the server but skip the Paris smoke test.