From 70c91f94a7745a19d210a34d361daa3cd060f88e Mon Sep 17 00:00:00 2001 From: ent Date: Wed, 24 Jun 2026 13:10:36 +1000 Subject: [PATCH] docs: GB10 memory is 128 GB (119 GiB), not 128 GiB --- README.md | 8 ++++---- docs/FINDINGS.md | 2 +- install.sh | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 7846449..c358d18 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [`Qwen3.5-122B-A10B`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) (hybrid GDN + mamba + 128-expert MoE, ~10B active) running on a single -**NVIDIA DGX Spark** (GB10 / SM121, 128 GiB unified) under **vLLM**, with +**NVIDIA DGX Spark** (GB10 / SM121, 128 GB / 119 GiB unified) under **vLLM**, with **[DFlash](https://modal.com/blog/spec-is-all-u-need) block-diffusion speculative decode** and an optional **dense-bandwidth patch stack** — measured end-to-end, with a per-token bandwidth model that explains every number. @@ -25,7 +25,7 @@ so it's a lever for *base / low-acceptance* serving, not for the agent path. - **Engine:** [`vLLM`](https://github.com/vllm-project/vllm) 0.23, sm121 build with the DFlash PRs, via the prebuilt image `ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix`. No host build — the four runtime patches in [`runtime/`](runtime/) are applied at serve time. - **Target:** [`Intel/Qwen3.5-122B-A10B-int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) — INT4 (AutoRound/GPTQ) routed experts + attention, BF16 shared experts/embeddings/head, ~62 GiB. (Safetensors, *not* GGUF — vLLM serves HF checkpoints directly.) - **Drafter:** [`z-lab/Qwen3.5-122B-A10B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) — 0.8B / 6-layer non-causal block-diffusion drafter (block 16), shares the target's `embed_tokens` + `lm_head`, ~1.6 GiB. -- **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GiB LPDDR5X unified, ~273 GB/s. +- **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GB LPDDR5X unified (~119 GiB usable), ~273 GB/s. ## Quick start @@ -58,12 +58,12 @@ Preview without running: `... | bash -s -- --help`. | | | |---|---| -| Validated on | NVIDIA DGX Spark (GB10, SM121, 128 GiB unified) | +| Validated on | NVIDIA DGX Spark (GB10, SM121, 128 GB / 119 GiB unified) | | Likely to work | other Blackwell with `--force` (untested) | | Runtime | Docker + NVIDIA container runtime (`docker run --gpus all`) | | Disk | ≥ 75 GiB free (image + weights); ≥ 150 GiB if `--build-hybrid` | | OS | aarch64 Linux (Grace) | -| Memory | 128 GiB unified is enough for the model + DFlash drafter + KV @ 16k | +| Memory | 128 GB / 119 GiB unified is enough for the model + DFlash drafter + KV @ 16k | GB10 is detected via `nvidia-smi --query-gpu=compute_cap` returning `12.1`; anything else needs `--force`. diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index 856fed3..c552346 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -1,7 +1,7 @@ # FINDINGS — DFlash + dense levers for Qwen3.5-122B-A10B on DGX Spark Single-stream (c=1) decode of `Qwen3.5-122B-A10B` (hybrid GDN + mamba + 128-expert -MoE, ~10B active) on GB10 / SM121, 128 GiB unified, ~273 GB/s. The agent this +MoE, ~10B active) on GB10 / SM121, 128 GB / 119 GiB unified, ~273 GB/s. The agent this backs (Hermes) is ~73 % tool-calls. All numbers temperature 0. > **Credit.** This builds on [albond's recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4) diff --git a/install.sh b/install.sh index 1b72421..1b6a87f 100755 --- a/install.sh +++ b/install.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # install.sh — Qwen3.5-122B-A10B + DFlash speculative decode on NVIDIA DGX Spark -# (GB10 / SM121, 128 GiB unified), via vLLM in Docker. +# (GB10 / SM121, 128 GB / 119 GiB unified), via vLLM in Docker. # # curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash # curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash -s -- --help @@ -92,7 +92,7 @@ Flags: --nspec N num_speculative_tokens (default 12 dflash/dense, 2 mtp, 0 base). --port N Server port (default: $PORT). --ctx N max-model-len (default: $CTX). - --gpu-mem F gpu-memory-utilization, keep <=0.84 on 128GiB (default: $GPU_MEM). + --gpu-mem F gpu-memory-utilization, keep <=0.84 on 128GB (default: $GPU_MEM). --force Skip the GB10/SM121 host check. --no-smoke Start the server but skip the Paris smoke test.