docs: GB10 memory is 128 GB (119 GiB), not 128 GiB
This commit is contained in:
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
[`Qwen3.5-122B-A10B`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound)
|
[`Qwen3.5-122B-A10B`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound)
|
||||||
(hybrid GDN + mamba + 128-expert MoE, ~10B active) running on a single
|
(hybrid GDN + mamba + 128-expert MoE, ~10B active) running on a single
|
||||||
**NVIDIA DGX Spark** (GB10 / SM121, 128 GiB unified) under **vLLM**, with
|
**NVIDIA DGX Spark** (GB10 / SM121, 128 GB / 119 GiB unified) under **vLLM**, with
|
||||||
**[DFlash](https://modal.com/blog/spec-is-all-u-need) block-diffusion speculative
|
**[DFlash](https://modal.com/blog/spec-is-all-u-need) block-diffusion speculative
|
||||||
decode** and an optional **dense-bandwidth patch stack** — measured end-to-end,
|
decode** and an optional **dense-bandwidth patch stack** — measured end-to-end,
|
||||||
with a per-token bandwidth model that explains every number.
|
with a per-token bandwidth model that explains every number.
|
||||||
@@ -25,7 +25,7 @@ so it's a lever for *base / low-acceptance* serving, not for the agent path.
|
|||||||
- **Engine:** [`vLLM`](https://github.com/vllm-project/vllm) 0.23, sm121 build with the DFlash PRs, via the prebuilt image `ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix`. No host build — the four runtime patches in [`runtime/`](runtime/) are applied at serve time.
|
- **Engine:** [`vLLM`](https://github.com/vllm-project/vllm) 0.23, sm121 build with the DFlash PRs, via the prebuilt image `ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix`. No host build — the four runtime patches in [`runtime/`](runtime/) are applied at serve time.
|
||||||
- **Target:** [`Intel/Qwen3.5-122B-A10B-int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) — INT4 (AutoRound/GPTQ) routed experts + attention, BF16 shared experts/embeddings/head, ~62 GiB. (Safetensors, *not* GGUF — vLLM serves HF checkpoints directly.)
|
- **Target:** [`Intel/Qwen3.5-122B-A10B-int4-AutoRound`](https://huggingface.co/Intel/Qwen3.5-122B-A10B-int4-AutoRound) — INT4 (AutoRound/GPTQ) routed experts + attention, BF16 shared experts/embeddings/head, ~62 GiB. (Safetensors, *not* GGUF — vLLM serves HF checkpoints directly.)
|
||||||
- **Drafter:** [`z-lab/Qwen3.5-122B-A10B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) — 0.8B / 6-layer non-causal block-diffusion drafter (block 16), shares the target's `embed_tokens` + `lm_head`, ~1.6 GiB.
|
- **Drafter:** [`z-lab/Qwen3.5-122B-A10B-DFlash`](https://huggingface.co/z-lab/Qwen3.5-122B-A10B-DFlash) — 0.8B / 6-layer non-causal block-diffusion drafter (block 16), shares the target's `embed_tokens` + `lm_head`, ~1.6 GiB.
|
||||||
- **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GiB LPDDR5X unified, ~273 GB/s.
|
- **Hardware:** NVIDIA DGX Spark, GB10, SM121, 128 GB LPDDR5X unified (~119 GiB usable), ~273 GB/s.
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -58,12 +58,12 @@ Preview without running: `... | bash -s -- --help`.
|
|||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Validated on | NVIDIA DGX Spark (GB10, SM121, 128 GiB unified) |
|
| Validated on | NVIDIA DGX Spark (GB10, SM121, 128 GB / 119 GiB unified) |
|
||||||
| Likely to work | other Blackwell with `--force` (untested) |
|
| Likely to work | other Blackwell with `--force` (untested) |
|
||||||
| Runtime | Docker + NVIDIA container runtime (`docker run --gpus all`) |
|
| Runtime | Docker + NVIDIA container runtime (`docker run --gpus all`) |
|
||||||
| Disk | ≥ 75 GiB free (image + weights); ≥ 150 GiB if `--build-hybrid` |
|
| Disk | ≥ 75 GiB free (image + weights); ≥ 150 GiB if `--build-hybrid` |
|
||||||
| OS | aarch64 Linux (Grace) |
|
| OS | aarch64 Linux (Grace) |
|
||||||
| Memory | 128 GiB unified is enough for the model + DFlash drafter + KV @ 16k |
|
| Memory | 128 GB / 119 GiB unified is enough for the model + DFlash drafter + KV @ 16k |
|
||||||
|
|
||||||
GB10 is detected via `nvidia-smi --query-gpu=compute_cap` returning `12.1`;
|
GB10 is detected via `nvidia-smi --query-gpu=compute_cap` returning `12.1`;
|
||||||
anything else needs `--force`.
|
anything else needs `--force`.
|
||||||
|
|||||||
+1
-1
@@ -1,7 +1,7 @@
|
|||||||
# FINDINGS — DFlash + dense levers for Qwen3.5-122B-A10B on DGX Spark
|
# FINDINGS — DFlash + dense levers for Qwen3.5-122B-A10B on DGX Spark
|
||||||
|
|
||||||
Single-stream (c=1) decode of `Qwen3.5-122B-A10B` (hybrid GDN + mamba + 128-expert
|
Single-stream (c=1) decode of `Qwen3.5-122B-A10B` (hybrid GDN + mamba + 128-expert
|
||||||
MoE, ~10B active) on GB10 / SM121, 128 GiB unified, ~273 GB/s. The agent this
|
MoE, ~10B active) on GB10 / SM121, 128 GB / 119 GiB unified, ~273 GB/s. The agent this
|
||||||
backs (Hermes) is ~73 % tool-calls. All numbers temperature 0.
|
backs (Hermes) is ~73 % tool-calls. All numbers temperature 0.
|
||||||
|
|
||||||
> **Credit.** This builds on [albond's recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)
|
> **Credit.** This builds on [albond's recipe](https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4)
|
||||||
|
|||||||
+2
-2
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# install.sh — Qwen3.5-122B-A10B + DFlash speculative decode on NVIDIA DGX Spark
|
# install.sh — Qwen3.5-122B-A10B + DFlash speculative decode on NVIDIA DGX Spark
|
||||||
# (GB10 / SM121, 128 GiB unified), via vLLM in Docker.
|
# (GB10 / SM121, 128 GB / 119 GiB unified), via vLLM in Docker.
|
||||||
#
|
#
|
||||||
# curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash
|
# curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash
|
||||||
# curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash -s -- --help
|
# curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/main/install.sh | bash -s -- --help
|
||||||
@@ -92,7 +92,7 @@ Flags:
|
|||||||
--nspec N num_speculative_tokens (default 12 dflash/dense, 2 mtp, 0 base).
|
--nspec N num_speculative_tokens (default 12 dflash/dense, 2 mtp, 0 base).
|
||||||
--port N Server port (default: $PORT).
|
--port N Server port (default: $PORT).
|
||||||
--ctx N max-model-len (default: $CTX).
|
--ctx N max-model-len (default: $CTX).
|
||||||
--gpu-mem F gpu-memory-utilization, keep <=0.84 on 128GiB (default: $GPU_MEM).
|
--gpu-mem F gpu-memory-utilization, keep <=0.84 on 128GB (default: $GPU_MEM).
|
||||||
--force Skip the GB10/SM121 host check.
|
--force Skip the GB10/SM121 host check.
|
||||||
--no-smoke Start the server but skip the Paris smoke test.
|
--no-smoke Start the server but skip the Paris smoke test.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user