Entrpi
2cb5033fb8
Reclaims over-reserved memory rather than spending headroom, so it honors the conservative 0.82 / 14 GB-OS-reserve choice (still ~15 GiB free under peak load) while closing most of the gap to the dflash pool (456k): - int8 lm-head: build int8 on first warmup forward, then free the dead bf16 copy (~1.4 GiB) + empty_cache() so the block lands in the KV pool. Safe: the DFlash drafter shares the one int8 lm_head module (verified one int8 build; drafter ckpt has no lm_head/embed) and tie_word_embeddings=False, so bf16 has no reader (bias fallback = int8-GEMV + add-bias). SPARK_KEEP_BF16_LMHEAD=1 restores keep-bf16. - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 in serve.sh/mtp_serve.sh: returns the CUDA-graph over-estimate (~0.6 GiB; actual capture ~0.14 GiB from the wide 0.82 headroom). Overridable with =1. - scripts/stress_mem.sh: drive the concurrency bank while sampling host RAM to prove a gpu-mem setting survives peak load without swapping. Validated: pool 426,610 tokens, coherent output, drafter acceptance 4-12, no new swap, throughput unchanged. max-batched-tokens 8192->4096 tested as a KV lever and rejected (only +2.5k tokens, costs prefill speed).
32 lines
1.4 KiB
Bash
Executable File
32 lines
1.4 KiB
Bash
Executable File
#!/bin/bash
|
|
# mtp_serve.sh — native qwen3_5 MTP-N head for the comparison (`--profile mtp`).
|
|
# The MTP head (1 layer, reuses target KV/embed/lm_head) is in the Intel
|
|
# checkpoint (mtp.layers.0) — no separate drafter, no unify patch needed.
|
|
# $1 = num_speculative_tokens (default 2 = the "MTP-2" recipe); $2 = backend.
|
|
set -euo pipefail
|
|
NSPEC="${1:-2}"
|
|
BACKEND="${2:-flash_attn}"
|
|
MODEL="${MODEL:-Intel/Qwen3.5-122B-A10B-int4-AutoRound}"
|
|
MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}"
|
|
GPU_MEM="${GPU_MEM:-0.82}"
|
|
MAX_NUM_SEQS="${MAX_NUM_SEQS:-3}"
|
|
MAX_BATCHED_TOKENS="${MAX_BATCHED_TOKENS:-8192}"
|
|
LOAD_FORMAT="${LOAD_FORMAT:-fastsafetensors}"
|
|
PORT="${PORT:-8000}"
|
|
# Reclaim the CUDA-graph memory over-estimate to KV (see serve.sh). Set =1 to restore.
|
|
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS="${VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS:-0}"
|
|
echo "[mtp] qwen3_5_mtp — backend=$BACKEND, num_speculative_tokens=$NSPEC, model=$MODEL"
|
|
exec vllm serve "$MODEL" \
|
|
--served-model-name qwen \
|
|
--host 0.0.0.0 --port "$PORT" \
|
|
--max-model-len "$MAX_MODEL_LEN" \
|
|
--max-num-seqs "$MAX_NUM_SEQS" \
|
|
--max-num-batched-tokens "$MAX_BATCHED_TOKENS" \
|
|
--gpu-memory-utilization "$GPU_MEM" \
|
|
--no-enable-prefix-caching \
|
|
--enable-chunked-prefill \
|
|
--trust-remote-code \
|
|
--load-format "$LOAD_FORMAT" \
|
|
--attention-backend "$BACKEND" \
|
|
--speculative-config "{\"method\":\"qwen3_5_mtp\",\"num_speculative_tokens\":$NSPEC,\"model\":\"$MODEL\"}"
|