Entrpi
2cb5033fb8
Reclaims over-reserved memory rather than spending headroom, so it honors the conservative 0.82 / 14 GB-OS-reserve choice (still ~15 GiB free under peak load) while closing most of the gap to the dflash pool (456k): - int8 lm-head: build int8 on first warmup forward, then free the dead bf16 copy (~1.4 GiB) + empty_cache() so the block lands in the KV pool. Safe: the DFlash drafter shares the one int8 lm_head module (verified one int8 build; drafter ckpt has no lm_head/embed) and tie_word_embeddings=False, so bf16 has no reader (bias fallback = int8-GEMV + add-bias). SPARK_KEEP_BF16_LMHEAD=1 restores keep-bf16. - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 in serve.sh/mtp_serve.sh: returns the CUDA-graph over-estimate (~0.6 GiB; actual capture ~0.14 GiB from the wide 0.82 headroom). Overridable with =1. - scripts/stress_mem.sh: drive the concurrency bank while sampling host RAM to prove a gpu-mem setting survives peak load without swapping. Validated: pool 426,610 tokens, coherent output, drafter acceptance 4-12, no new swap, throughput unchanged. max-batched-tokens 8192->4096 tested as a KV lever and rejected (only +2.5k tokens, costs prefill speed).
34 lines
1.3 KiB
Bash
34 lines
1.3 KiB
Bash
#!/bin/bash
|
|
# stress_mem.sh — drive the server with the concurrency bank while sampling host
|
|
# memory, to prove a gpu-memory-utilization setting survives peak load WITHOUT
|
|
# swapping (the failure mode that hard-freezes the Spark). Reports min available
|
|
# RAM and max swap observed across the run.
|
|
#
|
|
# usage: stress_mem.sh [LEVELS] (default "1,2,3")
|
|
# Run on the box; the server (container qwen-spark) must be READY on :8000.
|
|
set -uo pipefail
|
|
LEVELS="${1:-1,2,3}"
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
LOG=/tmp/stress_memlog.$$
|
|
|
|
# baseline
|
|
echo "[stress] swap before:"; free -m | awk '/Swap:/{print " swap_used_MiB",$3}'
|
|
echo "[stress] avail before:"; free -m | awk '/Mem:/{print " avail_MiB",$7}'
|
|
|
|
# background sampler: timestamp, avail MiB, swap-used MiB, every 1s
|
|
( for i in $(seq 1 900); do
|
|
free -m | awk -v t="$i" '/Mem:/{a=$7} /Swap:/{s=$3} END{print t, a, s}'
|
|
sleep 1
|
|
done ) > "$LOG" 2>/dev/null &
|
|
SPID=$!
|
|
|
|
echo "[stress] running conc_workloads --levels $LEVELS ..."
|
|
python3 "$HERE/conc_workloads.py" --levels "$LEVELS"
|
|
RC=$?
|
|
|
|
kill "$SPID" 2>/dev/null
|
|
echo "[stress] === memory envelope during load ==="
|
|
awk 'NF>=3 {if(min==""||$2<min)min=$2; if($3>maxs)maxs=$3} END{print " min_avail_MiB", min, " max_swap_MiB", maxs+0}' "$LOG"
|
|
rm -f "$LOG"
|
|
echo "[stress] conc_workloads exit=$RC"
|