load speed: persist compile cache at $HF_HOME/.vllm_cache (was wiped each container boot, -42s/boot); propagate --load-format to mtp_serve; document startup time

This commit is contained in:
ent
2026-06-24 16:32:02 +10:00
parent bdb57d776d
commit d3688cc833
3 changed files with 15 additions and 1 deletions
+12
View File
@@ -129,6 +129,18 @@ streams queue rather than fail) or a smaller `--ctx`.
> [`scripts/monitor.sh`](scripts/monitor.sh) (OOM auto-kill guard) the first time
> at any new `gpu-mem` or `ctx`.
### Startup time
Time to READY is **~3 min** (vs ~12 min on the default loader). `--load-format
fastsafetensors` (default) reads the 6271 GiB **straight to the device**
**~32 s** vs ~8 min for the default mmap-backed per-tensor read + redundant
CPU→GPU copy, which is pathologically slow on GB10's current kernel (the copy is
also physically pointless on unified memory). It falls back to `nogds`
automatically — no GPUDirect Storage hardware required; override with
`LOAD_FORMAT=auto`. The remaining ~2.5 min is `torch.compile` (~42 s) + CUDA-graph
capture (~2 min); the compile cache is persisted at `$HF_HOME/.vllm_cache`, so
boots after the first skip the compile.
## Profiles
Selected with `--profile`: