510a76c885
--gpus selection was NVIDIA_VISIBLE_DEVICES-only, which CDI runtimes (NixOS hardware.nvidia-container-toolkit, nvidia-ctk cdi) IGNORE — mog's 3x3090 rig got all cards exposed and vLLM took the wrong two. Index-based CUDA_VISIBLE_DEVICES can't fix it either: the classic runtime renumbers the exposed set in-container. UUIDs solve both at once: - launch.sh --gpus N,M resolves indices -> GPU UUIDs (nvidia-smi -i) and exports BOTH NVIDIA_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES as UUIDs. Classic runtime: exposes-by-UUID; CUDA mask agrees (renumbering-proof). CDI: exposure comes from device_ids (nvidia.com/gpu=all) and the CUDA UUID mask is what pins the cards. Index fallback if the query fails. - 26 composes gain a bare `- CUDA_VISIBLE_DEVICES` passthrough (unset -> absent -> zero change for existing users); stale commented hints removed. - preflight: selector consumers are host-index-based -> new _preflight_selector_normalize maps GPU-xxxx tokens back to indices at the single choke point (the first live boot caught this: preflight rejected the UUID selector). - new guard test-compose-gpu-mask-passthrough.sh: every compose with the NVIDIA env line must pass CUDA_VISIBLE_DEVICES through + launch.sh must keep the UUID resolution. - docs/HARDWARE.md: "Pinning specific GPUs (and CDI / NixOS runtimes)" — the runtime table, the CDI deploy-block swap, the in-container renumbering gotcha, manual recipes. LIVE-VERIFIED on this rig (classic runtime, 2x3090): launch.sh --variant vllm/minimal --gpus 1 -> host GPU0 = 1 MiB (empty), GPU1 = 20.7 GB (model), both env vars UUID-set in-container. CDI leg validation goes to mog (NixOS reporter). Follow-up (tracked on #610): estate ESTATE_GPUS UUID-ization for CDI multi-instance rigs; post-boot visible-GPU-count sanity check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
268 lines
15 KiB
YAML
268 lines
15 KiB
YAML
# ===========================================================================
|
||
# Profile (at-a-glance):
|
||
# Model: Gemma 4 31B (Intel AutoRound INT4)
|
||
# Topology: Dual 3090 (TP=2, NVLink auto-detected via NVLINK_MODE)
|
||
# Drafter: MTP n=4 (Google's official `gemma-4-31B-it-assistant`, BF16 0.5B)
|
||
# KV: int8_per_token_head (1 byte/token, via vendored PR #40391 overlay)
|
||
# Vision: yes
|
||
# Max ctx: 262K native default (dual-card max-ctx rule) @ 4-stream cap; INT8 PTH pool ~2× BF16
|
||
# Genesis: N/A — Genesis is Qwen3-Next-specific
|
||
# Status: 🗑️ Deprecated
|
||
# Caveats: Superseded by vllm/gemma-31b-dual (cyankiwi bf16 @224K, stock v0.24.0, overlay-free) in the
|
||
# v0.24.0 consolidation (2026-07-02). This was the 262K int8-PTH+#40391 path on v0.22.0 — NOT
|
||
# migrated to v0.24.0 (int8-PTH silently craters recall there; #40391 open/unmerged upstream).
|
||
# The 262K int8-PTH path returns when #40391 merges. Use vllm/gemma-31b-dual (bf16, ~224K).
|
||
# Quality: 105/150 (70%) thinking-OFF default — det 59/75 · sandbox 46/75
|
||
# TC 13/15 · IF 8/15 · SO 14/15 · DE 10/15 · RM 14/15 · BF 13/15 · HA 14/20 · CLI 19/40
|
||
# (--full, v0.22.0, 2026-05-31). Reasoning-ON A/B (with --reasoning-parser gemma4) = 107/150
|
||
# — a WASH (+2, within ±5-7 8-pack noise: IF+6/BF+2 offset by RM-2/HA-2/DE-1/CLI-1) at higher
|
||
# latency, so thinking-off stays the default; reasoning-on is a clean opt-in. Soak (continuous)
|
||
# PASS: 0 errors, 0 VRAM growth, 100% TPS retention, p50 87.96 TPS.
|
||
# Best for: Gemma 4 dual-card default — full 262K ctx + vision + 4 streams ⭐
|
||
# (stable-but-131K alternative without the overlay: vllm/gemma-bf16-mtp)
|
||
# ---------------------------------------------------------------------------
|
||
# Gemma-4-31B-it (Intel AutoRound INT4) + Google MTP "assistant" drafter
|
||
# 2× RTX 3090 TP=2 + per-token-head INT8 KV + ~120K ctx target.
|
||
#
|
||
# Default KV_DTYPE is int8_per_token_head because that's what runs on Ampere
|
||
# sm_86. fp8 PTH variants require Ada/Blackwell — see hardware compatibility
|
||
# table below. Override KV_DTYPE=fp8_per_token_head if you're on Ada/Blackwell
|
||
# and want to A/B against fp8 precision.
|
||
#
|
||
# Status: COMMUNITY-CONTRIBUTED, EXPERIMENTAL.
|
||
# Post-rebase of vLLM PR #40391 by @lisp19 (Gemma 4 KV cache page-size
|
||
# alignment for per-token-head quantization). PR is upstream-open + stalled
|
||
# on maintainer review; cross-rig validation by @cferra (sm_120 Blackwell
|
||
# TP=4) and @noonghunna (sm_86 Ampere — this rig). Local rebase resolves
|
||
# the conflict against post-Mamba-hybrid-support main.
|
||
#
|
||
# Companion composes (post-2026-05-31 rename):
|
||
# bf16-mtp.yml — vllm/gemma-bf16-mtp: BF16 KV, 131K (no overlays, stock v0.22.0)
|
||
# int8.yml — vllm/gemma-int8-mtp: INT8 PTH KV, 262K native (this file)
|
||
# single (TP=1) — boot-OOMs on Ampere 24 GB → vllm/gemma-mtp-tp1 DEPRECATED;
|
||
# single-card Gemma → beellama/gemma-dflash
|
||
#
|
||
# Hardware × per-token-head dtype compatibility:
|
||
#
|
||
# | | INT8 PTH | FP8 e4m3 PTH | FP8 e5m2 PTH | NVFP4 |
|
||
# | sm_90 Hopper | ✅ | ✅ | ✅ | ✅ |
|
||
# | sm_89 Ada | ✅ | ✅ | ✅ | ✅ |
|
||
# | sm_120 Black. | ✅ | ✅ | ✅ | ✅ |
|
||
# | sm_86 Ampere | ✅ | ❌ | ❌ | ❌ |
|
||
#
|
||
# Ampere blocker for fp8 PTH: Triton kernel uses `fp8e4nv` which sm_86
|
||
# doesn't implement (Ampere only supports `fp8e4b15` and `fp8e5`).
|
||
# ValueError: "type fp8e4nv not supported in this architecture"
|
||
# → fires from `_initialize_kv_caches` boot path. Confirmed 2026-05-08.
|
||
#
|
||
# INT8 PTH dispatches to standard PyTorch torch.int8 ops (not Triton fp8
|
||
# kernel) so it runs on every consumer GPU including Ampere. Same KV
|
||
# memory savings as FP8 (1 byte/element); slightly different precision
|
||
# characteristics (better near-zero, narrower dynamic range than e5m2).
|
||
#
|
||
# For Ada/Blackwell users: override KV_DTYPE=fp8_per_token_head if you
|
||
# want to benchmark FP8 vs INT8 PTH precision on your hardware.
|
||
#
|
||
# Why this exists (the upstream-blocker chain, summarized):
|
||
#
|
||
# Gemma 4 has interleaved attention with two head_dims:
|
||
# - sliding/local layers: head_dim=256
|
||
# - global/full layers: head_dim=512
|
||
# ANY per-token-head KV format adds scale metadata per token, breaking
|
||
# vLLM's `unify_kv_cache_spec_page_size()` because the resulting page
|
||
# sizes don't share a clean ratio. PR #40391 pre-pads the global layers
|
||
# to a 1040-byte factor and routes standard attention through a new
|
||
# `get_padded_attention_kv_cache_shape` helper to unblock this family.
|
||
#
|
||
# Without PR #40391, dual.yml is forced to bf16 KV + 32K ctx ceiling
|
||
# (KV pool exhausted at higher). With PR #40391 + INT8 PTH KV, ctx
|
||
# ceiling lifts to ~120K (4×) at the same TP=2 mem-util budget.
|
||
#
|
||
# Bench targets (TBD post-validation on this rig):
|
||
# narrative ~109 wall TPS (parity with bf16 path; KV format affects
|
||
# ctx ceiling, not per-token TPS materially)
|
||
# code ~142 wall TPS
|
||
# Max ctx ~120K (4× lift vs dual.yml's 32K)
|
||
#
|
||
# Models:
|
||
# target: Intel/gemma-4-31B-it-int4-AutoRound (21.2 GB, vision preserved)
|
||
# draft : google/gemma-4-31B-it-assistant (0.5B / 927 MB BF16)
|
||
#
|
||
# Vendored overlay — ONE remaining (2026-05-31 pin bump v0.21.0 → v0.22.0):
|
||
# vLLM PR #40391 — gemma4 per-token-head KV page-size alignment (head_dim
|
||
# 256 SWA vs 512 global). This is the ONLY reason the compose is pinned: it
|
||
# makes INT8 PTH KV bootable on Gemma 4. STILL OPEN upstream → REBASED onto
|
||
# stock v0.22.0 (the old v0.21.0-era full-module copies ImportError'd on
|
||
# v0.22.0 — they lacked v0.22.0's get_kv_cache_spec_kind) and re-delivered as
|
||
# a LEAN boot-time diff-apply (~240-line patch + 1 new file via install.sh),
|
||
# not 7 full-module mounts. Live-validated 2026-05-31 (boots, KV pool 447K @
|
||
# 262K, coherent generation). See: ../../../patches/vllm-pr40391-v0.22.0/README.md
|
||
#
|
||
# The 2 tool-parser/truncate overlays this compose used to carry are now IN
|
||
# stock v0.22.0 (PR #41800 merged 2026-05-06, PR #41991 merged 2026-05-08;
|
||
# both predate the 2026-05-15 v0.21.0 tag too) and PR #42006 was dropped by
|
||
# maintainer call (bf16-mtp ships prod without it). Verify before any pin bump:
|
||
# gh api repos/vllm-project/vllm/pulls/40391 --jq '.state, .merged_at' # drop overlay when MERGED
|
||
# gh api repos/vllm-project/vllm/pulls/42006 --jq '.state, .merged_at' # re-test trigger (docs/UPSTREAM.md)
|
||
# ===========================================================================
|
||
# Hardware metadata (parsed by scripts/preflight.sh):
|
||
# Requires-min-vram-gb: 24
|
||
# Engine-profile: vllm-gemma-stable
|
||
# Requires-min-gpu-count: 2
|
||
# Tensor-parallel: 2
|
||
services:
|
||
vllm-gemma-4-31b-mtp-int8:
|
||
image: ${VLLM_IMAGE:-vllm/vllm-openai:v0.22.0}
|
||
container_name: "${ESTATE_CONTAINER:-vllm-gemma-4-31b-mtp-int8}"
|
||
restart: ${CLUB3090_RESTART:-unless-stopped}
|
||
ports:
|
||
- "${ESTATE_PORT:-${PORT:-8032}}:8000"
|
||
volumes:
|
||
- ${MODEL_DIR:-../../../../../../models-cache}:/root/.cache/huggingface
|
||
# torch.compile + Triton kernel caches — first boot warms (~5-7 min);
|
||
# subsequent boots reuse cached graphs (~3 min). Separate cache dirs
|
||
# from dual.yml because the overlay file set differs (cudagraph
|
||
# capture would re-key on the patched kernel paths anyway).
|
||
- ../../../cache/torch_compile_int8:/root/.cache/vllm/torch_compile_cache
|
||
- ../../../cache/triton_int8:/root/.triton/cache
|
||
# ---- vLLM PR #40391 — LEAN diff-apply overlay (install_script) ----
|
||
# The ONLY remaining overlay. Mounts the patch dir (the ~240-line #40391
|
||
# diff + the one new helper file) and applies it to stock v0.22.0 at boot
|
||
# via install.sh — NOT 7 full-module file mounts (the old, drift-prone
|
||
# vllm-pr40391-rebased/ approach). Drop the whole dir when #40391 merges.
|
||
# See ../../../patches/vllm-pr40391-v0.22.0/README.md.
|
||
- ../../../patches/vllm-pr40391-v0.22.0:/etc/club3090/pr40391:ro
|
||
# vLLM PR #42006 — Gemma 4 MTP streaming multi-tool-call fix (both duals
|
||
# carry it; stock v0.22.0 drops non-last tool-call args when streaming).
|
||
- ../../../patches/vllm-pr42006-v0.22.0:/etc/club3090/pr42006:ro
|
||
# NVLink auto-detection — runs inside container at boot.
|
||
- ../../../../../../scripts/detect_nvlink.sh:/etc/club3090/detect_nvlink.sh:ro
|
||
# --------------------------------------------------------------------
|
||
# 2026-05-31: dropped 2 overlays that are now IN stock v0.22.0 (verified
|
||
# against the running image) + 1 by maintainer call, when bumping the pin
|
||
# v0.21.0 → v0.22.0 (only #40391 survives — still open upstream):
|
||
# - PR #41800 (truncate_prompt_tokens) — merged 2026-05-06, in v0.22.0 stock.
|
||
# - PR #41991 (tool-parser bounds) — merged 2026-05-08, in v0.22.0 stock.
|
||
# - PR #42006 (MTP streaming multi-tool) — still OPEN, but bf16-mtp ships
|
||
# production on v0.22.0 without it → dropped from both for an identical
|
||
# minimal overlay surface. Tracked in docs/UPSTREAM.md (re-test trigger:
|
||
# streaming multi-tool-call regression).
|
||
environment:
|
||
- NVIDIA_VISIBLE_DEVICES=${ESTATE_GPUS:-${NVIDIA_VISIBLE_DEVICES:-all}}
|
||
# CUDA-level GPU mask — the runtime-agnostic half of GPU selection
|
||
# (#610): launch.sh --gpus exports GPU UUIDs here. CDI runtimes (NixOS,
|
||
# nvidia-ctk cdi) IGNORE NVIDIA_VISIBLE_DEVICES, so this mask is what
|
||
# actually pins cards there; on the classic runtime the UUID form is
|
||
# renumbering-proof. Unset → absent (no masking).
|
||
- CUDA_VISIBLE_DEVICES
|
||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-${OFFLINE:-0}}
|
||
- TRANSFORMERS_OFFLINE=${TRANSFORMERS_OFFLINE:-${OFFLINE:-0}}
|
||
- VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||
- NVLINK_MODE=${NVLINK_MODE:-auto}
|
||
- NCCL_CUMEM_ENABLE=0
|
||
- NCCL_P2P_DISABLE=1
|
||
- VLLM_NO_USAGE_STATS=1
|
||
- OMP_NUM_THREADS=1
|
||
- PYTORCH_CUDA_ALLOC_CONF=${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True,max_split_size_mb:512}
|
||
- VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
|
||
- TRITON_CACHE_DIR=/root/.triton/cache
|
||
shm_size: "16gb"
|
||
ipc: host
|
||
deploy:
|
||
resources:
|
||
reservations:
|
||
devices:
|
||
- driver: nvidia
|
||
count: all
|
||
capabilities: [gpu]
|
||
entrypoint:
|
||
- /bin/bash
|
||
- -c
|
||
- |
|
||
# vLLM PR #40391 (Gemma 4 per-token-head KV) — apply the lean diff to
|
||
# stock v0.22.0 BEFORE vllm imports. Fail-loud: a non-zero exit aborts
|
||
# boot rather than serving a half-patched engine.
|
||
bash /etc/club3090/pr40391/install.sh
|
||
# vLLM PR #42006 (Gemma 4 streaming multi-tool-call fix).
|
||
bash /etc/club3090/pr42006/install.sh
|
||
# NVLink auto-detection (sets NCCL env vars, _NVLINK_ENABLED).
|
||
source /etc/club3090/detect_nvlink.sh
|
||
if [ "$${_NVLINK_ENABLED:-0}" = "1" ]; then
|
||
exec vllm serve "$$@"
|
||
else
|
||
exec vllm serve --disable-custom-all-reduce "$$@"
|
||
fi
|
||
- --
|
||
command:
|
||
- --override-generation-config
|
||
- '{"temperature":${TEMP:-${TEMPERATURE:-1.0}},"top_p":${TOP_P:-0.95},"top_k":${TOP_K:-64},"min_p":${MIN_P:-0.0},"repetition_penalty":${REPEAT_PENALTY:-1.0}}'
|
||
- --host
|
||
- 0.0.0.0
|
||
- --port
|
||
- "8000"
|
||
- --model
|
||
- /root/.cache/huggingface/gemma-4-31b-autoround-int4
|
||
- --served-model-name
|
||
- gemma-4-31b
|
||
- gemma-4-31b-autoround
|
||
- --tensor-parallel-size
|
||
- "${TP:-2}"
|
||
- --pipeline-parallel-size
|
||
- "${PP:-1}"
|
||
# ---- KV format ----
|
||
# Default `int8_per_token_head` runs on every consumer GPU including
|
||
# sm_86 Ampere (uses standard PyTorch torch.int8 ops, not the Triton
|
||
# fp8e4nv kernel that sm_86 doesn't implement).
|
||
# Ada/Blackwell users can override KV_DTYPE=fp8_per_token_head to A/B
|
||
# against fp8 PTH precision.
|
||
# NOT `fp8_e5m2` (legacy tensor-wide fp8, rejected by query_quant
|
||
# allowlist on Gemma 4 even with PR #40391 — that's a separate query
|
||
# quantization compatibility check). NOT `fp8_e4m3` either (Triton
|
||
# fp8e4nv kernel not supported on sm_86 Ampere).
|
||
- --kv-cache-dtype
|
||
- "${KV_DTYPE:-int8_per_token_head}"
|
||
# ---- max-model-len (TP=2, INT8 PTH KV) — DUAL-CARD RULE: prioritize max
|
||
# context over concurrency (see docs/DUAL_CARD.md). --max-num-seqs is a
|
||
# CAP, not a reservation; the INT8 PTH pool (~2× the BF16 pool, measured
|
||
# ~354K-455K tok @ 0.95) holds a single 262K request with room to spare,
|
||
# so defaulting the ceiling to the model's NATIVE 262144 is ~free for
|
||
# short-request concurrency (4-wide opportunistic) — it just lets one
|
||
# request reach full native context.
|
||
- --max-model-len
|
||
- "${MAX_MODEL_LEN:-${CTX:-262144}}"
|
||
- --gpu-memory-utilization
|
||
- "${GPU_MEMORY_UTILIZATION:-0.95}"
|
||
# Lower --max-num-seqs only to GUARANTEE long requests are never preempted:
|
||
# max-num-seqs=4 + 262144 (default — native ctx, 4-wide for shorter reqs)
|
||
# max-num-seqs=2 (2 concurrent long-ctx agents, un-preempted)
|
||
# max-num-seqs=1 (one long request, never queued)
|
||
# ⚠️ vision + near-max ctx: drop max-num-seqs/mem-util (thin headroom at 262K).
|
||
- --max-num-seqs
|
||
- "${MAX_NUM_SEQS:-4}"
|
||
# Vision tower: max_tokens_per_mm_item=2496 must fit in batched tokens.
|
||
- --max-num-batched-tokens
|
||
- "4096"
|
||
- --trust-remote-code
|
||
# Tool-call support — required for soak / agent traffic that includes
|
||
# `tools: [...]` array. The vendored tool-parser overlay above also
|
||
# contributes here.
|
||
- --enable-auto-tool-choice
|
||
- --tool-call-parser
|
||
- gemma4
|
||
# Route Gemma-4's <|channel>thought…<channel|> reasoning trace into
|
||
# reasoning_content (vLLM ships Gemma4ReasoningParser). Without it,
|
||
# enable_thinking responses leave the trace in `content` → breaks
|
||
# JSON/structured output. Safe with thinking OFF (no-ops when the model
|
||
# emits no channel tokens). Validated 2026-05-31.
|
||
- --reasoning-parser
|
||
- gemma4
|
||
- --chat-template
|
||
- /vllm-workspace/examples/tool_chat_template_gemma4.jinja
|
||
# SPEC_N_MAX overridable for matched-config head-to-head against
|
||
# qwen3.6-27b/vllm/compose/dual/autoround-int4/int8.yml (Qwen's built-in MTP caps at
|
||
# n=3 architecturally, so the canonical apples-to-apples bench uses
|
||
# SPEC_N_MAX=3 here).
|
||
- --speculative-config
|
||
- '{"model":"/root/.cache/huggingface/gemma-4-31b-it-assistant","num_speculative_tokens":${SPEC_N_MAX:-4}}'
|