7e9f5cb930
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises "autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b — after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV), yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift). Fix WITHOUT breaking anything, via vLLM multi-served-name: - Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY (honest /v1/models id); the quant-specific name is retained as a live ALIAS. - LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the `-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift. - Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3, tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`) untouched; CHANGELOG + results/ history left as-is. Retiring the `-autoround` alias entirely is a deliberate later step once nothing still asks for it. Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names (root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
179 lines
8.5 KiB
YAML
179 lines
8.5 KiB
YAML
# ===========================================================================
|
||
# Profile (at-a-glance):
|
||
# Model: Qwen3.6-27B (Lorbus AutoRound INT4 + BF16 mtp.fc preserved)
|
||
# Topology: Dual 3090 (TP=2, NVLink auto-detected via NVLINK_MODE)
|
||
# Drafter: MTP n=3 (built-in)
|
||
# KV: fp8_e5m2 (1 byte/token)
|
||
# Vision: yes
|
||
# Max ctx: 262K (237K single-prompt verified)
|
||
# Genesis: none — intentionally Genesis-free (isolation control + fallback)
|
||
# Status: ✅ Production
|
||
# Quality: 109/150 (det 64/75 · sandbox 45/75) (--full, current harness 2026-06-07).
|
||
# In the 3-way fast/balanced/max A/B (same harness, same day): fast 109 ·
|
||
# balanced 105 · max 110 — a TIE (the 8-pack doesn't separate the quants).
|
||
# ⚠️ Earlier 129/150 (--full, 2026-05-09 harness) is NOT comparable — benchlocal-cli
|
||
# verifier fixes since then tightened scoring (mostly the agentic HA/CLI packs).
|
||
# Best for: General-purpose dual-card vision + tools + long-ctx default ⭐
|
||
# ---------------------------------------------------------------------------
|
||
# Dual RTX 3090 — DEFAULT for 2× cards. Qwen3.6-27B + MTP n=3 + fp8 KV + vision.
|
||
#
|
||
# What this gives you (vs the single-card default):
|
||
# - TP=2 across both 3090s
|
||
# - max_num_seqs=2 → 2 concurrent agents at full 262K context (KV pool 2.36×)
|
||
# - max_model_len=262144 → model's natural max, no rope_scaling needed
|
||
# - vision + tools + MTP n=3 + recall — everything on, no compromises
|
||
# - Measured (re-bench 2026-04-28 on club-3090 substrate, dev205 + Genesis v7.51-stable):
|
||
# 69.05 narr (CV 2.3%) / 88.58 code (CV 3.4%) TPS single-stream, AL 3.4, VRAM 23.6 GB/card
|
||
#
|
||
# What's intentionally NOT enabled:
|
||
# - TurboQuant KV — sidesteps vllm#40831 entirely. fp8_e5m2 is plenty for
|
||
# 262K across two cards. For 4-stream concurrency at 262K, switch to
|
||
# dual/turbo.yml (which uses TurboQuant KV + Genesis P65).
|
||
#
|
||
# Dependencies:
|
||
# - 2× RTX 3090 (Ampere SM 8.6), PCIe-only (no NVLink — works fine)
|
||
# - vLLM STABLE v0.21.0 image (pulled from Docker Hub) — no source overlays
|
||
# required (validated 2026-05-25: marlin-pad / PR-35936 / PR-41800 all
|
||
# unnecessary on a current vLLM; AutoRound INT4 TP=2 + MTP + vision boot clean).
|
||
#
|
||
# Qwen3.6-27b vLLM dual/multi configs (post-2026-05-31 prune — settled on one
|
||
# dual config; the A/B variants were deprecated):
|
||
#
|
||
# Slug Ctx Streams KV Vision Topology Notes
|
||
# vllm/dual (this) 262K 2 fp8 ✅ 2× PCIe THE dual default ⭐
|
||
# vllm/dual4 262K 4 fp8 ✅ 4× PCIe 4-card
|
||
# vllm/dual4-dflash 262K 2 FP16 ✅ 4× PCIe 4-card, peak code TPS
|
||
# (deprecated 2026-05-31, recover from git if needed: dual-dflash, dual-dflash-noviz,
|
||
# dual-tq3-nomtp, dual-bf16, dual-int8 — A/B leftovers superseded by vllm/dual.
|
||
# dual-turbo / dual-tq3-mtp* were already deprecated.)
|
||
# (NVLink rigs: any dual compose auto-detects NVLink at boot — no separate
|
||
# nvlink-* variant. NVLINK_MODE=force_on forces it if auto-detect misses.)
|
||
#
|
||
# Run:
|
||
# cd <repo>/models/qwen3.6-27b/vllm/compose
|
||
# docker compose -f dual/docker-compose.yml up -d
|
||
# ===========================================================================
|
||
# Hardware metadata (parsed by scripts/preflight.sh):
|
||
# Requires-min-vram-gb: 24
|
||
# Engine-profile: vllm-stable
|
||
# Requires-min-gpu-count: 2
|
||
# Tensor-parallel: 2
|
||
services:
|
||
vllm-qwen36-27b-dual:
|
||
# Pinned to a vLLM STABLE release (immutable — never purged from Docker Hub,
|
||
# unlike nightlies; this is the #407 pin-drift fix). Genesis-free + fp8 KV
|
||
# means no patch overlays are anchored here, so we ride stable releases.
|
||
# Bump the tag to a newer stable after a verify-stress + soak gate.
|
||
# Override with VLLM_IMAGE=... if you need a specific build.
|
||
image: ${VLLM_IMAGE:-vllm/vllm-openai:v0.24.0}
|
||
container_name: "${ESTATE_CONTAINER:-vllm-qwen36-27b-dual}"
|
||
restart: ${CLUB3090_RESTART:-unless-stopped}
|
||
ports:
|
||
- "${BIND_HOST:-0.0.0.0}:${ESTATE_PORT:-${PORT:-8010}}:8000"
|
||
volumes:
|
||
- ${MODEL_DIR:-../../../../../../models-cache}:/root/.cache/huggingface
|
||
# torch.compile + Triton kernel caches — first boot warms (~60-90 sec);
|
||
# subsequent boots reuse cached graphs. Pattern from Sander's PROD launch.
|
||
# Closes club-3090 #22.
|
||
- ../../../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
|
||
- ../../../cache/triton:/root/.triton/cache
|
||
# NOTE: vLLM source overlays (marlin-pad #40361, PR-35936 required-tool
|
||
# fallback, PR-41800 truncate_prompt_tokens) were dropped 2026-05-25 — all
|
||
# confirmed unnecessary on stable v0.21.0 (PR-41800 merged upstream; the
|
||
# other two don't trip on a current vLLM). Keeping zero source mounts is
|
||
# what makes this path pin-drift-immune. See docs/UPSTREAM.md.
|
||
# froggeric/Qwen-Fixed-Chat-Templates qwen3.6 — fixes 7 default-template
|
||
# bugs (empty <think></think> spam, </thinking> hallucination, unclosed
|
||
# think before tool call, no-user-query crash, developer role, etc.).
|
||
# See docs/UPSTREAM.md "Community templates / model assets" + the row at
|
||
# https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
|
||
- ../../../patches/froggeric-chat-template/chat_template.jinja:/etc/qwen-froggeric-chat-template.jinja:ro
|
||
# NVLink auto-detection — runs inside container at boot.
|
||
- ../../../../../../scripts/detect_nvlink.sh:/etc/club3090/detect_nvlink.sh:ro
|
||
environment:
|
||
- NVIDIA_VISIBLE_DEVICES=${ESTATE_GPUS:-${NVIDIA_VISIBLE_DEVICES:-all}}
|
||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-${OFFLINE:-0}}
|
||
- TRANSFORMERS_OFFLINE=${TRANSFORMERS_OFFLINE:-${OFFLINE:-0}}
|
||
- VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||
- NVLINK_MODE=${NVLINK_MODE:-auto}
|
||
- NCCL_CUMEM_ENABLE=0
|
||
- NCCL_P2P_DISABLE=1
|
||
- VLLM_NO_USAGE_STATS=1
|
||
- VLLM_USE_FLASHINFER_SAMPLER=1
|
||
- OMP_NUM_THREADS=1
|
||
# expandable_segments:True crashes boot on some setups (likely cuMemMap path).
|
||
# Known: JusefPol on NVLink (PR #31), WSL2 single-card 3090 Ti.
|
||
# Override via .env: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False
|
||
- PYTORCH_CUDA_ALLOC_CONF=${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True,max_split_size_mb:512}
|
||
shm_size: "16gb"
|
||
ipc: host
|
||
deploy:
|
||
resources:
|
||
reservations:
|
||
devices:
|
||
- driver: nvidia
|
||
count: all
|
||
capabilities: [gpu]
|
||
entrypoint:
|
||
- bash
|
||
- -c
|
||
- |
|
||
# VLLM_ENFORCE_EAGER=1 in compose/.env disables CUDA graphs — use on
|
||
# hardware where Cliff 2 GDN activation spikes occur at runtime
|
||
# (~50-65K active context tokens). Costs ~20-30% TPS in exchange for
|
||
# stability. See docs/HARDWARE.md "Note for WSL2 / Windows users".
|
||
# NVLink auto-detection (sets NCCL env vars, _NVLINK_ENABLED).
|
||
source /etc/club3090/detect_nvlink.sh
|
||
if [ "$${_NVLINK_ENABLED:-0}" = "1" ]; then
|
||
exec vllm serve $${VLLM_ENFORCE_EAGER:+--enforce-eager} "$$@"
|
||
else
|
||
exec vllm serve $${VLLM_ENFORCE_EAGER:+--enforce-eager} --disable-custom-all-reduce "$$@"
|
||
fi
|
||
- --
|
||
command:
|
||
- --model
|
||
- /root/.cache/huggingface/qwen3.6-27b-autoround-int4
|
||
- --served-model-name
|
||
- qwen3.6-27b
|
||
- qwen3.6-27b-autoround
|
||
- --quantization
|
||
- auto_round
|
||
- --dtype
|
||
- float16
|
||
- --tensor-parallel-size
|
||
- "${TP:-2}"
|
||
- --pipeline-parallel-size
|
||
- "${PP:-1}"
|
||
- --max-model-len
|
||
- "${MAX_MODEL_LEN:-262144}"
|
||
- --gpu-memory-utilization
|
||
- "${GPU_MEMORY_UTILIZATION:-0.92}"
|
||
- --max-num-seqs
|
||
- "2"
|
||
- --max-num-batched-tokens
|
||
- "8192"
|
||
- --kv-cache-dtype
|
||
- "${KV_CACHE_DTYPE:-fp8_e5m2}"
|
||
- --trust-remote-code
|
||
# froggeric chat-template override (see volumes block + docs/UPSTREAM.md)
|
||
- --chat-template
|
||
- /etc/qwen-froggeric-chat-template.jinja
|
||
- --reasoning-parser
|
||
- qwen3
|
||
- --default-chat-template-kwargs
|
||
- '{"enable_thinking": false}'
|
||
- --enable-auto-tool-choice
|
||
- --tool-call-parser
|
||
- qwen3_coder
|
||
- --enable-prefix-caching
|
||
- --enable-chunked-prefill
|
||
- --speculative-config
|
||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||
- --override-generation-config
|
||
- '{"temperature":${TEMP:-${TEMPERATURE:-0.6}},"top_p":${TOP_P:-0.95},"top_k":${TOP_K:-20},"min_p":${MIN_P:-0.0},"repetition_penalty":${REPEAT_PENALTY:-1.0}}'
|
||
- --host
|
||
- 0.0.0.0
|
||
- --port
|
||
- "8000"
|