DUAL_CARD.md: annotate dual.yml as the "fast" tier (≡ vllm/qwen-27b-dual-fast)
and add the new "max accuracy" row (vllm/qwen-27b-dual-max — FP8 + int8-PTH KV,
🧪, boot/KV-validated, no TPS claim).
MULTI_CARD.md: the "Shipped TP=4 baselines" section documented vllm/dual4 +
vllm/dual4-dflash → composes that no longer exist (multi4 now has only
autoround-int4/mtp.yml + fp8/mtp.yml). Rewrote to vllm/qwen-27b-multi-fast +
-multi-max, carried Whamp's cross-rig fp8/MTP numbers onto multi-fast (same
config, renamed file; labelled cross-rig), and dropped the removed DFlash TP=4
variant (DFlash on Qwen3-Next vLLM is blocked, #39931 — kept its numbers as a
historical note). Fixed the recipe's stale filename (multi4.yml) + container/port
(8014 multi-fast / 8015 multi-max), and the "why one config" section → fast/max pair.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Post-release additive pass — the pre-existing high-traffic guides didn't
know `pull` exists. All edits additive, curated path untouched (same
discipline as the README migration nudge):
- FAQ.md: new Q "model not in the supported list — can I still run it?";
GGUF Q gets a v0.8.0 note (safetensors-only eval, GGUF→llama.cpp stays
curated/manual, cross-engine deferred); launch.sh answer points
non-catalog models at `pull`.
- SINGLE_CARD / DUAL_CARD / MULTI_CARD: one blockquote cross-link each to
docs/PULL.md ("not in the configs / any HF safetensors repo — both
paths work").
- ADDING_MODELS.md: reframed catalog-onboarding vs just-run-a-model
(`pull`); the doc is the heavier calibration-catalog promotion task,
not a prerequisite for serving.
- GLOSSARY.md: new "Universal pull (v0.8.0)" table (pull, dry-run,
confidence tier, boot-fit≠runtime, calibration backbone).
Leak-clean; all links resolve on master; CommonMark structure verified
(blockquotes/headings blank-line separated). Docs-only — triggers no CI
(only tags do); lands as post-v0.8.0 polish on master.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` /
`multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md,
MULTI_CARD.md). Compose filename topology prefix should match.
`dual4.yml` was an awkward outlier — it had `dual` in the prefix
but actually meant TP=4 (4-card config). The clean shape: GPU
count is implicit when there's no ambiguity (`single` always 1,
`dual` always 2), and explicit when there is (`multi3` / `multi4`
/ `multi8`).
Renames:
- docker-compose.dual4.yml → docker-compose.multi4.yml
- docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml
Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh
keep their existing names (backward compat for users running
`bash scripts/switch.sh vllm/dual4`); only the file paths in the
VARIANTS map are updated.
References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md,
models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md,
sibling-table cross-references in 7 other compose headers.
AGENTS.md "Topology prefix" row tightened: examples now read
`single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink`
which is actually an interconnect feature suffix; dropped `quad`
which doesn't exist as a name in our convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@lexhoefsloot's 3× 3090 setup uses CUDA_VISIBLE_DEVICES=1,2 to pin TP=2
to a same-PCIe-switch pair (with GPU 0 doing other work). That selection
pattern wasn't in MULTI_CARD.md — only the TP-table cell mentioning
"CUDA_VISIBLE_DEVICES=0,1" for the 3-card case.
Added a "Picking which cards to use on awkward counts" subsection right
after the valid-TP table covering:
- nvidia-smi topo -m interpretation
- Connectivity classes ranked for TP allreduce
(NV# > PIX > PXB > PHB > SYS)
- When same-switch (PIX) actually matters (server-class +
NUMA-aware boards) vs when it doesn't (consumer ATX,
everything is PHB anyway)
- The override compose pattern for CUDA_VISIBLE_DEVICES=1,2
- Cross-rig cite to @lexhoefsloot's 3-card rig
Refs: #49
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
After shipping report.sh --full / --soak / --stress flags in 8a29b95, the
bug report flow + troubleshooting guidance still pointed at older patterns.
Updated:
- .github/ISSUE_TEMPLATE/bug-report.yml — replaced single-flag suggestion
with a "pick the flag that matches your bug" decision table:
- boot crash / wrong output / tool-call regression → --verify (~2 min)
- OOM mid-conversation / agentic cliff → --soak (~25 min) — only test
that catches Cliff 2b
- TPS regression / cross-rig perf → --bench (~5 min)
- not sure / capture everything → --full (~35 min)
- docs/FAQ.md "Found a bug — what should I include?" — replaced the generic
"verify-full.sh output" note with the same flag decision table; updated
the troubleshooting-ladder pointers to use --verify (boot path) and
--full (multi-card TQ3+Genesis intersection bugs).
- docs/MULTI_CARD.md cross-rig contribution callouts (×2) — recommended
command upgraded from --bench to --full (or --bench fallback if soak
time-budget is tight, with explicit caveat that --bench skips Cliff 2b).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Maintainer rig is 2x 3090. Users with 3+ GPUs (4x 3090, 8x A6000,
mixed setups) have been asking "does club-3090 apply?" — yes, but we
can't ship pre-baked configs for hardware we can't validate.
New `docs/MULTI_CARD.md` explains:
- **What scales** going TP=2 → TP=4 → TP=8: per-card weights drop
proportionally, KV pool grows linearly, Cliff 2 disappears entirely
on TP=4+ (DeltaNet GDN forward state splits across cards).
- **What doesn't scale**: per-stream decode TPS without NVLink. PCIe
NCCL all-reduce overhead grows with TP count — TP=4 PCIe per-stream
may be lower than TP=2. Aggregate concurrent throughput still scales.
- **Valid TP values for Qwen3.6-27B**: 1, 2, 4, 5, 8, 10. Must divide
both 80 attention heads AND 5 KV heads cleanly. TP=3, 6, 7, 9 do NOT
work — vLLM errors at boot. Awkward GPU counts (3, 6, 7) need to use
the next-lower valid TP with idle cards.
- **Derivation recipe**: copy `dual.yml`, change three lines
(--tensor-parallel-size, --max-num-seqs, --max-num-batched-tokens),
pick a distinct container_name + port. Marlin pad-sub-tile-n patch
stays mounted (more relevant at higher TP, not less).
- **What to expect on TP=4** (4x 3090 PCIe): per-card peak ~16-18 GB
(vs 23.6 GB on TP=2), Cliff 2 doesn't apply, per-stream TPS likely
drops to ~50-65 narr / ~70-80 code from PCIe NCCL overhead, but KV
pool 2x larger means more concurrent streams fit.
- **What to expect on TP=8** (server-class): per-card pressure
essentially disappears. With NVLink fabric, per-stream TPS could
approach 1.6-1.8x single-card vs the ~1.0x we see on PCIe TP=2.
- **Cross-rig data ask**: TP=4 PCIe, TP=4 with NVLink, TP=8 server-
class, TP=4 mixed cards. Each is a coverage gap we'd love filled.
Honest disclaimer at the top: nothing in this doc is locally measured,
all derived from documented vLLM TP behavior + our TP=2 baseline +
Marlin pad math. Asks community contributors with 3+ card hardware to
share `bash scripts/report.sh --bench > my-rig.md` results.
Why no pre-baked composes:
1. Can't hardware-test them.
2. Hardware combinations explode (different VRAM, NVLink topology,
power profiles, allreduce characteristics — no single quad.yml
optimal for all).
3. Users at this scale are typically experienced — they need
methodology + constraints + dial, not hand-held tested compose.
If a community member contributes a tested compose for their topology
(verify-stress passing + bench numbers), we ship it under
models/qwen3.6-27b/vllm/compose/ with credit.
Cross-links: README.md "Pick your path" table now includes 3+ GPU row;
DUAL_CARD.md gets a header pointer to MULTI_CARD.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
