Audit + reconcile dual-card compose headers, patches README, setup output

After the user flagged "are you validating all composer files" — ran a full dry-run audit of all 9 composes via docker compose config, extracted key flags (TP, max_len, mem_util, KV dtype, spec-decode), and found several doc-vs-code mismatches inherited from the predecessor repos. Compose header fixes: - docker-compose.dual.yml — header described it as inheriting from "single-card project's default", said "fp8 is plenty for 64K" (stale — file actually does 262K). Updated to reflect: this IS the dual-card default, fp8 is plenty for full 262K, plus a variant matrix showing all 4 dual files with their actual TPS / streams / KV / vision. - docker-compose.dual-turbo.yml — header claimed kv-cache-dtype was `turboquant_3bit_nc` but the file actually ships `turboquant_k8v4`. This mismatch was in the predecessor too; we kept the file (not the header) since k8v4 is what was tested. Updated header to reflect reality + noted the predecessor doc claim for archaeology. - docker-compose.dual-dflash.yml — header said max_model_len "drops from 262K to 16K" (stale dev-cycle comment); actual is 185K. Fixed. Also added: KV cache is FP16 (DFlash + head_size=256 + non-causal has no fp8/turbo Ampere backend), the bfloat16 dtype workaround for vllm#40334, and clear positioning vs the noviz variant. - docker-compose.dual-dflash-noviz.yml — minor: file path in "to run" pointed at the old compose/ dir; updated to new layout path. patches/README.md — was framed as dual-card-only ("we don't run Genesis here") but the patches dir is now shared across single and dual variants. Rewrote with a per-patch + per-variant matrix: - patch_tolist_cudagraph.py: single-default + dual-turbo - patch_pr40798_workspace.py: research artifact, no compose mounts - genesis/: single-default + tools-text + dual-turbo - Marlin pad fork (external /opt/ai/vllm-src/): all 4 dual composes Added a Genesis env-opts table showing per-patch toggles and which composes enable each. scripts/setup.sh — final-output Next-steps block referenced the OLD relative path `cd compose && docker compose up -d`, which would fail in the new layout. Updated to: cd models/<model>/vllm/compose && docker compose up -d Plus added a clear note about the Marlin pad fork dependency for dual-card composes (with the git-clone command users need to run once before booting any dual-card variant). YAML validation: `docker compose config` passes for all 9 composes with MODEL_DIR set. Volume paths resolve, env vars substitute, no syntax errors. Single-card default smoke-tested earlier (10/10 verify-full.sh checks pass); dual-card composes pass YAML validation but require a 2× 3090 rig to actually boot — left for cross-rig users to confirm.
2026-04-28 11:09:16 +00:00
parent 7f00e52140
commit 0f33561b6b
6 changed files with 168 additions and 94 deletions
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml
@@ -1,15 +1,22 @@
 # ===========================================================================
-# DFlash spec-decode on TP=2 — TEXT-ONLY VARIANT (no vision tower).
+# Dual-card DFlash text-only — TP=2 + DFlash N=5 + 200K ctx + NO vision.
 #
-# vs docker-compose.dflash.yml (the with-vision variant): drops MoonViT to
-# free ~0.78 GiB per card → bumps max_model_len 185K → 200K (close to the
+# vs `docker-compose.dual-dflash.yml` (the with-vision variant): drops MoonViT
+# to free ~0.78 GiB per card → bumps max_model_len 185K → 200K (close to the
 # absolute DFlash ceiling on dual-3090 with FP16 KV).
 #
-# Numbers (D5 row in BENCHMARKS.md):
-#   77 narr / 124 code TPS, 200K ctx, 1 stream, AL 4.27 warm
+# Best for: long single-prompt text workloads (RAG / summarization / code
+# review across very long codebases) where you don't need image input.
+#
+# Measured: 77 narr / 124 code TPS, 200K ctx, 1 stream, code AL ~4.27 warm.
+#
+# Same DFlash caveats as dual-dflash.yml apply: single-stream only, FP16 KV
+# (no fp8/turbo on DFlash + Ampere), `--dtype bfloat16` for vllm#40334
+# workaround.
 #
 # To run:
-#   cd compose && docker compose -f docker-compose.dflash-noviz.yml up -d
+#   cd <repo>/models/qwen3.6-27b/vllm/compose
+#   docker compose -f docker-compose.dual-dflash-noviz.yml up -d
 # ===========================================================================
 services:
  vllm-qwen36-27b-dual-dflash:
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
@@ -1,18 +1,28 @@
 # ===========================================================================
-# DFlash spec-decode on TP=2 — z-lab Qwen3.6-27B-DFlash draft.
+# Dual-card DFlash — TP=2 + DFlash N=5 spec-decode + 185K ctx + vision.
 #
-# Hypothesis (per BENCHMARKS row 4 for Qwen3.5-27B on the same draft family):
-# DFlash N=4 unlocks ~+47% throughput vs no-spec on dense Qwen3-Next at TP=2.
-# Qwen3.6-27B is dense too (no MoE expert layers) so pad-Marlin should be a
-# no-op and the DFlash×AutoRound combo that crashed on Qwen3.6-35B-A3B (MoE)
-# shouldn't fire here.
+# The peak-single-stream code TPS variant. DFlash N=5 (z-lab's Qwen3.6-27B
+# draft model) unlocks 78 narr / 128 code TPS — substantially faster than
+# the fp8/MTP default's 71/89 on code-heavy workloads (code AL ~4.7 vs MTP's 3.4).
 #
 # Trades vs the fp8/MTP default:
-#   - max_model_len drops from 262K to 16K (DFlash + cudagraph + KV pool tight)
-#   - vision tower ON (verified 2026-04-25 — MoonViT works with DFlash on Qwen3.6-27B)
+#   - max_model_len 262K → 185K (DFlash draft model adds ~500 MB VRAM per card,
+#     and cudagraph + KV pool budget tightens at higher ctx)
+#   - Single-stream only (parallel draft + main model uses concurrency budget)
+#   - DFlash bias toward code hurts narrative AL (drops to ~2.8 on prose)
+#
+# Vision: ON (verified 2026-04-25 — MoonViT works with DFlash on Qwen3.6-27B).
+# Drop vision for 200K ctx via docker-compose.dual-dflash-noviz.yml.
+#
+# KV cache: FP16 (default — DFlash needs head_size=256 + non-causal attention,
+# and no Ampere backend supports that triple with fp8/turbo KV).
+#
+# Required workaround: vllm#40334 dtype mismatch fix is not yet upstream;
+# compose sets `--dtype bfloat16` to match the draft's training dtype.
 #
 # To run:
-#   cd compose && docker compose -f docker-compose.dflash.yml up -d
+#   cd <repo>/models/qwen3.6-27b/vllm/compose
+#   docker compose -f docker-compose.dual-dflash.yml up -d
 # ===========================================================================
 services:
  vllm-qwen36-27b-dual-dflash:
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml
@@ -1,20 +1,26 @@
 # ===========================================================================
-# EXPLORATORY: TP=2 + TurboQuant KV (turboquant_3bit_nc) + MTP n=3 + Genesis v7.14.
+# Dual-card Turbo — TP=2 + TurboQuant KV (turboquant_k8v4) + MTP n=3 + Genesis v7.14.
 #
-# Question: with v7.14 P65 (cudagraph downgrade for spec-decode) applied, does
-# MTP × TurboQuant work on dual-card TP=2 with vision + tools + 262K?
+# The 4-stream concurrent-serving variant. Trades ~25% per-stream TPS vs the
+# fp8 default for ~9× the KV pool capacity, unlocking 4 concurrent streams at
+# full 262K context. Aggregate throughput exceeds fp8 above ~3 concurrent users.
 #
-# If yes: this is a strict superset of the fp8 default (more KV pool → either
-# more concurrent streams at 262K or more ctx at same concurrency).
-#
-# Genesis v7.14 also includes:
+# Genesis v7.14 P65 (cudagraph downgrade for spec-decode) makes MTP × TurboQuant
+# work on dual-card TP=2 with vision + tools + 262K. Other relevant patches:
 #   - P4 hybrid turboquant support (replaces standalone PR #39931 patches)
 #   - P5 KV page-size unification for hybrid models
 #   - P64 streaming MTP tool-call edge case
 #   - P66 cudagraph_capture_sizes divisibility filter
-# Plus our patch_tolist_cudagraph.py for the .tolist() crash class.
+# Plus patch_tolist_cudagraph.py for the .tolist() crash class.
 #
-# To run: cd compose && docker compose -f docker-compose.turbo.yml up -d
+# Note: kv-cache-dtype is `turboquant_k8v4` (8-bit keys, 4-bit values; ~6 bits
+# average per token). Earlier predecessor docs claimed `turboquant_3bit_nc`,
+# but the file shipped k8v4 — we kept k8v4 since that's what the published
+# concurrency numbers were measured against.
+#
+# To run:
+#   cd <repo>/models/qwen3.6-27b/vllm/compose
+#   docker compose -f docker-compose.dual-turbo.yml up -d
 # ===========================================================================
 services:
  vllm-qwen36-27b-dual-turbo:
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
@@ -1,24 +1,35 @@
 # ===========================================================================
-# Dual RTX 3090 — Qwen3.6-27B Lorbus int4-AutoRound + MTP + fp8 KV + vision.
+# Dual RTX 3090 — DEFAULT for 2× cards. Qwen3.6-27B + MTP n=3 + fp8 KV + vision.
 #
-# What this gives you (vs the single-card project's default):
-#   - TP=2 across both 3090s — meaningfully higher TPS, much more headroom
-#   - max_num_seqs=2 → 2 concurrent agents at full 262K context (KV pool 168K/2.36×)
+# What this gives you (vs the single-card default):
+#   - TP=2 across both 3090s
+#   - max_num_seqs=2 → 2 concurrent agents at full 262K context (KV pool 2.36×)
 #   - max_model_len=262144 → model's natural max, no rope_scaling needed
 #   - vision + tools + MTP n=3 + recall — everything on, no compromises
+#   - Measured: 71 narr / 89 code TPS single-stream
 #
 # What's intentionally NOT enabled:
-#   - TurboQuant KV — sidesteps vllm#40831 entirely. fp8_e5m2 has none of
-#     that bug's caveats and is plenty for 64K context across both cards.
+#   - TurboQuant KV — sidesteps vllm#40831 entirely. fp8_e5m2 is plenty for
+#     262K across two cards. For 4-stream concurrency at 262K, switch to
+#     docker-compose.dual-turbo.yml (which uses TurboQuant KV + Genesis P65).
 #
 # Dependencies:
 #   - 2× RTX 3090 (Ampere SM 8.6), PCIe-only (no NVLink — works fine)
 #   - vLLM PR #40361 (Marlin pad-sub-tile-n) — patched files volume-mounted
 #     from /opt/ai/vllm-src/. PR is open upstream; drop the mount when it
-#     lands. See patches/README.md for context.
+#     lands. See ../patches/README.md for the git-clone command.
+#
+# All dual-card variants in this dir:
+#
+#   File                                Ctx    Streams   Narr/Code TPS   KV       Vision
+#   docker-compose.dual.yml (this)      262K   2         71 / 89          fp8      ✅
+#   docker-compose.dual-turbo.yml       262K   4         58 / 69          TQ k8v4  ✅
+#   docker-compose.dual-dflash.yml      185K   1         78 / 128         FP16     ✅
+#   docker-compose.dual-dflash-noviz... 200K   1         77 / 124         FP16     ❌
 #
 # Run:
-#   cd compose && docker compose up -d
+#   cd <repo>/models/qwen3.6-27b/vllm/compose
+#   docker compose -f docker-compose.dual.yml up -d
 # ===========================================================================
 services:
  vllm-qwen36-27b-dual:
--- a/models/qwen3.6-27b/vllm/patches/README.md
+++ b/models/qwen3.6-27b/vllm/patches/README.md
@@ -1,27 +1,40 @@
-# Patches
+# Patches for Qwen3.6-27B on vLLM

-This stack tracks `vllm:nightly` (latest dev branch) and adds patches
-only when something actively breaks for our TP=2 + Lorbus + fp8 + MTP
-configuration.
+This directory contains the model + engine-specific patches that different compose variants apply at boot. Each patch is opt-in — composes mount only what they need.

-Currently we have **one** patch dependency.
+| File | Used by | Purpose |
+|---|---|---|
+| `patch_tolist_cudagraph.py` | single-card default + dual-turbo | CUDA graph capture fix for TurboQuant continuation prefill |
+| `patch_pr40798_workspace.py` | (none — research artifact) | Negative-result reproducer for vllm#40798 |
+| `genesis/` | single-card default + dual-turbo | Sandermage's Genesis v7.14 patch tree (gitignored; fetched by setup.sh) |
+| External: `/opt/ai/vllm-src/` (Marlin pad fork) | all 4 dual-card composes | vLLM PR #40361 patched source, not a file in this repo |

-## vLLM PR #40361 — Marlin pad-sub-tile-n
+---

-**What it fixes:** Marlin's `GPTQ_MARLIN_MIN_THREAD_N=64` blocks any W4A16
-shard where per-rank out-dim falls below 64. Hits on Ampere sm_86 with
-AutoRound INT4 quants under TP=2 — Qwen3.6-27B's DeltaNet `linear_attn`
-projections are one such case (similar in shape to the Qwen3.5 GDN
-`in_proj_ba` issue tracked in #35924). No stock Ampere fallback kernel
-works (Machete/CutlassW4A8 are Hopper-only, AllSpark needs `group_size=-1`).
+## When you need each patch

-**Status:** PR open at https://github.com/vllm-project/vllm/pull/40361,
-labeled `bug`, awaiting maintainer `ready` label.
+- **Single-card default** (`docker-compose.yml`) — uses TurboQuant 3-bit KV + Genesis v7.14 P65 + tolist patch. Fetched by `setup.sh qwen3.6-27b`.
+- **Single-card fast-chat / no-genesis-mtp / minimal** — fp8 KV, no patches needed.
+- **Single-card tools-text** — fp8 KV + Genesis (for the qwen3-coder tool parser); no tolist patch (fp8 doesn't trip the bug).
+- **Dual-card default + DFlash variants** — fp8 / fp16 KV. Need only the Marlin pad fork (no Genesis, no tolist).
+- **Dual-card turbo** — TQ KV + Genesis v7.14 + tolist + Marlin pad fork.

-**How we apply it:** the patched source lives at `/opt/ai/vllm-src/`
-on branch `marlin-pad-sub-tile-n` (fork: `noonghunna/vllm`). The
-compose volume-mounts the two patched files over the nightly image's
-copies — no rebuild needed.
+---
+
+## vLLM PR #40361 — Marlin pad-sub-tile-n (dual-card requirement)
+
+**What it fixes:** Marlin's `GPTQ_MARLIN_MIN_THREAD_N=64` blocks any W4A16 shard where per-rank out-dim falls below 64. Hits on Ampere SM 8.6 with AutoRound INT4 quants under TP=2.
+
+**Status:** PR open at https://github.com/vllm-project/vllm/pull/40361, labeled `bug`, awaiting maintainer review.
+
+**Setup:** all 4 dual-card composes volume-mount the patched source from `/opt/ai/vllm-src/`. Clone the fork once before booting any dual-card compose:
+
+```bash
+sudo mkdir -p /opt/ai && sudo chown $USER /opt/ai
+git clone -b marlin-pad-sub-tile-n https://github.com/noonghunna/vllm.git /opt/ai/vllm-src
+```
+
+The compose then mounts two specific files over the nightly image's copies — no rebuild needed:

 ```yaml
 volumes:
@@ -29,49 +42,68 @@ volumes:
  - /opt/ai/vllm-src/vllm/model_executor/kernels/linear/mixed_precision/MPLinearKernel.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/MPLinearKernel.py:ro
 ```

-When the PR merges, drop both mounts and the compose just uses upstream
-nightly.
+When PR #40361 lands, drop both mounts and the dual composes just use upstream nightly.

-## What we DO NOT need (and why)
+### Brittleness note

- **Genesis patches** (Sandermage/genesis-vllm-patches) — only needed
-  when using TurboQuant KV on hybrid models. We're using fp8_e5m2 KV,
-  which goes through vLLM's stock attention backends. No hybrid-gate
-  bypass required.
- **`patch_tolist_cudagraph.py`** (from the single-card project) — the
-  bug it fixes is in `turboquant_attn.py`'s continuation-prefill path.
-  Doesn't fire on fp8 KV.
- **`cudagraph_mode=NONE` workaround for #40831** — the bug is in the
-  TurboQuant attention backend. Doesn't fire on fp8 KV. We get full
-  cudagraph + torch.compile speed.
- **A pinned image digest** — the single-card project pins
-  `vllm/vllm-openai@sha256:9bba4628...` because Genesis's anchor-text
-  patches need a specific upstream layout. We don't run Genesis here,
-  so we ride `vllm:nightly` and rebuild dependencies only when nightly
-  actively breaks our path.
-
-So this setup is meaningfully simpler than the single-card project:
-just one upstream patch dependency (Marlin pad), and even that drops
-out when #40361 lands.
-
-## Brittleness note
-
-The Marlin patch is a **file override** (volume-mount the entire
-patched `marlin.py` and `MPLinearKernel.py` over the container's copies),
-not an anchor-based disk-edit. If upstream refactors those files in
-nightly while #40361 is still open, our patched versions could fall
-out of sync with the rest of vLLM's import graph and crash at load
-time with `ImportError` or `AttributeError`.
+The Marlin patch is a **file override** (volume-mount the entire patched `marlin.py` and `MPLinearKernel.py` over the container's copies), not an anchor-based disk-edit. If upstream refactors those files in nightly while #40361 is still open, our patched versions could fall out of sync with the rest of vLLM's import graph and crash at load time with `ImportError` or `AttributeError`.

 If that happens:
+1. Pull the latest patched files from https://github.com/noonghunna/vllm/tree/marlin-pad-sub-tile-n into `/opt/ai/vllm-src/`.
+2. If the fork is also out of date, rebase it on current main and re-apply the pad-sub-tile-n change.
+3. Pin the image to the last-known-good digest as a fallback while you sort it out.

-1. Pull the latest patched files from
-   https://github.com/noonghunna/vllm/tree/marlin-pad-sub-tile-n into
-   `/opt/ai/vllm-src/`.
-2. If the fork is also out of date, rebase it on current main and
-   re-apply the pad-sub-tile-n change.
-3. Pin the image to the last-known-good digest as a fallback while
-   you sort it out.
+---

-This is the price we pay for "ride latest nightly" + a still-open patch.
-If/when #40361 lands upstream, this entire concern disappears.
+## `patch_tolist_cudagraph.py` (single-card default + dual-turbo)
+
+**What it fixes:** A `.tolist()` GPU→CPU sync in TurboQuant continuation-prefill that's illegal during CUDA graph capture. Trips when `--speculative-config` + `--enable-chunked-prefill` + `turboquant_*` KV are combined. Without this patch, vLLM crashes during engine warmup with:
+
+```
+turboquant_attn.py:570  qsl = query_start_loc.tolist()
+RuntimeError: Cannot copy between CPU and CUDA tensors during CUDA graph
+capture unless the CPU tensor is pinned.
+```
+
+**How:** disk-edit at container startup. Wraps both `.tolist()` sites with `torch.cuda.is_current_stream_capturing()` guards. Idempotent — re-running it on already-patched files is a no-op.
+
+**Why this lives here, not Genesis:** the equivalent functionality also ships as Genesis P78 (since v7.14, with attribution). Both running together is harmless. We keep our standalone version because it's pinned in this repo's git history, while Genesis is fetched fresh per setup.
+
+---
+
+## `patch_pr40798_workspace.py` (research artifact)
+
+**What this is NOT:** a fix.
+
+**What this IS:** a backport of vLLM PR #40798 (workspace-manager refactor) that we hypothesized would close [#40880](https://github.com/vllm-project/vllm/issues/40880) (TurboQuant × spec-decode × cudagraph corruption). Probe 8 in our 9-probe forensics ladder — see [INTERNALS.md](../INTERNALS.md) for full context. The backport was clean; the bug persisted.
+
+We keep this file in the repo for reproducibility of the negative result. No compose mounts it.
+
+---
+
+## Genesis tree (`genesis/`)
+
+Sandermage's [genesis-vllm-patches](https://github.com/Sandermage/genesis-vllm-patches) — a runtime monkey-patcher for vLLM that fixes several Qwen3-Next architectural bugs (hybrid-attention TurboQuant gate, MTP head loading, cudagraph spec-decode downgrade for #40880, etc).
+
+Setup:
+- Cloned by `bash scripts/setup.sh qwen3.6-27b` at the pinned tag (currently `v7.51-stable-2026-04-27`)
+- Override the pin via `GENESIS_PIN=<tag-or-commit>` env var
+- Gitignored from this repo (we don't vendor someone else's tree)
+
+Mounted by composes that need TurboQuant KV (single-default + dual-turbo) or the Qwen3 tool-parser fixes.
+
+---
+
+## Genesis env-opts (per-patch toggles)
+
+Genesis v7.14+ ships several patches as opt-in env flags. Each compose enables only the subset relevant to its config:
+
+| Env var | What it does | Used by |
+|---|---|---|
+| `GENESIS_ENABLE_P65_TURBOQUANT_SPEC_CG_DOWNGRADE=1` | Forces cudagraph PIECEWISE for spec-decode (closes #40880) | single-default, dual-turbo |
+| `GENESIS_ENABLE_P66_CUDAGRAPH_SIZE_FILTER=1` | Filters cudagraph capture sizes for spec-decode divisibility | single-default, dual-turbo |
+| `GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1` | Streaming MTP tool-call edge case fix | single-default, fast-chat, tools-text, dual-turbo |
+| `GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1` | Long-ctx tool-format adherence | single-default, fast-chat, tools-text, dual-turbo |
+| `GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1` | Long-ctx tool-format reminder | single-default, fast-chat, tools-text, dual-turbo |
+
+Composes that don't load Genesis (no-genesis-mtp, minimal, dual-default, dual-dflash, dual-dflash-noviz) ignore these env vars.
--- a/scripts/setup.sh
+++ b/scripts/setup.sh
@@ -163,14 +163,22 @@ if [[ "$count" == "0" ]]; then
 fi

 echo ""
-echo "Done. ${count} shards SHA-verified, ${GENESIS_DIR} in place."
+echo "[done]    ${count} shards SHA-verified."
+[[ -d "${GENESIS_DIR}/.git" ]] && echo "          Genesis pinned at ${GENESIS_PIN} ($(cd "${GENESIS_DIR}" && git rev-parse --short HEAD))."
 echo ""
-echo "Next:"
-echo "  cd compose && docker compose up -d"
+echo "Next — single-card vLLM (default):"
+echo "  cd models/${MODEL_NAME}/vllm/compose && docker compose up -d"
 echo "  docker logs -f vllm-qwen36-27b"
 echo ""
-echo "Wait for 'Application startup complete', then test with:"
+echo "For dual-card composes, you ALSO need the Marlin pad fork mounted at"
+echo "/opt/ai/vllm-src/ (vLLM PR #40361 — open upstream, drops out when it lands):"
+echo "  sudo mkdir -p /opt/ai && sudo chown \$USER /opt/ai"
+echo "  git clone -b marlin-pad-sub-tile-n https://github.com/noonghunna/vllm.git /opt/ai/vllm-src"
 echo ""
+echo "Then:"
+echo "  cd models/${MODEL_NAME}/vllm/compose && docker compose -f docker-compose.dual.yml up -d"
+echo ""
+echo "Sanity test (after 'Application startup complete'):"
 echo "  curl -sf http://localhost:8020/v1/chat/completions \\"
 echo "    -H 'Content-Type: application/json' \\"
 echo "    -d '{\"model\":\"qwen3.6-27b-autoround\",\"messages\":[{\"role\":\"user\",\"content\":\"Capital of France?\"}],\"max_tokens\":30}'"