feat(qwen3.6-27b/vllm): add dual4 + dual4-dflash composes (TP=4, 4×3090, #44)

First 4-card variants for Qwen3.6-27B vLLM. Two new composes for 4× RTX 3090 PCIe rigs: - `docker-compose.dual4.yml` — TP=4 fp8/MTP baseline. 63 narr / 76 code TPS, 6.77× concurrency at 262K, ~23.5 GB/card peak - `docker-compose.dual4-dflash.yml` — TP=4 + DFlash spec-decode. 64 narr / 104 code TPS, 2.27× concurrency at 262K, ~22 GB/card peak Both pass verify-full + verify-stress 7/7 (incl. Cliff 2 needle recall at 58K + 91K) on @Whamp's 4× RTX 3090 PCIe rig. Both pass v2 continuous soak (dual4: 20 sessions, 0 MiB growth, 90.8% TPS retention; dual4-dflash: 5 sessions, 0 MiB growth, 100% TPS retention) — first cross-rig confirmation that TP=4 escapes Cliff 2b. Closes #26 (4×3090 wishlist). Co-authored-by: Whamp
2026-05-03 17:26:31 -07:00
parent 85a6ea8c48
commit e3f8f02047
9 changed files with 426 additions and 65 deletions
--- a/docs/MULTI_CARD.md
+++ b/docs/MULTI_CARD.md
@@ -1,32 +1,33 @@
 # Multi-card (3+ GPUs) — derivation, constraints, scaling recipe

 You have **3 or more GPUs** and want to know if club-3090 applies. Short
-answer: yes, but we don't ship pre-baked configs because we can't
-hardware-test them. This page explains what scales (and what doesn't)
-when going beyond TP=2, the constraints to know, and the recipe for
-deriving your own compose from `dual.yml`.
+answer: yes. We ship one community-validated 4×3090 baseline and keep
+other 3+ GPU configs as derivation recipes until someone measures them.
+This page explains what scales (and what doesn't) when going beyond TP=2,
+the constraints to know, and how to derive your own compose when `dual4.yml`
+isn't your topology.

-> **Honest disclaimer:** the maintainer rig is **2× RTX 3090 PCIe**.
-> Everything below is derived from vLLM's documented tensor parallelism
-> behavior + our measured TP=2 baseline + Marlin pad math. None of it is
-> measured on 3+ card hardware locally. **If you have 4× / 8× hardware
-> and run any of these configs, please share results via the
-> [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml)
-> issue template** — `bash scripts/report.sh --bench > my-rig.md`
-> captures everything we'd want.
+> **Validation note:** the maintainer rig is **2× RTX 3090 PCIe**, but
+> Whamp's 4× RTX 3090 PCIe rig validated the TP=4 fp8/MTP baseline in
+> [discussion #26](https://github.com/noonghunna/club-3090/discussions/26)
+> on 2026-05-03. The TP=8+ sections remain derived expectations. If you
+> have 4× / 8× hardware and run additional configs, please share results via
+> the [Numbers from your rig](https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml)
+> issue template — `bash scripts/report.sh --bench > my-rig.md` captures
+> everything we'd want.

 ---

 ## TL;DR — what scales, what doesn't

-| Aspect | TP=1 | TP=2 (measured) | TP=4 (derived) | TP=8 (derived) |
+| Aspect | TP=1 | TP=2 (measured) | TP=4 (measured) | TP=8 (derived) |
 |---|---|---|---|---|
 | Per-card weight share | 100% (~14 GB) | 50% (~7 GB) | 25% (~3.5 GB) | 12.5% (~1.75 GB) |
 | KV pool capacity | smallest | 2× | ~4× | ~8× |
-| Per-card peak VRAM (262K target) | 23.5+ GB tight | 23.6 GB tight | ~16-18 GB | ~10-12 GB |
-| Cliff 2 single-prompt | fires at ~60K | doesn't fire (verified at 237K) | shouldn't fire | shouldn't fire |
-| Per-stream TPS (PCIe-only) | baseline | ~same as TP=1 | likely lower | lower still |
-| Concurrent throughput (multi-stream) | 1× | ~1.7-3.6× | derived ~2.5-7× | derived ~3-12× |
+| Per-card peak VRAM (262K target) | 23.5+ GB tight | 23.6 GB tight | **23.5 GB fp8 / 22.0 GB DFlash** | ~10-12 GB |
+| Cliff 2 single-prompt | fires at ~60K | doesn't fire (verified at 237K) | **passes 91K needle** | shouldn't fire |
+| Per-stream TPS (PCIe-only) | baseline | ~same as TP=1 | **63/76 fp8, 64/104 DFlash** | lower still |
+| Concurrent throughput (multi-stream) | 1× | ~1.7-3.6× | KV pre-check **6.77× fp8, 2.27× DFlash @ 262K** | derived ~3-12× |
 | Marlin pad-sub-tile-n patch | not needed | required | required | required |

 **Two key takeaways:**
@@ -70,9 +71,49 @@ the extras idle, or run separate stacks on different ports.

 ---

-## Recipe — derive your config from `dual.yml`
+## Shipped TP=4 baselines — `vllm/dual4` and `vllm/dual4-dflash`

-`dual.yml` is the tested baseline. To scale to TP=N, copy it and change
+For 4× RTX 3090 PCIe, start with the measured fp8/MTP compose:
+
+```bash
+bash scripts/switch.sh vllm/dual4
+```
+
+`docker-compose.dual4.yml` keeps the `dual.yml` fp8/MTP feature set and
+changes TP/streams from 2 → 4. Validation on Whamp's 4× 3090 PCIe rig:
+
+- boots at `max_model_len=262144`, `max_num_seqs=4`
+- vLLM reports GPU KV cache size **483,200 tokens** and **6.77×** maximum concurrency for 262K-token requests
+- `verify-full.sh` passes
+- `verify-stress.sh` passes 7/7; probe 7 recalls **58,569-token** and **91,070-token** needles
+- `bench.sh`: **63.01 narr / 76.25 code wall TPS**, peak **23,494 MiB/card**
+
+Use the DFlash variant when code throughput matters more than stream count
+and you can download the gated `z-lab/Qwen3.6-27B-DFlash` draft:
+
+```bash
+WITH_DFLASH_DRAFT=1 bash scripts/setup.sh qwen3.6-27b
+bash scripts/switch.sh vllm/dual4-dflash
+```
+
+`docker-compose.dual4-dflash.yml` keeps full 262K context but uses FP16 KV
+and admits two full-context streams:
+
+- boots at `max_model_len=262144`, `max_num_seqs=2`
+- vLLM reports GPU KV cache size **207,264 tokens** and **2.27×** maximum concurrency for 262K-token requests
+- `verify-full.sh` passes
+- `verify-stress.sh` passes 7/7; probe 7 recalls **58,570-token** and **91,070-token** needles
+- `bench.sh`: **64.00 narr / 104.40 code wall TPS**, peak **21,960 MiB/card**
+- DFlash AL during code bench: **4.43 / 4.37 / 4.35** last observed samples
+
+Single-stream TPS is lower than the 2-card DFlash variants on PCIe-only
+allreduce, so use TP=4 DFlash for full-262K code-heavy work and two admitted
+streams — not as a replacement for the fastest 2-card short-prompt DFlash path.
+
+## Recipe — derive your own config from `dual.yml`
+
+`dual.yml` is the tested 2-card baseline and `dual4.yml` is the measured
+4-card baseline. To scale to another TP=N, copy one of those and change
 **three lines**:

 ```diff
@@ -99,39 +140,42 @@ Everything else stays the same:
  needed, not less

 Container name + port: pick something distinct so it doesn't collide
-with your other variants:
+with your other variants. `dual4.yml` uses `vllm-qwen36-27b-dual4` and
+port `8015`; reserve a different name/port for further experiments:

 ```yaml
-container_name: vllm-qwen36-27b-quad     # or octa
+container_name: vllm-qwen36-27b-octa
 ports:
-  - "${PORT:-8014}:8000"                  # 8010-8013 are dual variants
+  - "${PORT:-8016}:8000"
 ```

 ---

-## What to expect on TP=4 (4× 3090 PCIe)
+## What we measured on TP=4 (4× 3090 PCIe)

-These are **derived expectations**, not measurements. Adjust to your
-rig's actual numbers when you bench:
+Measured 2026-05-03 on Whamp's 4× RTX 3090 PCIe rig:

- **Boot time:** longer than TP=2 (more NCCL handshakes, more weight
-  shards to load). 90s-3min cold boot typical for vLLM at this size.
- **Per-card VRAM:** roughly 16-18 GB at peak with `--max-model-len
-  262144 + max-num-seqs 4 + fp8 KV`. Significant headroom vs TP=2's
-  23.6 GB — you can push max-num-seqs higher or bump max-model-len if
-  your prompts need it.
- **Per-stream decode TPS:** likely lower than TP=2's ~70 narr / ~89
-  code (PCIe NCCL overhead). Could be ~50-65 narr / ~70-80 code as a
-  starting estimate — verify with `bench.sh`.
- **Aggregate concurrent throughput:** higher than TP=2 across multiple
-  streams. The KV pool 2× larger than TP=2 means more in-flight
-  requests fit, and each card's compute is freed by smaller weight
-  share.
- **Cliff 2:** doesn't apply. DeltaNet GDN forward state splits across
-  4 cards — single-prompt at 262K should work without the per-card 24
-  GB pressure that drives Cliff 2 on single-card.
- **NVLink bridges if available:** would substantially help per-stream
-  TPS. PCIe-only at TP=4 is functional but compute-bound on NCCL.
+- **fp8/MTP boot time:** 355s cold after model/image cache populated.
+- **fp8/MTP pre-check:** `max_model_len=262144`, `max_num_seqs=4`, GPU KV
+  cache size 483,200 tokens, max concurrency 6.77× at 262K.
+- **fp8/MTP VRAM:** 21,714 MiB idle after boot; 23,494 MiB/card peak
+  during canonical bench.
+- **fp8/MTP TPS:** 63.01 narrative / 76.25 code wall TPS.
+- **MTP AL:** last three code-bench metrics showed mean acceptance length
+  3.42 / 3.53 / 3.62.
+- **DFlash boot time:** 375s cold after model/image cache populated.
+- **DFlash pre-check:** `max_model_len=262144`, `max_num_seqs=2`, GPU KV
+  cache size 207,264 tokens, max concurrency 2.27× at 262K.
+- **DFlash VRAM:** 21,940 MiB idle after boot; 21,960 MiB/card peak during
+  canonical bench.
+- **DFlash TPS:** 64.00 narrative / 104.40 code wall TPS.
+- **DFlash AL:** last three code-bench metrics showed mean acceptance length
+  4.43 / 4.37 / 4.35.
+- **Cliff 2:** canonical `verify-stress.sh` probe 7 passes at both large
+  rungs on both TP=4 variants: ~58.6K tokens and 91K tokens recalled correctly.
+- **Trade-off:** PCIe allreduce makes single-stream decode slower than
+  TP=2, but TP=4 provides more full-context concurrency and the first
+  published 4×3090 Cliff 2 boundary data.

 ---

@@ -169,8 +213,8 @@ bash scripts/report.sh --bench > my-rig.md

 Specifically interested in:

- **TP=4 on 4× 3090 PCIe** — does Cliff 2 disappear entirely as
-  expected? What's per-stream TPS vs concurrent throughput?
+- **More TP=4 on 4× 3090 PCIe** — does your motherboard / power cap / PCIe
+  topology match or beat Whamp's 63 / 76 TPS baseline? What's concurrent throughput?
 - **TP=4 with NVLink topology** (e.g. NVLink across pairs) — how does
  per-stream TPS compare to PCIe-only TP=2?
 - **TP=8 on 8× A6000 / A100** — first server-class data point we'd
@@ -182,25 +226,27 @@ Specifically interested in:

 ---

-## Why we don't ship pre-baked configs
+## Why we ship only one pre-baked 4-card config

-Three reasons:
+We now ship `dual4.yml` because a community rig validated that exact
+4× RTX 3090 PCIe topology with `verify-full.sh`, `verify-stress.sh`, and
+`bench.sh`. We still avoid a broad matrix of untested 4+ GPU composes:

-1. **We can't hardware-test them.** Maintainer rig is 2× 3090. Pretending
-   we tested `quad.yml` would set false confidence.
-2. **Hardware combinations explode.** 4× 3090 vs 4× A5000 vs 4× A6000 vs
-   2×3090 + 2×4090 vs 4× modded 3080 — each has different VRAM, NVLink
-   topology, power profile, allreduce characteristics. A single
-   "quad.yml" can't be optimal for all.
+1. **Hardware combinations explode.** 4× 3090 vs 4× A5000 vs 4× A6000 vs
+   2×3090 + 2×4090 vs 4× modded 3080 — each has different VRAM, topology,
+   power profile, and allreduce characteristics.
+2. **Variant count needs discipline.** A single measured fp8/MTP TP=4
+   baseline is useful; a directory full of derived-but-unvalidated variants
+   would create false confidence.
 3. **Users at this scale are typically experienced.** If you have a
   workstation chassis or rack with 4-8 GPUs, you've already done the
   hardware homework. What you need from us is the methodology, the
-   constraints, and the dial — not a hand-held tested compose.
+   constraints, and one validated starting point.

-If a community member contributes a tested compose for their specific
-topology (with `verify-stress.sh` passing + `bench.sh` numbers), we'll
-ship it under `models/qwen3.6-27b/vllm/compose/` with credit and a
-header noting which rig validated it.
+If a community member contributes another tested compose for a specific
+topology or workload (with `verify-stress.sh` passing + `bench.sh`
+numbers), we'll ship it with credit and a header noting which rig
+validated it.

 ---

--- a/models/qwen3.6-27b/CHANGELOG.md
+++ b/models/qwen3.6-27b/CHANGELOG.md
@@ -2,6 +2,53 @@

 Dated history for Qwen3.6-27B configs in this repo. Combines the single-card and dual-card timelines (both were previously separate repos; consolidated here 2026-04-28).

+## 2026-05-03 late PM — `dual4-dflash.yml` TP=4 DFlash validated on 4× RTX 3090 PCIe ⭐
+
+Adds `docker-compose.dual4-dflash.yml`, a 4-card full-context DFlash variant validated on Whamp's 4× RTX 3090 PCIe rig for [club-3090 discussion #26](https://github.com/noonghunna/club-3090/discussions/26). This is a capacity / 262K-code variant, not a replacement for the faster 2-card DFlash short-prompt path.
+
+**Config accepted by vLLM pre-check:**
+- `tensor_parallel_size=4`
+- `max_model_len=262144`
+- `max_num_seqs=2`
+- `max_num_batched_tokens=8192`
+- `dtype=bfloat16`, FP16/default KV (required by DFlash on Ampere)
+- `speculative_config={"method":"dflash","num_speculative_tokens":5}`
+- reported GPU KV cache size: **207,264 tokens**
+- reported max concurrency at 262K/request: **2.27×**
+
+**Validation:**
+- Boot: clean, ready after 375s on a warm image/model cache.
+- `verify-full.sh`: PASS.
+- `verify-stress.sh`: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: **58,570 tokens** and **91,070 tokens**.
+- `bench.sh`: **64.00 narrative / 104.40 code wall TPS** (CV 2.8% / 3.0%), TTFT 143ms / 164ms.
+- DFlash AL during code bench: last three log samples **4.43 / 4.37 / 4.35**.
+- Peak VRAM during bench: **21,960 MiB/card**.
+
+**Interpretation:** TP=4 DFlash gives a useful code-speed uplift over `dual4.yml` (104 vs 76 code TPS) while retaining full 262K admission, but PCIe TP=4 allreduce keeps it below the 2-card DFlash variants' raw single-stream TPS. Use it for 4-card, full-context, code-heavy work with two admitted streams.
+
+## 2026-05-03 PM — `dual4.yml` TP=4 baseline validated on 4× RTX 3090 PCIe ⭐
+
+Adds `docker-compose.dual4.yml`, a measured 4-card fp8/MTP baseline derived from `dual.yml` by scaling tensor parallelism and streams from 2 → 4. Validation came from Whamp's 4× RTX 3090 PCIe rig in [club-3090 discussion #26](https://github.com/noonghunna/club-3090/discussions/26).
+
+**Config accepted by vLLM pre-check:**
+- `tensor_parallel_size=4`
+- `max_model_len=262144`
+- `max_num_seqs=4`
+- `max_num_batched_tokens=8192`
+- `kv_cache_dtype=fp8_e5m2`
+- reported GPU KV cache size: **483,200 tokens**
+- reported max concurrency at 262K/request: **6.77×**
+
+**Validation:**
+- Boot: clean, ready after 355s on a warm image/model cache.
+- `verify-full.sh`: PASS after warm retry (first Paris request hit a cold-path 30s script timeout; direct retry returned HTTP 200 in 0.2s and full rerun passed).
+- `verify-stress.sh`: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: **58,569 tokens** and **91,070 tokens**.
+- `bench.sh`: **63.01 narrative / 76.25 code wall TPS** (CV 2.1% / 4.0%), TTFT 111ms / 132ms.
+- MTP AL during code bench: last three log samples **3.42 / 3.53 / 3.62**.
+- Peak VRAM during bench: **23,494 MiB/card**.
+
+**Interpretation:** TP=4 gives the first published 4×3090 Cliff 2 boundary data and higher full-context concurrency headroom, but single-stream TPS is lower than TP=2 on PCIe-only allreduce (published TP=2 fp8/MTP baseline is ~69 / 89 TPS). Use `dual4.yml` for 4-card capacity / Cliff 2 margin, not for fastest single-user short-prompt decode.
+
 ## 2026-05-02 PM — Genesis v7.69 + vllm#35975 backport — Cliff 2 60K CLOSED ⭐⭐

 Genesis pin bump `fc89395` (v7.66) → `2db18df` (v7.69 dev tip). All three v7.66/v7.68 regressions we surfaced upstream landed in v7.69, plus a local backport of [vllm#35975](https://github.com/vllm-project/vllm/pull/35975) brings the Cliff 2 single-prompt envelope to **60K cleanly on TQ3 + MTP K=3 at 24 GB**. Two shippable single-card recipes ship at this pin.
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml
@@ -10,9 +10,10 @@
 #
 # Measured: 77 narr / 124 code TPS, 200K ctx, 1 stream, code AL ~4.27 warm.
 #
-# Same DFlash caveats as dual-dflash.yml apply: single-stream only, FP16 KV
-# (no fp8/turbo on DFlash + Ampere), `--dtype bfloat16` for vllm#40334
-# workaround.
+# Same DFlash caveats as dual-dflash.yml apply: single-stream only on the
+# 2-card variants, FP16 KV (no fp8/turbo on DFlash + Ampere), `--dtype
+# bfloat16` for vllm#40334 workaround. If you have 4× 3090 and need full
+# 262K context + 2 streams, use docker-compose.dual4-dflash.yml.
 #
 # ─── Prerequisite: download the DFlash draft model ──────────────────────
 # Same as dual-dflash.yml — needs `z-lab/Qwen3.6-27B-DFlash` at
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
@@ -47,6 +47,11 @@
 #
 # To run:
 #   cd <repo>/models/qwen3.6-27b/vllm/compose
+# Sibling DFlash variants:
+#   - docker-compose.dual-dflash.yml:      2× PCIe, 185K, 1 stream, 82/125 TPS
+#   - docker-compose.dual-dflash-noviz.yml: 2× PCIe, 200K, 1 stream, 78/127 TPS
+#   - docker-compose.dual4-dflash.yml:     4× PCIe, 262K, 2 streams, 64/104 TPS
+#
 #   docker compose -f docker-compose.dual-dflash.yml up -d
 # ===========================================================================
 services:
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink.yml
@@ -33,6 +33,8 @@
 #
 #   File                                Ctx    Streams   Narr/Code TPS   KV       Vision  NVLink
 #   docker-compose.dual.yml (DEFAULT)   262K   2         69 / 89          fp8      ✅      not used
+#   docker-compose.dual4.yml            262K   4         63 / 76          fp8      ✅      not used (4× PCIe)
+#   docker-compose.dual4-dflash.yml     262K   2         64 / 104         FP16     ✅      not used (4× PCIe)
 #   docker-compose.dual-nvlink.yml      262K   2         (community)      fp8      ✅      required
 #   docker-compose.dual-turbo.yml       262K   4         54 / 73          TQ3      ✅      not used
 #   docker-compose.dual-dflash.yml      185K   1         82 / 125         FP16     ✅      not used
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
@@ -22,11 +22,13 @@
 #
 # All dual-card variants in this dir:
 #
-#   File                                Ctx    Streams   Narr/Code TPS   KV       Vision
-#   docker-compose.dual.yml (this)      262K   2         69 / 89          fp8      ✅
-#   docker-compose.dual-turbo.yml       262K   4         54 / 73          TQ3      ✅
+#   File                                Ctx    Streams   Narr/Code TPS   KV       Vision  Topology
+#   docker-compose.dual.yml (this)      262K   2         69 / 89          fp8      ✅      2× PCIe
+#   docker-compose.dual4.yml            262K   4         63 / 76          fp8      ✅      4× PCIe
+#   docker-compose.dual4-dflash.yml     262K   2         64 / 104         FP16     ✅      4× PCIe
+#   docker-compose.dual-turbo.yml       262K   4         54 / 73          TQ3      ✅      2× PCIe
 #   docker-compose.dual-dflash.yml      185K   1         82 / 125         FP16     ✅
-#   docker-compose.dual-dflash-noviz... 200K   1         78 / 127         FP16     ❌
+#   docker-compose.dual-dflash-noviz... 200K   1         78 / 127         FP16     ❌      2× PCIe
 #
 # Run:
 #   cd <repo>/models/qwen3.6-27b/vllm/compose
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual4-dflash.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual4-dflash.yml
@@ -0,0 +1,125 @@
+# ===========================================================================
+# Quad RTX 3090 DFlash — TP=4 + DFlash N=5 + 262K ctx + vision.
+#
+# 4-card full-context DFlash variant validated on Whamp's 4× RTX 3090 PCIe
+# rig for discussion #26 / PR 2. This is not the fastest single-stream
+# DFlash path; 2-card dual-dflash still wins raw TPS. Use this when you want
+# DFlash's code-speed uplift while preserving full 262K admission and two
+# full-context streams on a 4×3090 PCIe box.
+#
+# What this gives you:
+#   - TP=4 across four 3090s
+#   - DFlash N=5 draft model (z-lab/Qwen3.6-27B-DFlash)
+#   - max_model_len=262144 with max_num_seqs=2
+#   - vision + tools + reasoning enabled, matching dual-dflash.yml
+#   - FP16 KV (default) because DFlash non-causal attention on Ampere cannot
+#     use fp8/turbo KV backends
+#
+# Measured on 4× RTX 3090 PCIe, vLLM nightly 0.20.1rc1.dev16+g7a1eb8ac2:
+#   - boot accepted max_model_len=262144 + max_num_seqs=2
+#   - GPU KV cache size: 207,264 tokens
+#   - Maximum concurrency for 262,144-token requests: 2.27×
+#   - verify-full.sh: PASS
+#   - verify-stress.sh: PASS 7/7, including 60K + 90K Cliff 2 needles
+#   - bench.sh: 64.00 narr (CV 2.8%) / 104.40 code (CV 3.0%) wall TPS
+#   - DFlash AL during code bench: last observed 4.43 / 4.37 / 4.35
+#   - Peak VRAM during bench: 21,960 MiB/card
+#
+# Trade-offs:
+#   - vs docker-compose.dual4.yml: similar narrative TPS but much faster code
+#     (104 vs 76 TPS) at lower stream count (2 vs 4) and FP16 KV.
+#   - vs docker-compose.dual-dflash.yml: slower raw single-stream TPS on
+#     PCIe TP=4 allreduce, but full 262K context and two admitted streams
+#     instead of 185K / one stream.
+#
+# ─── Prerequisite: download the DFlash draft model ──────────────────────
+# This compose loads `z-lab/Qwen3.6-27B-DFlash` from
+# `<MODEL_DIR>/qwen3.6-27b-dflash/` as the spec-decode draft. The default
+# setup does NOT fetch this model; rerun setup with:
+#
+#   WITH_DFLASH_DRAFT=1 bash scripts/setup.sh qwen3.6-27b
+#
+# If the draft is missing, vLLM may fall back or fail and TPS will not match
+# the numbers above.
+#
+# Required workaround: vllm#40334 dtype mismatch fix is not yet upstream;
+# compose sets `--dtype bfloat16` to match the draft's training dtype.
+#
+# Run:
+#   cd <repo>/models/qwen3.6-27b/vllm/compose
+#   docker compose -f docker-compose.dual4-dflash.yml up -d
+# ===========================================================================
+services:
+  vllm-qwen36-27b-dual4-dflash:
+    image: vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
+    container_name: vllm-qwen36-27b-dual4-dflash
+    restart: "no"
+    ports:
+      - "${PORT:-8016}:8000"
+    volumes:
+      - ${MODEL_DIR:-../../../../models-cache}:/root/.cache/huggingface
+      # torch.compile + Triton kernel caches — first boot warms (~60-90 sec);
+      # subsequent boots reuse cached graphs. Pattern from Sander's PROD launch.
+      # Closes club-3090 #22.
+      - ../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
+      - ../cache/triton:/root/.triton/cache
+      # Marlin pad-sub-tile-n (vLLM PR #40361) — vendored in this repo at
+      # ../patches/vllm-marlin-pad/. Drops out when vllm#40361 lands upstream.
+      - ../patches/vllm-marlin-pad/marlin.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/marlin.py:ro
+      - ../patches/vllm-marlin-pad/MPLinearKernel.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/MPLinearKernel.py:ro
+    environment:
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_WORKER_MULTIPROC_METHOD=spawn
+      - NCCL_CUMEM_ENABLE=0
+      - NCCL_P2P_DISABLE=1
+      - VLLM_NO_USAGE_STATS=1
+      - VLLM_USE_FLASHINFER_SAMPLER=1
+      - OMP_NUM_THREADS=1
+      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512
+    shm_size: "16gb"
+    ipc: host
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    command:
+      - --model
+      - /root/.cache/huggingface/qwen3.6-27b-autoround-int4
+      - --served-model-name
+      - qwen3.6-27b-autoround
+      - --quantization
+      - auto_round
+      - --dtype
+      - bfloat16
+      - --tensor-parallel-size
+      - "4"
+      - --disable-custom-all-reduce
+      - --max-model-len
+      - "262144"
+      - --gpu-memory-utilization
+      - "0.95"
+      - --max-num-seqs
+      - "2"
+      - --max-num-batched-tokens
+      - "8192"
+      # No --kv-cache-dtype: DFlash needs head_size=256 + non-causal attention,
+      # and no Ampere backend supports that triple with fp8/turbo KV. FP16 default
+      # is the only working choice (matches Qwen3.5-27B + DFlash row 4 = 89.7 TPS).
+      # --language-model-only removed to enable MoonViT vision tower (2026-04-25 test).
+      - --trust-remote-code
+      - --reasoning-parser
+      - qwen3
+      - --enable-auto-tool-choice
+      - --tool-call-parser
+      - qwen3_coder
+      - --enable-prefix-caching
+      - --enable-chunked-prefill
+      - --speculative-config
+      - '{"method":"dflash","model":"/root/.cache/huggingface/qwen3.6-27b-dflash","num_speculative_tokens":5}'
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual4.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual4.yml
@@ -0,0 +1,127 @@
+# ===========================================================================
+# Quad RTX 3090 — TP=4 baseline. Qwen3.6-27B + MTP n=3 + fp8 KV + vision.
+#
+# Community-validated on Whamp's 4× RTX 3090 PCIe rig (discussion #26),
+# 2026-05-03. Derived from docker-compose.dual.yml by scaling tensor
+# parallelism and concurrent streams from 2 → 4.
+#
+# What this gives you (vs the 2-card default):
+#   - TP=4 across four 3090s
+#   - max_num_seqs=4 with full 262K context admission
+#   - vLLM pre-check accepts max_model_len=262144 + max_num_seqs=4
+#   - KV pool reports 483,200 tokens and 6.77× max concurrency at 262K
+#   - Cliff 2 probe 7 passes: 58,569-token and 91,070-token needles recalled
+#   - vision + tools + MTP n=3 + recall — same feature set as dual.yml
+#
+# Measured on 4× RTX 3090 PCIe, vLLM nightly 0.20.1rc1.dev16+g7a1eb8ac2:
+#   - verify-full.sh: PASS
+#   - verify-stress.sh: PASS 7/7, including 60K + 90K Cliff 2 needles
+#   - bench.sh: 63.01 narr (CV 2.1%) / 76.25 code (CV 4.0%) wall TPS
+#   - MTP AL during code bench: last observed 3.42 / 3.53 / 3.62
+#   - Peak VRAM during bench: 23,494 MiB/card
+#
+# Trade-off vs TP=2 dual.yml:
+#   - More long-context/concurrency headroom and Cliff 2 boundary data.
+#   - Pick this over dual.yml when you need >2 concurrent streams or
+#     single-prompt context beyond the 60K Cliff 2 boundary.
+#   - Lower single-stream TPS on PCIe-only allreduce (63/76 vs ~69/89 TP=2).
+#     Use this when you need 4-card capacity or Cliff 2 margin, not when you
+#     only want fastest single-stream decode.
+#
+# Dependencies:
+#   - 4× RTX 3090 (Ampere SM 8.6), PCIe-only validated (no NVLink required)
+#   - vLLM PR #40361 (Marlin pad-sub-tile-n) — patched files volume-mounted
+#     from ../patches/vllm-marlin-pad/. At TP=4 the pad fork is still required
+#     because more output-dim shards fall below Marlin's sub-tile threshold.
+#
+# All fp8/MTP long-context variants in this dir:
+#
+#   File                                Ctx    Streams   Narr/Code TPS   KV       Vision  Topology
+#   docker-compose.dual.yml             262K   2         69 / 89          fp8      ✅      2× PCIe
+#   docker-compose.dual4.yml (this)     262K   4         63 / 76          fp8      ✅      4× PCIe
+#   docker-compose.dual4-dflash.yml     262K   2         64 / 104         FP16     ✅      4× PCIe
+#   docker-compose.dual-nvlink.yml      262K   2         (community)      fp8      ✅      2× NVLink
+#
+# Run:
+#   cd <repo>/models/qwen3.6-27b/vllm/compose
+#   docker compose -f docker-compose.dual4.yml up -d
+# ===========================================================================
+services:
+  vllm-qwen36-27b-dual4:
+    # Tracking latest nightly intentionally — this stack uses fp8 KV (not
+    # TurboQuant), so it doesn't accumulate the same anchor-drift exposure
+    # the single-card project does. Pin if you hit a regression; otherwise
+    # ride the wave.
+    image: vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
+    container_name: vllm-qwen36-27b-dual4
+    restart: "no"
+    ports:
+      - "${PORT:-8015}:8000"
+    volumes:
+      - ${MODEL_DIR:-../../../../models-cache}:/root/.cache/huggingface
+      # torch.compile + Triton kernel caches — first boot warms (~60-90 sec);
+      # subsequent boots reuse cached graphs. Pattern from Sander's PROD launch.
+      # Closes club-3090 #22.
+      - ../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
+      - ../cache/triton:/root/.triton/cache
+      # Marlin pad-sub-tile-n patch (vLLM PR #40361) — required for TP=4
+      # on AutoRound W4A16 models where out-dim shards fall below 64.
+      # Vendored in this repo at ../patches/vllm-marlin-pad/ (no host
+      # filesystem dependency). Drops out when vllm#40361 lands upstream.
+      - ../patches/vllm-marlin-pad/marlin.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/marlin.py:ro
+      - ../patches/vllm-marlin-pad/MPLinearKernel.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/MPLinearKernel.py:ro
+    environment:
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_WORKER_MULTIPROC_METHOD=spawn
+      # PCIe-only stack — disable NCCL features that assume NVLink.
+      - NCCL_CUMEM_ENABLE=0
+      - NCCL_P2P_DISABLE=1
+      - VLLM_NO_USAGE_STATS=1
+      - VLLM_USE_FLASHINFER_SAMPLER=1
+      - OMP_NUM_THREADS=1
+      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512
+    shm_size: "16gb"
+    ipc: host
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    command:
+      - --model
+      - /root/.cache/huggingface/qwen3.6-27b-autoround-int4
+      - --served-model-name
+      - qwen3.6-27b-autoround
+      - --quantization
+      - auto_round
+      - --dtype
+      - float16
+      - --tensor-parallel-size
+      - "4"
+      - --disable-custom-all-reduce
+      - --max-model-len
+      - "262144"
+      - --gpu-memory-utilization
+      - "0.92"
+      - --max-num-seqs
+      - "4"
+      - --max-num-batched-tokens
+      - "8192"
+      - --kv-cache-dtype
+      - fp8_e5m2
+      - --trust-remote-code
+      - --reasoning-parser
+      - qwen3
+      - --enable-auto-tool-choice
+      - --tool-call-parser
+      - qwen3_coder
+      - --enable-prefix-caching
+      - --enable-chunked-prefill
+      - --speculative-config
+      - '{"method":"mtp","num_speculative_tokens":3}'
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
--- a/scripts/switch.sh
+++ b/scripts/switch.sh
@@ -25,6 +25,8 @@
 #
 #   Dual-card vLLM (TP=2):
 #     vllm/dual             262K + fp8 + 2 streams + vision (recommended dual)
+#     vllm/dual4            262K + fp8 + 4 streams + vision (4× 3090 PCIe baseline)
+#     vllm/dual4-dflash     262K + FP16 + DFlash N=5 + 2 streams + vision (4× 3090 code)
 #     vllm/dual-turbo       262K + TQ3 + 4 streams + vision (multi-tenant)
 #     vllm/dual-dflash      185K + FP16 + DFlash N=5 + vision (peak code TPS)
 #     vllm/dual-dflash-noviz 200K + FP16 + DFlash N=5 + no vision (peak code, max ctx)
@@ -65,6 +67,8 @@ declare -A VARIANT_DEFAULT_PORT=(
  [vllm/tools-text]=8020
  [vllm/minimal]=8020
  [vllm/dual]=8010
+  [vllm/dual4]=8015
+  [vllm/dual4-dflash]=8016
  [vllm/dual-turbo]=8011
  [vllm/dual-dflash]=8012
  [vllm/dual-dflash-noviz]=8013
@@ -83,6 +87,8 @@ declare -A VARIANTS=(
  [vllm/tools-text]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.tools-text.yml"
  [vllm/minimal]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.minimal.yml"
  [vllm/dual]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual.yml"
+  [vllm/dual4]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual4.yml"
+  [vllm/dual4-dflash]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual4-dflash.yml"
  [vllm/dual-turbo]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual-turbo.yml"
  [vllm/dual-dflash]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual-dflash.yml"
  [vllm/dual-dflash-noviz]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual-dflash-noviz.yml"