Add docker-compose.dual-nvlink-dflash.yml (#92)

Adds NVLink-enabled DFlash compose for 2× 3090 with NVLink bridge. Mirrors `docker-compose.dual-dflash.yml` but enables NCCL P2P over NVLink (`NCCL_P2P_LEVEL=NVL`) and re-enables vLLM's custom all-reduce kernel. Drops `expandable_segments=True` per JusefPol's NVLink startup-crash report (#31). Validation (rig: 2× 3090 + NVLink, 230W cap, EDT — see PR body): - verify-full: 8/8 PASS - verify-stress: 7/7 PASS (incl. 91K Cliff 2 needle) - soak-continuous: PASS (0 errors, 0 silent-empty, 0 MiB growth, 100% TPS retention) - Canonical bench: narr 101.55 / code 163.33 wall TPS (CV 1.8% / 1.9%) NVLink lift vs his own PCIe baseline (`dual-dflash.yml` 86.62 / 141.02): +17% narr / +16% code — matches the +15-19% NVLink lift the controlled A/B in BENCHMARKS shows on DFlash paths (K+1 verify is heavily cross-card matmul). Port: 8018. Status: community-contributed, experimental.
2026-05-07 07:36:57 -04:00
parent 29e7de5a90
commit 0d199a1f50
6 changed files with 156 additions and 5 deletions
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
@@ -48,9 +48,10 @@
 # To run:
 #   cd <repo>/models/qwen3.6-27b/vllm/compose
 # Sibling DFlash variants:
-#   - docker-compose.dual-dflash.yml:      2× PCIe, 185K, 1 stream, 82/125 TPS
-#   - docker-compose.dual-dflash-noviz.yml: 2× PCIe, 200K, 1 stream, 78/127 TPS
-#   - docker-compose.dual4-dflash.yml:     4× PCIe, 262K, 2 streams, 64/104 TPS
+#   - docker-compose.dual-dflash.yml:         2× PCIe, 185K, 1 stream, 82/125 TPS
+#   - docker-compose.dual-dflash-noviz.yml:   2× PCIe, 200K, 1 stream, 78/127 TPS
+#   - docker-compose.dual-nvlink-dflash.yml:  2× NVLink, 185K, 1 stream, (community) TPS
+#   - docker-compose.dual4-dflash.yml:        4× PCIe, 262K, 2 streams, 64/104 TPS
 #
 #   docker compose -f docker-compose.dual-dflash.yml up -d
 # ===========================================================================
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink-dflash.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink-dflash.yml
@@ -0,0 +1,143 @@
+# ===========================================================================
+# Dual RTX 3090 with NVLink + DFlash — TP=2 + DFlash N=5 spec-decode + 185K ctx
+# + vision + NVLink P2P for faster allreduce.
+#
+# Mirrors docker-compose.dual-dflash.yml but enables NCCL P2P over NVLink and
+# re-enables vLLM's custom all-reduce kernel (which dual-dflash.yml disables
+# for PCIe-only stacks). Combines DFlash N=5 draft model with NVLink bridge
+# for maximum single-stream code throughput on 2x 3090.
+#
+# Status: COMMUNITY-CONTRIBUTED, EXPERIMENTAL.
+#   If you run this, please drop numbers in discussion #19 — paste-ready report
+#   via `bash scripts/report.sh`.
+#
+# What this gives you (vs the PCIe-only `dual-dflash.yml`):
+#   - NCCL P2P over NVLink (NCCL_P2P_LEVEL=NVL) — much faster allreduce on TP=2
+#   - Custom all-reduce ENABLED (--disable-custom-all-reduce removed) — NVLink
+#     makes vLLM's custom kernel a win where PCIe makes it a loss
+#   - PYTORCH_CUDA_ALLOC_CONF without `expandable_segments:True` — JusefPol
+#     reports it crashes on startup with NVLink wired up
+#
+# What's intentionally NOT enabled:
+#   - TurboQuant KV — DFlash needs head_size=256 + non-causal attention,
+#     and no Ampere backend supports that triple with fp8/turbo KV.
+#
+# KV cache: FP16 (default — DFlash needs head_size=256 + non-causal attention,
+# and no Ampere backend supports that triple with fp8/turbo KV).
+#
+# ─── Prerequisite: download the DFlash draft model ──────────────────────
+# Same as dual-dflash.yml — needs `z-lab/Qwen3.6-27B-DFlash` at
+# `<MODEL_DIR>/qwen3.6-27b-dflash/`. Get it via:
+#
+#   WITH_DFLASH_DRAFT=1 bash scripts/setup.sh qwen3.6-27b
+#
+# OR manually `hf download z-lab/Qwen3.6-27B-DFlash --local-dir <MODEL_DIR>/qwen3.6-27b-dflash`.
+# If missing, vLLM falls back silently to baseline bf16 decode (~25 TPS
+# instead of 125 TPS — reported by @lolren in club-3090#18). See
+# `dual-dflash.yml` header for the under-training caveat.
+#
+# Dependencies:
+#   - 2x RTX 3090 (Ampere SM 8.6) WITH NVLink bridge installed and `nvidia-smi
+#     topo -m` showing `NV*` between GPU0 and GPU1
+#   - vLLM PR #40361 (Marlin pad-sub-tile-n) — patched files vendored in-repo
+#     at ../patches/vllm-marlin-pad/. PR is open upstream; drop the mount when
+#     it lands. See ../patches/vllm-marlin-pad/README.md.
+#
+# All dual-card variants in this dir:
+#
+#   File                                    Ctx    Streams   Narr/Code TPS   KV       Vision  NVLink
+#   docker-compose.dual.yml (DEFAULT)       262K   2         69 / 89          fp8      ✓      not used
+#   docker-compose.dual4.yml                262K   4         63 / 76          fp8      ✓      not used (4x PCIe)
+#   docker-compose.dual4-dflash.yml         262K   2         64 / 104         FP16     ✓      not used (4x PCIe)
+#   docker-compose.dual-nvlink.yml          262K   2         (community)      fp8      ✓      required
+#   docker-compose.dual-nvlink-turbo.yml    262K   4         101 / 133        TQ3      ✓      required
+#   docker-compose.dual-turbo.yml           262K   4         54 / 73          TQ3      ✓      not used
+#   docker-compose.dual-dflash.yml          185K   1         82 / 125         FP16     ✓      not used
+#   docker-compose.dual-dflash-noviz.yml    200K   1         78 / 127         FP16     ✗      not used
+#   docker-compose.dual-nvlink-dflash.yml   185K   1         (community)      FP16     ✓      required
+#
+# To run:
+#   cd <repo>/models/qwen3.6-27b/vllm/compose
+#   docker compose -f docker-compose.dual-nvlink-dflash.yml up -d
+# ===========================================================================
+services:
+  vllm-qwen36-27b-dual-nvlink-dflash:
+    image: vllm/vllm-openai:nightly-01d4d1ad375dc5854779c593eee093bcebb0cada
+    container_name: vllm-qwen36-27b-dual-nvlink-dflash
+    restart: "no"
+    ports:
+      - "${PORT:-8018}:8000"
+    volumes:
+      - ${MODEL_DIR:-../../../../models-cache}:/root/.cache/huggingface
+      # torch.compile + Triton kernel caches — first boot warms (~60-90 sec);
+      # subsequent boots reuse cached graphs. Pattern from Sander's PROD launch.
+      # Closes club-3090 #22.
+      - ../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
+      - ../cache/triton:/root/.triton/cache
+      # Marlin pad-sub-tile-n (vLLM PR #40361) — vendored in this repo at
+      # ../patches/vllm-marlin-pad/. Drops out when vllm#40361 lands upstream.
+      - ../patches/vllm-marlin-pad/marlin.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/marlin.py:ro
+      - ../patches/vllm-marlin-pad/MPLinearKernel.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/kernels/linear/mixed_precision/MPLinearKernel.py:ro
+    environment:
+      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
+      - VLLM_WORKER_MULTIPROC_METHOD=spawn
+      # NVLink bridge present — let NCCL use P2P (don't disable it the way
+      # dual-dflash.yml does for PCIe-only stacks) and pin the P2P level to
+      # NVLink so NCCL doesn't fall back to PCIe paths if topology query is fuzzy.
+      - NCCL_CUMEM_ENABLE=0
+      - NCCL_P2P_LEVEL=NVL
+      - VLLM_NO_USAGE_STATS=1
+      - VLLM_USE_FLASHINFER_SAMPLER=1
+      - OMP_NUM_THREADS=1
+      # JusefPol report (PR #31): expandable_segments=True crashes on startup
+      # with NVLink wired in. Keep max_split_size_mb cap, drop the rest.
+      - PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512
+    shm_size: "16gb"
+    ipc: host
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    command:
+      - --model
+      - /root/.cache/huggingface/qwen3.6-27b-autoround-int4
+      - --served-model-name
+      - qwen3.6-27b-autoround
+      - --quantization
+      - auto_round
+      - --dtype
+      - bfloat16
+      - --tensor-parallel-size
+      - "2"
+      # Custom all-reduce ENABLED (no --disable-custom-all-reduce) — NVLink
+      # makes vLLM's custom kernel a win. dual-dflash.yml disables it because
+      # PCIe P2P bandwidth makes the NCCL fallback faster there.
+      - --max-model-len
+      - "${MAX_MODEL_LEN:-185000}"
+      - --gpu-memory-utilization
+      - "${GPU_MEMORY_UTILIZATION:-0.95}"
+      - --max-num-seqs
+      - "1"
+      - --max-num-batched-tokens
+      - "8192"
+      # No --kv-cache-dtype: DFlash needs head_size=256 + non-causal attention,
+      # and no Ampere backend supports that triple with fp8/turbo KV. FP16 default
+      # is the only working choice (matches Qwen3.5-27B + DFlash row 4 = 89.7 TPS).
+      # --language-model-only removed to enable MoonViT vision tower (2026-04-25 test).
+      - --trust-remote-code
+      - --reasoning-parser
+      - qwen3
+      - --enable-auto-tool-choice
+      - --tool-call-parser
+      - qwen3_coder
+      - --enable-prefix-caching
+      - --enable-chunked-prefill
+      - --speculative-config
+      - '{"method":"dflash","model":"/root/.cache/huggingface/qwen3.6-27b-dflash","num_speculative_tokens":5}'
+      - --host
+      - 0.0.0.0
+      - --port
+      - "8000"
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual-nvlink.yml
@@ -39,6 +39,7 @@
 #   docker-compose.dual-turbo.yml       262K   4         54 / 73          TQ3      ✅      not used
 #   docker-compose.dual-dflash.yml      185K   1         82 / 125         FP16     ✅      not used
 #   docker-compose.dual-dflash-noviz... 200K   1         78 / 127         FP16     ❌      not used
+#   docker-compose.dual-nvlink-dflash.. 185K   1         (community)      FP16     ✅      required
 #
 # Run:
 #   cd <repo>/models/qwen3.6-27b/vllm/compose
--- a/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
+++ b/models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
@@ -29,6 +29,7 @@
 #   docker-compose.dual-turbo.yml       262K   4         54 / 73          TQ3      ✅      2× PCIe
 #   docker-compose.dual-dflash.yml      185K   1         82 / 125         FP16     ✅
 #   docker-compose.dual-dflash-noviz... 200K   1         78 / 127         FP16     ❌      2× PCIe
+#   docker-compose.dual-nvlink-dflash.yml 185K   1         (community)      FP16     ✅      2× NVLink
 #
 # Run:
 #   cd <repo>/models/qwen3.6-27b/vllm/compose
--- a/scripts/launch.sh
+++ b/scripts/launch.sh
@@ -218,6 +218,7 @@ declare -A LAUNCH_DEFAULT_PORT=(
  [vllm/dual-dflash-noviz]=8013
  [vllm/dual-nvlink]=8014
  [vllm/dual-nvlink-turbo]=8017
+  [vllm/dual-nvlink-dflash]=8018
  [vllm/gemma-mtp]=8030
  [vllm/gemma-mtp-tp1]=8031
  [llamacpp/default]=8020
@@ -237,6 +238,7 @@ declare -A LAUNCH_DEFAULT_CONTAINER=(
  [vllm/dual-dflash-noviz]=vllm-qwen36-27b-dual-dflash-noviz
  [vllm/dual-nvlink]=vllm-qwen36-27b-dual-nvlink
  [vllm/dual-nvlink-turbo]=vllm-qwen36-27b-dual-nvlink-turbo
+  [vllm/dual-nvlink-dflash]=vllm-qwen36-27b-dual-nvlink-dflash
  [vllm/gemma-mtp]=vllm-gemma-4-31b-mtp
  [vllm/gemma-mtp-tp1]=vllm-gemma-4-31b-mtp-tp1
  [llamacpp/default]=llama-cpp-qwen36-27b
--- a/scripts/switch.sh
+++ b/scripts/switch.sh
@@ -30,8 +30,9 @@
 #     vllm/dual-turbo       262K + TQ3 + 4 streams + vision (multi-tenant)
 #     vllm/dual-dflash      185K + FP16 + DFlash N=5 + vision (peak code TPS)
 #     vllm/dual-dflash-noviz 200K + FP16 + DFlash N=5 + no vision (peak code, max ctx)
-#     vllm/dual-nvlink      262K + fp8 + 2 streams + vision (REQUIRES NVLink bridge — community/experimental)
-#     vllm/dual-nvlink-turbo 262K + TQ3 + 4 streams + vision (REQUIRES NVLink bridge — community/experimental)
+#     vllm/dual-nvlink          262K + fp8 + 2 streams + vision (REQUIRES NVLink bridge — community/experimental)
+#     vllm/dual-nvlink-turbo    262K + TQ3 + 4 streams + vision (REQUIRES NVLink bridge — community/experimental)
+#     vllm/dual-nvlink-dflash   185K + FP16 + DFlash N=5 + vision (REQUIRES NVLink bridge — community/experimental)
 #     vllm/gemma-mtp        Gemma-4-31B + Google MTP drafter (32K, bf16 KV, vision — community/experimental, pre-merge)
 #
 #   Single-card llama.cpp:
@@ -76,6 +77,7 @@ declare -A VARIANT_DEFAULT_PORT=(
  [vllm/dual-dflash-noviz]=8013
  [vllm/dual-nvlink]=8014
  [vllm/dual-nvlink-turbo]=8017
+  [vllm/dual-nvlink-dflash]=8018
  [vllm/gemma-mtp]=8030
  [vllm/gemma-mtp-tp1]=8031
  [vllm/gemma-dflash]=8032
@@ -100,6 +102,7 @@ declare -A VARIANTS=(
  [vllm/dual-dflash-noviz]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual-dflash-noviz.yml"
  [vllm/dual-nvlink]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual-nvlink.yml"
  [vllm/dual-nvlink-turbo]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual-nvlink-turbo.yml"
+  [vllm/dual-nvlink-dflash]="vllm|models/qwen3.6-27b/vllm/compose|docker-compose.dual-nvlink-dflash.yml"
  [vllm/gemma-mtp]="vllm|models/gemma-4-31b/vllm/compose|docker-compose.gemma-mtp.yml"
  [vllm/gemma-mtp-tp1]="vllm|models/gemma-4-31b/vllm/compose|docker-compose.gemma-mtp-tp1.yml"
  [vllm/gemma-dflash]="vllm|models/gemma-4-31b/vllm/compose|docker-compose.gemma-dflash.yml"