16 KiB
Dual 3090 — what changes when you add the second card
You have 2× RTX 3090s, PCIe-only (no NVLink). This page is the front door for picking a config and knowing what dual-card unlocks vs single. Model-specific deep dives (quants, Genesis, engine internals) live in the model directory — links at the bottom.
Have 3+ GPUs? See
MULTI_CARD.md— derivation of TP=4 / TP=8 configs fromdual.yml, valid TP values for Qwen3.6-27B (1, 2, 4, 5, 8, 10), and what scales vs what doesn't.
TL;DR — pick by workload
| What you're doing | Compose | Max ctx | Narr / Code TPS | VRAM per card | Why |
|---|---|---|---|---|---|
| Hermes agentic fine-tune (Carnice tool specialization) | carnice-bf16mtp.yml |
262K | 72 / 80 | ~22.3 / 24 GB | BF16 MTP overlay. Hermes-style assistant. Available on HF: wasifb/Carnice_V2_27B_INT4_BF16MTP |
| General-purpose default (vision + tools + long ctx) | dual.yml ⭐ |
262K (237K single-prompt verified) | 69 / 89 | ~23.6 / 24 GB | fp8 KV, 2 streams, full feature set |
| Multi-tenant (4 concurrent agents at full ctx) | dual-turbo.yml |
262K | 58 / 76 per-stream (269 TPS aggregate at 4 streams) | ~19.8 / 24 GB | TQ3 KV (3 bits/token) + full v7.69 PROD env-var stack — 4.67× concurrency. 20 GB Ampere users: override --kv-cache-dtype turboquant_3bit_nc → fp8_e5m2; see HARDWARE.md + #47. |
| Peak code TPS with vision | dual-dflash.yml |
185K | 82 / 125 | ~23.6 / 24 GB | DFlash N=5 + 1.75 GB draft per card, AL ~4.4 (vs MTP's 3.4) |
| Peak code TPS, no vision | dual-dflash-noviz.yml |
200K | 78 / 127 | ~23.8 / 24 GB | DFlash + no vision, +15K ctx vs dual-dflash |
VRAM column is per-card under TP=2 (each card holds half the weights + half the KV; both cards' totals are nearly identical). For a 2× 20 GB rig (e.g. 2× 3080-20GB / 40 GB combined),
dual.ymlanddual-turboshould fit;dual-dflash*won't (FP16 KV + DFlash draft pushes per-card past 20 GB). Component breakdown intools/charts/gen-vram.py.
Run any of these via bash scripts/launch.sh (interactive) or bash scripts/switch.sh <variant>.
Measured TPS on 2× 3090
Bench protocol: 3 warm + 5 measured runs of the canonical narrative + code prompts on each config. Substrate: vLLM nightly 0.20.1rc1.dev16+g7a1eb8ac2 + Genesis v7.69 dev tip (commit 2db18df), RTX 3090 sm_86 PCIe-only at 230 W. Cliff 2 doesn't apply on TP=2 (DeltaNet GDN forward state splits across cards — 237K single-prompt verified on dual.yml); the v7.69 cutover is mostly a hygiene bump for dual-turbo.yml (its old workspace_lock sidecar is now covered by Genesis PN34 env-gate). Per-config run-by-run + VRAM peaks: models/qwen3.6-27b/CHANGELOG.md.
VRAM budget on 2× 24 GB (TP=2)
Tensor parallelism (TP=2) splits weights AND KV symmetrically across both cards. Each card holds ~7 GB of weights (vs ~14 GB on single-card) plus its half of the KV pool. That's why dual unlocks what single can't:
- 262K context + vision + 2 streams fits at ~23.6 GB / card on
dual.yml(would need ~33 GB on a hypothetical single-card) - DFlash draft adds ~1.75 GB / card (manageable across two cards; would crowd out KV on single)
- 4 concurrent streams via
dual-turbouse TQ3 KV's compactness to fit 4 × full-context KV pools
For the single-card picture, see SINGLE_CARD.md.
Pick a config
General default — dual.yml
Workload: anything. Chat, tool agents, vision, mixed-modal. The recommended default for 2× 3090.
262K context, fp8 KV, MTP n=3, 2 streams, vision tower active. Genesis-less by design — fp8 KV doesn't trigger the cudagraph bug (#40880) that drove Genesis's existence on single-card. Pure vLLM nightly path. Tool calls work via --tool-call-parser qwen3_coder + --enable-auto-tool-choice. All verify-stress.sh checks pass clean.
When to pick: the obvious starting point. Unless one of the specialized variants below names your exact workload, this is right. Strongly recommended for IDE coding agents (Cline / OpenCode / Roo / Claude Code / Cursor) — fp8 KV avoids the inductor compile-path leak that affects all 4 TQ3-KV variants. See club-3090#16.
Multi-tenant — dual-turbo.yml
Workload: small team or agent farm running 2-4 concurrent sessions. Open WebUI multi-user, GitHub-Actions-with-AI-PRs flows, batch agent runs.
262K + TurboQuant 3-bit KV + Genesis v7.69 PROD env-var stack + 4 streams. TQ3 packs each KV slot to ~3 bits/token (vs fp8's ~8 bits), which is what makes 4 × 262K pools fit on 2 cards. KV pool 1.52M tokens, max concurrency 4.67×. Per-stream TPS lands at 58 narr / 76 code (n=5, CV 3-5%), AL 3.39-3.51, MTP avg accept 79-84%, VRAM 19.8 GB / card.
Concurrent throughput (n=4 streams of the canonical code prompt, 2026-05-01 PM, vLLM v0.20 + Genesis v7.65 dev tip — re-bench against v7.69 pending but decode TPS regime unchanged by the bump): aggregate code TPS 269 across 4 streams (3.63× speedup over single-stream 74 TPS), per-stream mean 74 (CV 3.1%) — true parallel decoding, not interleaved. See results/v0.20-migration/dual-turbo-concurrent.summary for the run-by-run.
When to pick: real concurrent load. Solo users won't see the win on the per-stream curve — but per-stream TPS at n=4 is essentially the same as n=1 here (74 vs 76 TPS code), so this is also a viable single-stream config if you want max KV pool. Pick this if you ever serve >1 request at a time, or want the biggest single-card-equivalent context.
Peak code TPS, with vision — dual-dflash.yml
Workload: code-heavy single-stream — fast iteration on quicksort-class problems, Cline going through a codebase, Cursor doing inline completions in a heavy file.
185K context (vs 262K — DFlash's draft model takes ~1.75 GB / card), FP16 KV (forced — DFlash's non-causal head_size=256 path requires fp16), DFlash N=5 draft model from Luce z-lab. Code TPS lands at 125 vs dual.yml's 89 — a real 40% jump on code prompts thanks to DFlash's higher acceptance length (AL ~4.4 vs MTP's 3.4).
When to pick: code is the dominant workload, you want TPS over context budget, vision is still required.
⚠️ Prereq before this compose works: download the DFlash draft model:
WITH_DFLASH_DRAFT=1 bash scripts/setup.sh qwen3.6-27b
# OR manually:
hf download z-lab/Qwen3.6-27B-DFlash --local-dir <MODEL_DIR>/qwen3.6-27b-dflash
Without it, vLLM falls back silently to baseline bf16 decode (~25 TPS, not 125). Reported by @lolren in #18.
Caveats:
- DFlash's per-position acceptance falls off faster than MTP — narrative TPS (82) is good but not dramatically better than
dual.yml's 69. The win is concentrated on code/repetitive prompts. - The z-lab draft is still under training (see UPSTREAM.md). Published 125 TPS code is against the 2026-04-26 snapshot at peak code-prompt conditions; agent traffic with mixed code + narrative + tool schemas will see lower per-stream TPS until z-lab tags training-complete. For autonomous coding agents (Cline / OpenCode / Pi / Claude Code) prefer
dual.yml(FP8 + MTP) until then — its 89 code TPS is robust across prompt shapes.
Peak code TPS, no vision — dual-dflash-noviz.yml
Workload: same as above, but no images. Squeezes another 15K of context out of the vision-tower's space.
200K context, FP16 KV, DFlash N=5, --language-model-only. Best code TPS in the lineup at 127. Narrative is 78 (slight drop vs vision variant from compute distribution).
When to pick: pure-text code work where you'd rather have 200K than 185K. Drop vision wherever you don't need it.
What dual-card unlocks (vs single)
| Want | Single-card status | Dual-card status |
|---|---|---|
| 262K context + vision | Works on long-vision.yml (192K) but Cliff 1 fires on big tool prefills |
dual.yml — clean, 262K, no Cliff 1 |
| 4 concurrent streams at full context | Single-card serializes; can't fit | dual-turbo.yml — 4 streams, 262K each |
| DFlash N=5 spec-decode | Blocked: DFlash needs head_size=256 + non-causal which doesn't fit single-card head-dim split | dual-dflash.yml / dual-dflash-noviz.yml |
| Code TPS >100 | Best single-card is 67 code (default) | 125-127 code (DFlash variants) |
| Long single prompts safely | Cliff 2 fires at 50-60K on vLLM single-card (forces llama.cpp fallback at 21 TPS) | TP=2 splits activation across cards — 237K single-prompt verified on dual.yml 2026-04-29 (~830 tok/s prefill, no OOM, peak 23.5 GB / card) |
| Big tool returns at 192K context | Cliff 1 fires on TQ3 paths regardless | dual.yml is below the cliff at 262K — activation budget is bigger per-card after split |
Common pitfalls (dual-card specifics)
Marlin pad-sub-tile-n mount dependency
The dual variants currently mount /opt/ai/vllm-src/vllm/model_executor/kernels/linear/mixed_precision/marlin.py (and one neighbor) read-only into the container. This is our patched fork of vllm#40361 — required for AutoRound W4A16 at TP=2 where output-dim shards fall below 64. You need to clone vLLM source to /opt/ai/vllm-src/ for these composes to boot. When the upstream PR lands, we'll drop the mount.
If you don't have /opt/ai/vllm-src/:
sudo mkdir -p /opt/ai && sudo chown $USER /opt/ai
git clone https://github.com/vllm-project/vllm.git /opt/ai/vllm-src
cd /opt/ai/vllm-src && git checkout main
PCIe allreduce overhead (no NVLink)
--disable-custom-all-reduce is set in all dual composes. Without it, vLLM tries to use a custom CUDA path that assumes NVLink topology and crashes. The trade is some allreduce latency on every layer, hence the per-stream TPS being lower than you'd see on an A100/A5000 dual setup with NVLink. Don't bother with NVLink bridges; this stack is intentionally PCIe-tested.
dual.yml is Genesis-less by design
The single-card cliffs (Cliff 1 / Cliff 2) and the cudagraph bug (#40880) that drove Genesis's existence don't fire on dual.yml — fp8 KV + 2 streams + 262K has plenty of headroom. So dual.yml runs plain vLLM nightly without any patch tree. If you want Genesis on dual (e.g. for dual-turbo's TQ3 spec-verify path), it's structurally enabled there but absent from dual.yml.
DFlash variants are FP16 KV (forced)
DFlash's combine_hidden_states path needs head_size=256 + non-causal, which forces FP16 KV on Ampere — there's no fp8 / TurboQuant alternative for this path right now. Tracked at vllm#40334. When that lands you can drop --dtype bfloat16 and let dtype auto-detect.
DFlash's vision compatibility
The DFlash draft + ViT path is documented and works (--language-model-only was historically required, now optional). dual-dflash.yml keeps vision; dual-dflash-noviz.yml drops it for an extra 15K ctx.
Single-stream user on dual = small win
If you're solo-using on dual, you're paying for hardware that mostly sits idle on alternate GPUs during single-stream decode. The win shows up at concurrency or when you need DFlash. For solo users, single-card is often the better cost choice.
Quick start
# 1. Setup (downloads model, clones Genesis + vllm-src, ~20 min cold)
bash scripts/setup.sh qwen3.6-27b
git clone https://github.com/vllm-project/vllm.git /opt/ai/vllm-src # required for dual variants
# 2. Pick + boot via wizard (asks GPU count + workload)
bash scripts/launch.sh
# 3. Or skip the wizard:
bash scripts/launch.sh --variant vllm/dual # general default
bash scripts/launch.sh --variant vllm/dual-turbo # 4 streams
bash scripts/launch.sh --variant vllm/dual-dflash # peak code + vision
bash scripts/launch.sh --variant vllm/dual-dflash-noviz # peak code, no vision
# 4. Sanity test
curl -sf http://localhost:8020/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"qwen3.6-27b-autoround","messages":[{"role":"user","content":"Capital of France?"}],"max_tokens":200}'
# 5. Switch later without re-running setup
bash scripts/switch.sh vllm/dual-dflash # for example
bash scripts/switch.sh --list # show all variants
Performance summary
For variance, AL / accept rates, per-config row docstrings: see each compose YAML, plus the TPS chart for the full lineup in the top-level README.
| Compose | Max ctx | Narr / Code TPS | TTFT | Concurrency | Vision | Best for |
|---|---|---|---|---|---|---|
dual.yml |
262K | 69 / 89 | ~145 ms | 2 | ✅ | general default |
dual-turbo.yml |
262K | 58 / 76 per stream (269 agg @ 4) | ~110 ms | 4 | ✅ | multi-tenant |
dual-dflash.yml |
185K | 82 / 125 | ~140 ms | 1 | ✅ | code + vision |
dual-dflash-noviz.yml |
200K | 78 / 127 | ~145 ms | 1 | ❌ | pure text code |
All four dual variants re-benched 2026-05-01 PM on the v0.20 + Genesis v7.65 dev tip substrate (n=5 measured + 3 warmup per prompt; v7.69 re-bench pending — decode TPS regime unchanged by the bump, which targets Cliff 2 prefill envelope on single-card):
| Variant | Narr / Code wall_TPS (CV) | vs prior chart |
|---|---|---|
dual.yml |
68.61 / 90.71 (CV 1.8% both) | flat (within noise) |
dual-turbo.yml |
58.33 / 76.01 (n=1) · 269 TPS aggregate at n=4 streams | matches prior |
dual-dflash.yml |
77.12 / 125.97 (CV 2-4%) | code flat, narr -5.9% (slight) |
dual-dflash-noviz.yml |
78.94 / 123.18 (CV 2-3%) | flat (within noise) |
Code TPS held within bench variance across all 4 variants — no v0.20 regression on fp8 / FP16 paths. Run-by-run + per-config summaries in results/v0.20-migration/.
Models supported on dual 3090
- Qwen3.6-27B — primary model. Quant choices, Genesis patch surface (single-card), engine internals all in the model directory.
- More models coming. As they're added, this section will list which dual-card configs each one supports.
Deep dives
- Model README — quant choices (AutoRound INT4 / GGUF), Genesis patch surface (mostly single-card relevant), what's working / what's not.
- INTERNALS.md — engineering rationale: AutoRound vs GPTQ, DFlash forensics, Marlin pad fork, MTP, upstream tracker.
- VRAM allocation diagram — full per-config breakdown across single + dual.
- FAQ.md — common questions (NVLink? AMD/Intel? Why fp8 not TQ3 on dual.yml? etc.).
- EXAMPLES.md — Python / TS / curl client snippets + IDE connection settings.
- HARDWARE.md — Ampere SM 8.6 specifics, NVLink (declined), power caps, PCIe topology.
- SINGLE_CARD.md — when one card is enough.