docs: surface Gemma 4 31B + add at-a-glance profile schemas to canonical composes
Gemma 4 31B has been production-ready on dual 3090 since 2026-05-08 (5 shipped composes, full benched data in BENCHMARKS.md, disc #67 thread posted) but the user-facing surfaces still framed the stack as Qwen3.6-only. This commit fixes that without renaming any compose files (Option A: document, don't rename — keeps backward compat with external GitHub issue/disc references to specific filenames). Changes: 1. README.md "Supported models" — add Gemma 4 31B row alongside Qwen3.6 with dual-card-only caveat (single-card boots OOM on Ampere 24 GB even at 8K ctx; needs 32 GB+ — validated on RTX 5090 by @apnar). 2. docs/DUAL_CARD.md - Split TL;DR into two model sections (Qwen3.6-27B + Gemma 4 31B) - Added 6 Gemma config rows: gemma-mtp.yml (32K balanced), gemma-mtp-int8.yml (262K + multi-stream variant), gemma-dflash.yml (peak code TPS 105/177), gemma-dflash-int8.yml (262K code-optimal, gated on PR #42102), gemma-awq.yml (118K AWQ-4bit weights) - "Models supported on dual 3090" expanded with substantive Gemma entry - "Deep dives" split into per-model + cross-cutting sections, adds Gemma model README + disc #67 link 3. Profile schema header added to 8 canonical composes for at-a-glance scanning — every compose now declares Model / Topology / Drafter / KV / Vision / Max ctx / Genesis / Best-for in a structured comment block before the existing free-form description: - Qwen: dual.yml, dual-turbo.yml, dual-dflash.yml, dual-dflash-noviz.yml - Gemma: gemma-mtp.yml, gemma-mtp-int8.yml, gemma-mtp-tp1.yml, gemma-dflash.yml Schema added to remaining composes (gemma-awq, gemma-dflash-int8, plus 15 Qwen variants) in follow-up commits. Refs: noonghunna/club-3090#67 Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
noonghunna
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
fa9df49ef2
commit
4d7356aac6
@@ -43,6 +43,9 @@ Each hardware page lists every supported model with the working composes for tha
|
||||
| Model | Status | Card counts | Engines | Highlights |
|
||||
|---|---|---|---|---|
|
||||
| **[Qwen3.6-27B](models/qwen3.6-27b/)** | Production-ready ⭐ | 1× / 2× 3090 | vLLM ✅ · llama.cpp ✅ · SGLang ❌ blocked | Vision · tools · MTP n=3 · up to 262K ctx · vLLM dual = 89/127 TPS · llama.cpp single = full 262K, no prefill cliffs |
|
||||
| **[Gemma 4 31B](models/gemma-4-31b/)** | Production-ready (dual-card only on Ampere 24 GB) | 2× 3090 only ¹ | vLLM ✅ · llama.cpp ❌ · SGLang ❌ | Vision · tools · MTP n=3 (Google official drafter) **OR** DFlash n=7 (z-lab drafter) · up to 262K ctx via INT8 PTH KV (PR [#40391](https://github.com/vllm-project/vllm/pull/40391) vendored) · MTP dual = 106/141 TPS at 32K, 95/126 at 262K · DFlash dual = 105/177 TPS at 32K (code-optimal), 87/146 at 262K via [PR #42102](https://github.com/vllm-project/vllm/pull/42102) (our patch, OPEN) |
|
||||
|
||||
¹ Single-card boot OOMs on Ampere 24 GB regardless of KV format (weights + drafter + profiling at 8K ctx leaves no KV pool). Single-card Gemma 4 is feasible on 32 GB+ GPUs (validated on RTX 5090 32 GB by [@apnar](https://github.com/noonghunna/club-3090/discussions/67#discussioncomment-16832042)).
|
||||
|
||||
More models coming. The repo structure scales — when we add Qwen3.5-27B / GLM-4.6 / etc., they go under `models/<name>/` with the same internal pattern.
|
||||
|
||||
|
||||
+29
-3
@@ -8,6 +8,8 @@ You have **2× RTX 3090s, PCIe-only (no NVLink)**. This page is the front door f
|
||||
|
||||
## TL;DR — pick by workload
|
||||
|
||||
### Qwen3.6-27B (default model, also runs single-card)
|
||||
|
||||
| What you're doing | Compose | Max ctx | Narr / Code TPS | VRAM per card | Why |
|
||||
|---|---|---|---|---|---|
|
||||
| **Hermes agentic fine-tune** (Carnice tool specialization) | [`carnice-bf16mtp.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.carnice-bf16mtp.yml) | **262K** | **72 / 80** | ~22.3 / 24 GB | BF16 MTP overlay. Hermes-style assistant. Available on HF: [wasifb/Carnice_V2_27B_INT4_BF16MTP](https://huggingface.co/wasifb/Carnice_V2_27B_INT4_BF16MTP) |
|
||||
@@ -16,6 +18,19 @@ You have **2× RTX 3090s, PCIe-only (no NVLink)**. This page is the front door f
|
||||
| Peak code TPS with vision | [`dual-dflash.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml) | **185K** | **82 / 125** | ~23.6 / 24 GB | DFlash N=5 + 1.75 GB draft per card, AL ~4.4 (vs MTP's 3.4) |
|
||||
| Peak code TPS, no vision | [`dual-dflash-noviz.yml`](../models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml) | **200K** | **78 / 127** | ~23.8 / 24 GB | DFlash + no vision, +15K ctx vs dual-dflash |
|
||||
|
||||
### Gemma 4 31B (dual-card only on Ampere 24 GB ¹)
|
||||
|
||||
| What you're doing | Compose | Max ctx | Narr / Code TPS | VRAM per card | Why |
|
||||
|---|---|---|---|---|---|
|
||||
| General-purpose default (vision + tools + 32K ctx) | [`gemma-mtp.yml`](../models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml) ⭐ | 32K | **106 / 141** | ~22 / 24 GB | bf16 KV, MTP n=3 (Google's official `gemma-4-31B-it-assistant` drafter). PR [#41745](https://github.com/vllm-project/vllm/pull/41745) merged upstream. |
|
||||
| Long-context default (262K ctx, balanced TPS) | [`gemma-mtp-int8.yml`](../models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-int8.yml) | **262K** | **95 / 126** | ~22.1 / 24 GB | INT8 PTH KV via vendored PR [#40391](https://github.com/vllm-project/vllm/pull/40391) overlay. **8.2× context lift** for ~10% TPS cost. NIAH PASS at 137K. |
|
||||
| Multi-stream long-context (3.6× concurrency at 98K) | [`gemma-mtp-int8.yml`](../models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-int8.yml) (override `MAX_NUM_SEQS=4`) | 98K | **96 / 127** per-stream | ~22.2 / 24 GB | INT8 PTH KV pool 354K tokens → 3.6× concurrency. |
|
||||
| Peak code TPS with vision | [`gemma-dflash.yml`](../models/gemma-4-31b/vllm/compose/docker-compose.gemma-dflash.yml) | 32K | **105 / 177** | ~22.3 / 24 GB | z-lab Gemma 4 DFlash drafter, n=7. **+18% code TPS over MTP** (177 vs 141). bf16 KV. |
|
||||
| Long-context code-optimal (262K + DFlash) | [`gemma-dflash-int8.yml`](../models/gemma-4-31b/vllm/compose/docker-compose.gemma-dflash-int8.yml) ⚠️ | **262K** | **87 / 146** | ~22.0 / 24 GB | DFlash drafter + INT8 PTH KV target via vLLM PR [#42102](https://github.com/vllm-project/vllm/pull/42102) (our patch — currently OPEN). **+16% code TPS at 262K vs `gemma-mtp-int8.yml`** (146 vs 126). Boots after #42102 lands or with the vendored vllm-src override mounts. |
|
||||
| Multi-tenant agent fleet (118K + AWQ-4bit weights) | [`gemma-awq.yml`](../models/gemma-4-31b/vllm/compose/docker-compose.gemma-awq.yml) | 118K | **101 / 142** | ~19.8 / 24 GB | AWQ-4bit weights (`cyankiwi/gemma-4-31B-it-AWQ-4bit`) — bypasses PR #40391's per-token-head bug entirely. NIAH PASS at 88K. |
|
||||
|
||||
¹ Single-card boot OOMs on Ampere 24 GB regardless of KV format. Single-card Gemma 4 is feasible on 32 GB+ GPUs (validated on RTX 5090 32 GB by [@apnar](https://github.com/noonghunna/club-3090/discussions/67#discussioncomment-16832042) — 160/215 TPS at 32K MTP, 150/261 at 12K DFlash). Tracked in [`docs/UPSTREAM.md`](UPSTREAM.md) row 78 + [#67](https://github.com/noonghunna/club-3090/discussions/67).
|
||||
|
||||
> **VRAM column is per-card** under TP=2 (each card holds half the weights + half the KV; both cards' totals are nearly identical). For a 2× 20 GB rig (e.g. 2× 3080-20GB / 40 GB combined), `dual.yml` and `dual-turbo` should fit; `dual-dflash*` won't (FP16 KV + DFlash draft pushes per-card past 20 GB). Component breakdown in [`tools/charts/gen-vram.py`](../tools/charts/gen-vram.py).
|
||||
|
||||
Run any of these via `bash scripts/launch.sh` (interactive) or `bash scripts/switch.sh <variant>`.
|
||||
@@ -197,17 +212,28 @@ Code TPS held within bench variance across all 4 variants — no v0.20 regressio
|
||||
|
||||
## Models supported on dual 3090
|
||||
|
||||
- **[Qwen3.6-27B](../models/qwen3.6-27b/)** — primary model. Quant choices, Genesis patch surface (single-card), engine internals all in the model directory.
|
||||
- More models coming. As they're added, this section will list which dual-card configs each one supports.
|
||||
- **[Qwen3.6-27B](../models/qwen3.6-27b/)** — primary model. Runs single-card AND dual-card. Quant choices (AutoRound INT4, GGUF Q3_K_XL / Q4_K_M), Genesis patch surface (mostly single-card relevant), engine internals all in the model directory.
|
||||
- **[Gemma 4 31B](../models/gemma-4-31b/)** — dual-card only on Ampere 24 GB (single-card boot OOMs even at 8K ctx; needs 32 GB+ per card). Two drafter paths (MTP via Google's official `gemma-4-31B-it-assistant` + DFlash via z-lab) × two KV strategies (bf16 / 32K vs INT8 PTH / 262K) + AWQ-4bit-weights variant. Genesis doesn't apply (Genesis patches are Qwen3-Next-specific).
|
||||
|
||||
As more models land, they'll show up here with their dual-card compose set.
|
||||
|
||||
---
|
||||
|
||||
## Deep dives
|
||||
|
||||
### Qwen3.6-27B
|
||||
- **[Model README](../models/qwen3.6-27b/)** — quant choices (AutoRound INT4 / GGUF), Genesis patch surface (mostly single-card relevant), what's working / what's not.
|
||||
- **[INTERNALS.md](../models/qwen3.6-27b/INTERNALS.md)** — engineering rationale: AutoRound vs GPTQ, DFlash forensics, Marlin pad fork, MTP, upstream tracker.
|
||||
- **[VRAM allocation diagram](../models/qwen3.6-27b/README.md#vram-allocation-across-configs)** — full per-config breakdown across single + dual.
|
||||
|
||||
### Gemma 4 31B
|
||||
- **[Model README](../models/gemma-4-31b/)** — quants (BF16 source, AWQ-4bit, INT8 PTH KV via PR #40391 vendored overlay), drafter options (MTP / DFlash), upstream PR tracker.
|
||||
- **[Discussion #67](https://github.com/noonghunna/club-3090/discussions/67)** — first Ampere consumer cross-rig data thread. MTP, DFlash, INT8 PTH long-context, single-card 5090 numbers.
|
||||
|
||||
### Cross-cutting
|
||||
- **[FAQ.md](FAQ.md)** — common questions (NVLink? AMD/Intel? Why fp8 not TQ3 on dual.yml? etc.).
|
||||
- **[EXAMPLES.md](EXAMPLES.md)** — Python / TS / curl client snippets + IDE connection settings.
|
||||
- **[HARDWARE.md](HARDWARE.md)** — Ampere SM 8.6 specifics, NVLink (declined), power caps, PCIe topology.
|
||||
- **[SINGLE_CARD.md](SINGLE_CARD.md)** — when one card is enough.
|
||||
- **[CLIFFS.md](CLIFFS.md)** — single-card Cliff 1 / Cliff 2 mechanisms (mostly Qwen3-Next-specific; Gemma 4 doesn't have these because it's dense attention without DeltaNet).
|
||||
- **[UPSTREAM.md](UPSTREAM.md)** — every upstream PR / issue we filed or watch (vLLM, Genesis, lucebox-hub, transformers, llama.cpp, SGLang).
|
||||
- **[SINGLE_CARD.md](SINGLE_CARD.md)** — when one card is enough (Qwen3.6-27B only — Gemma 4 needs ≥32 GB single-card).
|
||||
|
||||
@@ -1,5 +1,15 @@
|
||||
# ===========================================================================
|
||||
# Gemma-4-31B-it (Intel AutoRound INT4) + Google MTP "assistant" drafter
|
||||
# Profile (at-a-glance):
|
||||
# Model: Gemma 4 31B (Intel AutoRound INT4)
|
||||
# Topology: Dual 3090 PCIe (TP=2, no NVLink)
|
||||
# Drafter: z-lab Gemma 4 DFlash n=7 (block-diffusion drafter, vLLM PR #41703 vendored overlay)
|
||||
# KV: bfloat16 (forced for drafter compatibility)
|
||||
# Vision: yes
|
||||
# Max ctx: 32K (BF16 ceiling — for 262K code-optimal use gemma-dflash-int8.yml)
|
||||
# Genesis: N/A — Genesis is Qwen3-Next-specific
|
||||
# Best for: Peak code TPS on Gemma 4 (105/177 — +18% code over MTP) ⭐
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gemma-4-31B-it (Intel AutoRound INT4) + z-lab Gemma 4 DFlash drafter
|
||||
# 2× RTX 3090 TP=2 + bfloat16 KV + 32K ctx + 4 streams.
|
||||
#
|
||||
# Status: COMMUNITY-CONTRIBUTED, EXPERIMENTAL.
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: Gemma 4 31B (Intel AutoRound INT4)
|
||||
# Topology: Dual 3090 PCIe (TP=2, no NVLink)
|
||||
# Drafter: MTP n=3 (Google's official `gemma-4-31B-it-assistant`, BF16 0.5B)
|
||||
# KV: int8_per_token_head (1 byte/token, via vendored PR #40391 overlay)
|
||||
# Vision: yes
|
||||
# Max ctx: 262K (8.2× over bf16 ceiling) — override MAX_NUM_SEQS=4 for 98K + 3.6× concurrency
|
||||
# Genesis: N/A — Genesis is Qwen3-Next-specific
|
||||
# Best for: Gemma 4 long-context default — full 262K ctx ⭐
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gemma-4-31B-it (Intel AutoRound INT4) + Google MTP "assistant" drafter
|
||||
# 2× RTX 3090 TP=2 + per-token-head INT8 KV + ~120K ctx target.
|
||||
#
|
||||
|
||||
@@ -1,6 +1,17 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: Gemma 4 31B (Intel AutoRound INT4)
|
||||
# Topology: Single 3090 (TP=1) — ⚠️ BOOT-OOMs on 24 GB Ampere; for 32 GB+ only
|
||||
# Drafter: MTP n=3 (Google's official `gemma-4-31B-it-assistant`)
|
||||
# KV: bfloat16 (fp8 dispatch blocked on Ampere — Triton fp8e4nv unsupported on sm_86)
|
||||
# Vision: yes
|
||||
# Max ctx: ~16K (theoretical) — boot fails before serving on 24 GB
|
||||
# Genesis: N/A — Genesis is Qwen3-Next-specific
|
||||
# Best for: Single-card 32 GB+ GPUs (RTX 5090 32 GB validated by @apnar disc #67)
|
||||
# For 24 GB Ampere, use gemma-mtp.yml (dual-card) instead.
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gemma-4-31B-it (Intel AutoRound INT4) + Google MTP "assistant" drafter
|
||||
# 2× RTX 3090 TP=2 + bfloat16 KV + 32K ctx + 4 streams.
|
||||
# Single-card TP=1 + bfloat16 KV — preserved for 32 GB+ GPUs (5090, A6000).
|
||||
#
|
||||
# Status: COMMUNITY-CONTRIBUTED, EXPERIMENTAL.
|
||||
# First Ampere consumer cross-rig data on Google's Gemma 4 MTP drafters
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: Gemma 4 31B (Intel AutoRound INT4)
|
||||
# Topology: Dual 3090 PCIe (TP=2, no NVLink) — single-card boot-OOMs on 24 GB
|
||||
# Drafter: MTP n=3 (Google's official `gemma-4-31B-it-assistant`, BF16 0.5B)
|
||||
# KV: bfloat16 (2 bytes/token)
|
||||
# Vision: yes
|
||||
# Max ctx: 32K (BF16 ceiling — for 262K use gemma-mtp-int8.yml)
|
||||
# Genesis: N/A — Genesis is Qwen3-Next-specific
|
||||
# Best for: Gemma 4 dual-card default — vision + tools + balanced TPS ⭐
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gemma-4-31B-it (Intel AutoRound INT4) + Google MTP "assistant" drafter
|
||||
# 2× RTX 3090 TP=2 + bfloat16 KV + 32K ctx + 4 streams.
|
||||
#
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: Qwen3.6-27B (Lorbus AutoRound INT4 + BF16 mtp.fc preserved)
|
||||
# Topology: Dual 3090 PCIe (TP=2, no NVLink)
|
||||
# Drafter: z-lab DFlash N=5 (block-diffusion drafter)
|
||||
# KV: fp8_e5m2 (forced bf16 for drafter via --dtype bfloat16)
|
||||
# Vision: no (vision tower dropped — frees ~0.5 GB/card for ctx)
|
||||
# Max ctx: 200K (+15K vs dual-dflash.yml)
|
||||
# Genesis: none — DFlash drafter handles spec-decode independently
|
||||
# Best for: Peak code TPS, no-vision, with extra context headroom ⭐
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dual-card DFlash text-only — TP=2 + DFlash N=5 + 200K ctx + NO vision.
|
||||
#
|
||||
# vs `docker-compose.dual-dflash.yml` (the with-vision variant): drops MoonViT
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: Qwen3.6-27B (Lorbus AutoRound INT4 + BF16 mtp.fc preserved)
|
||||
# Topology: Dual 3090 PCIe (TP=2, no NVLink)
|
||||
# Drafter: z-lab DFlash N=5 (block-diffusion drafter)
|
||||
# KV: fp8_e5m2 (forced bf16 for drafter via --dtype bfloat16)
|
||||
# Vision: yes
|
||||
# Max ctx: 185K
|
||||
# Genesis: none — DFlash drafter handles spec-decode independently
|
||||
# Best for: Peak code TPS with vision (AL ~4.4 vs MTP's 3.4) ⭐
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dual-card DFlash — TP=2 + DFlash N=5 spec-decode + 185K ctx + vision.
|
||||
#
|
||||
# The peak-single-stream code TPS variant. DFlash N=5 (z-lab's Qwen3.6-27B
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: Qwen3.6-27B (Lorbus AutoRound INT4 + BF16 mtp.fc preserved)
|
||||
# Topology: Dual 3090 PCIe (TP=2, no NVLink)
|
||||
# Drafter: MTP n=3 (built-in)
|
||||
# KV: turboquant_3bit_nc (TQ3, 0.375 bytes/token)
|
||||
# Vision: yes
|
||||
# Max ctx: 262K
|
||||
# Genesis: v7.72.2 (PN59 streaming-GDN + PN34 workspace-lock-relax + PN12)
|
||||
# Best for: Multi-tenant agent fleet — 4 streams at full ctx (4.67× concurrency vs dual.yml) ⭐
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dual-card Turbo — TP=2 + TurboQuant KV (turboquant_3bit_nc) + MTP n=3 + Genesis v7.69 dev.
|
||||
#
|
||||
# 2026-05-02 PM note: setup.sh now pins Genesis to v7.69 (commit 2db18df) which
|
||||
|
||||
@@ -1,4 +1,14 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: Qwen3.6-27B (Lorbus AutoRound INT4 + BF16 mtp.fc preserved)
|
||||
# Topology: Dual 3090 PCIe (TP=2, no NVLink)
|
||||
# Drafter: MTP n=3 (built-in)
|
||||
# KV: fp8_e5m2 (1 byte/token)
|
||||
# Vision: yes
|
||||
# Max ctx: 262K (237K single-prompt verified)
|
||||
# Genesis: none — intentionally Genesis-free (isolation control + fallback)
|
||||
# Best for: General-purpose dual-card vision + tools + long-ctx default ⭐
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dual RTX 3090 — DEFAULT for 2× cards. Qwen3.6-27B + MTP n=3 + fp8 KV + vision.
|
||||
#
|
||||
# What this gives you (vs the single-card default):
|
||||
|
||||
Reference in New Issue
Block a user