alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

composes: formalize Status enum + Caveats field (100% coverage)

Browse Source 
The previous "Status optional, only when not production" convention
left readers guessing whether absence-of-status meant "validated
production" or "author forgot to fill it in." Making Status required +
enumerated removes that ambiguity.

Schema enhancement (codified in AGENTS.md):
- Status: now REQUIRED, exactly one of:
    ✅ Production              — verify-full + stress + bench + soak PASS
    ⚠️ Production w/ caveats   — works under documented constraints
    🧪 Experimental             — under active validation
    👁️ Preview                  — known quality issues, not production
    ⏸️ Upstream-gated           — blocked by external action (PR/driver)
    🗑️ Deprecated               — kept for historical reference
- Caveats: REQUIRED if Status is ⚠️/👁️/⏸️/🗑️, OMITTED for ✅/🧪.
  Single-line summary or short bullet list with issue/PR links.

Status applied to all 27 composes (vLLM + llama-cpp, all topologies):
  ✅ Production:              20  (canonical Qwen + Gemma + llama.cpp + gemma-awq)
  ⚠️ Production w/ caveats:    4  (long-text*, long-vision, carnice-bf16mtp)
  ⏸️ Upstream-gated:           2  (gemma single boot-OOM, gemma dflash-int8 needs #42102)
  👁️ Preview:                  1  (qwopus-bf16mtp)
  🧪 Experimental:             0
  🗑️ Deprecated:               0

llama-cpp composes also gained `Genesis: N/A — llama.cpp engine` for
parity with the vLLM schema (Genesis is Qwen3-Next-specific and
vLLM-only; explicit N/A prevents future readers from looking for
patches that don't exist).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-09 12:19:44 +00:00
parent 9fbce96120
commit e1137d6889
25 changed files with 48 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
AGENTS.md
View File

@@ -113,12 +113,28 @@ Every compose starts with a `Profile (at-a-glance)` block declaring the (Model,
# Vision: <yes | no>
# Max ctx: <e.g. 262K>
# Genesis: <none | v7.72.2 | N/A — Genesis is Qwen3-Next-specific>
# Status: <optional — only if not production: ⚠️ PREVIEW / BOOT-OOM / etc.>
# Status: <REQUIRED — exactly one of the enum values below>
# Caveats: <REQUIRED if Status is ⚠️ / 👁️ / ⏸️ / 🗑️; otherwise omit>
# Best for: <one short phrase — what workload this serves; ⭐ for canonical>
# ---------------------------------------------------------------------------
# (existing free-form description continues below)
```
**Status enum** — pick exactly one:
| Value | Meaning | Validation gate |
|---|---|---|
| `✅ Production` | Recommended for users. | verify-full 8/8 + verify-stress 7/7 + bench (BENCHMARKS row) + soak-continuous PASS. |
| `⚠️ Production w/ caveats` | Works under documented constraints; not the same as broken. | Same gates as Production, but a known-and-disclosed limitation exists (e.g., Cliff 2b at >50K). Caveats line MUST list the constraint. |
| `🧪 Experimental` | Under active validation; may not boot or pass all tests. | Typically untracked in git. No production guarantee. |
| `👁️ Preview` | Known quality issues; tracked but not for production. | E.g., quality regressions in soak / NIAH. Caveats line MUST list specific issues. |
| `⏸️ Upstream-gated` | Exists but blocked by external action (PR merge, driver fix, hardware ceiling). | Boots only with vendored override OR doesn't boot until external dep lands. Caveats line MUST point at the external dep. |
| `🗑️ Deprecated` | Kept for historical reference; will be removed. | N/A — flagged for cleanup. |
**Why this enum exists**: the previous "Status optional, only when not production" convention left readers guessing whether absence-of-status meant "validated production" or "author forgot to fill it in." Making Status required + enumerated removes that ambiguity. Users picking a config can scan to one field and know the lifecycle stage instantly; new contributors must consciously declare it when authoring.
The `Caveats:` line is REQUIRED whenever Status is ⚠️ / 👁️ / ⏸️ / 🗑️, OMITTED for ✅ / 🧪. Format: a single-line summary or a short bullet list, with links to issues / discussions / upstream PRs where relevant.
This rule applies to **shipped composes AND local-only test composes** — apply the convention even before deciding whether to ship; it avoids a rename later if the experiment graduates.
When testing a new model, create the directory hierarchy from the start: `models/<new-model>/<engine>/compose/<topology>/docker-compose.yml`. The hierarchy enforces the convention; filenames encode only the feature stack within that topology. When the model isn't Qwen3-Next, write `Genesis: N/A — Genesis is Qwen3-Next-specific` in the profile schema so readers don't expect Genesis-style perf folds where they don't apply.

1
models/gemma-4-31b/vllm/compose/dual/dflash.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 32K (BF16 ceiling — for 262K code-optimal use dual-dflash-int8.yml)
# Genesis: N/A — Genesis is Qwen3-Next-specific
# Status: ✅ Production
# Best for: Peak code TPS on Gemma 4 (105/177 — +18% code over MTP) ⭐
# ---------------------------------------------------------------------------
# Gemma-4-31B-it (Intel AutoRound INT4) + z-lab Gemma 4 DFlash drafter

1
models/gemma-4-31b/vllm/compose/dual/docker-compose.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 32K (BF16 ceiling — for 262K use dual-int8.yml)
# Genesis: N/A — Genesis is Qwen3-Next-specific
# Status: ✅ Production
# Best for: Gemma 4 dual-card default — vision + tools + balanced TPS ⭐
# ---------------------------------------------------------------------------
# Gemma-4-31B-it (Intel AutoRound INT4) + Google MTP "assistant" drafter

1
models/gemma-4-31b/vllm/compose/dual/int8.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 262K (8.2× over bf16 ceiling) — override MAX_NUM_SEQS=4 for 98K + 3.6× concurrency
# Genesis: N/A — Genesis is Qwen3-Next-specific
# Status: ✅ Production
# Best for: Gemma 4 long-context default — full 262K ctx ⭐
# ---------------------------------------------------------------------------
# Gemma-4-31B-it (Intel AutoRound INT4) + Google MTP "assistant" drafter

2
models/gemma-4-31b/vllm/compose/single/docker-compose.yml
View File

@@ -7,6 +7,8 @@
# Vision: yes
# Max ctx: ~16K (theoretical) — boot fails before serving on 24 GB
# Genesis: N/A — Genesis is Qwen3-Next-specific
# Status: ⏸️ Upstream-gated
# Caveats: Boot-OOMs on Ampere 24 GB regardless of KV format (weights+drafter+profiling at 8K ctx leaves no KV pool). Compose preserved for 32 GB+ GPUs (RTX 5090 32 GB validated by @apnar disc #67 — 160/215 TPS at 32K MTP). Re-evaluate when (a) vLLM adds Ampere-aware fp8 dispatch OR (b) PR #41745-style relaxation lands.
# Best for: Single-card 32 GB+ GPUs (RTX 5090 32 GB validated by @apnar disc #67)
# For 24 GB Ampere, use dual.yml (dual-card) instead.
# ---------------------------------------------------------------------------

2
models/qwen3.6-27b/llama-cpp/compose/single/concurrent.yml
View File

@@ -7,6 +7,8 @@
# KV: q4_0 (4-bit packed)
# Vision: yes (mmproj F16)
# Max ctx: 192K pool / 4 parallel slots
# Genesis: N/A — llama.cpp engine; Genesis is vLLM/Qwen3-Next-specific
# Status: ✅ Production
# Best for: Single-card multi-tenant llama.cpp — 4 concurrent agents
# at smaller per-stream ctx; trade max-ctx for parallelism
# ---------------------------------------------------------------------------

2
models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml
View File

@@ -7,6 +7,8 @@
# KV: q4_0 (4-bit packed)
# Vision: yes (mmproj F16)
# Max ctx: 262K (full model native — no Cliff 1 / Cliff 2)
# Genesis: N/A — llama.cpp engine; Genesis is vLLM/Qwen3-Next-specific
# Status: ✅ Production
# Best for: Bulletproof single-card path — slow decode (~21 TPS) but
# cliff-immune; recommended fallback when vLLM hits OOM at long ctx
# ---------------------------------------------------------------------------

3
models/qwen3.6-27b/vllm/compose/dual/carnice-bf16mtp.yml
View File

@@ -8,7 +8,8 @@
# Vision: yes
# Max ctx: 262K
# Genesis: v7.72.2 (full PROD env stack)
# Status: AL=2.0 (vs Lorbus's 3.4-3.8) — TPS 72/80, working but suboptimal
# Status: ⚠️ Production w/ caveats
# Caveats: MTP AL=2.0 vs Lorbus's 3.4-3.8 (TPS 72/80, working but suboptimal). Carnice was uploaded with stripped MTP keys; we delta-merged the head. Three diagnostic steps queued in /opt/ai/club-3090-todo.md (Tier 2 Carnice plan).
# Best for: Hermes-style agentic tool-calling specialization
# ---------------------------------------------------------------------------
# Carnice-V2-27B + BF16 MTP overlay — dual RTX 3090 variant.

1
models/qwen3.6-27b/vllm/compose/dual/dflash-noviz.yml
View File

@@ -7,6 +7,7 @@
# Vision: no (vision tower dropped — frees ~0.5 GB/card for ctx)
# Max ctx: 200K (+15K vs dual-dflash.yml)
# Genesis: none — DFlash drafter handles spec-decode independently
# Status: ✅ Production
# Best for: Peak code TPS, no-vision, with extra context headroom ⭐
# ---------------------------------------------------------------------------
# Dual-card DFlash text-only — TP=2 + DFlash N=5 + 200K ctx + NO vision.

1
models/qwen3.6-27b/vllm/compose/dual/dflash.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 185K
# Genesis: none — DFlash drafter handles spec-decode independently
# Status: ✅ Production
# Best for: Peak code TPS with vision (AL ~4.4 vs MTP's 3.4) ⭐
# ---------------------------------------------------------------------------
# Dual-card DFlash — TP=2 + DFlash N=5 spec-decode + 185K ctx + vision.

1
models/qwen3.6-27b/vllm/compose/dual/docker-compose.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 262K (237K single-prompt verified)
# Genesis: none — intentionally Genesis-free (isolation control + fallback)
# Status: ✅ Production
# Best for: General-purpose dual-card vision + tools + long-ctx default ⭐
# ---------------------------------------------------------------------------
# Dual RTX 3090 — DEFAULT for 2× cards. Qwen3.6-27B + MTP n=3 + fp8 KV + vision.

1
models/qwen3.6-27b/vllm/compose/dual/nvlink-dflash-noviz.yml
View File

@@ -7,6 +7,7 @@
# Vision: no (frees ~0.5 GB/card for ctx)
# Max ctx: 200K (+15K vs dual-nvlink-dflash.yml; NVLink lifts max to 188K stable)
# Genesis: none
# Status: ✅ Production
# Best for: NVLink + peak code TPS, no-vision (max ctx headroom)
# ---------------------------------------------------------------------------
# Dual RTX 3090 with NVLink + DFlash text-only — TP=2 + DFlash N=5 + 200K ctx

1
models/qwen3.6-27b/vllm/compose/dual/nvlink-dflash.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 185K
# Genesis: none — DFlash drafter handles spec-decode independently
# Status: ✅ Production
# Best for: NVLink + peak code TPS — +17% over PCIe-only dual-dflash.yml
# ---------------------------------------------------------------------------
# Dual RTX 3090 with NVLink + DFlash — TP=2 + DFlash N=5 spec-decode + 185K ctx

1
models/qwen3.6-27b/vllm/compose/dual/nvlink-turbo.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 262K
# Genesis: v7.72.2 (full PROD env stack)
# Status: ✅ Production
# Best for: NVLink + multi-tenant — +11% narr / +12% code over PCIe-only dual-turbo.yml
# ---------------------------------------------------------------------------
# Dual-card Turbo with NVLink — TP=2 + TurboQuant KV (turboquant_3bit_nc) + MTP n=3

1
models/qwen3.6-27b/vllm/compose/dual/nvlink.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 262K
# Genesis: v7.72.2
# Status: ✅ Production
# Best for: Users with the NVLink bridge — +15% narr / +15% code over PCIe-only dual.yml
# ---------------------------------------------------------------------------
# Dual RTX 3090 with NVLink — opt-in variant for users with the bridge installed.

1
models/qwen3.6-27b/vllm/compose/dual/turbo.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 262K
# Genesis: v7.72.2 (PN59 streaming-GDN + PN34 workspace-lock-relax + PN12)
# Status: ✅ Production
# Best for: Multi-tenant agent fleet — 4 streams at full ctx (4.67× concurrency vs dual.yml) ⭐
# ---------------------------------------------------------------------------
# Dual-card Turbo — TP=2 + TurboQuant KV (turboquant_3bit_nc) + MTP n=3 + Genesis v7.69 dev.

1
models/qwen3.6-27b/vllm/compose/multi4/dflash.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 262K
# Genesis: none — DFlash drafter handles spec-decode independently
# Status: ✅ Production
# Best for: 4-card peak code TPS path
# ---------------------------------------------------------------------------
# Quad RTX 3090 DFlash — TP=4 + DFlash N=5 + 262K ctx + vision.

1
models/qwen3.6-27b/vllm/compose/multi4/docker-compose.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 262K
# Genesis: v7.72.2 (full PROD env stack)
# Status: ✅ Production
# Best for: 4-card TP=4 baseline — high concurrency, max KV pool size
# ---------------------------------------------------------------------------
# Quad RTX 3090 — TP=4 baseline. Qwen3.6-27B + MTP n=3 + fp8 KV + vision.

1
models/qwen3.6-27b/vllm/compose/single/bounded-thinking.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 180K
# Genesis: v7.72.2 (full PROD env stack)
# Status: ✅ Production
# Best for: Reasoning/coding workloads with bounded thinking budget
# via FSM-constrained scratchpad grammar — caps think budget,
# improves HE+/LCB scores, +24pp accuracy on max_tokens=4096

1
models/qwen3.6-27b/vllm/compose/single/docker-compose.yml
View File

@@ -7,6 +7,7 @@
# Vision: yes
# Max ctx: 48K (mem-util 0.92 — safe single-card default)
# Genesis: v7.72.2 (full PROD env stack)
# Status: ✅ Production
# Best for: Single-card default — vision + tools + MTP + Cliff-safe at
# <48K ctx; recommended starting point for 1× 3090
# ---------------------------------------------------------------------------

3
models/qwen3.6-27b/vllm/compose/single/long-text-no-mtp.yml
View File

@@ -16,7 +16,8 @@
# Vision: no (text-only)
# Max ctx: 200K (max-context single-card path)
# Genesis: v7.72.2
# Status: ⚠️ Cliff 2b applies at >50K single-prompt — see warning above
# Status: ⚠️ Production w/ caveats
# Caveats: Cliff 2b at >50K single-prompt context. Sandermage/genesis-vllm-patches#22. Workaround: dual.yml or llamacpp/default.
# Best for: Max single-card context window for one-shot summarization /
# long-document Q&A; slower decode (no spec-decode) is the trade
# ---------------------------------------------------------------------------

3
models/qwen3.6-27b/vllm/compose/single/long-text.yml
View File

@@ -16,7 +16,8 @@
# Vision: no (text-only — frees VRAM for ctx)
# Max ctx: 180K (was 214K pre-Cliff-2b; reduced for safety)
# Genesis: v7.72.2
# Status: ⚠️ NOT IDE-agent safe (Cliff 2b at >50K — see warning above)
# Status: ⚠️ Production w/ caveats
# Caveats: Cliff 2b at >50K single-prompt context (Genesis PN59 doesn't engage on chunked-prefill). NOT IDE-agent safe. Filed at Sandermage/genesis-vllm-patches#22. Workaround: dual.yml or llamacpp/default.
# Best for: RAG / single-shot summarization at <25K accumulated ctx;
# for IDE agents use tools-text.yml or dual.yml instead
# ---------------------------------------------------------------------------

3
models/qwen3.6-27b/vllm/compose/single/long-vision.yml
View File

@@ -16,7 +16,8 @@
# Vision: yes (vision tower active)
# Max ctx: 145K (vision tower costs ~500 MiB activation budget vs long-text)
# Genesis: v7.72.2
# Status: ⚠️ Cliff 2b applies at >50K single-prompt — see warning above
# Status: ⚠️ Production w/ caveats
# Caveats: Cliff 2b at >50K single-prompt context. Sandermage/genesis-vllm-patches#22. Workaround: dual.yml or llamacpp/default.
# Best for: Single-card vision workloads (image Q&A, multimodal RAG)
# ---------------------------------------------------------------------------
# Long-vision variant — 198K context with vision on a single 3090.

1
models/qwen3.6-27b/vllm/compose/single/minimal.yml
View File

@@ -7,6 +7,7 @@
# Vision: no
# Max ctx: 65K (mem-util 0.95)
# Genesis: none — bare vLLM, isolation control
# Status: ✅ Production
# Best for: Debugging baseline / 20 GB Ampere fallback when TQ3 paths
# don't fit / first-time setup verification
# ---------------------------------------------------------------------------

1
models/qwen3.6-27b/vllm/compose/single/tools-text.yml
View File

@@ -7,6 +7,7 @@
# Vision: no
# Max ctx: 75K (Cliff 1 mech B safe via Genesis P8 + fp8 KV path)
# Genesis: v7.72.2 (P8 active for IDE-agent prefill safety)
# Status: ✅ Production
# Best for: IDE-agent workloads (Cline / OpenCode / Roo / Claude Code /
# Cursor) — fp8 KV escape valve from the long-text TQ3 leak (see #16)
# ---------------------------------------------------------------------------