8ae205bcce
* baselines: backfill dual-max quality_8pk_think_on (110/150) The vllm/qwen-27b-dual-max row logged only the thinking-off 8-pack (107/150); the thinking-enabled run existed since the 2026-06-07 #515 quant-A/B (fp8 dual-max full 8-pack = 110/150, pack-defaults regime, autoround sibling 109) but was never promoted. Backfilled with provenance so it isn't re-derived. Guards: test-baselines, test-quality-thinking, test-quality-baseline all green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm * preflight: warn on single-card GPU_MEMORY_UTILIZATION override above default Two 5090 testers (#617 paul, and the same trap earlier) OOMed the tool-prefill step of verify-stress by setting GPU_MEMORY_UTILIZATION=0.92 on the nvfp4 single-card slug whose validated default is 0.85. On one GPU a higher util steals the free VRAM a large tool-response prefill needs for its activation peak, so vLLM OOMs mid-prefill even though boot succeeds — a confusing failure (boot ok, then HTTP 500 at step 2/8) that vLLM's own error hints at ("lower --gpu-memory-utilization"). Add preflight_single_card_util(): an advisory WARN that fires only when the user overrode GPU_MEMORY_UTILIZATION *above* the compose's shipped default on a TP<=1 config. Runs even under --force (the nvfp4 slug launches with --force), and no-ops for dual/multi-card, non-vLLM engines (no util default in the compose), a plain default run, or util at/below the default. Wired into switch.sh alongside the other force-independent hints. Verified: behavior matrix (single>default WARN; single=default/below/none and dual all silent) + test-preflight-compose-deps / test-model-switch / test-switch-registry-parity / test-launch-compat all green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
512 lines
26 KiB
YAML
512 lines
26 KiB
YAML
# baselines.yml — the shipped catalog baseline ("the bar").
|
||
# ===========================================================================
|
||
# One row per registry slug: the ACCEPTED display projection of a validated
|
||
# gate run (full evidence stays in results/rebench/<source_tag>/ or the
|
||
# BENCHMARKS.md row it was reviewed from). PR-reviewed; written by
|
||
# scripts/catalog-baseline.sh (slice 2) or by hand at promotion time.
|
||
# Consumers read the registry-emit join, NEVER this file directly, and NEVER
|
||
# BENCHMARKS.md (which stays the public human/cross-rig ledger — publication,
|
||
# not machine source).
|
||
#
|
||
# Conventions:
|
||
# narr_tps / code_tps DECODE-class TPS means (bench.sh canonical prompts,
|
||
# warm, n>=3; the per-token rate, not wall).
|
||
# quality_8pk thinking-OFF arm "P/150"; _think_on = thinking-ON.
|
||
# 8-pack noise band is ±5–7 — separates classes, not
|
||
# fine rankings (badge semantics, design §2.1.4).
|
||
# quality_env the harness fingerprint the quality numbers were
|
||
# produced ON: { harness: "benchlocal-cli X.Y.Z",
|
||
# sandbox_digest?: "sha256:…" } — §2.1.4 provenance
|
||
# (the half-deployed-sandbox lesson). Written by
|
||
# catalog-baseline.sh from the results JSON; digest
|
||
# rides along once benchlocal emits it.
|
||
# ctx_validated highest ctx EXERCISED by the gate, with the recall
|
||
# verdict: {tokens: N, niah: "clean@XK" | "allocation-
|
||
# only"} — allocation alone is NOT validation (§2.1.2).
|
||
# engine_pin the pin the numbers were measured ON. If it differs
|
||
# from the slug's current engine pin, the row is
|
||
# PROVABLY STALE → c3 badges "re-bench owed" (§2.2).
|
||
# rig / power_cap_w hardware fingerprint class + per-card caps at bench.
|
||
# source_tag results/rebench/<tag>/ when a tag dir exists;
|
||
# omitted for rows reviewed from a BENCHMARKS.md entry
|
||
# (the row comment names the source).
|
||
# tier provenance trust tier (slice 3):
|
||
# local — measured on the maintainer reference
|
||
# rig via the full gate (primary rows
|
||
# are ALWAYS tier: local)
|
||
# submitted — foreign bundle, ingested via
|
||
# catalog-baseline.sh --from-bundle;
|
||
# NOT independently verified
|
||
# reproduced — a submission independently re-run
|
||
# (second submitter on the same
|
||
# rig-class, or maintainer on matching
|
||
# hardware)
|
||
# submissions OPTIONAL map keyed by rig_class (e.g. 2x5090-pcie):
|
||
# cross-rig rows ingested from volunteer bundles.
|
||
# Same fields as a primary row PLUS required `source:`
|
||
# (the disc/PR link) and tier ∈ {submitted,reproduced}.
|
||
# ONE row per rig_class — newest-by-date wins, older
|
||
# stays in git history. Submissions NEVER become the
|
||
# slug's default bar (consumers show them rig-labeled);
|
||
# a slug MAY carry submissions with no primary row
|
||
# (measured on hardware we don't have).
|
||
#
|
||
# SEED WAVE 1 (2026-07-04): rows with airtight traceability only — a rebench
|
||
# tag on disk or an unambiguous decode-class BENCHMARKS.md row. Slugs still
|
||
# owed a row (evidence exists but needs review/archaeology) are listed at the
|
||
# bottom; DON'T guess numbers into them.
|
||
# SEED WAVE 2 (2026-07-04): the reviewed dispositions of the wave-1 gap list —
|
||
# 5 slugs seeded (each from a named BENCHMARKS.md row), the rest converted to
|
||
# explicit no-row gaps with their re-gate status (footer).
|
||
# ===========================================================================
|
||
schema_version: 1
|
||
baselines:
|
||
|
||
# ── Qwen3.6-27B ───────────────────────────────────────────────────────────
|
||
vllm/dual:
|
||
# BENCHMARKS.md 2026-06-30 row — the v0.22.0→v0.24.0 pin-bump validation
|
||
# gate (decode 70.7/93.5; MTP n=3 accept 3.51, KV pool 622K/2.37×,
|
||
# verify-full 8/8, verify-stress NIAH→240K, soak-continuous PASS).
|
||
# Measured ON the current vllm-stable pin → FRESH. On-rig quality gate was
|
||
# `--quick` (toolcall 11/15 · IF 15/15); the 8-pack below is CARRIED from
|
||
# multi-fast #584 (ryan, 4×3090, same AutoRound-INT4 weights, current harness)
|
||
# — TP-invariant (4-card = 2-card; TP is not a quality lever, confirmed on #584).
|
||
# Replace with an on-rig 2-card --full number if/when one is run.
|
||
narr_tps: 70.7
|
||
code_tps: 93.5
|
||
quality_8pk: "108/150"
|
||
quality_env: { harness: "carried from multi-fast #584 (ryan 4×3090, TP-invariant); on-rig 2-card --full pending" }
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-06-30
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
vllm/qwen-27b-dual-fast:
|
||
# alias of vllm/dual (same compose_path dual/autoround-int4/fp8-mtp.yml,
|
||
# same port — the #340 tier naming). Row mirrors vllm/dual verbatim;
|
||
# update both together. 8-pack carried from multi-fast #584 (TP-invariant).
|
||
narr_tps: 70.7
|
||
code_tps: 93.5
|
||
quality_8pk: "108/150"
|
||
quality_env: { harness: "carried from multi-fast #584 (ryan 4×3090, TP-invariant); on-rig 2-card --full pending" }
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-06-30
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
llamacpp/default:
|
||
# BENCHMARKS.md 2026-05-23 row (decode, n=3, CV 0.6%/1.4%).
|
||
# Measured on the pre-2026-05-26 ROLLING tag (unknowable build) — the
|
||
# compose has pinned server-cuda-b9246 since → BORN-STALE, re-bench owed.
|
||
narr_tps: 50.27
|
||
code_tps: 58.92
|
||
date: 2026-05-23
|
||
engine_pin: "ghcr.io/ggml-org/llama.cpp:server-cuda"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
llamacpp/mtp:
|
||
# BENCHMARKS.md 2026-06-18 row (decode, n=5, CV <2%, thinking-off).
|
||
narr_tps: 47.9
|
||
code_tps: 55.3
|
||
date: 2026-06-18
|
||
engine_pin: "ghcr.io/ggml-org/llama.cpp:server-cuda-b9246"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
llamacpp/mtp-vision:
|
||
# BENCHMARKS.md 2026-05-20 row (decode, n=5, CV 1.6%/1.7%).
|
||
# Rolling-tag era (pre-b9246 pin) → BORN-STALE, re-bench owed.
|
||
narr_tps: 56.52
|
||
code_tps: 66.17
|
||
date: 2026-05-20
|
||
engine_pin: "ghcr.io/ggml-org/llama.cpp:server-cuda"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/iq4ks-mtp:
|
||
# BENCHMARKS.md 2026-05-23 row (decode 60.39/72.40, n=3; set+readback 370 W).
|
||
narr_tps: 60.39
|
||
code_tps: 72.4
|
||
date: 2026-05-23
|
||
# ik composes pin by digest; measurement-era pin presumed == current cu13
|
||
# digest (verify + exactify on the next gate via the induction tool).
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/iq4ks-two-stage:
|
||
# BENCHMARKS.md 2026-05-24 row (decode, n=3, CV 1.9%/5.3%).
|
||
narr_tps: 59.4
|
||
code_tps: 97.8
|
||
date: 2026-05-24
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
beellama/dflash:
|
||
# BENCHMARKS.md 2026-05-30 row (decode 50.4/101.3, n=5, CV 5.6%/7.4%).
|
||
# Measured on the pre-#296 noonghunna multiarch image → PROVABLY STALE vs
|
||
# the current v0.3.2-preview engine pin (never re-benched on it) — the
|
||
# staleness badge is CORRECT here, keep it until a re-bench.
|
||
narr_tps: 50.4
|
||
code_tps: 101.3
|
||
date: 2026-05-30
|
||
engine_pin: "ghcr.io/noonghunna/beellama-cpp:multiarch-b9459-07ac3ce"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/iq4ks-mtp-vision:
|
||
# BENCHMARKS.md 2026-05-25 vision re-tune row (PR #227/#437): decode
|
||
# CARRIED from ik-llama/iq4ks-mtp (≈60/72 — decode is ctx-independent;
|
||
# never re-benched with the mmproj loaded). verify-stress 8/8 @160K+1M-px
|
||
# was measured directly (recall to 147K, vision functional). Carried
|
||
# numbers qualify because the delta config doesn't touch the decode path;
|
||
# replace with measured values on the next gate.
|
||
narr_tps: 60.39
|
||
code_tps: 72.4
|
||
date: 2026-05-25
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/byteshape-iq4xs-mtp:
|
||
# BENCHMARKS.md row (decode 115.60/137.07, n=5) + 8-pack 110/150 off —
|
||
# community intake #293/#299 reproduced on our rig 2026-06-02.
|
||
narr_tps: 115.6
|
||
code_tps: 137.07
|
||
quality_8pk: "110/150"
|
||
date: 2026-06-02
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Qwen3.6-35B-A3B ──────────────────────────────────────────────────────
|
||
ik-llama/apex-fit-q8q5:
|
||
# BENCHMARKS.md 2026-05-28 row (decode 105.63/156.80, n=5, CV 3.0%/1.5%;
|
||
# @laurimyllari's --fit + asymmetric q8/q5 KV from #241). verify-full 8/8,
|
||
# verify-stress 8/8 incl. 180K NIAH, soak-continuous PASS. Quality in the
|
||
# row is deterministic-6 76/90 + sandbox packs — not an 8-pack /150, so no
|
||
# quality_8pk here (add on the next full gate).
|
||
narr_tps: 105.63
|
||
code_tps: 156.8
|
||
ctx_validated: { tokens: 184320, niah: "clean@180K" }
|
||
date: 2026-05-28
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
vllm/qwen-35b-a3b-dual:
|
||
# BENCHMARKS.md promotion row (#259, decode 182.3/182.3; NIAH-clean 240K).
|
||
# Measured on v0.22.0; the slug now pins v0.24.0 (fast/max-tier era) →
|
||
# PROVABLY STALE — the born-stale demo row for the guard (§2.2).
|
||
# quality_8pk_think_on from the fresher #480-era measurement (incumbent
|
||
# baseline used in the Agents-A1 comparison).
|
||
narr_tps: 182.3
|
||
code_tps: 182.3
|
||
quality_8pk_think_on: "110/150"
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-05-30
|
||
engine_pin: "vllm/vllm-openai:v0.22.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Agents-A1 ────────────────────────────────────────────────────────────
|
||
vllm/agents-a1-dual:
|
||
# rebench tag agents-a1-fp8-dual (2026-07-03 gate): bench n=3 CV 0.1%
|
||
# (n=3 came from rebench-full.sh's old RUNS:-3 default — NOT an env leak
|
||
# as first attributed; fixed to protocol n=5 2026-07-04. n=5 was
|
||
# misstated in early publications; artifact-
|
||
# verified by catalog-baseline.sh 2026-07-04. think-on 110 = the #81
|
||
# rescored total, MATERIALIZED into the tag JSONs 2026-07-04 via
|
||
# `benchlocal-cli rescore --in-place` — artifact and publication agree;
|
||
# see docs/QUALITY_TEST.md "Rescoring saved results");
|
||
# verify-stress 8/8 staggered NIAH to 240,635 tok exact-recall, VRAM Δ0;
|
||
# soak PASS (99.8% retention). Quality = post benchlocal #79+#81 harness:
|
||
# OFF 105/150 · ON 110/150 (cli-40 ON 23/40 fresh-image).
|
||
# Cross-rig confirmed 2026-07-04 (#552 @sumo-dandan, decode −1%).
|
||
# Decode pair order corrected in wave-2 from the tag artifact
|
||
# (narrative 153.97 / code 153.78 — the wave-1 seed had them swapped).
|
||
narr_tps: 154.0
|
||
code_tps: 153.8
|
||
ttft_ms: 130
|
||
quality_8pk: "105/150"
|
||
quality_8pk_think_on: "110/150"
|
||
ctx_validated: { tokens: 240635, niah: "clean@240K" }
|
||
date: 2026-07-03
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
source_tag: "agents-a1-fp8-dual"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Gemma-4-31B ──────────────────────────────────────────────────────────
|
||
vllm/gemma-31b-dual:
|
||
# rebench tag gemma-31b-dual-bf16 (v0.24.0 consolidation gate #538/#539):
|
||
# decode 59.07/59.06, TTFT ~72 ms, soak PASS p50 58.69.
|
||
# TODO-review: quality_8pk + ctx_validated (gate artifacts have the NIAH
|
||
# ladder — extract the verdict; 224K claim pending #40391 for 262K).
|
||
narr_tps: 59.07
|
||
code_tps: 59.06
|
||
ttft_ms: 72
|
||
date: 2026-07-02
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
source_tag: "gemma-31b-dual-bf16"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Gemma-4-26B-A4B ──────────────────────────────────────────────────────
|
||
vllm/gemma-26ba4b-single:
|
||
# BENCHMARKS.md 2026-06-06 row (decode 169.2/219.9, CV 1.9%/2.0%; rebench
|
||
# tag gemma-26ba4b-int8r): single-3090 INT8-PTH KV via the vendored
|
||
# #40391 overlay on stock v0.22.0 (`vllm-gemma-stable`) + external MTP
|
||
# drafter n=4. verify-full ✓, NIAH-clean to 161K (91% of 176K), soak PASS
|
||
# 0-growth. gemma-stable still pins v0.22.0 → FRESH (the v0.24.0
|
||
# consolidation moved the gemma-TEXT slugs; this one kept the overlay pin).
|
||
narr_tps: 169.2
|
||
code_tps: 219.9
|
||
quality_8pk: "98/150"
|
||
quality_8pk_think_on: "109/150"
|
||
ctx_validated: { tokens: 164864, niah: "clean@161K" }
|
||
date: 2026-06-06
|
||
engine_pin: "vllm/vllm-openai:v0.22.0"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
source_tag: "gemma-26ba4b-int8r"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
beellama/gemma-dflash:
|
||
# inducted by catalog-baseline.sh from rebench tag gemma-dflash-regate (2026-07-04);
|
||
# evidence: verify-full pass · bench n=5 · quality both arms · NIAH ladder
|
||
narr_tps: 44.91
|
||
code_tps: 79.76
|
||
ttft_ms: 115
|
||
prefill_tps: { 10k: 949, 90k: 603 }
|
||
quality_8pk: "108/150"
|
||
quality_8pk_think_on: "113/150"
|
||
ctx_validated: { tokens: 117513, niah: "clean@118K" }
|
||
date: 2026-07-04
|
||
engine_pin: "ghcr.io/anbeeld/beellama.cpp@sha256:858e7cfbfb0d5d3d605723a82ace1474698de368b81bc310f4df4dadc6c1b7d2"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
source_tag: "gemma-dflash-regate"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
vllm/qwen-27b-dual-max:
|
||
# BENCHMARKS.md 2026-06-30 row (line 122) + disc #515 controlled A/B
|
||
# (c17490565, FP8-vs-W8A8, same compose isolated): decode 83.1/108.2,
|
||
# wall 82.0/104.6, TTFT 158/167 ms, KV pool 295K/1.13×, ~21.4 GB/card,
|
||
# NIAH clean@240K, 8-pack --full 107/150 (thinking-off). Measured ON the
|
||
# current v0.24.0 pin → FRESH. Corrects the stale "~56 TPS" probe.
|
||
# Promoted to ✅ Production 2026-07-06: soak-continuous PASS re-run on the
|
||
# v0.24.0 pin (0 err / 0 growth / 100% retention, p50 decode 85) + verify-
|
||
# full 9/9 + verify-stress fillable to 240,636 tok — completes the gate.
|
||
narr_tps: 83.1
|
||
code_tps: 108.2
|
||
ttft_ms: 158
|
||
prefill_tps: { 10k: 1364, 90k: 875 }
|
||
quality_8pk: "107/150"
|
||
# thinking-ON here is pack-defaults (mixed regime), NOT force-on: the 2026-06-07
|
||
# #515 quant-A/B ran the fp8 dual-max config full 8-pack = 110/150 (autoround
|
||
# sibling 109/150); measured then but never promoted to this row until 2026-07-08.
|
||
# Source: results/quality/quality-2026-06-07T15-18-18.json (endpoint :8013).
|
||
quality_8pk_think_on: "110/150"
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-06-30
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
submissions:
|
||
2x5090-pcie:
|
||
# inducted by catalog-baseline.sh --from-bundle from rebench tag 246-ab-fp8w (2026-07-05);
|
||
# evidence: verify-full pass · bench n=5 · NIAH ladder. QUALITY added
|
||
# 2026-07-06 (disc #571 c17542975): full 8-pack thinking-off 102/150 —
|
||
# within ±5-7 noise of our 2×3090 fp8 (107), so FP8 is near-lossless on
|
||
# native Blackwell too (closes the "route Blackwell → FP8 weights" gate).
|
||
# One dip: dataextract 9/15 (vs 13 elsewhere) — mostly verifier_fail, the
|
||
# known DE brittleness cluster, not an obvious fp8 regression. thinking-on
|
||
# run pending.
|
||
narr_tps: 134.52
|
||
code_tps: 165.11
|
||
ttft_ms: 56
|
||
prefill_tps: { 10k: 3693, 90k: 1906 }
|
||
# 8-pack: n=2 on-rig --full runs — 102 (2026-07-05) and 109 (2026-07-06,
|
||
# full per-pack + JSON in the source thread). Both inside the ±5-7
|
||
# noise band of the 2x3090 primary's 109/150 (#594) → cross-arch
|
||
# quality PARITY (Blackwell native FP8 GEMM vs Ampere Marlin W8A16).
|
||
# Row carries the better-evidenced latest run.
|
||
quality_8pk: "109/150"
|
||
quality_env: { harness: "benchlocal-cli --full on-rig (2026-07-06, thinking=mixed pack-defaults, timeout-scaling 1.37); n=2 runs 102->109, both within noise of the 2x3090 primary 109; per-pack: TC 14/15 IF 15/15 SO 14/15 DE 9/15 RM 13/15 BF 14/15 HA 11/20 CLI 19/40; https://github.com/noonghunna/club-3090/discussions/571#discussioncomment-17553023" }
|
||
ctx_validated: { tokens: 240635, niah: "clean@241K" }
|
||
date: 2026-07-05
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x5090-pcie"
|
||
power_cap_w: [575, 575]
|
||
source: "https://github.com/noonghunna/club-3090/discussions/571#discussioncomment-17536412"
|
||
source_tag: "246-ab-fp8w"
|
||
submitted_by: "guybrush01"
|
||
tier: submitted
|
||
|
||
vllm/qwen-27b-multi-fast:
|
||
# Cross-rig submission only — the reference bench has 2 cards, so there is
|
||
# no local 4-card row (and can't be). This is the FIRST measured data for
|
||
# the 4-card fast tier; promotion to ✅ Production rests on it (the compose
|
||
# header's own experimental-unblock: "until a real ≥4× 3090 host validates").
|
||
submissions:
|
||
4x3090-pcie:
|
||
# cross-rig report #584 (@ryanmpelletier, 4× 3090 Proxmox VM passthrough,
|
||
# v0.24.0, caps 350 W, all x16): verify-full 9/9 · verify-stress needles
|
||
# clean to 240K · soak-continuous PASS (0 err / 0 silent / <200 MiB) ·
|
||
# bench n=5 CV 4.2%/3.0%. KV pool 1.77M tok / 6.77× @262K (vs 2-card
|
||
# vllm/dual 622K/2.37×) — the 4-card win is concurrency, not single-stream
|
||
# speed (decode ~flat, TP=4 all-reduce over PCIe). ⚠ ceiling margin thin
|
||
# (851 MB @240K on driver 595, #149). QUALITY confirmed 2026-07-06 (#584
|
||
# follow-up): full 8-pack thinking-off 108/150 — within ±1 of the carried
|
||
# vllm/dual proxy (109), so TP-invariance is now MEASURED, not assumed.
|
||
narr_tps: 74.76
|
||
code_tps: 90.83
|
||
ttft_ms: 141
|
||
prefill_tps: { 10k: 1288, 90k: 1175 }
|
||
quality_8pk: "108/150"
|
||
quality_env: { harness: "benchlocal-cli @ club-3090 087d094 (sandboxes rebuilt 2026-07-06)" }
|
||
ctx_validated: { tokens: 240633, niah: "clean@240K" }
|
||
date: 2026-07-05
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "4x3090-pcie"
|
||
power_cap_w: [350, 350, 350, 350]
|
||
source: "https://github.com/noonghunna/club-3090/issues/584"
|
||
submitted_by: "ryanmpelletier"
|
||
tier: submitted
|
||
|
||
vllm/qwen-27b-multi-max:
|
||
# Cross-rig submission only — 4-card; our 2-card reference rig can't run it.
|
||
# FRESH v0.24.0 fp8/e4m3-KV gate (the #584 re-test the header requested) —
|
||
# supersedes @Whamp's doubly-stale #446 (int8-PTH KV, pre-v0.24.0, 85/102).
|
||
# Promoted vllm/qwen-27b-multi-max → ✅ Production on this report (2026-07-07).
|
||
submissions:
|
||
4x3090-pcie:
|
||
# #584 (@ryanmpelletier, 4× 3090 x16, Proxmox passthrough, v0.24.0, 350 W):
|
||
# attention backend FLASHINFER (the #595 fp8/e4m3 flip live at TP=4).
|
||
# verify-full 9/9 · verify-stress needles clean to 240,634 tok · soak-
|
||
# continuous PASS (0 err / 0 silent / <200 MiB, p50 124.94) · bench n=5 ·
|
||
# full 8-pack 111/150 (@ryanmpelletier built the benchlocal sandboxes) —
|
||
# ties the dual-max proxy 109 within noise → max-tier quality TP-invariant.
|
||
# Corroborated by a 2nd 4-card rig — #625 (@MoppelMat, bare metal, mixed
|
||
# x4/x8 lanes, 300 W): decode 79.2/101.6 (higher — bare metal), prefill
|
||
# 964/887 (lower — narrow lanes vs ryan's x16), soak PASS.
|
||
narr_tps: 74.10
|
||
code_tps: 91.30
|
||
ttft_ms: 133
|
||
prefill_tps: { 10k: 1275, 90k: 1141 }
|
||
quality_8pk: "111/150"
|
||
quality_env: { harness: "benchlocal-cli @ryanmpelletier, sandboxes built 2026-07-07 (cli-40 v1.0.2 · hermesagent-20 v1.0.0 · bugfind-15 v1.0.1); thinking-off" }
|
||
ctx_validated: { tokens: 240634, niah: "clean@240K" }
|
||
date: 2026-07-07
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "4x3090-pcie"
|
||
power_cap_w: [350, 350, 350, 350]
|
||
source: "https://github.com/noonghunna/club-3090/issues/584" # + #625 (@MoppelMat, corroborating 4-card)
|
||
submitted_by: "ryanmpelletier" # + MoppelMat (#625)
|
||
tier: submitted
|
||
|
||
vllm/qwen-27b-single-nvfp4:
|
||
# Cross-rig submission only — NVFP4 needs Hopper/Blackwell; our sm_86 rig
|
||
# can't run it. Community-validated on two 5090s (#613, #617).
|
||
submissions:
|
||
1x5090-pcie:
|
||
# #613 (@guybrush01, 5090, 185 GB RAM): the config sweep pinned MTP as the
|
||
# OOM cause — MTP-on OOMs at 98K, but PASSES at 65K + util 0.85. THIS is the
|
||
# 65K + MTP-on WIN: verify-stress all-pass (60K/90K needle clean, above-ctx
|
||
# cleanly rejected), MTP accept ~3.2 (draft ~74%), ~1.4 GB VRAM free — ~2x
|
||
# the SPEC=off throughput (71/71). Drove the default re-derate (98K/util-0.92
|
||
# → 65K/util-0.85, MTP kept). SPEC=off is the max-ctx (81K/98K @ 71) + tight-
|
||
# system-RAM path (@paulp83 28 GB host OOM-kills MTP load, #617). 8-pack owed.
|
||
narr_tps: 129.16
|
||
code_tps: 155.29
|
||
ttft_ms: 76
|
||
prefill_tps: { 10k: 4437 }
|
||
ctx_validated: { tokens: 58569, niah: "clean@58K (65K max-ctx)" }
|
||
date: 2026-07-08
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "1x5090-pcie"
|
||
power_cap_w: [600] # 5090 uncapped (default=max=600 W); single-card → no allreduce
|
||
source: "https://github.com/noonghunna/club-3090/issues/613" # + #617 (host-RAM axis)
|
||
submitted_by: "guybrush01"
|
||
tier: submitted
|
||
|
||
vllm/qwen-35b-a3b-single-nvfp4:
|
||
# Cross-rig submission only — NVFP4 needs Hopper/Blackwell; our sm_86 rig
|
||
# can't run it. FIRST community validation of the whole NVFP4 tier.
|
||
submissions:
|
||
1x5090-pcie:
|
||
# TWO independent 5090 validations, agreeing to within noise:
|
||
# · #612 (@guybrush01, 5090, 185 GB RAM): decode 257.3/258.0 TPS,
|
||
# verify-stress needle-clean to 120K (fillable 120,320 = 91% of
|
||
# 131K), soak PASS (0 growth). THE DEEPER run → ctx_validated below.
|
||
# · #619 (@paulp83, laptop 5090, 28 GB RAM): decode 255.8/257.9 TPS,
|
||
# verify-full 9/9, stress 9.8K+29K, soak PASS (15 MiB / p50 311).
|
||
# Both vLLM v0.24.0 modelopt_mixed, fp8 KV @131K, MTP off (as shipped).
|
||
# VRAM ~30.5/32 GB @131K — tight but flat. 8-pack quality NOT run on
|
||
# either (benchlocal sandboxes not built) → quality_8pk owed.
|
||
narr_tps: 255.79
|
||
code_tps: 257.93
|
||
ttft_ms: 60
|
||
ctx_validated: { tokens: 120320, niah: "clean@120K" }
|
||
date: 2026-07-07
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "1x5090-pcie"
|
||
power_cap_w: [600] # uncapped (5090 default=max=600 W); PCIe x4 slot (single-card → no allreduce, decode unaffected)
|
||
source: "https://github.com/noonghunna/club-3090/issues/612" # + #619 (two 5090s, see comment)
|
||
submitted_by: "guybrush01" # + paulp83 (#619)
|
||
tier: submitted
|
||
|
||
# ───────────────────────────────────────────────────────────────────────────
|
||
# KNOWN GAPS — wave-2 dispositions (2026-07-04): each slug below has NO row
|
||
# on purpose; the disposition says what unlocks one. Do NOT guess numbers in.
|
||
# vllm/minimal — only ~32/~33 approximations exist (never a protocol bench);
|
||
# needs a proper gate run. No row until then.
|
||
# vllm/gemma-12b-dual-bf16-mtp · vllm/gemma-12b-single-int8-mtp —
|
||
# conflicting-era rows (2026-06-04) with ambiguous decode-vs-wall class,
|
||
# AND both slugs re-pinned in the v0.24.0 gemma consolidation (#538/#539)
|
||
# → any seed would be born-stale AND class-uncertain. Re-gate owed.
|
||
# beellama/gemma-dflash — no direct BENCHMARKS row under the slug (numbers
|
||
# live in promotion-era notes). PRIORITY re-gate: this is the single-card
|
||
# gemma DEFAULT — the catalog's recommended path with no bar is the worst
|
||
# gap on this list.
|
||
# llamacpp/deckard40B-dual-mtp — 41.6 tok/s MTP exists only in registry
|
||
# notes; no BENCHMARKS row. Needs a gate run.
|
||
# ───────────────────────────────────────────────────────────────────────────
|