noonghunnaandClaude Opus 4.8
b0c5bc1d47
The catalog join (services.py) skipped submission-only slugs, so the main TPS column stayed blank for 4-card slugs (multi-fast, multi-max) — their numbers only appeared in the detail panel. Now the join falls back to the BEST cross-rig submission, tagged submission_rig, and Measurement.tps_label renders it ⑂-labelled (a submission is NOT this rig's own on-rig bar). Updated the slice-3 test to assert the new behaviour. Also seed multi-max as a stale-flagged @Whamp #446 submission (int8-PTH, pre-#595 fp8 flip) so it shows *something* with the caveat until the pending 4-card fp8 re-test (#584) lands. Result: multi-fast -> "75/91 ⑂ · 8pk 108/150"; multi-max -> "85/102 ⑂ †". 239 c3 tests + baselines guards green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
443 lines
22 KiB
YAML
443 lines
22 KiB
YAML
# baselines.yml — the shipped catalog baseline ("the bar").
|
||
# ===========================================================================
|
||
# One row per registry slug: the ACCEPTED display projection of a validated
|
||
# gate run (full evidence stays in results/rebench/<source_tag>/ or the
|
||
# BENCHMARKS.md row it was reviewed from). PR-reviewed; written by
|
||
# scripts/catalog-baseline.sh (slice 2) or by hand at promotion time.
|
||
# Consumers read the registry-emit join, NEVER this file directly, and NEVER
|
||
# BENCHMARKS.md (which stays the public human/cross-rig ledger — publication,
|
||
# not machine source).
|
||
#
|
||
# Conventions:
|
||
# narr_tps / code_tps DECODE-class TPS means (bench.sh canonical prompts,
|
||
# warm, n>=3; the per-token rate, not wall).
|
||
# quality_8pk thinking-OFF arm "P/150"; _think_on = thinking-ON.
|
||
# 8-pack noise band is ±5–7 — separates classes, not
|
||
# fine rankings (badge semantics, design §2.1.4).
|
||
# quality_env the harness fingerprint the quality numbers were
|
||
# produced ON: { harness: "benchlocal-cli X.Y.Z",
|
||
# sandbox_digest?: "sha256:…" } — §2.1.4 provenance
|
||
# (the half-deployed-sandbox lesson). Written by
|
||
# catalog-baseline.sh from the results JSON; digest
|
||
# rides along once benchlocal emits it.
|
||
# ctx_validated highest ctx EXERCISED by the gate, with the recall
|
||
# verdict: {tokens: N, niah: "clean@XK" | "allocation-
|
||
# only"} — allocation alone is NOT validation (§2.1.2).
|
||
# engine_pin the pin the numbers were measured ON. If it differs
|
||
# from the slug's current engine pin, the row is
|
||
# PROVABLY STALE → c3 badges "re-bench owed" (§2.2).
|
||
# rig / power_cap_w hardware fingerprint class + per-card caps at bench.
|
||
# source_tag results/rebench/<tag>/ when a tag dir exists;
|
||
# omitted for rows reviewed from a BENCHMARKS.md entry
|
||
# (the row comment names the source).
|
||
# tier provenance trust tier (slice 3):
|
||
# local — measured on the maintainer reference
|
||
# rig via the full gate (primary rows
|
||
# are ALWAYS tier: local)
|
||
# submitted — foreign bundle, ingested via
|
||
# catalog-baseline.sh --from-bundle;
|
||
# NOT independently verified
|
||
# reproduced — a submission independently re-run
|
||
# (second submitter on the same
|
||
# rig-class, or maintainer on matching
|
||
# hardware)
|
||
# submissions OPTIONAL map keyed by rig_class (e.g. 2x5090-pcie):
|
||
# cross-rig rows ingested from volunteer bundles.
|
||
# Same fields as a primary row PLUS required `source:`
|
||
# (the disc/PR link) and tier ∈ {submitted,reproduced}.
|
||
# ONE row per rig_class — newest-by-date wins, older
|
||
# stays in git history. Submissions NEVER become the
|
||
# slug's default bar (consumers show them rig-labeled);
|
||
# a slug MAY carry submissions with no primary row
|
||
# (measured on hardware we don't have).
|
||
#
|
||
# SEED WAVE 1 (2026-07-04): rows with airtight traceability only — a rebench
|
||
# tag on disk or an unambiguous decode-class BENCHMARKS.md row. Slugs still
|
||
# owed a row (evidence exists but needs review/archaeology) are listed at the
|
||
# bottom; DON'T guess numbers into them.
|
||
# SEED WAVE 2 (2026-07-04): the reviewed dispositions of the wave-1 gap list —
|
||
# 5 slugs seeded (each from a named BENCHMARKS.md row), the rest converted to
|
||
# explicit no-row gaps with their re-gate status (footer).
|
||
# ===========================================================================
|
||
schema_version: 1
|
||
baselines:
|
||
|
||
# ── Qwen3.6-27B ───────────────────────────────────────────────────────────
|
||
vllm/dual:
|
||
# BENCHMARKS.md 2026-06-30 row — the v0.22.0→v0.24.0 pin-bump validation
|
||
# gate (decode 70.7/93.5; MTP n=3 accept 3.51, KV pool 622K/2.37×,
|
||
# verify-full 8/8, verify-stress NIAH→240K, soak-continuous PASS).
|
||
# Measured ON the current vllm-stable pin → FRESH. On-rig quality gate was
|
||
# `--quick` (toolcall 11/15 · IF 15/15); the 8-pack below is CARRIED from
|
||
# multi-fast #584 (ryan, 4×3090, same AutoRound-INT4 weights, current harness)
|
||
# — TP-invariant (4-card = 2-card; TP is not a quality lever, confirmed on #584).
|
||
# Replace with an on-rig 2-card --full number if/when one is run.
|
||
narr_tps: 70.7
|
||
code_tps: 93.5
|
||
quality_8pk: "108/150"
|
||
quality_env: { harness: "carried from multi-fast #584 (ryan 4×3090, TP-invariant); on-rig 2-card --full pending" }
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-06-30
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
vllm/qwen-27b-dual-fast:
|
||
# alias of vllm/dual (same compose_path dual/autoround-int4/fp8-mtp.yml,
|
||
# same port — the #340 tier naming). Row mirrors vllm/dual verbatim;
|
||
# update both together. 8-pack carried from multi-fast #584 (TP-invariant).
|
||
narr_tps: 70.7
|
||
code_tps: 93.5
|
||
quality_8pk: "108/150"
|
||
quality_env: { harness: "carried from multi-fast #584 (ryan 4×3090, TP-invariant); on-rig 2-card --full pending" }
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-06-30
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
llamacpp/default:
|
||
# BENCHMARKS.md 2026-05-23 row (decode, n=3, CV 0.6%/1.4%).
|
||
# Measured on the pre-2026-05-26 ROLLING tag (unknowable build) — the
|
||
# compose has pinned server-cuda-b9246 since → BORN-STALE, re-bench owed.
|
||
narr_tps: 50.27
|
||
code_tps: 58.92
|
||
date: 2026-05-23
|
||
engine_pin: "ghcr.io/ggml-org/llama.cpp:server-cuda"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
llamacpp/mtp:
|
||
# BENCHMARKS.md 2026-06-18 row (decode, n=5, CV <2%, thinking-off).
|
||
narr_tps: 47.9
|
||
code_tps: 55.3
|
||
date: 2026-06-18
|
||
engine_pin: "ghcr.io/ggml-org/llama.cpp:server-cuda-b9246"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
llamacpp/mtp-vision:
|
||
# BENCHMARKS.md 2026-05-20 row (decode, n=5, CV 1.6%/1.7%).
|
||
# Rolling-tag era (pre-b9246 pin) → BORN-STALE, re-bench owed.
|
||
narr_tps: 56.52
|
||
code_tps: 66.17
|
||
date: 2026-05-20
|
||
engine_pin: "ghcr.io/ggml-org/llama.cpp:server-cuda"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/iq4ks-mtp:
|
||
# BENCHMARKS.md 2026-05-23 row (decode 60.39/72.40, n=3; set+readback 370 W).
|
||
narr_tps: 60.39
|
||
code_tps: 72.4
|
||
date: 2026-05-23
|
||
# ik composes pin by digest; measurement-era pin presumed == current cu13
|
||
# digest (verify + exactify on the next gate via the induction tool).
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/iq4ks-two-stage:
|
||
# BENCHMARKS.md 2026-05-24 row (decode, n=3, CV 1.9%/5.3%).
|
||
narr_tps: 59.4
|
||
code_tps: 97.8
|
||
date: 2026-05-24
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
beellama/dflash:
|
||
# BENCHMARKS.md 2026-05-30 row (decode 50.4/101.3, n=5, CV 5.6%/7.4%).
|
||
# Measured on the pre-#296 noonghunna multiarch image → PROVABLY STALE vs
|
||
# the current v0.3.2-preview engine pin (never re-benched on it) — the
|
||
# staleness badge is CORRECT here, keep it until a re-bench.
|
||
narr_tps: 50.4
|
||
code_tps: 101.3
|
||
date: 2026-05-30
|
||
engine_pin: "ghcr.io/noonghunna/beellama-cpp:multiarch-b9459-07ac3ce"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/iq4ks-mtp-vision:
|
||
# BENCHMARKS.md 2026-05-25 vision re-tune row (PR #227/#437): decode
|
||
# CARRIED from ik-llama/iq4ks-mtp (≈60/72 — decode is ctx-independent;
|
||
# never re-benched with the mmproj loaded). verify-stress 8/8 @160K+1M-px
|
||
# was measured directly (recall to 147K, vision functional). Carried
|
||
# numbers qualify because the delta config doesn't touch the decode path;
|
||
# replace with measured values on the next gate.
|
||
narr_tps: 60.39
|
||
code_tps: 72.4
|
||
date: 2026-05-25
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
ik-llama/byteshape-iq4xs-mtp:
|
||
# BENCHMARKS.md row (decode 115.60/137.07, n=5) + 8-pack 110/150 off —
|
||
# community intake #293/#299 reproduced on our rig 2026-06-02.
|
||
narr_tps: 115.6
|
||
code_tps: 137.07
|
||
quality_8pk: "110/150"
|
||
date: 2026-06-02
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Qwen3.6-35B-A3B ──────────────────────────────────────────────────────
|
||
ik-llama/apex-fit-q8q5:
|
||
# BENCHMARKS.md 2026-05-28 row (decode 105.63/156.80, n=5, CV 3.0%/1.5%;
|
||
# @laurimyllari's --fit + asymmetric q8/q5 KV from #241). verify-full 8/8,
|
||
# verify-stress 8/8 incl. 180K NIAH, soak-continuous PASS. Quality in the
|
||
# row is deterministic-6 76/90 + sandbox packs — not an 8-pack /150, so no
|
||
# quality_8pk here (add on the next full gate).
|
||
narr_tps: 105.63
|
||
code_tps: 156.8
|
||
ctx_validated: { tokens: 184320, niah: "clean@180K" }
|
||
date: 2026-05-28
|
||
engine_pin: "ghcr.io/ikawrakow/ik-llama-cpp@sha256:5f914f1ccade922417af58c94bd1cbb558052c8852d86678ead3fe693eec0143"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
vllm/qwen-35b-a3b-dual:
|
||
# BENCHMARKS.md promotion row (#259, decode 182.3/182.3; NIAH-clean 240K).
|
||
# Measured on v0.22.0; the slug now pins v0.24.0 (fast/max-tier era) →
|
||
# PROVABLY STALE — the born-stale demo row for the guard (§2.2).
|
||
# quality_8pk_think_on from the fresher #480-era measurement (incumbent
|
||
# baseline used in the Agents-A1 comparison).
|
||
narr_tps: 182.3
|
||
code_tps: 182.3
|
||
quality_8pk_think_on: "110/150"
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-05-30
|
||
engine_pin: "vllm/vllm-openai:v0.22.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Agents-A1 ────────────────────────────────────────────────────────────
|
||
vllm/agents-a1-dual:
|
||
# rebench tag agents-a1-fp8-dual (2026-07-03 gate): bench n=3 CV 0.1%
|
||
# (n=3 came from rebench-full.sh's old RUNS:-3 default — NOT an env leak
|
||
# as first attributed; fixed to protocol n=5 2026-07-04. n=5 was
|
||
# misstated in early publications; artifact-
|
||
# verified by catalog-baseline.sh 2026-07-04. think-on 110 = the #81
|
||
# rescored total, MATERIALIZED into the tag JSONs 2026-07-04 via
|
||
# `benchlocal-cli rescore --in-place` — artifact and publication agree;
|
||
# see docs/QUALITY_TEST.md "Rescoring saved results");
|
||
# verify-stress 8/8 staggered NIAH to 240,635 tok exact-recall, VRAM Δ0;
|
||
# soak PASS (99.8% retention). Quality = post benchlocal #79+#81 harness:
|
||
# OFF 105/150 · ON 110/150 (cli-40 ON 23/40 fresh-image).
|
||
# Cross-rig confirmed 2026-07-04 (#552 @sumo-dandan, decode −1%).
|
||
# Decode pair order corrected in wave-2 from the tag artifact
|
||
# (narrative 153.97 / code 153.78 — the wave-1 seed had them swapped).
|
||
narr_tps: 154.0
|
||
code_tps: 153.8
|
||
ttft_ms: 130
|
||
quality_8pk: "105/150"
|
||
quality_8pk_think_on: "110/150"
|
||
ctx_validated: { tokens: 240635, niah: "clean@240K" }
|
||
date: 2026-07-03
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
source_tag: "agents-a1-fp8-dual"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Gemma-4-31B ──────────────────────────────────────────────────────────
|
||
vllm/gemma-31b-dual:
|
||
# rebench tag gemma-31b-dual-bf16 (v0.24.0 consolidation gate #538/#539):
|
||
# decode 59.07/59.06, TTFT ~72 ms, soak PASS p50 58.69.
|
||
# TODO-review: quality_8pk + ctx_validated (gate artifacts have the NIAH
|
||
# ladder — extract the verdict; 224K claim pending #40391 for 262K).
|
||
narr_tps: 59.07
|
||
code_tps: 59.06
|
||
ttft_ms: 72
|
||
date: 2026-07-02
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
source_tag: "gemma-31b-dual-bf16"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
# ── Gemma-4-26B-A4B ──────────────────────────────────────────────────────
|
||
vllm/gemma-26ba4b-single:
|
||
# BENCHMARKS.md 2026-06-06 row (decode 169.2/219.9, CV 1.9%/2.0%; rebench
|
||
# tag gemma-26ba4b-int8r): single-3090 INT8-PTH KV via the vendored
|
||
# #40391 overlay on stock v0.22.0 (`vllm-gemma-stable`) + external MTP
|
||
# drafter n=4. verify-full ✓, NIAH-clean to 161K (91% of 176K), soak PASS
|
||
# 0-growth. gemma-stable still pins v0.22.0 → FRESH (the v0.24.0
|
||
# consolidation moved the gemma-TEXT slugs; this one kept the overlay pin).
|
||
narr_tps: 169.2
|
||
code_tps: 219.9
|
||
quality_8pk: "98/150"
|
||
quality_8pk_think_on: "109/150"
|
||
ctx_validated: { tokens: 164864, niah: "clean@161K" }
|
||
date: 2026-06-06
|
||
engine_pin: "vllm/vllm-openai:v0.22.0"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
source_tag: "gemma-26ba4b-int8r"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
beellama/gemma-dflash:
|
||
# inducted by catalog-baseline.sh from rebench tag gemma-dflash-regate (2026-07-04);
|
||
# evidence: verify-full pass · bench n=5 · quality both arms · NIAH ladder
|
||
narr_tps: 44.91
|
||
code_tps: 79.76
|
||
ttft_ms: 115
|
||
prefill_tps: { 10k: 949, 90k: 603 }
|
||
quality_8pk: "108/150"
|
||
quality_8pk_think_on: "113/150"
|
||
ctx_validated: { tokens: 117513, niah: "clean@118K" }
|
||
date: 2026-07-04
|
||
engine_pin: "ghcr.io/anbeeld/beellama.cpp@sha256:858e7cfbfb0d5d3d605723a82ace1474698de368b81bc310f4df4dadc6c1b7d2"
|
||
rig: "1x3090-pcie"
|
||
power_cap_w: [370]
|
||
source_tag: "gemma-dflash-regate"
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
|
||
vllm/qwen-27b-dual-max:
|
||
# BENCHMARKS.md 2026-06-30 row (line 122) + disc #515 controlled A/B
|
||
# (c17490565, FP8-vs-W8A8, same compose isolated): decode 83.1/108.2,
|
||
# wall 82.0/104.6, TTFT 158/167 ms, KV pool 295K/1.13×, ~21.4 GB/card,
|
||
# NIAH clean@240K, 8-pack --full 107/150 (thinking-off). Measured ON the
|
||
# current v0.24.0 pin → FRESH. Corrects the stale "~56 TPS" probe.
|
||
# Promoted to ✅ Production 2026-07-06: soak-continuous PASS re-run on the
|
||
# v0.24.0 pin (0 err / 0 growth / 100% retention, p50 decode 85) + verify-
|
||
# full 9/9 + verify-stress fillable to 240,636 tok — completes the gate.
|
||
narr_tps: 83.1
|
||
code_tps: 108.2
|
||
ttft_ms: 158
|
||
prefill_tps: { 10k: 1364, 90k: 875 }
|
||
quality_8pk: "107/150"
|
||
ctx_validated: { tokens: 245760, niah: "clean@240K" }
|
||
date: 2026-06-30
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x3090-pcie"
|
||
power_cap_w: [370, 420]
|
||
submitted_by: "noonghunna"
|
||
tier: local
|
||
submissions:
|
||
2x5090-pcie:
|
||
# inducted by catalog-baseline.sh --from-bundle from rebench tag 246-ab-fp8w (2026-07-05);
|
||
# evidence: verify-full pass · bench n=5 · NIAH ladder. QUALITY added
|
||
# 2026-07-06 (disc #571 c17542975): full 8-pack thinking-off 102/150 —
|
||
# within ±5-7 noise of our 2×3090 fp8 (107), so FP8 is near-lossless on
|
||
# native Blackwell too (closes the "route Blackwell → FP8 weights" gate).
|
||
# One dip: dataextract 9/15 (vs 13 elsewhere) — mostly verifier_fail, the
|
||
# known DE brittleness cluster, not an obvious fp8 regression. thinking-on
|
||
# run pending.
|
||
narr_tps: 134.52
|
||
code_tps: 165.11
|
||
ttft_ms: 56
|
||
prefill_tps: { 10k: 3693, 90k: 1906 }
|
||
quality_8pk: "102/150"
|
||
quality_env: { harness: "benchlocal-cli @ club-3090 master (2026-07-06)" }
|
||
ctx_validated: { tokens: 240635, niah: "clean@241K" }
|
||
date: 2026-07-05
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "2x5090-pcie"
|
||
power_cap_w: [575, 575]
|
||
source: "https://github.com/noonghunna/club-3090/discussions/571#discussioncomment-17536412"
|
||
source_tag: "246-ab-fp8w"
|
||
submitted_by: "guybrush01"
|
||
tier: submitted
|
||
|
||
vllm/qwen-27b-multi-fast:
|
||
# Cross-rig submission only — the reference bench has 2 cards, so there is
|
||
# no local 4-card row (and can't be). This is the FIRST measured data for
|
||
# the 4-card fast tier; promotion to ✅ Production rests on it (the compose
|
||
# header's own experimental-unblock: "until a real ≥4× 3090 host validates").
|
||
submissions:
|
||
4x3090-pcie:
|
||
# cross-rig report #584 (@ryanmpelletier, 4× 3090 Proxmox VM passthrough,
|
||
# v0.24.0, caps 350 W, all x16): verify-full 9/9 · verify-stress needles
|
||
# clean to 240K · soak-continuous PASS (0 err / 0 silent / <200 MiB) ·
|
||
# bench n=5 CV 4.2%/3.0%. KV pool 1.77M tok / 6.77× @262K (vs 2-card
|
||
# vllm/dual 622K/2.37×) — the 4-card win is concurrency, not single-stream
|
||
# speed (decode ~flat, TP=4 all-reduce over PCIe). ⚠ ceiling margin thin
|
||
# (851 MB @240K on driver 595, #149). QUALITY confirmed 2026-07-06 (#584
|
||
# follow-up): full 8-pack thinking-off 108/150 — within ±1 of the carried
|
||
# vllm/dual proxy (109), so TP-invariance is now MEASURED, not assumed.
|
||
narr_tps: 74.76
|
||
code_tps: 90.83
|
||
ttft_ms: 141
|
||
prefill_tps: { 10k: 1288, 90k: 1175 }
|
||
quality_8pk: "108/150"
|
||
quality_env: { harness: "benchlocal-cli @ club-3090 087d094 (sandboxes rebuilt 2026-07-06)" }
|
||
ctx_validated: { tokens: 240633, niah: "clean@240K" }
|
||
date: 2026-07-05
|
||
engine_pin: "vllm/vllm-openai:v0.24.0"
|
||
rig: "4x3090-pcie"
|
||
power_cap_w: [350, 350, 350, 350]
|
||
source: "https://github.com/noonghunna/club-3090/issues/584"
|
||
submitted_by: "ryanmpelletier"
|
||
tier: submitted
|
||
|
||
vllm/qwen-27b-multi-max:
|
||
# Cross-rig submission only — 4-card; our 2-card reference rig can't run it.
|
||
# ⚠ STALE: the only 4-card data (@Whamp #446) was measured on the OLD int8-PTH
|
||
# KV, before #595 flipped multi-max to fp8/e4m3 (and on a pre-v0.24.0 engine).
|
||
# Kept stale-flagged so the catalog shows something with the caveat — REPLACE
|
||
# with the pending 4-card fp8 re-test (the #584 request) when it lands.
|
||
submissions:
|
||
4x3090-pcie:
|
||
# #446 (@Whamp, 4× 3090 PCIe x4/x16/x8/x16, aikitoria P2P kernel, 230 W):
|
||
# verify-full + verify-stress 7/7 + soak-continuous PASS, 85.09/102.43.
|
||
# int8-PTH KV (pre-#595 fp8 flip) + pre-v0.24.0 engine → doubly STALE.
|
||
narr_tps: 85.09
|
||
code_tps: 102.43
|
||
ctx_validated: { tokens: 240633, niah: "clean@240K" }
|
||
date: 2026-06-19
|
||
engine_pin: "vllm/vllm-openai:v0.22.0"
|
||
rig: "4x3090-pcie"
|
||
power_cap_w: [230, 230, 230, 230]
|
||
source: "https://github.com/noonghunna/club-3090/issues/446"
|
||
submitted_by: "Whamp"
|
||
tier: submitted
|
||
stale: true
|
||
|
||
# ───────────────────────────────────────────────────────────────────────────
|
||
# KNOWN GAPS — wave-2 dispositions (2026-07-04): each slug below has NO row
|
||
# on purpose; the disposition says what unlocks one. Do NOT guess numbers in.
|
||
# vllm/minimal — only ~32/~33 approximations exist (never a protocol bench);
|
||
# needs a proper gate run. No row until then.
|
||
# vllm/gemma-12b-dual-bf16-mtp · vllm/gemma-12b-single-int8-mtp —
|
||
# conflicting-era rows (2026-06-04) with ambiguous decode-vs-wall class,
|
||
# AND both slugs re-pinned in the v0.24.0 gemma consolidation (#538/#539)
|
||
# → any seed would be born-stale AND class-uncertain. Re-gate owed.
|
||
# beellama/gemma-dflash — no direct BENCHMARKS row under the slug (numbers
|
||
# live in promotion-era notes). PRIORITY re-gate: this is the single-card
|
||
# gemma DEFAULT — the catalog's recommended path with no bar is the worst
|
||
# gap on this list.
|
||
# llamacpp/deckard40B-dual-mtp — 41.6 tok/s MTP exists only in registry
|
||
# notes; no BENCHMARKS row. Needs a gate run.
|
||
# ───────────────────────────────────────────────────────────────────────────
|