catalog-baselines slice 1: baselines.yml + registry-emit join + guards; catalog drops the BENCHMARKS scrape

The catalog's measured columns (TPS / 8pk) now come from ONE productized
source: scripts/lib/profiles/baselines.yml — the shipped, PR-reviewed
'bar' (accepted display projection of a validated gate run, with pin/rig/
power provenance) — joined per-slug into registry-emit --json with an
emit-computed staleness verdict. Consumers never read baselines.yml or
BENCHMARKS.md directly (BENCHMARKS stays the public human/cross-rig
ledger). Design: the catalog-baselines note (2026-07-02, §2/§5 slice 1).

- baselines.yml SEED WAVE 1: 10 rows with airtight traceability only
  (rebench tags on disk: agents-a1 golden specimen + gemma-31b-dual;
  unambiguous decode-class BENCHMARKS rows for the rest). 4 rows are
  HONESTLY born-stale with documented reasons (llamacpp rolling-tag-era
  x2, beellama pre-#296 image, 35B v0.22.0) — the guard's demo cases.
  9 slugs still owed rows are listed as wave-2 gaps, no guessed numbers.
- registry-emit join: per-variant 'baseline' field; current pin resolved
  the way launchers actually resolve it (engine-profile install.spec,
  compose-image-default fallback for ik/llama.cpp) → 'stale' =
  measured-pin != current-pin, null when undeterminable.
- test-baselines.sh: schema + slug-membership + ctx-parity (compose ctx
  default == registry max_ctx, functional slugs) RED; pin-staleness
  WARN-only (pin bumps must not block on immediate re-bench — the debt
  stays visible). test-registry-json gains the 'baseline' contract key.
- c3: enrich_measurements = pure in-memory map off the joined field —
  deletes BOTH the per-slug --explain fan-out (~4s/slug, the #439
  option-3 leg) and the BENCHMARKS.md scrape from the catalog path
  (Explain modal + cross-rig explorer keep their readers). Stale rows
  render a † on the TPS cell + a status-line legend; full badge/overlay
  treatment is slice 2. Measurement gains source='baseline' + stale.
- results/baselines/README: the three-store relationship (regression
  corpus vs measurement records vs display bar) + the no-drift rule the
  slice-2 induction tool enforces.

Verified: guard test green (10 rows, 4 stale-warned); live join emits
57 variants / 10 baselines; live c3 catalog shows all 10 with daggers on
exactly the stale 4 (A1 serving during the check: 154/154 - 105/150);
c3 suite 764/764; full scripts gate green serial. (First gate pass
tripped classifier/dedup by running CONCURRENTLY with the c3 suite —
.pull-captures pollution class; serial rerun clean.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
noonghunna
2026-07-04 15:25:36 +00:00
parent c2c06e47a7
commit 75b29556c1
10 changed files with 513 additions and 50 deletions

View File

@@ -9,6 +9,15 @@ This is distinct from the runtime **measurement-record** TPS corpus
(`scripts/lib/profiles/measurement_record.py`) — that tracks throughput; this tracks
*behavioral quality* (ToolCall / InstructFollow / StructOutput / DataExtract / …).
It is ALSO distinct from the **shipped catalog baseline**
(`scripts/lib/profiles/baselines.yml`) — the slug-keyed *display* bar (decode TPS ·
8-pack headline · ctx-validated · pin provenance) joined into `registry-emit --json`
for c3/switch. The three stores answer different questions (regression-diff vs raw
corpus vs accepted display row) but must not drift: the induction tool
(`catalog-baseline.sh`, slice 2) refreshes `baselines.yml`'s `quality_8pk` from the
same capture that lands here, so a slug's headline never disagrees with its
regression baseline.
## What lives here
One JSON per `(registry-slug, thinking-mode)`, named: