catalog-baselines slice 1: baselines.yml + registry-emit join + guards; catalog drops the BENCHMARKS scrape
The catalog's measured columns (TPS / 8pk) now come from ONE productized source: scripts/lib/profiles/baselines.yml — the shipped, PR-reviewed 'bar' (accepted display projection of a validated gate run, with pin/rig/ power provenance) — joined per-slug into registry-emit --json with an emit-computed staleness verdict. Consumers never read baselines.yml or BENCHMARKS.md directly (BENCHMARKS stays the public human/cross-rig ledger). Design: the catalog-baselines note (2026-07-02, §2/§5 slice 1). - baselines.yml SEED WAVE 1: 10 rows with airtight traceability only (rebench tags on disk: agents-a1 golden specimen + gemma-31b-dual; unambiguous decode-class BENCHMARKS rows for the rest). 4 rows are HONESTLY born-stale with documented reasons (llamacpp rolling-tag-era x2, beellama pre-#296 image, 35B v0.22.0) — the guard's demo cases. 9 slugs still owed rows are listed as wave-2 gaps, no guessed numbers. - registry-emit join: per-variant 'baseline' field; current pin resolved the way launchers actually resolve it (engine-profile install.spec, compose-image-default fallback for ik/llama.cpp) → 'stale' = measured-pin != current-pin, null when undeterminable. - test-baselines.sh: schema + slug-membership + ctx-parity (compose ctx default == registry max_ctx, functional slugs) RED; pin-staleness WARN-only (pin bumps must not block on immediate re-bench — the debt stays visible). test-registry-json gains the 'baseline' contract key. - c3: enrich_measurements = pure in-memory map off the joined field — deletes BOTH the per-slug --explain fan-out (~4s/slug, the #439 option-3 leg) and the BENCHMARKS.md scrape from the catalog path (Explain modal + cross-rig explorer keep their readers). Stale rows render a † on the TPS cell + a status-line legend; full badge/overlay treatment is slice 2. Measurement gains source='baseline' + stale. - results/baselines/README: the three-store relationship (regression corpus vs measurement records vs display bar) + the no-drift rule the slice-2 induction tool enforces. Verified: guard test green (10 rows, 4 stale-warned); live join emits 57 variants / 10 baselines; live c3 catalog shows all 10 with daggers on exactly the stale 4 (A1 serving during the check: 154/154 - 105/150); c3 suite 764/764; full scripts gate green serial. (First gate pass tripped classifier/dedup by running CONCURRENTLY with the c3 suite — .pull-captures pollution class; serial rerun clean.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
@@ -9,6 +9,15 @@ This is distinct from the runtime **measurement-record** TPS corpus
|
||||
(`scripts/lib/profiles/measurement_record.py`) — that tracks throughput; this tracks
|
||||
*behavioral quality* (ToolCall / InstructFollow / StructOutput / DataExtract / …).
|
||||
|
||||
It is ALSO distinct from the **shipped catalog baseline**
|
||||
(`scripts/lib/profiles/baselines.yml`) — the slug-keyed *display* bar (decode TPS ·
|
||||
8-pack headline · ctx-validated · pin provenance) joined into `registry-emit --json`
|
||||
for c3/switch. The three stores answer different questions (regression-diff vs raw
|
||||
corpus vs accepted display row) but must not drift: the induction tool
|
||||
(`catalog-baseline.sh`, slice 2) refreshes `baselines.yml`'s `quality_8pk` from the
|
||||
same capture that lands here, so a slug's headline never disagrees with its
|
||||
regression baseline.
|
||||
|
||||
## What lives here
|
||||
|
||||
One JSON per `(registry-slug, thinking-mode)`, named:
|
||||
|
||||
Reference in New Issue
Block a user