alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions 38 Packages Projects Releases Wiki Activity

catalog-baseline --from-bundle: ingest volunteer bundles (slice 3b)

Browse Source 
Bundle mode inducts a volunteer's rebench bundle into the slug's
submissions: map with provenance FROM THE BUNDLE — rig/power from rig.txt,
engine pin from container-config.json Config.Image — never from this rig
(nvidia-smi / resolve_variant_pin would stamp our fingerprint onto foreign
numbers). --source and --submitted-by are required, no $USER default.

- splice safety both directions: a primary re-induction preserves an
  existing submissions map; bundle mode never touches the primary row
- one row per rig_class (newest replaces; history stays in git)
- multi-tag bundles refuse without --from-tag selection
- test-catalog-baseline: synthetic-bundle fixture covering refusals,
  bundle-derived provenance, add/replace, submission-only entries,
  splice preservation

Dogfood: first external row — guybrush01's #571 fp8w bundle lands as
vllm/qwen-27b-dual-max submissions[2x5090-pcie] (134.52/165.11 decode,
NIAH-clean 240,635 tok, v0.24.0, tier: submitted, TPS-only pending his
quality run).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
This commit is contained in:
noonghunna
2026-07-05 09:35:19 +00:00
parent 3841582728
commit 3970549ff7
3 changed files with 367 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

289
scripts/catalog-baseline.sh
View File

@@ -3,9 +3,17 @@
# baseline row in scripts/lib/profiles/baselines.yml (catalog-baselines §2.3).
#
# bash scripts/catalog-baseline.sh <slug> --from-tag <rebench-tag> [options]
# bash scripts/catalog-baseline.sh <slug> --from-bundle <tgz|dir> \
# --source <disc/PR link> --submitted-by <handle> [options]
#
# One mechanism, three entry points: ⑤ Promote (c3 producer lane) · maintainer
# CLI · the rebench-full completion prompt (re-validation after pin bumps).
# PLUS the slice-3 cross-rig path: --from-bundle ingests a VOLUNTEER's rebench
# bundle into the slug's `submissions:` map (keyed by rig_class, tier:
# submitted). In bundle mode provenance comes FROM THE BUNDLE — rig/power from
# rig.txt, engine pin from container-config.json — NEVER from this rig
# (nvidia-smi / resolve_variant_pin would stamp OUR fingerprint onto foreign
# numbers). --source and --submitted-by are REQUIRED (no $USER default).
#
# It VALIDATES the tag dir covers the gates (verify-full pass · bench n>=3,
# n<5 warned · quality run present), extracts the display projection (decode
@@ -20,16 +28,24 @@
# saved results".
#
# Options:
# --from-tag <tag> rebench tag under results/rebench/ (required)
# --from-tag <tag> rebench tag under results/rebench/ (required unless
# --from-bundle; WITH --from-bundle it selects the tag
# when the bundle carries more than one)
# --tag-dir <dir> explicit tag dir (overrides --from-tag lookup; for
# tags living in another checkout/worktree)
# --from-bundle <p> a volunteer's rebench bundle (.tgz or an extracted
# dir) → ingest into the slug's submissions: map
# --source <url> (bundle mode, REQUIRED) provenance link — the
# discussion comment / PR the bundle was submitted in
# --engine-pin <spec> override the measured pin (default: the slug's CURRENT
# resolved pin — correct when inducting right after the
# gate ran on it)
# resolved pin; bundle mode: container-config.json)
# --rig <class> hardware fingerprint class (default: derived from
# nvidia-smi, e.g. 2x3090-pcie)
# --power-cap-w <csv> per-card caps, e.g. "370,420" (default: nvidia-smi)
# --submitted-by <who> provenance (default: $USER)
# nvidia-smi, e.g. 2x3090-pcie; bundle mode: rig.txt.
# In bundle mode this is also the submissions map key)
# --power-cap-w <csv> per-card caps, e.g. "370,420" (default: nvidia-smi;
# bundle mode: rig.txt)
# --submitted-by <who> provenance (default: $USER; bundle mode: REQUIRED —
# the volunteer's handle, never defaulted)
# --tps-only induct without a quality run (WARNS; the 8-pack gate
# is normally required)
# --dry-run print the diff, do not write
@@ -47,12 +63,15 @@ SLUG="${1:-}"
}
shift
TAG="" TAG_DIR="" ENGINE_PIN="" RIG="" POWER="" SUBMITTED_BY="${USER:-unknown}"
TPS_ONLY=0 DRY_RUN=0 BASELINES_FILE="scripts/lib/profiles/baselines.yml"
TAG="" TAG_DIR="" BUNDLE="" SOURCE="" ENGINE_PIN="" RIG="" POWER=""
SUBMITTED_BY="" TPS_ONLY=0 DRY_RUN=0
BASELINES_FILE="scripts/lib/profiles/baselines.yml"
while [[ $# -gt 0 ]]; do
case "$1" in
--from-tag) TAG="$2"; shift 2 ;;
--tag-dir) TAG_DIR="$2"; shift 2 ;;
--from-bundle) BUNDLE="$2"; shift 2 ;;
--source) SOURCE="$2"; shift 2 ;;
--engine-pin) ENGINE_PIN="$2"; shift 2 ;;
--rig) RIG="$2"; shift 2 ;;
--power-cap-w) POWER="$2"; shift 2 ;;
@@ -63,12 +82,46 @@ while [[ $# -gt 0 ]]; do
*) echo "[catalog-baseline] unknown option: $1" >&2; exit 2 ;;
esac
done
[[ -n "$TAG" || -n "$TAG_DIR" ]] || { echo "[catalog-baseline] --from-tag (or --tag-dir) is required" >&2; exit 2; }
TAG_DIR="${TAG_DIR:-$ROOT_DIR/results/rebench/$TAG}"
TAG="${TAG:-$(basename "$TAG_DIR")}"
FROM_BUNDLE=0
if [[ -n "$BUNDLE" ]]; then
# ── bundle mode (slice 3): extract + locate the tag dir INSIDE the bundle ──
FROM_BUNDLE=1
[[ -n "$SOURCE" ]] || { echo "[catalog-baseline] bundle mode: --source <disc/PR link> is required" >&2; exit 2; }
[[ -n "$SUBMITTED_BY" ]] || { echo "[catalog-baseline] bundle mode: --submitted-by <handle> is required (never defaulted to \$USER)" >&2; exit 2; }
[[ -z "$TAG_DIR" ]] || { echo "[catalog-baseline] --tag-dir and --from-bundle are mutually exclusive" >&2; exit 2; }
if [[ -d "$BUNDLE" ]]; then
BUNDLE_DIR="$BUNDLE"
else
[[ -f "$BUNDLE" ]] || { echo "[catalog-baseline] bundle not found: $BUNDLE" >&2; exit 2; }
BUNDLE_DIR="$(mktemp -d)"
trap 'rm -rf "$BUNDLE_DIR"' EXIT
tar xzf "$BUNDLE" -C "$BUNDLE_DIR"
fi
# a tag dir is any dir carrying _internal.json (the rebench-report record)
mapfile -t _tag_dirs < <(find "$BUNDLE_DIR" -name _internal.json -printf '%h\n' | sort)
if [[ ${#_tag_dirs[@]} -eq 0 ]]; then
echo "[catalog-baseline] no tag dir (_internal.json) inside the bundle" >&2; exit 1
elif [[ ${#_tag_dirs[@]} -eq 1 ]]; then
TAG_DIR="${_tag_dirs[0]}"
else
[[ -n "$TAG" ]] || { echo "[catalog-baseline] bundle carries ${#_tag_dirs[@]} tags — select one with --from-tag:" >&2
printf ' %s\n' "${_tag_dirs[@]##*/}" >&2; exit 2; }
TAG_DIR=""
for d in "${_tag_dirs[@]}"; do [[ "$(basename "$d")" == "$TAG" ]] && TAG_DIR="$d"; done
[[ -n "$TAG_DIR" ]] || { echo "[catalog-baseline] tag $TAG not in the bundle" >&2; exit 2; }
fi
TAG="${TAG:-$(basename "$TAG_DIR")}"
else
[[ -n "$TAG" || -n "$TAG_DIR" ]] || { echo "[catalog-baseline] --from-tag (or --tag-dir / --from-bundle) is required" >&2; exit 2; }
SUBMITTED_BY="${SUBMITTED_BY:-${USER:-unknown}}"
TAG_DIR="${TAG_DIR:-$ROOT_DIR/results/rebench/$TAG}"
TAG="${TAG:-$(basename "$TAG_DIR")}"
fi
SLUG="$SLUG" TAG="$TAG" TAG_DIR="$TAG_DIR" ENGINE_PIN="$ENGINE_PIN" RIG="$RIG" \
POWER="$POWER" SUBMITTED_BY="$SUBMITTED_BY" TPS_ONLY="$TPS_ONLY" DRY_RUN="$DRY_RUN" \
FROM_BUNDLE="$FROM_BUNDLE" SOURCE="$SOURCE" \
BASELINES_FILE="$BASELINES_FILE" \
python3 - <<'PY'
import difflib
@@ -204,39 +257,74 @@ if bench_log:
"warm-vs-cold or config drift; investigate before trusting the depth curve")
# ── provenance ────────────────────────────────────────────────────────────────
engine_pin = os.environ["ENGINE_PIN"]
if not engine_pin:
try:
exports = resolve_variant_pin(load_profiles(), slug)
if "VLLM_NIGHTLY_SHA" not in exports:
engine_pin = next(iter(exports.values()))
except ProfileError:
pass
if not engine_pin:
# compose-image default (ik/llama.cpp class) — same rule as the emit join
txt = (Path(COMPOSE_REGISTRY[slug]["compose_path"])).read_text(errors="replace")
m = re.search(r"^\s*image:\s*[\"']?(?:\$\{[A-Z_0-9]+:-)?([^\s}\"']+)\}?", txt, re.M)
engine_pin = m.group(1) if m else ""
if not engine_pin:
die("could not resolve the engine pin — pass --engine-pin explicitly")
# Two regimes: local induction reads THIS rig (nvidia-smi + resolved pin);
# bundle mode reads THE BUNDLE (rig.txt + container-config.json) and NEVER
# falls back to local state — that would stamp our fingerprint onto foreign
# numbers (the slice-3 trust boundary).
from_bundle = os.environ["FROM_BUNDLE"] == "1"
source = os.environ["SOURCE"]
engine_pin = os.environ["ENGINE_PIN"]
rig = os.environ["RIG"]
power = os.environ["POWER"]
if not rig or not power:
try:
out = subprocess.run(
["nvidia-smi", "--query-gpu=name,power.limit", "--format=csv,noheader,nounits"],
capture_output=True, text=True, timeout=10,
).stdout.strip().splitlines()
names = [l.split(",")[0].strip() for l in out]
caps = [str(round(float(l.split(",")[1]))) for l in out]
if not rig and names:
short = re.sub(r"NVIDIA GeForce RTX\s*", "", names[0]).strip().replace(" ", "").lower()
rig = f"{len(names)}x{short}-pcie"
if not power and caps:
power = ",".join(caps)
except Exception:
pass
if from_bundle:
if not engine_pin:
cc = tag_dir / "container-config.json"
if cc.is_file():
try:
data = json.loads(cc.read_text(errors="replace"))
engine_pin = ((data[0].get("Config") or {}).get("Image") or "").strip()
except (ValueError, IndexError, AttributeError, TypeError):
pass
if not engine_pin:
die("bundle mode: no readable Config.Image in container-config.json — pass --engine-pin")
rig_txt = (tag_dir / "rig.txt").read_text(errors="replace") if (tag_dir / "rig.txt").is_file() else ""
names = re.findall(r"^GPU \d+: (.+?) \(UUID", rig_txt, re.M)
if not rig:
if not names:
die("bundle mode: no GPU lines in rig.txt — pass --rig")
short = re.sub(r"NVIDIA\s+(GeForce\s+)?(RTX\s+)?", "", names[0]).strip().replace(" ", "").lower()
rig = f"{len(names)}x{short}-pcie"
if not power:
caps = re.findall(r"^power_cap_w:\s*([\d.]+)", rig_txt, re.M)
if not caps:
die("bundle mode: no power_cap_w in rig.txt — pass --power-cap-w")
vals = [str(round(float(c))) for c in caps]
if len(vals) == 1 and len(names) > 1:
vals = vals * len(names) # one global cap line → replicate per card
power = ",".join(vals)
else:
if not engine_pin:
try:
exports = resolve_variant_pin(load_profiles(), slug)
if "VLLM_NIGHTLY_SHA" not in exports:
engine_pin = next(iter(exports.values()))
except ProfileError:
pass
if not engine_pin:
# compose-image default (ik/llama.cpp class) — same rule as the emit join
txt = (Path(COMPOSE_REGISTRY[slug]["compose_path"])).read_text(errors="replace")
m = re.search(r"^\s*image:\s*[\"']?(?:\$\{[A-Z_0-9]+:-)?([^\s}\"']+)\}?", txt, re.M)
engine_pin = m.group(1) if m else ""
if not engine_pin:
die("could not resolve the engine pin — pass --engine-pin explicitly")
if not rig or not power:
try:
out = subprocess.run(
["nvidia-smi", "--query-gpu=name,power.limit", "--format=csv,noheader,nounits"],
capture_output=True, text=True, timeout=10,
).stdout.strip().splitlines()
names = [l.split(",")[0].strip() for l in out]
caps = [str(round(float(l.split(",")[1]))) for l in out]
if not rig and names:
short = re.sub(r"NVIDIA GeForce RTX\s*", "", names[0]).strip().replace(" ", "").lower()
rig = f"{len(names)}x{short}-pcie"
if not power and caps:
power = ",".join(caps)
except Exception:
pass
if not rig:
die("could not derive --rig (nvidia-smi unavailable) — pass it explicitly")
if not power:
@@ -249,34 +337,45 @@ row_date = date.fromtimestamp(
).isoformat()
# ── build the row text (matches the file's hand-written shape) ───────────────
lines = [f" {slug}:"]
lines.append(f" # inducted by catalog-baseline.sh from rebench tag {tag} ({datetime.now().date().isoformat()});")
lines.append(f" # evidence: verify-full pass · bench n={n_runs or '?'} · "
+ ("quality both arms" if (q_off and q_on) else ("quality one arm" if (q_off or q_on) else "TPS-ONLY (no quality)"))
+ (" · NIAH ladder" if ctx_tokens else ""))
lines.append(f" narr_tps: {round(float(narr['decode_tps_mean']), 2)}")
lines.append(f" code_tps: {round(float(code['decode_tps_mean']), 2)}")
# The SAME fields serve both shapes; only indent + head + tier/source differ:
# local: " <slug>:" + 4-space fields + tier: local
# bundle: " <rig>:" + 8-space fields + source: + tier: submitted
# (spliced under the slug's " submissions:" map)
tool = "catalog-baseline.sh --from-bundle" if from_bundle else "catalog-baseline.sh"
fields = [f"# inducted by {tool} from rebench tag {tag} ({datetime.now().date().isoformat()});"]
fields.append(f"# evidence: verify-full pass · bench n={n_runs or '?'} · "
+ ("quality both arms" if (q_off and q_on) else ("quality one arm" if (q_off or q_on) else "TPS-ONLY (no quality)"))
+ (" · NIAH ladder" if ctx_tokens else ""))
fields.append(f"narr_tps: {round(float(narr['decode_tps_mean']), 2)}")
fields.append(f"code_tps: {round(float(code['decode_tps_mean']), 2)}")
if ttft is not None:
lines.append(f" ttft_ms: {round(float(ttft))}")
fields.append(f"ttft_ms: {round(float(ttft))}")
if prefill_by_ctx:
inner = ", ".join(
f"{int(k) // 1000}k: {v:.0f}"
for k, v in sorted(prefill_by_ctx.items(), key=lambda x: int(x[0]))
)
lines.append(f" prefill_tps: {{ {inner} }}")
fields.append(f"prefill_tps: {{ {inner} }}")
if q_off:
lines.append(f' quality_8pk: "{q_off}"')
fields.append(f'quality_8pk: "{q_off}"')
if q_on:
lines.append(f' quality_8pk_think_on: "{q_on}"')
fields.append(f'quality_8pk_think_on: "{q_on}"')
if ctx_tokens:
lines.append(f' ctx_validated: {{ tokens: {ctx_tokens}, niah: "{niah}" }}')
lines.append(f" date: {row_date}")
lines.append(f' engine_pin: "{engine_pin}"')
lines.append(f' rig: "{rig}"')
lines.append(f" power_cap_w: {power_list}")
lines.append(f' source_tag: "{tag}"')
lines.append(f' submitted_by: "{os.environ["SUBMITTED_BY"]}"')
row_text = "\n".join(lines) + "\n"
fields.append(f'ctx_validated: {{ tokens: {ctx_tokens}, niah: "{niah}" }}')
fields.append(f"date: {row_date}")
fields.append(f'engine_pin: "{engine_pin}"')
fields.append(f'rig: "{rig}"')
fields.append(f"power_cap_w: {power_list}")
if from_bundle:
fields.append(f'source: "{source}"')
fields.append(f'source_tag: "{tag}"')
fields.append(f'submitted_by: "{os.environ["SUBMITTED_BY"]}"')
fields.append("tier: submitted" if from_bundle else "tier: local")
if from_bundle:
row_text = f" {rig}:\n" + "".join(f" {f}\n" for f in fields)
else:
row_text = f" {slug}:\n" + "".join(f" {f}\n" for f in fields)
# ── upsert (textual splice — comments elsewhere in the file preserved) ────────
bl_path = Path(os.environ["BASELINES_FILE"])
@@ -284,22 +383,74 @@ old = bl_path.read_text()
block_re = re.compile(
rf"^ {re.escape(slug)}:\n(?:^(?: .*|\s*)\n)*?(?=^ \S|^#|^\S|\Z)", re.M
)
if block_re.search(old):
new = block_re.sub(row_text + "\n", old, count=1)
action = "replaced"
# an existing submissions: sub-map inside a slug block (6+-space content lines)
SUBMAP_RE = re.compile(
r"^ submissions:\n(?:^(?: .*|\s*)\n)*?(?=^ \S|^ \S|^#|^\S|\Z)", re.M
)
m_block = block_re.search(old)
if from_bundle:
# bundle mode touches ONLY the submissions map — the primary row (if any)
# is the local bar and stays byte-identical.
if m_block:
block = m_block.group(0)
body = block.rstrip("\n") + "\n" # block content
trail = block[len(body):] # blank-line row separator(s)
sub_re = re.compile(
rf"^ {re.escape(rig)}:\n(?:^(?: .*|\s*)\n)*?(?=^ \S|^ \S|^ \S|^#|^\S|\Z)",
re.M,
)
m_hdr = re.search(r"^ submissions:\n", body, re.M)
if m_hdr:
m_sub = sub_re.search(body)
if m_sub:
# ONE row per rig_class — newest replaces; older stays in git
new_body = body[:m_sub.start()] + row_text + body[m_sub.end():]
action = f"replaced submission [{rig}] in"
else:
new_body = body[:m_hdr.end()] + row_text + body[m_hdr.end():]
action = f"added submission [{rig}] to"
else:
new_body = body + " submissions:\n" + row_text
action = f"added submissions map [{rig}] to"
new = old[:m_block.start()] + new_body + trail + old[m_block.end():]
else:
# submission-only entry: a slug measured on hardware we don't have
entry_text = (
f" {slug}:\n"
" # cross-rig submissions only — no local baseline row yet\n"
" submissions:\n" + row_text
)
m = re.search(r"^# ─+\n# (?:KNOWN GAPS|SEED WAVE 2)", old, re.M)
ins = m.start() if m else len(old)
new = old[:ins] + entry_text + "\n" + old[ins:]
action = f"added submission-only entry [{rig}] as"
else:
# append before the top-level gap-list footer comment block (or at EOF)
m = re.search(r"^# ─+\n# (?:KNOWN GAPS|SEED WAVE 2)", old, re.M)
ins = m.start() if m else len(old)
new = old[:ins] + row_text + "\n" + old[ins:]
action = "added"
if m_block:
# preserve an existing submissions: sub-map across the primary-row
# replace (the block regex would otherwise swallow it)
m_sub = SUBMAP_RE.search(m_block.group(0))
keep = (m_sub.group(0).rstrip("\n") + "\n") if m_sub else ""
new = old[:m_block.start()] + row_text + keep + "\n" + old[m_block.end():]
action = "replaced"
else:
# append before the top-level gap-list footer comment block (or at EOF)
m = re.search(r"^# ─+\n# (?:KNOWN GAPS|SEED WAVE 2)", old, re.M)
ins = m.start() if m else len(old)
new = old[:ins] + row_text + "\n" + old[ins:]
action = "added"
# self-check: the produced file must still parse + the row must round-trip
import yaml # noqa: E402
parsed = yaml.safe_load(new)
got = (parsed.get("baselines") or {}).get(slug)
assert got and isinstance(got.get("narr_tps"), (int, float)), "upsert self-check failed"
if from_bundle:
sub_got = (got or {}).get("submissions", {}).get(rig)
assert sub_got and isinstance(sub_got.get("narr_tps"), (int, float)), "upsert self-check failed (submission)"
assert sub_got.get("tier") == "submitted" and sub_got.get("source"), "upsert self-check failed (provenance)"
else:
assert got and isinstance(got.get("narr_tps"), (int, float)), "upsert self-check failed"
diff = "".join(
difflib.unified_diff(

20
scripts/lib/profiles/baselines.yml
View File

@@ -311,6 +311,26 @@ baselines:
submitted_by: "noonghunna"
tier: local
vllm/qwen-27b-dual-max:
# cross-rig submissions only — no local baseline row yet
submissions:
2x5090-pcie:
# inducted by catalog-baseline.sh --from-bundle from rebench tag 246-ab-fp8w (2026-07-05);
# evidence: verify-full pass · bench n=5 · TPS-ONLY (no quality) · NIAH ladder
narr_tps: 134.52
code_tps: 165.11
ttft_ms: 56
prefill_tps: { 10k: 3693, 90k: 1906 }
ctx_validated: { tokens: 240635, niah: "clean@241K" }
date: 2026-07-05
engine_pin: "vllm/vllm-openai:v0.24.0"
rig: "2x5090-pcie"
power_cap_w: [575, 575]
source: "https://github.com/noonghunna/club-3090/discussions/571#discussioncomment-17536412"
source_tag: "246-ab-fp8w"
submitted_by: "guybrush01"
tier: submitted
# ───────────────────────────────────────────────────────────────────────────
# KNOWN GAPS — wave-2 dispositions (2026-07-04): each slug below has NO row
# on purpose; the disposition says what unlocks one. Do NOT guess numbers in.

132
scripts/tests/test-catalog-baseline.sh
View File

@@ -18,17 +18,20 @@ mkdir -p "$TAGD"
BL="$TMP/baselines.yml"
cp scripts/lib/profiles/baselines.yml "$BL"
REAL_SUM_BEFORE="$(sha256sum scripts/lib/profiles/baselines.yml | cut -d' ' -f1)"
# guarantee the ADD path below: strip any real vllm/dual row from the copy
# (the real file seeds one since wave-2)
# guarantee the ADD paths below: strip any real vllm/dual row (the real file
# seeds one since wave-2) AND any real vllm/qwen-27b-dual-max entry (the real
# file carries a submission-only entry since slice 3 — section 5's
# submission-only case needs the slug absent)
python3 - "$BL" <<'PY'
import re
import sys
p = sys.argv[1]
old = open(p).read()
new = re.sub(r"^ vllm/dual:\n(?:^(?: .*|\s*)\n)*?(?=^ \S|^#|^\S|\Z)",
"", old, count=1, flags=re.M)
open(p, "w").write(new)
for slug in ("vllm/dual", "vllm/qwen-27b-dual-max"):
old = re.sub(rf"^ {re.escape(slug)}:\n(?:^(?: .*|\s*)\n)*?(?=^ \S|^#|^\S|\Z)",
"", old, count=1, flags=re.M)
open(p, "w").write(old)
PY
fail() { echo "FAIL: $1" >&2; exit 1; }
@@ -148,6 +151,125 @@ if bash scripts/catalog-baseline.sh vllm/__nope__ "${ARGS[@]}" --dry-run >/dev/n
fail "unknown slug must refuse"
fi
# ── 5. bundle mode (slice 3): provenance FROM the bundle → submissions map ───
# Synthetic volunteer bundle: the same gate artifacts + rig.txt +
# container-config.json (a 2×5090 rig, one global power line, a pin that is
# NOT this rig's resolved pin — so any local-fallback leak fails loudly).
BSRC="$TMP/bundle-src/results/rebench/synth-fp8w"
mkdir -p "$BSRC"
cp "$TAGD"/verify-full.log "$TAGD"/bench.log "$TAGD"/_internal.json \
"$TAGD"/verify-stress.log "$BSRC"/
cat > "$BSRC/rig.txt" <<'EOF'
hostname: volunteer-box
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-aaa)
GPU 1: NVIDIA GeForce RTX 5090 (UUID: GPU-bbb)
power_cap_w: 575.00
EOF
cat > "$BSRC/container-config.json" <<'EOF'
[{"Config": {"Image": "vllm/test-pin:v9"}}]
EOF
tar czf "$TMP/bundle.tgz" -C "$TMP/bundle-src" results
BARGS=(--from-bundle "$TMP/bundle.tgz" --source "https://example.test/disc#42"
--submitted-by volunteer1 --tps-only --baselines-file "$BL")
# refusals: --source and --submitted-by are the trust boundary — REQUIRED
if bash scripts/catalog-baseline.sh vllm/dual --from-bundle "$TMP/bundle.tgz" \
--submitted-by v --tps-only --baselines-file "$BL" --dry-run >/dev/null 2>&1; then
fail "bundle mode without --source must refuse"
fi
if bash scripts/catalog-baseline.sh vllm/dual --from-bundle "$TMP/bundle.tgz" \
--source x --tps-only --baselines-file "$BL" --dry-run >/dev/null 2>&1; then
fail "bundle mode without --submitted-by must refuse (no \$USER default)"
fi
# dry-run: every provenance field derived FROM the bundle, none from this rig
sum_before="$(sha256sum "$BL" | cut -d' ' -f1)"
out="$(bash scripts/catalog-baseline.sh vllm/dual "${BARGS[@]}" --dry-run 2>&1)"
grep -q 'rig: "2x5090-pcie"' <<<"$out" || fail "rig not derived from rig.txt: $out"
grep -q 'power_cap_w: \[575, 575\]' <<<"$out" || fail "one global cap line not replicated per card"
grep -q 'engine_pin: "vllm/test-pin:v9"' <<<"$out" || fail "pin not read from container-config.json"
grep -q 'tier: submitted' <<<"$out" || fail "tier: submitted missing"
[[ "$(sha256sum "$BL" | cut -d' ' -f1)" == "$sum_before" ]] || fail "bundle dry-run WROTE the file"
# write: the submissions map lands; the primary row is untouched
out="$(bash scripts/catalog-baseline.sh vllm/dual "${BARGS[@]}" 2>&1)" || fail "bundle add failed"
grep -q "added submissions map" <<<"$out" || fail "first bundle induction did not add the map"
python3 - "$BL" <<'PY'
import sys
import yaml
d = yaml.safe_load(open(sys.argv[1]))
row = d["baselines"]["vllm/dual"]
assert row["narr_tps"] == 153.9 and row["engine_pin"] == "test/pin:v2", \
"primary row disturbed by a bundle induction"
s = row["submissions"]["2x5090-pcie"]
assert s["narr_tps"] == 153.9 and s["tier"] == "submitted"
assert s["source"] == "https://example.test/disc#42" and s["submitted_by"] == "volunteer1"
assert s["power_cap_w"] == [575, 575] and s["engine_pin"] == "vllm/test-pin:v9"
assert s["rig"] == "2x5090-pcie"
PY
# same rig_class re-induction REPLACES (one row per rig_class, newest wins)
out="$(bash scripts/catalog-baseline.sh vllm/dual "${BARGS[@]}" 2>&1)" || fail "bundle replace failed"
grep -q "replaced submission" <<<"$out" || fail "same-rig_class re-induction did not replace"
python3 - "$BL" <<'PY'
import sys
import yaml
d = yaml.safe_load(open(sys.argv[1]))
assert len(d["baselines"]["vllm/dual"]["submissions"]) == 1
PY
# primary re-induction PRESERVES the submissions map (the block regex would
# otherwise swallow it — the slice-3 splice-safety guard)
out="$(bash scripts/catalog-baseline.sh vllm/dual "${ARGS[@]}" --tps-only 2>&1)" \
|| fail "primary re-induction over a row with submissions failed"
python3 - "$BL" <<'PY'
import sys
import yaml
d = yaml.safe_load(open(sys.argv[1]))
row = d["baselines"]["vllm/dual"]
assert row.get("tier") == "local", "primary induction must stamp tier: local"
assert row["submissions"]["2x5090-pcie"]["tier"] == "submitted", \
"primary re-induction dropped the submissions map"
PY
# submission-only entry: a slug measured on hardware we don't have
out="$(bash scripts/catalog-baseline.sh vllm/qwen-27b-dual-max "${BARGS[@]}" 2>&1)" \
|| fail "submission-only add failed"
grep -q "submission-only" <<<"$out" || fail "did not create a submission-only entry"
python3 - "$BL" <<'PY'
import sys
import yaml
d = yaml.safe_load(open(sys.argv[1]))
row = d["baselines"]["vllm/qwen-27b-dual-max"]
assert "narr_tps" not in row, "submission-only entry must carry no primary fields"
assert row["submissions"]["2x5090-pcie"]["tier"] == "submitted"
txt = open(sys.argv[1]).read()
assert txt.index(" vllm/qwen-27b-dual-max:") < txt.index("KNOWN GAPS"), \
"submission-only entry landed after the gap-list footer"
PY
# multi-tag bundle: refuse without --from-tag, select with it
mkdir -p "$TMP/bundle-src/results/rebench/synth-second"
cp "$BSRC"/* "$TMP/bundle-src/results/rebench/synth-second/"
tar czf "$TMP/bundle2.tgz" -C "$TMP/bundle-src" results
if bash scripts/catalog-baseline.sh vllm/dual --from-bundle "$TMP/bundle2.tgz" \
--source x --submitted-by v --tps-only --baselines-file "$BL" --dry-run >/dev/null 2>&1; then
fail "multi-tag bundle without --from-tag must refuse"
fi
out="$(bash scripts/catalog-baseline.sh vllm/dual --from-bundle "$TMP/bundle2.tgz" \
--from-tag synth-second --source x --submitted-by v --tps-only \
--baselines-file "$BL" --dry-run 2>&1)" || fail "multi-tag --from-tag selection failed"
grep -q 'source_tag: "synth-second"' <<<"$out" || fail "--from-tag did not select the tag in the bundle"
# real file untouched throughout (checksum across the run — git state may
# legitimately be dirty in a working checkout)
[[ "$(sha256sum scripts/lib/profiles/baselines.yml | cut -d' ' -f1)" == "$REAL_SUM_BEFORE" ]] \