The Profile (at-a-glance) header + required Status field is gate-tested
(test-compose-status-drift) but was documented only in CLAUDE.md/AGENTS.md
(the agent guide) — the human-contributor surfaces never stated it, so a
submission could omit Status and only fail in CI. Surface the requirement
(link, don't duplicate the schema) on every path:
- CONTRIBUTING: new "Profile header on every compose" rule next to one-per-PR.
- PR template: a "Profile header complete" checkbox for compose PRs.
- ADDING_MODELS Step 3: a "Profile header (mandatory)" subsection.
- BRING_YOUR_OWN §4: a header bullet for contributed composes.
Also reciprocal-link the two onboarding docs: ADDING_MODELS now points paths
1-2 (serve/tune/validate without the catalog) to BRING_YOUR_OWN, framing
itself as the promotion step after a config validates there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The one-model-per-PR rule only covered new-compose PRs; #301 violated
scope a different way — an engine-image bump + sampling-default change +
benchmark dump + pin update in one PR. Extend the rule: one model OR one
feature/concern per PR, compose or not. A grab-bag stalls on its weakest part.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new-compose PR must add exactly one model. Multiple models in one PR
can't each get their own validation gate, issue thread, and merge
decision — a blocker on any one stalls all of them (lived this on #298,
4 models with 4 different fates). Sibling variants of one model stay fine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Calibration fix in 29e7de5 + apnar's re-run on disc #86 surfaced that
the canonical anchor-data invocation needs to be more explicit. Two
risks: (1) default --step-size 10 is right but coarse-step suggestions
(--step-size 50) leave too few data points to find the knee, (2) without
--bench-runs 3, single-batch variance can produce 10-30% adjacent-cap
deltas that are noise rather than signal.
HARDWARE.md: adds full canonical command with all four flags (cooling,
load-mode, concurrency auto, bench-runs 3) plus rationale for each.
Explicitly notes --step-size 10 is the default and not to override
unless you know why.
CONTRIBUTING.md: bumps the inline example to the same canonical form.
Two minimal pointers added now that scripts are engine-agnostic:
- README.md: host-build invocation pattern + link to disc #88
- CONTRIBUTING.md: same + new bullet for power-cap-sweep.sh ask
(anchors for cards we don't have yet — A5000/A6000, 4080, 5060
Ti/5080, modded variants)
No structural changes — keeps the existing flow intact for vLLM-stack
users; just opens the door for non-Docker contributors who would have
been silently turned away before #87 / #88.
Adds a "Where to file what" table to CONTRIBUTING.md before the
"Process for non-trivial changes" section. Codifies the convention
that bug-shaped problems (logs, tracebacks, report.sh dumps) belong
in issues, while design/welcome/uncertain questions belong in
discussions.
Motivated by accumulating log-heavy comments in discussions
(notably disc #51 5090 NVFP4 debug, disc #33 k8v4 exploration)
that would have been more discoverable as issues. Issues have
state machines (open/closed, labels, assignees) and proper search
surface that discussions don't. Maintainers may now ask folks to
fork bug-shaped pieces into issues with a cross-link back.
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/report.sh now has four optional sections:
--verify verify-full.sh (~1-2 min)
--stress verify-stress.sh (~5-10 min, 7/7 incl. Cliff 2 needles)
--soak SOAK_MODE=continuous + summary.md embed (~25 min, catches Cliff 2b)
--bench bench.sh (~3 min, canonical TPS)
--full all four (~35 min total, the canonical cross-rig pass)
Why soak as its own flag: verify-full + verify-stress + bench all PASS on
configs that FAIL the multi-turn continuous soak (Cliff 2b at ~25K accumulated
tokens). Until upstream lands a fix, soak is the only test that catches the
agentic-workload failure mode that bit issues #41 / #42 / #43 / #45.
Propagated --full as the recommended single-command path through:
- README.md (bug/bench callout)
- CONTRIBUTING.md (Numbers from your rig + new compose variant gate)
- .github/PULL_REQUEST_TEMPLATE.md (one checkbox covers verify+stress+soak+bench)
- .github/ISSUE_TEMPLATE/numbers-from-your-rig.yml (single paste includes soak)
- BENCHMARKS.md ("How to add a row for your rig")
Backward compatible: existing --verify and --bench flags unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add .github/PULL_REQUEST_TEMPLATE.md with the rig-report / verify /
soak-continuous / bench / BENCHMARKS row checklist that PR #44 surfaced
as missing. New compose variants now have an explicit gate list.
- CONTRIBUTING.md: new "Submitting a new compose variant — full gate list"
section explaining the why behind each gate, cross-linked to docs/CLIFFS.md
and #41 for the soak-continuous rationale.
- numbers-from-your-rig.yml: optional v2 SOAK_MODE=continuous summary field
for cross-rig Cliff 2b validation.
- docs/STRUCTURED_COT.md: replace "Phase 2 pending" stub with measured
results — Holiday tagline (4/6 rescue, 23-token think), DeepSeek scratchpad
(5/6 rescue, 387-token think), PROMPT_TERSE (5/6 rescue at 75 tokens with
no FSM mask). Headline reframe: PROMPT_TERSE rescuing 5/6 with the same
G/A/E shape as current suggests FSM enforcement is the mechanism causing
those regressions, not absence of structure. Phase 3 plan included.
- tools/grammar-eval: deepseek-scratchpad.gbnf (PLAN/NOTE×0-15/VERDICT)
and subset-bench.py wired for the 5-condition harness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug report and numbers-from-your-rig issue templates previously asked for
6+ separate fields each (docker logs, verify-full, nvidia-smi, GPU config,
compose variant, commit, etc.). The new scripts/report.sh captures all
of that in one paste-ready dump, plus the data we kept asking individually
(power caps + default vs current, NVLink topology, OS, system RAM, idle
GPU VRAM detection, container runtime versions, cached vLLM image SHAs).
Templates restructured to lead with `bash scripts/report.sh > my-rig.md`:
- bug-report.yml: 6 required + 2 optional fields → 3 required (what
happened, repro, rig report) + 1 optional (extras / fallback). Net -38%
lines, but more importantly: one command instead of six manual asks.
- numbers-from-your-rig.yml: 5 required + 2 optional fields → 1 required
(rig + bench report via `--bench`) + 1 optional (notes / fallback). Net
-38% lines.
Both templates retain manual-fallback guidance for users who can't run
report.sh (no shell access, different rig environment, etc.).
CONTRIBUTING.md "Numbers from your rig" + "Bug reports" entries updated
to reference report.sh as the primary path. README.md repo layout adds
report.sh to scripts/ list, plus a one-liner in the docs-and-extras
paragraph pointing affected users at the script.
Net effect: future cross-rig contributors share more standardized data
with less effort. Triage threads stop bouncing on "could you also send
me X?" follow-ups for the surface report.sh covers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
We had upstream issue / PR links scattered across CHANGELOG, INTERNALS,
FAQ, per-compose comments — drifting independently. Centralizing in
one place with a status convention so the tracker stays current.
- docs/UPSTREAM.md — single source of truth, categorized by upstream
(vLLM / Genesis / fla-org / FlashQLA / llama.cpp / transformers /
SGLang), with status emoji + what unblocks for us + workaround.
- AGENTS.md (repo root) — AI-coding-agent guidance. The rule:
before filing or referencing an upstream issue, check + update
docs/UPSTREAM.md. Also captures today's Genesis-opt-in vetting
lesson (behavioral mitigations need streaming + large-prompt repro
before shipping default-on).
- README, CONTRIBUTING, INTERNALS "See also" sections cross-link the
two new files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three sections:
1. What's welcome — bench numbers from new rigs, bug reports with
the data we always need, upstream-bug minimum repros, new compose
variants with verify-full + verify-stress + bench output, new
model support (with the canonical learnings/<model>.md template),
patch experiments (Genesis-style file replacements + idempotent
patcher), genuine doc clarity wins, cross-link to your published
numbers (Reddit / blog / Twitter).
2. What's NOT — doc style nitpicks, untested config knobs (every
flag we ship has a measurement attached), removing the two-routes
editorial framing without new data, vendoring upstream, marketing-
style README rewrites, driveby PRs that don't run verify-full.
3. Process for non-trivial PRs — issue first / branch off master /
verify-full + verify-stress / bench numbers / four-question PR
description (what / impact / compared-to / trade-off) / upstream
author attribution.
Closes with the ground rules: don't claim numbers you didn't measure,
always capture VRAM during benchmarks, pin everything, differentiate
"shipped" from "measured."
Linked from README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
