alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Results Card doc — standard format for sharing config results

Browse Source 
Three-panel format (Serving · Quality · Takeaways) for posting a config's
measured serving + quality numbers. Spec-dec is its own column; core 8-pack
stays /150 with optional humaneval+/lcb/aider in a separate table. Linked
from the docs index; first instance is the Qwen3.6-27B thinking A/B in #221.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-06-01 19:04:44 +00:00
parent 8cb50273d9
commit cabdf1286b
2 changed files with 68 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/README.md
View File

@@ -76,5 +76,6 @@ These are cross-cutting references both tracks reach for.
| [`DTYPE_MATRIX.md`](DTYPE_MATRIX.md) | Supported dtype × model × engine matrix. |
| [`KERNEL_MATRIX.md`](KERNEL_MATRIX.md) | Quant-kernel availability and alignment constraints. |
| [`QUALITY_TEST.md`](QUALITY_TEST.md) | The quality-test harness and what it measures. |
| [`RESULTS_CARD.md`](RESULTS_CARD.md) | The standard 3-panel format (Serving · Quality · Takeaways) for sharing a config's measured results. |
| [`STRUCTURED_COT.md`](STRUCTURED_COT.md) | The bounded-thinking / structured-CoT compose path. |
| [`TQ3_MTP_GENESIS.md`](TQ3_MTP_GENESIS.md) | TQ3 KV × MTP × Genesis-patch results and config. |

67
docs/RESULTS_CARD.md Normal file
View File

@@ -0,0 +1,67 @@
# Results Card
A small, fixed format for sharing a config's **measured** results — in a discussion, an issue (see the [`numbers-from-your-rig`](../.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml) template), a `learnings/` note, or a PR description.
It's the empirical counterpart to a compose's `Profile (at-a-glance)` header: the header declares *what the config is*, the Results Card reports *what it measured*. Three panels, always in this order — **Serving → Quality → Takeaways**.
## When to use it
Any time you post serving/quality numbers for a `(model, engine, topology, spec-dec, KV)` config. If you're A/B-ing one knob (thinking on/off, KV format, drafter `n`, …), use two value columns in the Quality table and bold the winner per row.
## Template
```markdown
### Serving — <engine + version>, <topology>
| Config | Spec-dec | KV / ctx | decode TPS (narr / code) | TTFT | VRAM / card |
|--------|----------|----------|--------------------------|------|-------------|
| <model + quant> | <MTP / DFlash / ngram / EAGLE / none> (draft, n=) | <k/v quant> · <max ctx> | <narr> / <code> | <ms> | <GB> |
_(engine-internal decode TPS; 3 warm + 5 measured; temp/top-k/top-p; image/pin; tracking issue. Note any spec-dec accept-rate or balance caveat.)_
### Quality — 8-pack suite (`benchlocal-cli`, verifier-backed, n=<N>)
| Pack | <setting A> | <setting B> |
|------|-------------|-------------|
| toolcall-15 | x/15 | x/15 |
| instructfollow-15 | x/15 | x/15 |
| structoutput-15 | x/15 | x/15 |
| dataextract-15 | x/15 | x/15 |
| reasonmath-15 | x/15 | x/15 |
| bugfind-15 | x/15 | x/15 |
| hermesagent-20 | x/20 | x/20 |
| cli-40 | x/40 | x/40 |
| **TOTAL (8-pack)** | **x/150 (y%)** | **x/150 (y%)** |
**Optional reasoning/code packs** _(on top of the core 8-pack — kept separate so /150 stays intact)_:
| Pack | <setting A> | <setting B> |
|------|-------------|-------------|
| humaneval-plus-30 | x/30 | x/30 |
| lcb-v6-30 | x/30 | x/30 |
| aider-polyglot-30 | x/30 | x/30 |
### Takeaways
- **<headline verdict>** — the single most important finding (bold lead).
- <comparison / tradeoff bullet>
- <production / stability note>
- _tl;dr — one line._
```
## Rules that keep cards comparable
- **The core 8-pack is exactly `/150`:** `toolcall-15 + instructfollow-15 + structoutput-15 + dataextract-15 + reasonmath-15 + bugfind-15 + hermesagent-20 + cli-40` (75 + 75). **Never fold the optional packs into the 150** — `humaneval-plus-30` / `lcb-v6-30` / `aider-polyglot-30` go in their own table below. (They're 30-sample subsets; compare the A/B **delta**, not absolute % against fuller external runs.)
- **Spec-dec is its own column** — MTP / DFlash / ngram / EAGLE / none. Don't rely on the config name to carry it; the same config name can run different spec methods.
- **State the n** (warm/measured runs for TPS; `n=` for quality). Don't present a single run as ranked truth — pack noise is ±5–7, so a small total delta is a tie.
- **Reproduce the conditions** in the Serving footnote: sampling params, engine image/pin, and `thinking on/off` (benchlocal sends `enable_thinking=false` unless you pass `--enable-thinking`).
## Posting to a public discussion/issue
- **No internal paths or secrets** — grep your draft for absolute host paths, model-store paths, and tokens before posting.
- **Don't link files that aren't on a public branch** (experimental/untracked composes 404 — describe them in prose, or link the image tag instead).
- **Verify every link resolves** (repo paths via `git ls-tree`, image tags via `docker manifest inspect`).
## Worked example
The first Results Card — Qwen3.6-27B Q8 on beellama v0.3.0 DFlash, thinking ON-vs-OFF — is [club-3090 discussion #221](https://github.com/noonghunna/club-3090/discussions/221#discussioncomment-17140596).