club-3090/CONTRIBUTING.md

Contributing to club-3090

Thanks for being here. This repo collects working recipes for serving big LLMs on RTX 3090s. Most of its value is in measured numbers and tested configs — not prose. The contributions that move the most needle are also the easiest: bench your rig, file the data.

---

What kind of contributions work best

✅ Yes please

• Numbers from your rig. Different power caps, different motherboards, different models — we want all of it. Use the Numbers from your rig issue template (no PR needed). The template asks for bash scripts/report.sh --full > my-rig.md — one ~35-min pass captures hardware (incl. power caps + NVLink topology), stack version, verify-full + verify-stress 7/7, SOAK_MODE=continuous summary (catches Cliff 2b), AND the canonical bench numbers. If you're unsure what to run before measuring, start with bash scripts/setup.sh and bash scripts/launch.sh; both wizards mark hardware-fit. High-signal contributions land in BENCHMARKS with attribution. Not running our Docker composes? All scripts now work on non-Docker host builds (llama.cpp host server, SGLang, etc.) via URL=... CONTAINER=none MODEL=... bash scripts/... — engine is auto-detected, vLLM-specific checks skip cleanly. See discussion #88 for the full host-build contributor flow.
• Power-cap efficiency curves. sudo bash scripts/power-cap-sweep.sh --cooling air|water|aio --load-mode decode-concurrent --concurrency auto --bench-runs 3 produces cross-rig efficiency-knee data (discussion #86). ~15-20 min for a 30-cap sweep on a 3090/4090/5090. Especially valuable on cards we don't have anchors for yet (A5000/A6000, 4080, 5060 Ti / 5080, modded variants). Keep --step-size 10 (the default). Larger step-sizes (e.g. --step-size 50) are too coarse for the efficiency knee and only useful for quick smoke tests. See docs/HARDWARE.md for the full canonical command and rationale.
• Bug reports with the data we ask for. The bug report template leads with bash scripts/report.sh > my-rig.md (add --verify to include verify-full output, --soak to also run SOAK_MODE=continuous if you suspect a multi-turn agent cliff) — single command captures the rig state we'd otherwise ask for individually (hardware, container state, Genesis patches, KV pool sizing, engine config). With that paste, the first reply is usually a fix or a clear next step instead of "can you send me…".
• Bug reproductions / minimum repros for upstream issues. vLLM / llama.cpp / Genesis bugs that affect this stack are most useful when they have a one-paragraph reduction. Drop them in an issue or open a draft PR adding a reproducer to verify-stress.sh.
• New compose variants with measured numbers. If you've found a config combination that beats one we ship — better TPS, lower VRAM, cleaner stress profile — open a PR with: (a) the docker-compose.<name>.yml, (b) verify-full.sh output passing, (c) verify-stress.sh output passing, (d) a bench.sh run (3 warm + 5 measured) showing the delta against the closest existing variant. Bonus points: a footer in the compose file explaining which existing variant you compared against and why this one is better for which workload.
• New models. Adding a model is a real lift but well-defined: clone the models/qwen3.6-27b/ directory structure, populate the engine subdirs, follow the canonical learnings template layout (this repo doesn't ship that file but the convention is documented in models/qwen3.6-27b/INTERNALS.md). Open an issue first to scope.
• Patch experiments. If you've found a working file-replacement patch for an upstream issue (Genesis-style), the PR shape we expect: a script or .py that does the patch idempotently, a verify-full.sh pass on a fresh container with the patch applied, and a CHANGELOG entry naming the upstream issue + tracking link. We're happy to ship "buggy-but-fast" patches as opt-in variants if they're cleanly fenced.
• Doc improvements where there's a genuine clarity win. "I read this section and was confused; here's what I tried first" is a great PR opener. We're tone-conservative (terse / technical / no marketing fluff), but not allergic to clearer writing.
• Cross-link to your rig's own published numbers (Reddit, blog, Twitter). Adding a row to BENCHMARKS or a footnote in the relevant doc with attribution is welcome.

❌ Not really

• Doc style nitpicks. Reformatting tables, changing list bullets, "this would read better as bullet points" — usually no. We optimize for "reader finds the answer" not "prose is consistent." Open an issue if you really want to make the case.
• Untested config knobs. "Adding --foo-bar 42 because the vLLM docs mention it" without a before/after measurement on this hardware = no. Every flag in our composes is there because we measured it changing something. PRs that add or change flags need numbers.
• Removing the two-routes framing or the cliffs language without new data. The "vLLM dual = max throughput, llama.cpp single = max robustness" framing is editorial — built on stress-test findings (no Cliff 1, no Cliff 2 on llama.cpp). Not up for revision unless the underlying data changes (e.g., vllm#40914 lands and Cliff 1 closes).
• Vendoring upstream packages. We pin specific commits/SHAs of vLLM, Genesis, llama.cpp images — we don't fork them into the repo. PRs adding vendor/ directories or copying upstream source get redirected.
• Marketing-style README rewrites. "What if we added emojis here?" type changes. The launch tweet handles the marketing surface; the README is for users who already clicked through.
• Driveby PRs that don't run verify-full. If a config change affects what the server does, run bash scripts/verify-full.sh on it. A passing verify is the entry ticket for compose / patch / script changes.

---

Issues vs Discussions — where to file what

Two GitHub channels, two different shapes of conversation. Picking the right one keeps the right people seeing the right thing:

You have	File as	Why
A reproducible failure (boot OOM, 500, silent-empty, stuck engine) — anything with a stack trace, error log, or report.sh dump	Issue	Bug-tracking lives in issues — open/closed state, labels, assignees, searchable when others hit the same symptom. The bug report template leads with bash scripts/report.sh > my-rig.md so the rig context is in the issue, not split across replies
A measured rig + bench (cross-rig data, BENCHMARKS row request)	Numbers from your rig issue	Same reasoning — structured one-shot data point lands cleanly as a closeable issue, ends up in BENCHMARKS with attribution
A design question ("should the launch wizard ask X?", "why does dual-turbo use TQ3 not k8v4?")	Discussion	Open-ended, branchy, no single resolution state
A welcome / first-impression / sharing your rig	Discussion	Tree-of-comments shape fits casual interaction; doesn't pollute the issue tracker
Asking for help and not sure if it's a bug	Discussion, until clarity emerges; then fork the bug-shaped piece into an issue with a cross-link	Avoid prematurely treating an unclear question as a bug; but once the bug is clear, move it to issues so the resolution state is trackable

Please don't post log dumps / report.sh output / stack traces inside discussions. It buries the bug under non-bug content (and the reverse: a bug-shaped problem buried inside a "great success!" thread doesn't reach maintainers' issue queue). If a discussion thread accumulates a log dump or a reproducer, the maintainers may ask you to fork the bug-shaped piece into an issue with a cross-link back — not banishment, just keeping each channel doing what it's good at.

---

Submitting your bench

The matrix is hand-curated — the canonical path is to file an issue with your rig + numbers; we'll review, ask clarifying questions, and integrate.

After running bash scripts/rebench-full.sh, generate a paste-ready row:

bash scripts/submit-bench.sh --tag <your-tag>

The script writes results/rebench/<tag>/BENCHMARKS-row.md. To submit:

Path A — Auto-issue (recommended, requires gh auth login)

bash scripts/submit-bench.sh --tag <your-tag> --auto-submit

Opens an issue via gh issue create with your rig + row pre-filled.

Path B — Manual issue (no tools beyond browser)

Open https://github.com/noonghunna/club-3090/issues/new?template=numbers-from-your-rig.yml and paste the row + your rig.txt into the body.

Path C — Direct PR (advanced)

bash scripts/submit-bench.sh --tag <your-tag> --auto-submit --as-pr

For contributors who know the BENCHMARKS.md section structure and want to propose the exact row. The maintainer may still redirect to an issue thread for context-gathering before merge — direct PRs aren't a fast-path bypass.

---

Process for non-trivial changes

1. Open an issue first for anything bigger than a typo fix or a one-line measurement contribution. We'll either align on shape or explain why we'd land it differently — saves you a wasted afternoon.
2. Branch off master, work in your fork.
3. Run the verify suite before pushing:

   bash scripts/verify-full.sh    # fast functional smoke (~1-2 min)
   bash scripts/verify-stress.sh  # boundary tests (~5-10 min)
   

   For compose / patch / script changes, both should pass against your changes. Include the output in the PR description.
4. Bench the relevant config if you're touching anything that could move TPS:

   bash scripts/bench.sh
   

   Drop the run-by-run output in the PR — wall_TPS, decode_TPS, PP tok/s, TTFT, MTP AL (where applicable). Mean + CV + n=5 minimum.
5. Open the PR with a description that answers four questions:
   • What problem does this solve? (One paragraph.)
   • What's the measured impact? (Numbers.)
   • What did you compare it against? (Specific existing variant or BENCHMARKS row.)
   • What's the trade-off? (TPS vs VRAM vs ctx vs feature support.)
   • The PR template walks you through these — fill it in, don't delete it.
6. Sign-off: if your patch involves upstream code (vLLM source, Genesis tree, llama.cpp), credit the upstream author in the docstring/CHANGELOG. We hold a high bar on attribution because Sandermage / the vLLM maintainers / the llama.cpp folks do most of the actual heavy lifting; we just package and bench.

---

Submitting a new compose variant — full gate list

New compose files (models/<model>/<engine>/compose/docker-compose.<name>.yml) get a tighter checklist than other PRs because they ship as a "supported" path that other people boot blind. The PR template enumerates these — bullets here are the why:

One model — or one feature/concern — per PR. A new-compose PR must add exactly one model (one set of sibling variants for a single model is fine; multiple different models is not). Each model needs its own validation gate, its own issue thread, and its own merge decision — bundling several into one PR means a blocker on any one of them stalls all of them, and per-model issues can't be tracked. The same applies to non-compose changes: keep each PR to a single feature or concern — don't bundle an engine-image bump, a sampling-default change, a benchmark-results dump, and a pin update into one PR. A small focused PR clears its gate on its own timeline; a grab-bag stalls entirely on its weakest part. If you've got several, open one PR each and link them.

Profile header on every compose. A new/changed compose must carry the # Profile (at-a-glance): block with a Status: field — exactly one of ✅ Production / ⚠️ Production w/ caveats / 🧪 Experimental / 👁️ Preview / ⏸️ Upstream-gated / 🗑️ Deprecated — and a Caveats: line whenever Status is ⚠️/👁️/⏸️/🗑️. This is gate-tested: test-compose-status-drift asserts the header status ↔ registry status, so a missing or mismatched Status fails CI. Full field list + schema: CLAUDE.md → "Profile schema header (every compose, every time)."

Single command captures all of (1)–(5) in one paste:

bash scripts/report.sh --full > my-rig.md

That runs ~35 min and captures rig + verify-full + verify-stress + soak-continuous + bench. Paste the file contents as a PR comment.

Or run the steps individually if you'd rather:

1. Rig report — bash scripts/report.sh > my-rig.md, paste as a PR comment. Captures GENESIS_PIN, vLLM image SHA, container CUDA/Python, PCIe lanes per card, power caps, NVLink topology in one pass. Without it future readers can't tell whether your numbers are reproducible against their environment or rig-specific. This is a merge gate, not a nice-to-have.
2. verify-full.sh PASS — fast functional smoke. Confirms the variant boots and serves correctly on your rig.
3. verify-stress.sh 7/7 PASS — boundary tests including Cliff 2 needle recall (probe 7: 60K + 90K needles). Required for any variant claiming long-context support.
4. SOAK_MODE=continuous summary (single-card variants: required) — verify-stress catches single-prompt cliffs but not the multi-turn accumulating-context cliff (Cliff 2b) that fires at ~25K accumulated tokens on single-card vLLM paths. Soak-continuous is the only test that exercises that regime. See docs/CLIFFS.md for the byte-level explanation and #41 for the validation matrix. Multi-card variants: strongly recommended (TP=2 and TP=4 escape Cliff 2b, but expectation isn't evidence — verify on your rig).

   SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 \
     CONTAINER=<container-name> ENDPOINT=<http://localhost:port> \
     bash scripts/soak-test.sh
   

5. bench.sh run — 3 warmups + 5 measured runs of narrative + code prompts. Report wall_TPS, decode_TPS, PP tok/s, TTFT, peak VRAM/card per run, MTP/DFlash AL where applicable. For llama.cpp or other engines without vLLM prompt-throughput logs, include PP=1 bash scripts/bench.sh so the long-prompt fallback captures prompt-processing throughput.
6. BENCHMARKS row — under the appropriate model section, mirroring existing column shape (incl. Rig column). Attribution is automatic.
7. CHANGELOG entry — in models/<model>/CHANGELOG.md.

If any of (1)–(5) don't apply to your variant, say so explicitly in the PR ("N/A — short-prompt-only path; soak-continuous would not exercise the multi-turn regime"). "Forgot to run" gets the PR put on hold; "explained why N/A" gets it merged.

---

Honesty and reproducibility — the ground rules

• Don't claim a number you didn't measure. "Should be ~80 TPS" is fine if labeled as estimate. "Is 80 TPS" needs a bench.
• Always capture VRAM during benchmarks. Per-card peak VRAM is a load-bearing piece of information for any TPS comparison. Skip it and the PR will be asked to add it.
• Pin everything. vLLM image SHA, Genesis commit, llama.cpp commit. If you bumped one between bench runs, say so.
• Differentiate "we shipped" from "we measured." If a config can run a workload at 70 TPS but only with a known-buggy patch combination, that's measured but not shipped. We label it accordingly in BENCHMARKS.

---

License

Apache-2.0 (see LICENSE). By submitting a PR you agree your contribution is offered under that license.

---

See also

• README — top-level overview
• AGENTS.md — guidance for AI coding agents (Claude Code, Cursor, etc.) working in this repo
• docs/FAQ.md — common questions
• docs/EXAMPLES.md — client snippets
• docs/UPSTREAM.md — every upstream issue / PR we depend on or have filed (file new issues here first)
• models/qwen3.6-27b/INTERNALS.md — engineering deep dive (where most "why is X this way" answers live)
• .github/ISSUE_TEMPLATE/ — bug report + numbers-from-your-rig templates