76dbc009a9fe6a301a95abc1930c778012123895
1309 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
noonghunna
|
ec704e4e2e |
Pin Genesis to exact tested commit + add .env.example + issue templates
- setup.sh: GENESIS_PIN now defaults to commit bf667c7 (Genesis HEAD as of 2026-04-27, semver "v7.54"). This is the exact tree our published TPS numbers were measured against; tagged v7.51-stable was one minor older but came up first because the SHA isn't durable. Switch to commit pin removes the doc-vs-runtime mismatch. Clone strategy adjusted since --branch + --depth 1 doesn't accept SHAs. - .env.example: documents MODEL_DIR / HF_TOKEN / CUDA_VISIBLE_DEVICES / MEM_UTIL / MAX_MODEL_LEN / GENESIS_PIN / SKIP_GENESIS / URL / WARMUPS / RUNS with the same defaults the composes ship. Pure opt-in. - .github/ISSUE_TEMPLATE/: bug-report.yml requires docker logs --tail 100, verify-full.sh output, nvidia-smi, GPU config, compose variant, repo commit. numbers-from-your-rig.yml structures cross-rig TPS contributions with rig spec, bench output, VRAM, max ctx, and notes. config.yml routes Q&A to Discussions. - .gitignore: drop trailing slash on genesis pattern so it also ignores local symlinks that some of us point at out-of-tree clones. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
5060e22a6c |
Split verify-full.sh → verify-full.sh (fast functional) + verify-stress.sh (boundary)
Recent additions to verify-full.sh (#8 tool-prefill OOM, #9 cascade detection, #10 MTP AL) made the script slow — the longctx needle ladder (#7) alone could run 5+ min, and the full 10-check suite was approaching 10 min. Awkward for "is the stack functional" iteration during dev work. verify-full.sh (8 fast checks, ~1-2 min) 1. Server reachable 2. Genesis patches applied 3. Basic completion (Paris) 4. Tool calling 5. Streaming (SSE) 6. Thinking / reasoning mode 7. Output quality / cascade detection (was #9) 8. MTP acceptance length threshold (was #10) Run: after every config change to confirm the stack still serves cleanly. verify-stress.sh (2 boundary checks, ~5-10 min) 1. Long-context needle ladder (4 depths, 10K / 30K / 60K / 90K) — was #7 2. Tool response prefill OOM (~25K-token mock tool message) — was #8 Run: before publishing or when investigating prefill-OOM regressions specifically. Smoke-tested against dual.yml on dual-card: verify-full.sh: 8/8 green in 65 seconds verify-stress.sh: 2/2 green (skipped longctx for this smoke), 15s Same env-var conventions (URL, MODEL, CONTAINER, SKIP_LONGCTX, SKIP_TOOL_PREFILL, PREFILL_TARGET_CHARS). Doc updates: - top-level README repo layout: lists both scripts with timing/scope - docs/ARCHITECTURE.md: scripts/ section + design rules updated - models/qwen3.6-27b/USE_CASES.md: tool-prefill reference points at verify-stress.sh now - models/qwen3.6-27b/CHANGELOG.md: dated entry documenting the split |
||
|
noonghunna
|
c70147426d |
Dual-card re-bench on club-3090 substrate + fix dual-turbo mount path
The published dual-card TPS numbers (T1, DT1, D5, D7 in BENCHMARKS.md)
were measured pre-v714-formalization on a different nightly + Genesis
tree. Re-benched all 4 dual composes today on the unified club-3090
substrate (dev205 + Genesis v7.51-stable + Marlin pad fork mounted).
Also caught + fixed a stale mount path in dual-turbo.yml — predecessor
referenced patch_tolist_cudagraph.py at `../patches/genesis/` (old
qwen36-dual-3090 layout); club-3090 has it at `../patches/` (top-level).
Container died at boot with 'cant find __main__ module' before fix.
Audited all 9 composes — only dual-turbo had the bug.
Re-bench numbers (3 warm + 5 measured per prompt arm):
Compose Narr (CV) Code (CV) AL VRAM/card vs claimed
dual.yml 69.05 (2.3%) 88.58 (3.4%) 3.4 23.6 GB -3% / -1%
dual-turbo.yml (now TQ3) 53.65 (2.7%) 72.93 (2.7%) 3.4 24.1 GB -8% / +6% vs k8v4
dual-dflash.yml 81.94 (4.3%) 124.93 (5.8%) 4.1-4.4 23.6 GB +5% / -2%
dual-dflash-noviz.yml 78.19 (2.5%) 126.99 (2.2%) 4.2-4.4 23.8 GB +2% / +2%
Net: most numbers within run-to-run variance. dual-turbo's TQ3 swap (from
k8v4) cost ~8% narrative but recovered ~6% code — net trade for ~9× the
KV pool capacity (which is what the compose exists for).
verify-full.sh: dual.yml passes 10/10 incl. 90K needle. dual-turbo passes
10/10 too. DFlash variants passed all checks except longctx (skipped for
time; recall path validated previously).
Doc updates:
- models/qwen3.6-27b/README.md: dual-card variant table updated with
measured numbers (was 71/89, 58/69, 78/128, 77/124 → 69/89, 54/73,
82/125, 78/127). Also corrected DFlash variants to FP16 KV (was
written as fp8 in the table but file uses default FP16).
- models/qwen3.6-27b/USE_CASES.md: quick map updated with measured
dual TPS for each workload.
- dual.yml header: measured-numbers line replaces predecessor's claim;
variant matrix in dual.yml updated.
- dual-turbo.yml header: TPS regression vs fp8 noted as ~22% (was
"~25% trade" claim).
- CHANGELOG.md: dated entry documenting the re-bench + path fix.
|
||
|
noonghunna
|
b641719eb8 |
Add long-vision + long-text composes (formalize R3' / R3''' bench rows)
The v714 formalization round (2026-04-27) measured these as opt-in tiers edit-able into docker-compose.yml. That made reproducibility fragile: users who wanted to boot the exact 192K-vision or 205K-text-only configs had to hand-edit max-model-len, gpu-memory-utilization, and --language-model-only. Promoting both to dedicated compose files so each published bench row boots with a single -f flag. New composes: docker-compose.long-vision.yml 192K + 0.98 + vision (R3', 51/68 TPS) docker-compose.long-text.yml 205K + 0.98 + no vision (R3''', 50/66 TPS) Both validate via `docker compose config`. Both carry the same prefill cliff caveats as the edit-the-default approach did: - Cliff 1: ≥25K-token tool-message prefills OOM (ampersandru #1 class) - Cliff 2: ≥50-60K single prompts OOM (DeltaNet GDN forward, hardware-bound) The full 192K/205K is for steady-state context accumulation across many small turns, NOT for stuffing 192K of fresh tokens in one request. Header / docs alignment: - models/qwen3.6-27b/README.md: variant table now lists long-vision + long-text rows; Quick map updated. - models/qwen3.6-27b/USE_CASES.md: Frontier 192K-262K section now references both composes directly instead of "edit the default". - vllm/README.md: "Pick a compose" code block adds two boot lines. - default's header variant matrix lists the new files. - CHANGELOG.md: dated entry documenting the addition + rationale. Did NOT add separate composes for 64K / 96K / 128K opt-in tiers. Those are interpolation points between the safe default (48K) and the frontier (192K/205K); users can edit if they want a midpoint, but it's not worth a separate file per benchmarked combination. |
||
|
noonghunna
|
3e1f5f61c0 |
dual-turbo: switch kv-cache-dtype k8v4 → 3bit_nc to align with test findings
The predecessor dual-3090 README documented `turboquant_3bit_nc` for the
turbo variant, but the actual compose shipped `turboquant_k8v4` (likely
config drift that was never reconciled). Earlier this session I kept the
file as-is and updated the doc to match — but that was the wrong call.
What our test findings actually say:
- Single-card v714 default ships turboquant_3bit_nc, validated extensively
(51 narr / 68 code TPS, 10/10 verify-full.sh checks pass)
- Memory entry: "Lorbus Qwen3.6-27B + MTP + TurboQuant = 85 TPS / 125K
single-card ⭐" — TQ3, the same _3bit_nc preset
- Predecessor dual-3090 README claimed 3bit_nc for turbo too
So 3bit_nc IS the tested config across the project; k8v4 was the drift.
Aligning the dual-turbo file with that. Trade-offs:
- TQ3 is 3 bits/token avg (vs k8v4's ~6 bits/token avg) — smaller KV per
token → larger KV pool capacity at same mem-util
- The published "4.59× concurrency at 262K" claim should hold at MINIMUM,
likely improve modestly (more KV pool → more streams or longer tail)
- Per-stream TPS should be roughly similar (KV bandwidth and compute paths
are similar between the two TurboQuant variants)
Risk: I haven't booted dual-card to confirm. The change is internally
consistent with single-card test data + predecessor docs. Cross-rig users
will validate via verify-full.sh on actual dual hardware.
Updated dual.yml's variant-matrix table accordingly (TQ k8v4 → TQ3).
|
||
|
noonghunna
|
0f33561b6b |
Audit + reconcile dual-card compose headers, patches README, setup output
After the user flagged "are you validating all composer files" — ran a
full dry-run audit of all 9 composes via docker compose config, extracted
key flags (TP, max_len, mem_util, KV dtype, spec-decode), and found
several doc-vs-code mismatches inherited from the predecessor repos.
Compose header fixes:
- docker-compose.dual.yml — header described it as inheriting from
"single-card project's default", said "fp8 is plenty for 64K"
(stale — file actually does 262K). Updated to reflect: this IS the
dual-card default, fp8 is plenty for full 262K, plus a variant matrix
showing all 4 dual files with their actual TPS / streams / KV / vision.
- docker-compose.dual-turbo.yml — header claimed kv-cache-dtype was
`turboquant_3bit_nc` but the file actually ships `turboquant_k8v4`.
This mismatch was in the predecessor too; we kept the file (not the
header) since k8v4 is what was tested. Updated header to reflect
reality + noted the predecessor doc claim for archaeology.
- docker-compose.dual-dflash.yml — header said max_model_len "drops
from 262K to 16K" (stale dev-cycle comment); actual is 185K. Fixed.
Also added: KV cache is FP16 (DFlash + head_size=256 + non-causal
has no fp8/turbo Ampere backend), the bfloat16 dtype workaround for
vllm#40334, and clear positioning vs the noviz variant.
- docker-compose.dual-dflash-noviz.yml — minor: file path in "to run"
pointed at the old compose/ dir; updated to new layout path.
patches/README.md — was framed as dual-card-only ("we don't run
Genesis here") but the patches dir is now shared across single and
dual variants. Rewrote with a per-patch + per-variant matrix:
- patch_tolist_cudagraph.py: single-default + dual-turbo
- patch_pr40798_workspace.py: research artifact, no compose mounts
- genesis/: single-default + tools-text + dual-turbo
- Marlin pad fork (external /opt/ai/vllm-src/): all 4 dual composes
Added a Genesis env-opts table showing per-patch toggles and which
composes enable each.
scripts/setup.sh — final-output Next-steps block referenced the OLD
relative path `cd compose && docker compose up -d`, which would fail
in the new layout. Updated to:
cd models/<model>/vllm/compose && docker compose up -d
Plus added a clear note about the Marlin pad fork dependency for
dual-card composes (with the git-clone command users need to run
once before booting any dual-card variant).
YAML validation: `docker compose config` passes for all 9 composes
with MODEL_DIR set. Volume paths resolve, env vars substitute, no
syntax errors. Single-card default smoke-tested earlier (10/10
verify-full.sh checks pass); dual-card composes pass YAML validation
but require a 2× 3090 rig to actually boot — left for cross-rig users
to confirm.
|
||
|
noonghunna
|
7f00e52140 |
Pin Genesis version + fix MODEL_DIR defaults + clean stale headers
Three related fixes for the post-restructure layout to actually work: 1. Pin Genesis to a tested tag (addresses walmis #8) - setup.sh now does `git clone --branch v7.51-stable-2026-04-27 --depth 1` instead of plain `git clone` (= latest HEAD). Re-runs `git checkout` on the pinned tag if the dir already exists. - GENESIS_PIN env var lets users opt into a different tag/commit. - Sanity-check the v7.14 layout (vllm/_genesis package) and bail with a clear error if missing, rather than silently shipping a broken compose-genesis combination. 2. MODEL_DIR default in all 9 composes (smoke-test fix) - Old default was ${MODEL_DIR:-../models}, which from the new compose dir at models/qwen3.6-27b/vllm/compose/ resolved to a non-existent path. Composes silently created an empty mount target → vLLM couldn't find the model on first boot. - Updated all 9 composes (single + dual variants) to: ${MODEL_DIR:-../../../../models-cache} This resolves to repo-root/models-cache/ which is exactly where setup.sh now downloads. Booting works zero-arg if you ran setup.sh. - Users with model weights elsewhere can still set MODEL_DIR via env. - Validated: from the new paths, MODEL_DIR=/mnt/models/huggingface docker compose up -d boots cleanly and verify-full.sh passes all 10 checks. 3. Clean stale header comments - tools-text.yml: header still self-described as alternate to old "20K default" + referenced deleted longctx-experimental.yml. Updated to current variant matrix (default 48K, this 75K text-only). - minimal.yml: similar — "20K default" + longctx-experimental refs. Updated. - fast-chat.yml: already fixed in previous commit. Smoke test: verify-full.sh from the new club-3090 paths passes 10/10 (including #4 tool calling, #8 tool-response prefill OOM, #10 MTP AL). |
||
|
noonghunna
|
2511a98110 |
Fix .gitignore + add the entire models/ tree (initial commit was incomplete)
The old repos used models/ as the model-weights download directory, so their .gitignore excluded it. We copied that .gitignore into club-3090 without updating, which silently dropped the entire models/ subtree from the initial commit on GitHub (everything still on disk locally, just not tracked). Fixes: - .gitignore: remove models/, add models-cache/ (the new default for weights download). Update genesis patches path to the new location models/<model>/vllm/patches/genesis/. Update compose-state ignores to use **/compose/ glob since composes are now nested. Add vllm-src/ for the dual-card Marlin pad fork mount. - Add the missing models/qwen3.6-27b/ tree: - README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md - vllm/README.md + 9 docker-compose.yml variants + patches dir - llama-cpp/README.md + 2 launch recipes - sglang/README.md (currently blocked status) Also fixes: stale comment header in fast-chat.yml that referenced the deleted longctx-experimental.yml and self-described as "default" when it's the chat-only variant. |
||
|
noonghunna
|
3fa33332ce |
Initial commit — club-3090: model-agnostic LLM serving recipes for RTX 3090
Consolidates and supersedes:
- noonghunna/qwen36-27b-single-3090
- noonghunna/qwen36-dual-3090
The two predecessor repos partitioned by card count (1× vs 2×). This
repo partitions by engine instead, which matches how users actually
decide ("vLLM or llama.cpp?" before "1 card or 2"). Card count becomes
a config variant within each engine.
Structure (model-agnostic from day 1):
docs/ cross-model engine + hardware docs
engines/ vLLM / llama.cpp / SGLang comparison + per-engine deep dives
HARDWARE.md Ampere SM 8.6+, NVLink, power, VRAM ceilings
GLOSSARY.md plain-language definitions
img/ illustrations (vram-budget.svg)
ARCHITECTURE.md how this stack thinks about LLM serving on 24 GB
models/<model-name>/ everything specific to a model
qwen3.6-27b/ today's only model
README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md
vllm/ vLLM-specific configs for this model
compose/ docker-compose files (single + dual variants)
patches/ tolist_cudagraph + Marlin pad notes
llama-cpp/ llama.cpp recipes for this model
recipes/ shell scripts (single-card default + 262K max-ctx)
sglang/ SGLang status (currently blocked)
scripts/ shared, model-aware
setup.sh bash setup.sh <model> → downloads + verifies
verify.sh / verify-full.sh smoke + functional tests
bench.sh canonical TPS bench
vLLM compose variants (all under models/qwen3.6-27b/vllm/compose/):
Single-card:
docker-compose.yml ⭐ DEFAULT — TQ3 + Genesis P65, 48K, 51/68 TPS
docker-compose.fast-chat.yml fp8 + 20K, 55/70 TPS — fastest at small ctx
docker-compose.tools-text.yml fp8 + 75K, 53/70 TPS — best for long single prompts
docker-compose.no-genesis-mtp.yml control variant
docker-compose.minimal.yml no spec-decode
Dual-card:
docker-compose.dual.yml ⭐ fp8 + 262K + MTP + vision, 71/89 TPS
docker-compose.dual-turbo.yml TQ3 + Genesis v7.14 — 4-stream concurrency
docker-compose.dual-dflash.yml DFlash N=5 + 185K + vision — 78/128 TPS
docker-compose.dual-dflash-noviz.yml DFlash + 200K text-only
llama.cpp recipes (under models/qwen3.6-27b/llama-cpp/recipes/):
single-card-default.sh Q4_K_M + 65K
single-card-max-ctx.sh Q4_K_M + q4_0 KV at full 262K — the standout recipe
Old repos remain readable for issue history + external links (Medium,
Reddit, Twitter, Sandermage's PR threads). New issues should be filed
here.
Credits in README. Apache 2.0.
|