master
1355 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
noonghunna
|
3e1f5f61c0 |
dual-turbo: switch kv-cache-dtype k8v4 → 3bit_nc to align with test findings
The predecessor dual-3090 README documented `turboquant_3bit_nc` for the
turbo variant, but the actual compose shipped `turboquant_k8v4` (likely
config drift that was never reconciled). Earlier this session I kept the
file as-is and updated the doc to match — but that was the wrong call.
What our test findings actually say:
- Single-card v714 default ships turboquant_3bit_nc, validated extensively
(51 narr / 68 code TPS, 10/10 verify-full.sh checks pass)
- Memory entry: "Lorbus Qwen3.6-27B + MTP + TurboQuant = 85 TPS / 125K
single-card ⭐" — TQ3, the same _3bit_nc preset
- Predecessor dual-3090 README claimed 3bit_nc for turbo too
So 3bit_nc IS the tested config across the project; k8v4 was the drift.
Aligning the dual-turbo file with that. Trade-offs:
- TQ3 is 3 bits/token avg (vs k8v4's ~6 bits/token avg) — smaller KV per
token → larger KV pool capacity at same mem-util
- The published "4.59× concurrency at 262K" claim should hold at MINIMUM,
likely improve modestly (more KV pool → more streams or longer tail)
- Per-stream TPS should be roughly similar (KV bandwidth and compute paths
are similar between the two TurboQuant variants)
Risk: I haven't booted dual-card to confirm. The change is internally
consistent with single-card test data + predecessor docs. Cross-rig users
will validate via verify-full.sh on actual dual hardware.
Updated dual.yml's variant-matrix table accordingly (TQ k8v4 → TQ3).
|
||
|
noonghunna
|
0f33561b6b |
Audit + reconcile dual-card compose headers, patches README, setup output
After the user flagged "are you validating all composer files" — ran a
full dry-run audit of all 9 composes via docker compose config, extracted
key flags (TP, max_len, mem_util, KV dtype, spec-decode), and found
several doc-vs-code mismatches inherited from the predecessor repos.
Compose header fixes:
- docker-compose.dual.yml — header described it as inheriting from
"single-card project's default", said "fp8 is plenty for 64K"
(stale — file actually does 262K). Updated to reflect: this IS the
dual-card default, fp8 is plenty for full 262K, plus a variant matrix
showing all 4 dual files with their actual TPS / streams / KV / vision.
- docker-compose.dual-turbo.yml — header claimed kv-cache-dtype was
`turboquant_3bit_nc` but the file actually ships `turboquant_k8v4`.
This mismatch was in the predecessor too; we kept the file (not the
header) since k8v4 is what was tested. Updated header to reflect
reality + noted the predecessor doc claim for archaeology.
- docker-compose.dual-dflash.yml — header said max_model_len "drops
from 262K to 16K" (stale dev-cycle comment); actual is 185K. Fixed.
Also added: KV cache is FP16 (DFlash + head_size=256 + non-causal
has no fp8/turbo Ampere backend), the bfloat16 dtype workaround for
vllm#40334, and clear positioning vs the noviz variant.
- docker-compose.dual-dflash-noviz.yml — minor: file path in "to run"
pointed at the old compose/ dir; updated to new layout path.
patches/README.md — was framed as dual-card-only ("we don't run
Genesis here") but the patches dir is now shared across single and
dual variants. Rewrote with a per-patch + per-variant matrix:
- patch_tolist_cudagraph.py: single-default + dual-turbo
- patch_pr40798_workspace.py: research artifact, no compose mounts
- genesis/: single-default + tools-text + dual-turbo
- Marlin pad fork (external /opt/ai/vllm-src/): all 4 dual composes
Added a Genesis env-opts table showing per-patch toggles and which
composes enable each.
scripts/setup.sh — final-output Next-steps block referenced the OLD
relative path `cd compose && docker compose up -d`, which would fail
in the new layout. Updated to:
cd models/<model>/vllm/compose && docker compose up -d
Plus added a clear note about the Marlin pad fork dependency for
dual-card composes (with the git-clone command users need to run
once before booting any dual-card variant).
YAML validation: `docker compose config` passes for all 9 composes
with MODEL_DIR set. Volume paths resolve, env vars substitute, no
syntax errors. Single-card default smoke-tested earlier (10/10
verify-full.sh checks pass); dual-card composes pass YAML validation
but require a 2× 3090 rig to actually boot — left for cross-rig users
to confirm.
|
||
|
noonghunna
|
7f00e52140 |
Pin Genesis version + fix MODEL_DIR defaults + clean stale headers
Three related fixes for the post-restructure layout to actually work: 1. Pin Genesis to a tested tag (addresses walmis #8) - setup.sh now does `git clone --branch v7.51-stable-2026-04-27 --depth 1` instead of plain `git clone` (= latest HEAD). Re-runs `git checkout` on the pinned tag if the dir already exists. - GENESIS_PIN env var lets users opt into a different tag/commit. - Sanity-check the v7.14 layout (vllm/_genesis package) and bail with a clear error if missing, rather than silently shipping a broken compose-genesis combination. 2. MODEL_DIR default in all 9 composes (smoke-test fix) - Old default was ${MODEL_DIR:-../models}, which from the new compose dir at models/qwen3.6-27b/vllm/compose/ resolved to a non-existent path. Composes silently created an empty mount target → vLLM couldn't find the model on first boot. - Updated all 9 composes (single + dual variants) to: ${MODEL_DIR:-../../../../models-cache} This resolves to repo-root/models-cache/ which is exactly where setup.sh now downloads. Booting works zero-arg if you ran setup.sh. - Users with model weights elsewhere can still set MODEL_DIR via env. - Validated: from the new paths, MODEL_DIR=/mnt/models/huggingface docker compose up -d boots cleanly and verify-full.sh passes all 10 checks. 3. Clean stale header comments - tools-text.yml: header still self-described as alternate to old "20K default" + referenced deleted longctx-experimental.yml. Updated to current variant matrix (default 48K, this 75K text-only). - minimal.yml: similar — "20K default" + longctx-experimental refs. Updated. - fast-chat.yml: already fixed in previous commit. Smoke test: verify-full.sh from the new club-3090 paths passes 10/10 (including #4 tool calling, #8 tool-response prefill OOM, #10 MTP AL). |
||
|
noonghunna
|
2511a98110 |
Fix .gitignore + add the entire models/ tree (initial commit was incomplete)
The old repos used models/ as the model-weights download directory, so their .gitignore excluded it. We copied that .gitignore into club-3090 without updating, which silently dropped the entire models/ subtree from the initial commit on GitHub (everything still on disk locally, just not tracked). Fixes: - .gitignore: remove models/, add models-cache/ (the new default for weights download). Update genesis patches path to the new location models/<model>/vllm/patches/genesis/. Update compose-state ignores to use **/compose/ glob since composes are now nested. Add vllm-src/ for the dual-card Marlin pad fork mount. - Add the missing models/qwen3.6-27b/ tree: - README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md - vllm/README.md + 9 docker-compose.yml variants + patches dir - llama-cpp/README.md + 2 launch recipes - sglang/README.md (currently blocked status) Also fixes: stale comment header in fast-chat.yml that referenced the deleted longctx-experimental.yml and self-described as "default" when it's the chat-only variant. |
||
|
noonghunna
|
3fa33332ce |
Initial commit — club-3090: model-agnostic LLM serving recipes for RTX 3090
Consolidates and supersedes:
- noonghunna/qwen36-27b-single-3090
- noonghunna/qwen36-dual-3090
The two predecessor repos partitioned by card count (1× vs 2×). This
repo partitions by engine instead, which matches how users actually
decide ("vLLM or llama.cpp?" before "1 card or 2"). Card count becomes
a config variant within each engine.
Structure (model-agnostic from day 1):
docs/ cross-model engine + hardware docs
engines/ vLLM / llama.cpp / SGLang comparison + per-engine deep dives
HARDWARE.md Ampere SM 8.6+, NVLink, power, VRAM ceilings
GLOSSARY.md plain-language definitions
img/ illustrations (vram-budget.svg)
ARCHITECTURE.md how this stack thinks about LLM serving on 24 GB
models/<model-name>/ everything specific to a model
qwen3.6-27b/ today's only model
README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md
vllm/ vLLM-specific configs for this model
compose/ docker-compose files (single + dual variants)
patches/ tolist_cudagraph + Marlin pad notes
llama-cpp/ llama.cpp recipes for this model
recipes/ shell scripts (single-card default + 262K max-ctx)
sglang/ SGLang status (currently blocked)
scripts/ shared, model-aware
setup.sh bash setup.sh <model> → downloads + verifies
verify.sh / verify-full.sh smoke + functional tests
bench.sh canonical TPS bench
vLLM compose variants (all under models/qwen3.6-27b/vllm/compose/):
Single-card:
docker-compose.yml ⭐ DEFAULT — TQ3 + Genesis P65, 48K, 51/68 TPS
docker-compose.fast-chat.yml fp8 + 20K, 55/70 TPS — fastest at small ctx
docker-compose.tools-text.yml fp8 + 75K, 53/70 TPS — best for long single prompts
docker-compose.no-genesis-mtp.yml control variant
docker-compose.minimal.yml no spec-decode
Dual-card:
docker-compose.dual.yml ⭐ fp8 + 262K + MTP + vision, 71/89 TPS
docker-compose.dual-turbo.yml TQ3 + Genesis v7.14 — 4-stream concurrency
docker-compose.dual-dflash.yml DFlash N=5 + 185K + vision — 78/128 TPS
docker-compose.dual-dflash-noviz.yml DFlash + 200K text-only
llama.cpp recipes (under models/qwen3.6-27b/llama-cpp/recipes/):
single-card-default.sh Q4_K_M + 65K
single-card-max-ctx.sh Q4_K_M + q4_0 KV at full 262K — the standout recipe
Old repos remain readable for issue history + external links (Medium,
Reddit, Twitter, Sandermage's PR threads). New issues should be filed
here.
Credits in README. Apache 2.0.
|