fcd85073238a1ffdcee52fa1072c00a85365f64d
32 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
noonghunna
|
2c904d3d40 |
ik-llama: migrate spec-dec flags to --spec-type + digest-pin the image (#401)
The rolling ghcr.io/ikawrakow/ik-llama-cpp:cu13-server tag moved (2026-05-23 -> 2026-06-10, digest 5f914f1c) and the new build REJECTS the legacy speculative-decode flags: "legacy speculative option '--multi-token-prediction' is disabled; use --spec-type mtp:n_max=1,p_min=0.0". Every ik-llama MTP compose crash-restart-loops on a fresh pull (reported by furfix, Discord — single 3090 WSL). The binary's own --help confirms `-mtp`/`--draft-*`/`--spec-stage` are all rejected in favour of the unified `--spec-type SPEC[:k=v,...]`. Flag migration (13 composes): - 11 MTP composes: --multi-token-prediction + --draft-max N + --draft-p-min P -> --spec-type mtp:n_max=N,p_min=P (per-compose N preserved). - 2 two-stage composes: the ngram-mod + mtp cascade used the rejected --spec-stage with legacy keys -> --spec-type ngram-mod:...,ngram_size_n=... + --spec-type mtp:...,p_min=... (spec-ngram-size-n -> ngram_size_n, draft-p-min -> p_min, the canonical keys). Digest-pin (decided to stop the recurrence — this is the 2nd flag churn from the moving tag; ik-llama has no engine profile, so the per-compose image: default is the only pin point): :cu13-server -> @sha256:5f914f1c... in all 13 ik composes + the preflight test fixture + 3 docs (IK_LLAMA / SINGLE_CARD / INFERENCE_ENGINES). VLLM-style central injection isn't available for ik-llama; a follow-up could add an engine profile so future pins are one-line. Live-validated on the pinned image (single 3090): - ik-llama/iq4ks-mtp: healthy, 0 restarts, no legacy error, MTP context ready, served a completion, draft acceptance 0.62 (26/42). - ik-llama/iq4ks-two-stage: healthy, 0 restarts, both stages load (ngram_mod n=16 + MTP context ready), speculative decoding initialized. Full test suite green (test-compose-registry-disk pre-existing on an unrelated untracked model dir). The 3 untracked nex-n2-mini ik composes also carry the rolling tag but are out of scope here (in-progress catalog work). Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> |
||
|
noonghunna
|
eaa7a8c1ad |
docs+scripts: finish <quant>/ path migration across full repo sweep
Repo-wide follow-up to the compose quant-layer move (
|
||
|
noonghunna
|
9821c94efb |
refactor(compose): insert <quant> layer + make the registry the single source of truth
Restructure all 47 composes to models/<model>/<engine>/compose/<topology>/<quant-slug>/<serving>.yml and centralize defaults as registry pointers. Move + path-rewire only — no compose runtime-config changes (plus the +1 ../ depth bump each moved file requires, and descriptive names for the former docker-compose.yml defaults). - <quant-slug> = 1:1 with a weights artifact == compose_registry weights_variant == weights.py key (provider-quant form). Fixes the prior coarse `gguf` and the bf16/int8-files-mislabeled-as-autoround_int4 weights_variant. - default = DEFAULTS[(model,engine,topology)] pointer in compose_registry.py; no default.yml / docker-compose.yml. KV-dtype variants (bf16/int8) stay as files under autoround-int4/. - +1 ../ on every relative mount in moved composes (cache/patches/scripts/models-cache). - Rewired: compose_registry (47 compose_path + weights_variant + DEFAULTS), launch.sh, gpu-mode.sh, bench-row-formatter, profile *.yml, generate_compose.py, weights.py (+ back-compat ALIASES), AGENTS.md, ADDING_MODELS.md, BENCHMARKS + doc path refs. - New guard tests: test-compose-registry-disk.sh, test-compose-mounts-resolve.sh. - Variant keys (vllm/dual, ik-llama/iq4ks-mtp, ...) unchanged — launch/switch UX preserved. Validation: docker compose config 47/47 · parity/launch-compat/registry-disk/ mounts-resolve/profiles-compat/model-weights-registry PASS · kv-calc 22/22 · leak-clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
c7fc9ca638 |
docs: correct single-card llama.cpp/ik_llama ctx 262K -> 200K (shipped default)
The single-card MTP defaults were bumped 262144 -> 200000 (PR #201 + the ik commit) after the "boots != fills" finding: 262K boots and passes verify-stress but only fills ~125K before the FA-scratch-at-fill OOM; 200K fills ~183K with ~1.1 GB margin (see docs/CLIFFS.md). The docs lagged. Corrected to 200K (the shipped, fillable default) across the Q4_K_M-MTP + ik IQ4_KS single-card paths: - README.md (quick-start + TL;DR robustness pitch) - docs/SINGLE_CARD.md (cliff-warning row, config table, llamacpp/default + ik narratives, fallback table, Cliff 2 explainer) - docs/engines/IK_LLAMA.md (defaults + max-context override) - models/qwen3.6-27b/llama-cpp/README.md (single/mtp.yml heading + workhorse line) - scripts/switch.sh (--list/--help comment) Also reconciled an older drift: llamacpp/default = llamacpp/mtp alias (collapsed 2026-05-22), Q4_K_M MTP, no vision, ~52/61 TPS -- several spots still described it as the retired Q3_K_XL @ ~21 TPS + vision. Left for a separate fill-test (smaller-quant / no-MTP headroom may genuinely exceed 200K): the "going-to-262K" max-ctx sections in docs/engines/LLAMA_CPP.md + VLLM.md, and the Q3_K_XL legacy rows in models/qwen3.6-27b/llama-cpp/README.md. Dual-card vLLM 262K, the model's 262144 max_position_embeddings, and the CLIFFS "boots!=fills" explainer are correct as-is and untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
b7353daa52 |
docs: correct ik_llama verdict — ~18-20% FASTER than mainline, not a "tie" (#184)
The 2026-05-22 "ik ties mainline at matched power" verdict was a wrong-engine measurement artifact: its "ik ~50/58" is exactly mainline@370, a number ik produces at NO power setting. A 2026-05-23 set-and-readback power-cap A/B + 5 independent ik runs (all ~70 code) confirm ik IQ4_KS is ~18-20% faster than llamacpp/mtp Q4_K_M on decode TPS at matched 370 W (~60/69 wall vs ~50/58), quality-tied (8-pack 101 vs 100) and ~0.5-0.8 GB leaner — faster AND leaner. Both llama.cpp engines are power-sensitive on this rig (ik -29% / mainline -42% from 370->230 W), so the comparison is only valid at matched power; the original confusion traced to unmatched/mismeasured power + a stale-container mixup. Corrected: IK_LLAMA.md (headline + measured table + bench note), SINGLE_CARD.md (picker row + prose), iq4ks-mtp.yml (Perf header), + new BENCHMARKS ik row. discussions/184 writeup updated separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
c470d9a32d |
docs: correct ik_llama claims to the matched-power tie (#184)
The "+18-20% / ~62-69 TPS" ik figure was the pre-correction power/card artifact; #184 established that at verified matched power ik ties llamacpp/mtp on TPS (~50/58), quality (103 vs 102) and context (both 262K) — its real edge is a ~0.5-0.8 GB leaner footprint. Fixed across IK_LLAMA.md (one-liner + measured table), SINGLE_CARD.md (table row + section), engines/README.md (matrix + recommendation), and the iq4ks-mtp.yml header (also un-stale: it's shipped, not "EVAL ONLY"). Reframes ik as the VRAM-tight pick, not the speed pick. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
412315d9cc |
docs: surface WEIGHTS=gguf + switch.sh ik-llama paths (match #189/#191)
setup.sh now has WEIGHTS=gguf (fetches the GGUF, not the vLLM model) and ik_llama is registered for switch.sh — point the engine docs at both instead of only the manual hf-download / raw-docker-compose paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
28cff3f559 |
refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias) (#181)
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias) Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve + verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no longer earns a separate profile. Collapse the llama.cpp single-card tree to two compose files: - `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh + compose_registry repointed. All references (estate-CLI default, tests, docs) keep resolving; the variant name survives. - Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml. - mtp.yml: document the -ub 512 -> 262K recipe in the header. - mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via -ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline previously left image bounds at the model default). Validated: full-res 2048^2 image @ ~22.2 GB / 24 (~2.4 GB headroom). - switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml link repointed. Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub independently, unlike ik_llama which forces n_ubatch=n_batch). Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note is now disproven (froggeric v19 + --reasoning off works on mainline b9246). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections - Correct stale "llama.cpp MTP via community PR / not merged" across INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged on mainline 2026-05-16 (the b9246 image we bench is post-merge). - Add ik_llama.cpp coverage that was missing: engine list / repo tree / supported-models table (README), workload picker (SINGLE_CARD), choose table (engines/README), model engines line (qwen3.6-27b/README). - Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad, --recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md). - IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall tied) so the ik composes default native; froggeric stays vLLM-only. - New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots + two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf bench pending. Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K self-contradiction in llama-cpp/README, a now-stale "mainline MTP still open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures. External PR links verified resolving; no internal paths leaked. Docs drafted via the Qwen coding agent; reviewed, corrected, and the new compose boot-validated by Claude. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(composes): promote native template default + bump llama-cpp context ceilings llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512 llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024 ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt- mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed) Retire stale artifacts: ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by compose files (canonical path since v0.8.0) 8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise, 6/8 packs within ±1. Delta is cli-40 variance, not a flag effect (-vhad and --recurrent-ckpt-mode are not output-quality levers). * docs(composes): guard -np 1 with hardware-conditional rationale Add inline ⚠ comment to all 5 single-card composes explaining why -np 1 is intentional on a single 24 GB card (compute-bound, not memory-bound — extra slots divide throughput, don't multiply it). Also expose NP as an env var on the two mainline composes (was hardcoded; ik composes already had ${NP:-1}). The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput card or multi-GPU the trade may flip — re-validate before raising.' Prevents future agents from blindly parallelizing slots on Ampere. * fix(composes): move -np guard comment out of folded scalar The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5 single-card composes. YAML folded scalars treat # lines as content, not comments — docker compose config failed on all 5 files. Move the comment block to YAML-level (between volumes: and command:). All 5 now pass 'docker compose -f <file> config'. Lesson learned: always validate compose edits with 'docker compose -f <file> config >/dev/null' before committing. * docs(FAQ): expand WSL2 section with GPU overhead guidance The existing FAQ entry only mentioned TDR and expandable_segments gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead, how it affects each engine path differently (dual-card: noise, single-card vLLM: one env var, single-card llama.cpp: lower ctx), and provide a concrete VRAM budget table showing which composes OOM on WSL2 vs which fit at defaults. Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama composes (IQ4_KS, smaller weights) fit at defaults on WSL2. No compose file changes — this is FAQ docs only. The llama.cpp composes already expose CTX_SIZE as an env var override. --------- Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
554b85b8ea |
docs: add ik_llama engine page + QUANTIZATION primer; surface IQK quants
- docs/engines/IK_LLAMA.md: ik_llama as the advanced-quant track (IQK imatrix, -khad/--merge-qkv/-ser, 262K single-card, MTP, cu13-server image) - docs/QUANTIZATION.md: community quant primer (k-quants -> imatrix i-quants -> IQK; vLLM AWQ/GPTQ/AutoRound; KV-cache quants; engine x quant matrix) - INFERENCE_ENGINES.md: correct stale "no official Docker image" claim + IQK pointer - engines/README.md: ik_llama At-a-glance row + IQ4_KS quant row + per-engine link - README.md: engine list + quant-names pointer Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
b1b851318f |
sglang: park EAGLE-3 path for Qwen3-Next (MTP wins everywhere) (#176)
The SGLang+EAGLE-3 path for Qwen3-Next is parked. Three independent findings, each sufficient on its own: 1. EAGLE-3 is sub-MTP for Qwen3-Next, even on Blackwell where it works. Ex0bit's own published numbers on the PRISM-PRO-DQ model card: native MTP = 121 TPS (1.51x) vs EAGLE-3 chain = 111 TPS (1.39x). The model family has a strong built-in MTP head; routing through an external drafter is structurally slower. 2. CUTE_DSL capture-hang on Ampere. SGLang v0.5.12's get_version() does pkgutil.walk_packages during cuda-graph capture, hits `cutlass.cute.experimental` which raises NotImplementedError under CUDA<13.1, and deadlocks against the locked capture stream. Three patch iterations (pre-import, sys.modules stub at engine init, per-process sys.modules stub at sglang/__init__.py) all failed — the walk re-fires during capture regardless of cache state. Workaround `--disable-cuda-graph` caps decode at ~15-18 TPS. 3. vLLM-MTP-dual already beats this path on the same rig. vllm/dual/turbo.yml delivers ~85 TPS on dual-3090; this SGLang path tops at ~15-18 TPS with the cuda-graph workaround. Changes: - docs/engines/SGLANG.md: status flipped from "experimental, validated to boot" to "PARKED 2026-05-21" with the 3-finding rationale. - models/qwen3.6-27b/sglang/README.md: parking banner, full "Why parked" section, updated TL;DR statuses, expanded re-test triggers list. - compose/dual/eagle3-experimental.yml: PARKED banner above the existing profile header. Internal config unchanged. - compose/single/eagle3-experimental.yml: PARKED banner. Single-card never reached first forward pass (OffloaderV1 tied-weights bug); now both blocked AND structurally inferior. The compose tree + vendored patches stay for archival reference. Re-test triggers if SGLang ships MTP for Qwen3-Next OR upstream lands a CUTE_DSL Ampere fallback. Production paths for Qwen3-Next remain vllm/dual/turbo.yml and llamacpp/mtp.yml — both MTP-based, both validated. Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
941fa0644e |
sglang: experimental EAGLE-3 + Qwen3-Next dual-3090 path (Codex-led patch)
First validated SGLang + EAGLE-3 + Qwen3-Next AutoRound INT4 boot on consumer Ampere. Dual 3090 (TP=2) boots cleanly and serves coherent output with two vendored patches: - patch_sglang_eagle3.py — provided by Ex0bit/Qwen3.6-27B-PRISM-EAGLE3, adds set_eagle3_layers_to_capture hook to Qwen3_5ForConditionalGeneration - patch_sglang_autoround_fused_bf16.py — local fix for the AutoRound + Qwen3-Next loader bug (Codex-led 5h diagnosis). Root cause: SGLang's auto-round loader drops packed_modules_mapping, so fused linear_attn.in_proj_ba routes to GPTQ-Marlin instead of staying BF16. Was previously cached as "same kernel-pad bug as vLLM PR #40361" — wrong layer; this is name-mapping. Composes: - compose/single/eagle3-experimental.yml — boots past Marlin but blocked on SGLang OffloaderV1 tied-weights bug; kept as reference - compose/dual/eagle3-experimental.yml — boots + serves; TPS/accept-rate pending prolonged bench Critical Ampere caveats baked into the dual compose: - --disable-cuda-graph (CUTLASS CUTE capture hangs on Ampere) - --speculative-draft-model-quantization unquant (BF16 drafter must opt out of target's INT4 quant) - --disable-custom-all-reduce (PCIe-only, per CLAUDE.md hardware truth) - --kv-cache-dtype fp8_e5m2 (smallest practical KV on Ampere; SGLang's fp4_e2m1 falls back to slow un-fused dequant) Docs rewritten: - docs/engines/SGLANG.md — engine-level pros/cons + KV cache options + Ampere quirks (previously "blocked, watch list") - models/qwen3.6-27b/sglang/README.md — quick recipe + validated vs not-yet-validated breakdown (previously "re-test pending") Status: ⚠️ EXPERIMENTAL. Boots + serves coherent output verified 2026-05-20. TPS / accept-rate / quality 8-pack / verify-stress / soak pending the next testing session. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
28bd0e8970 | docs: fix stale NVLINK_MODE comment, INTERNALS.md cliff status, and dead companion repo link | ||
|
noonghunna
|
fbf343129c |
docs: use \$MODEL_DIR placeholder, not the dev rig's /mnt/models/huggingface/
User docs were hardcoding the dev rig path (/mnt/models/huggingface/...) as
if it was canonical. It's not — cross-rig users have models at /data/models,
~/models, /mnt/nvme/llms, etc. Setting MODEL_DIR per their setup is the
intended UX (the compose already supports it via env-var default).
Updates:
- models/qwen3.6-27b/llama-cpp/README.md: download examples now use
\$MODEL_DIR/qwen3.6-27b-gguf/ instead of /mnt/models/huggingface/...
- models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml: header
comment uses \$MODEL_DIR/qwen3.6-27b-gguf/ for download examples + says
"MODEL_DIR=/your/models/dir docker compose up -d" instead of our path.
- docs/engines/LLAMA_CPP.md: same treatment + cleaned up Qwen3.5 + DFlash
draft path examples to also use \$MODEL_DIR.
- scripts/preflight.sh: hf download hint shows literal \${MODEL_DIR} so user
knows what to set, plus explicit "set MODEL_DIR first" line. Previously
echoed the resolved relative path (../../../../models-cache) which lands
outside the repo if pwd isn't the compose dir.
Caught by RobH589 in club-3090#116 — they hit the path-resolved-to-root-of-drive
case from the relative-path default. Closes the doc UX side; the compose's
env-var override mechanism was already correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
noonghunna
|
00366a58d7 |
reorg: services/ consolidation + gpu-mode under git + ComfyUI + pin tracker + path updates
Some checks failed
Release / release (push) Failing after 53s
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
noonghunna
|
acd7ffb67c |
restructure: promote topology to a directory level (single/dual/multi4)
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`, with topology as a folder rather than a filename prefix. Solves all 7 inconsistencies surfaced in the post-rename audit (single-card composes without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity, fine-tunes encoding model name in filename, etc.) by making the directory hierarchy enforce the convention. Layout: models/<model>/<engine>/compose/<topology>/<feature>.yml Where: - <model>: qwen3.6-27b, gemma-4-31b - <engine>: vllm, llama-cpp, sglang - <topology>: single, dual, multi3, multi4, multi8 - <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc. Each topology subdir has a `docker-compose.yml` for the recommended starter — bare `cd <topology> && docker compose up` works because docker compose finds that filename automatically. Variants drop the `docker-compose.` prefix since they're invoked via `-f` flag. 27 compose file moves total: - 18 Qwen vLLM composes redistributed across single/dual/multi4 - 2 Qwen llama-cpp composes into single/ - 6 Gemma vLLM composes redistributed across single/dual - 1 untracked qwopus-bf16mtp moved to dual/ Inside each compose: relative paths to `../patches/` and `../cache/` bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache` to `../../../../../models-cache` (one extra `..` for the new depth). Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs, sibling-table cross-references in compose headers, scripts, patch READMEs, .github issue templates, tools/residency-instrument). scripts/switch.sh VARIANTS map updated; tags themselves unchanged (`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`, `vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.). AGENTS.md "Compose layout" section rewritten to describe the new hierarchy, with concrete examples and the fine-tune exception (`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename prefix until the fine-tune graduates to its own model directory). All switch.sh paths verified to resolve to actual files post-move. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
a7a1d591d5 |
docs(hardware): reconcile 230W vs 290W vs 330W sweet-spot story
The "230W is the sweet spot" lore was stale — it traces to coarse 3-cap-resolution data. Dense 10W-resolution sweeps on this rig now show: - 290W: actual air-cooled decode sweet spot (0.111 TPS/W) - 330W: water-cooled sweet spot (per @syangsao 3-cap data) - 230W: NOT a sweet spot — costs ~16% efficiency vs 290W; just a low-power cap Add decode-concurrent N=4 row to cross-rig table (290W also peaks here on this rig, matching decode-single — concurrency doesn't move the knee). Add per-workload-class table showing decode at 290W vs prefill at 250W on the same card. Update vLLM engine doc to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
2fe017f88d |
docs(power): revise default cap 230W → 330W per @syangsao cross-rig data
@syangsao's three-point sweep (230W/330W/388W stock) on 1× water-cooled 3090 + llama.cpp + Qwen3.6 27B Q3_K_XL revealed 230W costs ~34% TPS (25 vs 38) on this engine path — far larger than the "<10%" framing in prior docs. The chunked_gated_delta_rule kernel is genuinely compute-bound on GDN-attention models, so power cap throttles SM clocks ~linearly. 330W is the actual sweet spot: peak TPS/W efficiency, only ~5% TPS loss vs 388W stock, and 388W is *less* efficient than 330W on this kernel mix. Updates docs/HARDWARE.md power section with the cross-rig data table and flips the recommended default. docs/engines/VLLM.md gets a smaller note flagging the engine-specific difference. Source: https://github.com/noonghunna/club-3090/issues/58#issuecomment-4388766174 |
||
|
noonghunna
|
dec0f22dac |
docs(lucebox): record PRs #78 + #80 — dual-GPU PFlash + DFlash split shipped (May 2026)
Two @weicj PRs merged that change the lucebox-hub serving topology: - PR #78 (PFlash phase-split, merged 2026-05-02) — --pflash-gpu flag, persistent pflash_daemon. Validation: passing NIAH source ctx 24K → 262K (10.7× over single-card co-resident) on dual RTX 2080 Ti 22 GB. - PR #80 (DFlash target/draft split, merged 2026-05-04) — --target-gpu / --draft-gpu flags. Validation: 51.86 tok/s HE 10-prompt, AL 7.09, 44.3% accept on Qwen3.5-27B Q4 target + z-lab DFlash draft. This is heterogeneous spec-decode (each model on its own card), not weight-sharded TP. Removes the single-card co-residency limit that was the binding blocker for 2× 3090 users (target + draft + KV all competing for 24 GB → 65K max_ctx ceiling). Updated: - docs/UPSTREAM.md — Luce DFlash section gains a "🆕 Dual-GPU split landed" subsection with both PR links + @weicj's measured numbers. PFlash row status icon flipped from 🟡 to 🟢; "Re-evaluate" criteria reworked to focus on reproducing the 262K NIAH claim on 2× 3090. - docs/engines/LLAMA_CPP.md — added "🆕 Dual-GPU split" subsection under the existing DFlash recipe with the new flag-based recipe and carry-over caveat (Qwen3.6-27B draft still under training; the benefit applies primarily to Qwen3.5-27B + DFlash today). Memory updates (gitignored, not in this commit): - pflash_future_exploration.md — type=project, status flipped from "co-residency blocker" to "co-residency blocker addressed via dual-GPU; bench task #229 queued" - pflash_x_bounded_thinking_intersection.md — added 2026-05-04 update noting the new dual-GPU path and that the parked exploration is more concrete now Bench tracked at task #229 (queued, not executed yet — these PRs are hours old as of this commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
ecc2d747ae |
docs(sglang): refresh per-engine + comparison pages — DFlash + MTP native upstream as of May 2026
The historical "currently blocked" framing is partially out-of-date. SGLang upstream has moved since we last tested: - DFlash spec-decode: native, recent (z-lab confirmed --speculative-algorithm DFLASH; Qwen3.6-27B draft published at z-lab/Qwen3.6-27B-DFlash) - MTP: native, first-class for Qwen3-Next family per LMSYS Jul 2025 blog - TurboQuant: WIP only (Issue #21618, not merged) - Marlin pad-sub-tile-n fix (the binding INT4 + TP=2 boot blocker): status unknown — needs re-test on current SGLang main Updated: - models/qwen3.6-27b/sglang/README.md — full rewrite. New TL;DR table with per-feature status + a 4-step re-test plan: (1) smoke-boot AutoRound INT4 + TP=2 with fp8/q4 KV (NOT TurboQuant — WIP), (2) verify-stress 7/7 if boots, (3) add DFlash spec-decode (preferred over MTP — higher accept rate + z-lab actively maintains SGLang integration), (4) ship as docker-compose.dual.yml if competitive vs vllm dual-dflash.yml. Watch list now anchored to specific upstream issues. - docs/engines/README.md — comparison table row + cons section + "How to choose" entry refreshed to reflect "re-test pending" not "blocked." Specific re-test steps inlined; full plan cross-referenced to the per-engine page. Status of physical work: re-test queued as task #227, not yet executed. The decision tree in the README will let any contributor with current SGLang main attempt the boot. If a cross-rig contributor reports a clean boot, that's the green light to bench properly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
e9c658cbc6 |
fix(docs): replace dead luce-spec/llama-cpp-dflash links with Luce-Org/lucebox-hub
Reported in #39 by @clort81 — the `luce-spec/llama-cpp-dflash` repo returns 404. The DFlash work consolidated into Luce-Org/lucebox-hub (verified: github.com/Luce-Org/lucebox-hub returns 200, contains dflash/ + pflash/ subdirs and dflash/deps/llama.cpp submodule). Affected files: - docs/engines/README.md (2 link sites in comparison table) - docs/engines/LLAMA_CPP.md (4 sites: intro, "Pros" table, build clone command, "See also" links) - models/qwen3.6-27b/llama-cpp/README.md (2 link sites) Plus collateral updates: - Build clone path /opt/llama-cpp-dflash → /opt/lucebox-hub (matches the new repo name; was a 3-replace via path globbing) - HF model path luce-spec/dflash-qwen3.6-27b-N5 (401 gated) → z-lab/Qwen3.6-27B-DFlash (200 public, the actually-shipping draft) + local-dir adjusted to /mnt/models/huggingface/z-lab/... matching the canonical HF model path convention - `git clone --recurse-submodules` flag added since lucebox-hub uses submodules for its bundled llama.cpp fork (in dflash/deps/llama.cpp) Updates URL framing in user-facing prose to acknowledge that lucebox-hub is a separate harness containing a llama.cpp fork rather than just being a llama.cpp fork. The recipe section build commands should be re-verified against the lucebox-hub README before treating them as canonical — this commit only updates the URL/path; the multi- step build instructions in docs/engines/LLAMA_CPP.md may need a follow-up walkthrough. CHANGELOG references to luce-spec preserved as historical context (the links were valid at the time the CHANGELOG entries were written). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
f8c9c365e0 |
docs: full sync to v7.69 + Cliff 2 60K closure recipes
Sweep all stale v7.66 / fc89395 substrate references to v7.69 (commit 2db18df) + local vllm#35975 inputs_embeds backport. Ship the Balanced MTP (long-text.yml, 180K + 0.93) and Max-context (long-text-no-mtp.yml, 200K + 0.95, no MTP) variants as the Cliff 2 closure recipes — both PASS the 60K single-prompt envelope (623s and 537s wall respectively). Updates: - CHANGELOGs (root + model) — new v7.69 PM entry above v7.66 - README + SINGLE_CARD + HARDWARE + EXAMPLES + FAQ + INTERNALS + VLLM engine doc — Cliff 2 status, substrate pins, mem-util defaults, variant table, sidecar list - vllm/README.md compose menu refreshed for the new ctx envelopes - model README patch surface table — added PN30 part3, PN32, P103, PN34 rows; collapsed P98 reference to PN34 env-gate - tools/charts/gen-perf.py + gen-vram.py — substrate label bumped to v7.69 + #35975, panel labels for the long-text variants updated, long-text-no-mtp 200K Max-context noted as bench-pending in chart - All performance + VRAM charts (svg + png) regenerated Cliff 2 60K closure: Genesis v7.69 (PN32 GDN chunked-prefill + P103 worker self-install + PN30 part3 + PN34 workspace_lock relax) plus local backport of vllm#35975 (~444 MiB freed on text-only paths). 3 sidecars dropped on long-text variants; 2 sidecars retained on master (patch_inputs_embeds_optional.py, patch_tolist_cudagraph.py). >60K single-prompt still hits the 24 GB hardware-physical wall on single-card. For those: dual-card TP=2 (verified at 237K) or llama.cpp single-card (262K, different engine). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
ae4846fd63 |
docs + charts: v7.66 + Cliff 1 mech B closed across all 4 TQ3 composes
Phase 3 of the v7.66 migration: documents the new state, regenerates performance/VRAM charts, posts cross-rig data to Sander on discussion #19 + issues #15/#16/#17. What changed ------------ **docs/SINGLE_CARD.md** - Updated TL;DR table: long-text 180K + 0.95, long-vision 145K + 0.95, bounded-thinking 180K + 0.95. - Removed Cliff 1 mech B "limitation to know" — now closed. - Added "What was Cliff 1 mech B (now closed) ✅" historical note. - Updated activation budget rationale to reflect PN12+PN25 pool residence. **docs/DUAL_CARD.md** - Bench protocol substrate: Genesis v7.65 → v7.66 dev tip. **docs/CLIFFS.md** - "vLLM pin compatibility status" rewritten for v7.66 + Cliff 1 mech B closure. - Replaced "v0.20 unblock" prose with "Cliff 1 mech B — what closed it" section explaining the two compounding fixes (PN25 v3 + PN30 dst-shaped). - Updated Genesis patches table with PN30 + PN33 + Sander v7.66 PN25. - Added "Local sidecars retained on master" table — 4 sidecars, why each one is still needed. - Updated "Validation across all 4 TQ3 variants" to 2026-05-02 numbers (180K / 145K / 180K / 262K — all 6/7 probes pass). **docs/UPSTREAM.md** - Genesis issue tracker updated with v7.66 cross-rig findings: - #16 PN25: both v7.65 and v7.66 mechanisms fail on TP=1 - #17 PN30: layout-correctness diagnosis + our corrected fix - #15 PN31: doesn't fit on 24 GB (lower mem-util sufficient) - PN33 partial (boot-time closes, runtime decode still fires) **docs/engines/VLLM.md, README.md, model README** - Genesis pin references bumped d89a089 → fc89395. **models/qwen3.6-27b/CHANGELOG.md** - New entry: "2026-05-02 — Genesis v7.66 + Cliff 1 mech B closed ⭐" with full validation matrix, sidecar inventory, and links to per-config result summaries. **tools/charts/gen-perf.py + gen-vram.py** - Substrate footnote: v7.65 dev (d89a089) → v7.66 dev (fc89395) - Compose ctx labels: long-text 214K → 180K, long-vision 198K → 145K, bounded-thinking 214K → 180K, mem-util 0.985 → 0.95 - Regenerated all 14 chart files (performance + vram, single + dual + combined). Cross-rig data posted to Sander ------------------------------- - [discussion #19 reply](https://github.com/noonghunna/club-3090/discussions/19#discussioncomment-16785590) — comprehensive update covering all 4 patches (PN33 partial, PN25 still TP=1-broken, PN30 layout diagnosis + corrected fix, PN31 still 24 GB-incompatible) - [genesis-vllm-patches#16 update](https://github.com/Sandermage/genesis-vllm-patches/issues/16#issuecomment-4362835267) — v7.66 mechanism still TP=1-broken - [genesis-vllm-patches#17 update](https://github.com/Sandermage/genesis-vllm-patches/issues/17#issuecomment-4362836196) — PN30 layout-correctness diagnosis + corrected fix offered - [genesis-vllm-patches#15 update](https://github.com/Sandermage/genesis-vllm-patches/issues/15#issuecomment-4362836881) — PN31 cross-rig confirmation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Codex CLI (ChatGPT) <noreply@openai.com> |
||
|
noonghunna
|
5aa97a25d9 |
v0.20 migration + Genesis v7.65 dev tip + cold-start cache + env-var alignment
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
|
||
|
noonghunna
|
df91d641c4 |
push long-text/bounded-thinking back to 185K + 0.975; long-vision stays 140K + 0.95
After
|
||
|
noonghunna
|
383b5cc381 |
long-text/long-vision/bounded-thinking: middle-ground recovery 130K → 175K / 120K → 140K
After
|
||
|
noonghunna
|
d803278ebc |
docs + bounded-thinking: roll new context defaults across user-facing surfaces
Following
|
||
|
noonghunna
|
427d2f8aa9 |
docs+scripts+charts: propagate new ceilings (long-vision 198K, long-text 218K)
Sweep across all user-facing docs reflecting the post-PN12-anchor-fix ceilings established in |
||
|
noonghunna
|
17aff4ce05 |
LLAMA_CPP.md: add structural explanation of why prefill cliffs don't fire
User asked the obvious question: vLLM at 192K hits Cliff 1 on 25K tool prefills, but llama.cpp at 262K processes the same message cleanly — why? Three structural reasons documented: 1. ggml-cuda attention has no max_seqlen parameter; FA2 does 2. Static KV slab + dynamic workspace vs paged + varlen pre-alloc 3. Cudagraph capture is decode-only; no path for cap-leak Plus Cliff 2 doesn't fire because llama.cpp's Qwen3-Next GDN implementation uses online state updates instead of materializing the chunk_gated_delta_rule O(seq_len * chunk_size) intermediate. Reframes the 3-4× TPS gap as the necessary trade for batched worst-case-workspace optimization vs dynamic-shape per-call serving. This is the architectural defense of the two-routes launch frame. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
f4a28b19eb |
Remove no-genesis-mtp.yml (research artifact, not user-facing)
This was a control variant used internally to A/B-test whether MTP worked without Genesis (it does, on fp8+MTP paths). No reason for end users to pick it over tools-text.yml (fp8+MTP+Genesis fixes+75K, strictly better) or minimal.yml (no Genesis at all, simplest). Wizard already didn't surface it. switch.sh map, sibling compose "see also" tables, patches/README, engines/VLLM.md all updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
37a4895f6d |
Remove fast-chat.yml; extend P68/P69 disable to default
fast-chat (20K, fp8, vision) and default docker-compose.yml (48K, TQ3, vision) had effectively the same TPS post-PN8. fast-chat's only remaining differentiator was "smaller context = ~3s faster boot," and 20K is actively bad for IDE-agent users (Copilot tool-schema preamble alone hits 20K). Net negative — removed. Default compose was missed in the previous P68/P69 fix — it had the same env vars enabled and the same silent-stop bug above 8000 chars. Both now disabled with the same explanatory comment. Updated: - scripts/switch.sh, scripts/launch.sh — drop the variant - docs/SINGLE_CARD.md, FAQ.md, engines/VLLM.md, model + vllm + patches READMEs — references removed or pointed to default/tools-text - All sibling compose YAML "see also" tables — fast-chat row removed, tools-text row repurposed for IDE-agent guidance - CHANGELOG entry; old historical entries kept as-is (append-only) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
26ac8118de |
Restructure docs around hardware axis: SINGLE_CARD.md + DUAL_CARD.md
User feedback: navigating to relevant docs was cumbersome. The natural first decision is "1 GPU or 2 GPUs?" and the existing docs mixed model- specific reference with deployment guidance. New navigation: - README.md adds a "Pick your path" pivot pointing at hardware-axis pages - docs/SINGLE_CARD.md — 1× 3090 deployment menu (workload → compose → TPS) with all single-card configs (vLLM + llama.cpp), VRAM budget, prefill cliffs explained operationally, what single-card can't do - docs/DUAL_CARD.md — 2× 3090 mirror (4 dual variants + TP=2 explainer + what dual unlocks vs single + Marlin pad fork dependency) Slimming: - models/qwen3.6-27b/README.md: dropped duplicated variant tables (now in GPU-count pages); kept model-specific content (quants, Genesis patch surface table, what's working / not, VRAM diagram) - models/qwen3.6-27b/USE_CASES.md: deleted. Per-workload content absorbed into the GPU-count pages (deduplicated). Troubleshooting list moved to docs/FAQ.md as a new "Troubleshooting" subsection. Image-token cost / vision specifics absorbed into SINGLE_CARD.md. Reference updates: 8 files updated (engines/VLLM.md, engines/README.md, COMPARISONS.md, ARCHITECTURE.md, EXAMPLES.md, FAQ.md, INTERNALS.md, top-level README) — all USE_CASES.md links re-pointed to SINGLE_CARD/ DUAL_CARD where appropriate. Net delta: -167 lines (was 197 in USE_CASES + duplicated tables in model README; now 342 lines split between SINGLE_CARD + DUAL_CARD with content deduplicated against each other). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
3fa33332ce |
Initial commit — club-3090: model-agnostic LLM serving recipes for RTX 3090
Consolidates and supersedes:
- noonghunna/qwen36-27b-single-3090
- noonghunna/qwen36-dual-3090
The two predecessor repos partitioned by card count (1× vs 2×). This
repo partitions by engine instead, which matches how users actually
decide ("vLLM or llama.cpp?" before "1 card or 2"). Card count becomes
a config variant within each engine.
Structure (model-agnostic from day 1):
docs/ cross-model engine + hardware docs
engines/ vLLM / llama.cpp / SGLang comparison + per-engine deep dives
HARDWARE.md Ampere SM 8.6+, NVLink, power, VRAM ceilings
GLOSSARY.md plain-language definitions
img/ illustrations (vram-budget.svg)
ARCHITECTURE.md how this stack thinks about LLM serving on 24 GB
models/<model-name>/ everything specific to a model
qwen3.6-27b/ today's only model
README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md
vllm/ vLLM-specific configs for this model
compose/ docker-compose files (single + dual variants)
patches/ tolist_cudagraph + Marlin pad notes
llama-cpp/ llama.cpp recipes for this model
recipes/ shell scripts (single-card default + 262K max-ctx)
sglang/ SGLang status (currently blocked)
scripts/ shared, model-aware
setup.sh bash setup.sh <model> → downloads + verifies
verify.sh / verify-full.sh smoke + functional tests
bench.sh canonical TPS bench
vLLM compose variants (all under models/qwen3.6-27b/vllm/compose/):
Single-card:
docker-compose.yml ⭐ DEFAULT — TQ3 + Genesis P65, 48K, 51/68 TPS
docker-compose.fast-chat.yml fp8 + 20K, 55/70 TPS — fastest at small ctx
docker-compose.tools-text.yml fp8 + 75K, 53/70 TPS — best for long single prompts
docker-compose.no-genesis-mtp.yml control variant
docker-compose.minimal.yml no spec-decode
Dual-card:
docker-compose.dual.yml ⭐ fp8 + 262K + MTP + vision, 71/89 TPS
docker-compose.dual-turbo.yml TQ3 + Genesis v7.14 — 4-stream concurrency
docker-compose.dual-dflash.yml DFlash N=5 + 185K + vision — 78/128 TPS
docker-compose.dual-dflash-noviz.yml DFlash + 200K text-only
llama.cpp recipes (under models/qwen3.6-27b/llama-cpp/recipes/):
single-card-default.sh Q4_K_M + 65K
single-card-max-ctx.sh Q4_K_M + q4_0 KV at full 262K — the standout recipe
Old repos remain readable for issue history + external links (Medium,
Reddit, Twitter, Sandermage's PR threads). New issues should be filed
here.
Credits in README. Apache 2.0.
|