722f998ff31100a9bee2dfd5484e7f435b24b1cd
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 noonghunnaandClaude Opus 4.7
|
ecc2d747ae |
docs(sglang): refresh per-engine + comparison pages — DFlash + MTP native upstream as of May 2026
The historical "currently blocked" framing is partially out-of-date. SGLang upstream has moved since we last tested: - DFlash spec-decode: native, recent (z-lab confirmed --speculative-algorithm DFLASH; Qwen3.6-27B draft published at z-lab/Qwen3.6-27B-DFlash) - MTP: native, first-class for Qwen3-Next family per LMSYS Jul 2025 blog - TurboQuant: WIP only (Issue #21618, not merged) - Marlin pad-sub-tile-n fix (the binding INT4 + TP=2 boot blocker): status unknown — needs re-test on current SGLang main Updated: - models/qwen3.6-27b/sglang/README.md — full rewrite. New TL;DR table with per-feature status + a 4-step re-test plan: (1) smoke-boot AutoRound INT4 + TP=2 with fp8/q4 KV (NOT TurboQuant — WIP), (2) verify-stress 7/7 if boots, (3) add DFlash spec-decode (preferred over MTP — higher accept rate + z-lab actively maintains SGLang integration), (4) ship as docker-compose.dual.yml if competitive vs vllm dual-dflash.yml. Watch list now anchored to specific upstream issues. - docs/engines/README.md — comparison table row + cons section + "How to choose" entry refreshed to reflect "re-test pending" not "blocked." Specific re-test steps inlined; full plan cross-referenced to the per-engine page. Status of physical work: re-test queued as task #227, not yet executed. The decision tree in the README will let any contributor with current SGLang main attempt the boot. If a cross-rig contributor reports a clean boot, that's the green light to bench properly. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
e9c658cbc6 |
fix(docs): replace dead luce-spec/llama-cpp-dflash links with Luce-Org/lucebox-hub
Reported in #39 by @clort81 — the `luce-spec/llama-cpp-dflash` repo returns 404. The DFlash work consolidated into Luce-Org/lucebox-hub (verified: github.com/Luce-Org/lucebox-hub returns 200, contains dflash/ + pflash/ subdirs and dflash/deps/llama.cpp submodule). Affected files: - docs/engines/README.md (2 link sites in comparison table) - docs/engines/LLAMA_CPP.md (4 sites: intro, "Pros" table, build clone command, "See also" links) - models/qwen3.6-27b/llama-cpp/README.md (2 link sites) Plus collateral updates: - Build clone path /opt/llama-cpp-dflash → /opt/lucebox-hub (matches the new repo name; was a 3-replace via path globbing) - HF model path luce-spec/dflash-qwen3.6-27b-N5 (401 gated) → z-lab/Qwen3.6-27B-DFlash (200 public, the actually-shipping draft) + local-dir adjusted to /mnt/models/huggingface/z-lab/... matching the canonical HF model path convention - `git clone --recurse-submodules` flag added since lucebox-hub uses submodules for its bundled llama.cpp fork (in dflash/deps/llama.cpp) Updates URL framing in user-facing prose to acknowledge that lucebox-hub is a separate harness containing a llama.cpp fork rather than just being a llama.cpp fork. The recipe section build commands should be re-verified against the lucebox-hub README before treating them as canonical — this commit only updates the URL/path; the multi- step build instructions in docs/engines/LLAMA_CPP.md may need a follow-up walkthrough. CHANGELOG references to luce-spec preserved as historical context (the links were valid at the time the CHANGELOG entries were written). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
5aa97a25d9 |
v0.20 migration + Genesis v7.65 dev tip + cold-start cache + env-var alignment
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
|
||
|
noonghunnaandClaude Opus 4.7
|
427d2f8aa9 |
docs+scripts+charts: propagate new ceilings (long-vision 198K, long-text 218K)
Sweep across all user-facing docs reflecting the post-PN12-anchor-fix ceilings established in |
||
|
noonghunnaandClaude Opus 4.7
|
26ac8118de |
Restructure docs around hardware axis: SINGLE_CARD.md + DUAL_CARD.md
User feedback: navigating to relevant docs was cumbersome. The natural first decision is "1 GPU or 2 GPUs?" and the existing docs mixed model- specific reference with deployment guidance. New navigation: - README.md adds a "Pick your path" pivot pointing at hardware-axis pages - docs/SINGLE_CARD.md — 1× 3090 deployment menu (workload → compose → TPS) with all single-card configs (vLLM + llama.cpp), VRAM budget, prefill cliffs explained operationally, what single-card can't do - docs/DUAL_CARD.md — 2× 3090 mirror (4 dual variants + TP=2 explainer + what dual unlocks vs single + Marlin pad fork dependency) Slimming: - models/qwen3.6-27b/README.md: dropped duplicated variant tables (now in GPU-count pages); kept model-specific content (quants, Genesis patch surface table, what's working / not, VRAM diagram) - models/qwen3.6-27b/USE_CASES.md: deleted. Per-workload content absorbed into the GPU-count pages (deduplicated). Troubleshooting list moved to docs/FAQ.md as a new "Troubleshooting" subsection. Image-token cost / vision specifics absorbed into SINGLE_CARD.md. Reference updates: 8 files updated (engines/VLLM.md, engines/README.md, COMPARISONS.md, ARCHITECTURE.md, EXAMPLES.md, FAQ.md, INTERNALS.md, top-level README) — all USE_CASES.md links re-pointed to SINGLE_CARD/ DUAL_CARD where appropriate. Net delta: -167 lines (was 197 in USE_CASES + duplicated tables in model README; now 342 lines split between SINGLE_CARD + DUAL_CARD with content deduplicated against each other). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunna
|
3fa33332ce |
Initial commit — club-3090: model-agnostic LLM serving recipes for RTX 3090
Consolidates and supersedes:
- noonghunna/qwen36-27b-single-3090
- noonghunna/qwen36-dual-3090
The two predecessor repos partitioned by card count (1× vs 2×). This
repo partitions by engine instead, which matches how users actually
decide ("vLLM or llama.cpp?" before "1 card or 2"). Card count becomes
a config variant within each engine.
Structure (model-agnostic from day 1):
docs/ cross-model engine + hardware docs
engines/ vLLM / llama.cpp / SGLang comparison + per-engine deep dives
HARDWARE.md Ampere SM 8.6+, NVLink, power, VRAM ceilings
GLOSSARY.md plain-language definitions
img/ illustrations (vram-budget.svg)
ARCHITECTURE.md how this stack thinks about LLM serving on 24 GB
models/<model-name>/ everything specific to a model
qwen3.6-27b/ today's only model
README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md
vllm/ vLLM-specific configs for this model
compose/ docker-compose files (single + dual variants)
patches/ tolist_cudagraph + Marlin pad notes
llama-cpp/ llama.cpp recipes for this model
recipes/ shell scripts (single-card default + 262K max-ctx)
sglang/ SGLang status (currently blocked)
scripts/ shared, model-aware
setup.sh bash setup.sh <model> → downloads + verifies
verify.sh / verify-full.sh smoke + functional tests
bench.sh canonical TPS bench
vLLM compose variants (all under models/qwen3.6-27b/vllm/compose/):
Single-card:
docker-compose.yml ⭐ DEFAULT — TQ3 + Genesis P65, 48K, 51/68 TPS
docker-compose.fast-chat.yml fp8 + 20K, 55/70 TPS — fastest at small ctx
docker-compose.tools-text.yml fp8 + 75K, 53/70 TPS — best for long single prompts
docker-compose.no-genesis-mtp.yml control variant
docker-compose.minimal.yml no spec-decode
Dual-card:
docker-compose.dual.yml ⭐ fp8 + 262K + MTP + vision, 71/89 TPS
docker-compose.dual-turbo.yml TQ3 + Genesis v7.14 — 4-stream concurrency
docker-compose.dual-dflash.yml DFlash N=5 + 185K + vision — 78/128 TPS
docker-compose.dual-dflash-noviz.yml DFlash + 200K text-only
llama.cpp recipes (under models/qwen3.6-27b/llama-cpp/recipes/):
single-card-default.sh Q4_K_M + 65K
single-card-max-ctx.sh Q4_K_M + q4_0 KV at full 262K — the standout recipe
Old repos remain readable for issue history + external links (Medium,
Reddit, Twitter, Sandermage's PR threads). New issues should be filed
here.
Credits in README. Apache 2.0.
|