acd7ffb67c07a1df4b34ec11a7ec52087f249d96
20
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Erik LaBiancaandClaude Opus 4.7
|
276ab89291 |
composes: PYTORCH_CUDA_ALLOC_CONF env-override knob + WSL2 boot-crash docs (#84)
A single-card RTX 3090 Ti rig on WSL2 (driver 596.36) hits RuntimeError: CUDA driver error: device not ready from gptq_marlin_repack immediately after weight load on the v7.72.2-uplift nightly pin. Bisect ruled out Genesis, spec-decode, TQ3 KV, async-residual error, and TDR (registry already extended + Windows rebooted). PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False resolves it. Same env-var workaround as JusefPol's NVLink boot-crash (PR #31), already hardcoded in the dual-nvlink*.yml composes. Replaces the hardcoded PYTORCH_CUDA_ALLOC_CONF line in 14 single-card and PCIe dual-card composes with a ${PYTORCH_CUDA_ALLOC_CONF:-...} override (defaults preserved). Pattern matches existing MAX_MODEL_LEN / GPU_MEMORY_UTILIZATION overrides from #79. The two dual-nvlink*.yml composes are unchanged — their existing JusefPol-driven default already has expandable_segments off. Documentation: - docs/HARDWARE.md: new "disable PyTorch expandable_segments" subsection alongside the TDR fix, with stack trace, what was ruled out, override recipe, and a single uncontrolled observation about weight-load time (32 sec → 13 sec). - docs/FAQ.md: WSL2 question now cross-links both the TDR and expandable_segments fix subsections. - .env.example: documents the override under "vLLM tuning knobs". - CHANGELOG.md (top-level + per-model): dated 2026-05-06 entries. The exact failing call hasn't been isolated. The cuMemMap virtual-memory API used by expandable_segments:True is the suspected culprit since both known occurrences respond to the same workaround, but no specific call has been proven to return cudaErrorNotReady. Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
|
c6e6163901 |
release(v7.72.2-uplift): Genesis pin + vLLM pin + sidecar consolidation (#59)
Aligns club-3090 with Sandermage's Genesis v7.72.2 release (2026-05-05) which shipped 7 new patches (PN59-PN67), v7.72.1 P68 xgrammar-incompat auto-skip, and v7.72.2 PN70 schema-subset filter. Pin bumps: - scripts/setup.sh GENESIS_PIN: 2db18df (v7.69) → 7b9fd319 (v7.72.2) - All 16 composes vLLM image: nightly-7a1eb8ac2 → nightly-01d4d1ad3 (Sander's PROD-validated pin, Genesis allowlist entry #2) 6 local sidecars deleted (Genesis natives supersede): - patch_inputs_embeds_optional.py → PN35 (vllm#35975 backport) - patch_pn30_dst_shaped_temp_fix.py → PN30 v7.68 - patch_pn25_genesis_register_fix.py → PN25 - patch_tolist_cudagraph.py → P78 (Sander's v7.72 CHANGELOG retires this) - patch_workspace_lock_disable.py → PN34 - patch_pr40798_workspace.py → research artifact, no compose mounted it 7 Genesis-loaded composes (yml, dual-turbo, long-text, long-text-no-mtp, long-vision, bounded-thinking, tools-text) had volume mounts and entry- point shell invocations cleaned up; GENESIS_ENABLE_PN59_STREAMING_GDN=1 added to all 7 for consistency. dual.yml left intentionally Genesis-free as a debugging fallback for cross-engine bisect — useful when isolating "is this Genesis or upstream vLLM" during silent-empty / OOM triage. Bench (dual-turbo, 2× 3090, single-stream, 5 measured runs each): - Narrative wall TPS: 81.21 (CV 2.3%), AL 3.46 - Code wall TPS: 108.20 (CV 0.9%) - VRAM/card: 20.0 GB (-2.1 GB vs v7.69 baseline of 22.1) - All 8/8 verify-full checks pass - verify-stress 6/7 (probe 7 = known TQ3 borderline at 60K, container alive throughout — not a regression) Cross-rig finding filed as Sandermage/genesis-vllm-patches#22 — PN59 streaming-GDN doesn't engage on chunked-prefill on Ampere consumer: its eligibility check rejects calls with chunk_indices/chunk_offsets populated, which vLLM's mandatory --max-num-batched-tokens 4128 always sets on 24 GB single-card configs. PN59 falls back to _vanilla_path which OOMs at the same chunk_o.py:161 site PN59 was meant to eliminate. Single-card 24 GB Cliff 2b is therefore unchanged — workaround is dual.yml/dual-turbo.yml (TP=2) or llamacpp/default. Docs warnings placed across README, docs/CLIFFS, docs/HARDWARE, docs/SINGLE_CARD, docs/FAQ, docs/UPSTREAM, BENCHMARKS, and the 3 affected single-card compose YAMLs. v7.72.1 closes #57 (lex's xgrammar-patternProperties fire on long-prompt agentic IDE traffic). Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
b85963033a |
docs: align bug-report + FAQ + MULTI_CARD with report.sh --full / --soak
After shipping report.sh --full / --soak / --stress flags in
|
||
|
noonghunna
|
04764c5f28 |
docs: surface Cliff 2b multi-turn envelope + WHY TP=2 / llama.cpp escape
Today's full validation matrix exposed Cliff 2b — DeltaNet GDN forward fires under accumulated multi-turn context (~21-26K), not just at single prompts >50-60K. All 6 single-card vLLM variants FAIL v2 continuous; only vllm/dual (TP=2) and llamacpp/default survive cleanly. Three issues filed today (#41, #42, #43) all map to the same class. Docs needed to reflect this so users don't keep landing on long-text/long-vision/ tools-text and burning time tuning what won't fix. docs/SINGLE_CARD.md - New ⚠️ section at top: single-card vLLM unsafe for hermes/openhands/ OpenCode/Cline/OpenClaw/Aider/Cursor with retained context. Routing tree to dual.yml or llamacpp/default. - Existing "One limitation" split into Cliff 2a (single-prompt, mostly closed v7.69) and Cliff 2b (multi-turn, NOT closed). Reasons each can/can't be tuned at config layer. docs/CLIFFS.md - TL;DR table extended from 2 cliffs to 3 (Cliff 2a + 2b separated). - New section "Why TP=2 escapes" — per-card head sharding halves the GDN live-tensor sizes (48 MiB → 24 MiB on v/u/o etc., 97 → 49 on h). Concrete byte math; total per-card live FLA set drops from ~500 MiB to ~250 MiB. Validated 0 MiB growth on dual.yml v2 continuous. - New section "Why llama.cpp escapes" — three concrete differences: (1) different GDN kernel (own CUDA, smaller per-step working buffers vs FLA Triton), (2) ggml manual allocator (no PyTorch caching layer fragmentation), (3) no JIT/Triton autotune (pre-compiled, static layout from boot). Trade ~3× decode speed for cliff-immunity. docs/FAQ.md - New troubleshooting entry: "My hermes / openhands / OpenCode / Cline / OpenClaw / Cursor session OOMs after a few turns. What do I do?" Lists every dead-end we tested today so users don't repeat them: mem-util tuning, MTP-off, max-num-batched-tokens (Mamba block_size floor blocks <4128), TRITON_CACHE_AUTOTUNING (Blackwell-only recovery), expandable_segments (already on), empty_cache (reclaims but cliff fires next turn). Routing answer + soak-test repro command. CHANGELOG.md - Dated entry above the soak-test v2 entry summarizing the docs sweep. Codex residency pilot data backs every claim: results/residency-20260503-codex-pilot3/ (initial) results/residency-20260503-empty-cache-idle/ (empty_cache experiment) Investigation memo: docs/diagnostics/cliff2-handoff-results.md (gitignored). |
||
|
noonghunna
|
bca5a063c9 |
docs: surface scripts/update.sh + repo-drift detection
Companion docs update for
|
||
|
noonghunnaandClaude Opus 4.7
|
3bf7da7501 |
docs(FAQ): add WSL2 RAM-constraint failure mode to troubleshooting
[@RossNE99]'s case in #32 surfaced a failure mode that wasn't in the FAQ: vLLM throws a misleading "GPU OOM, tried to allocate 44 MiB" error at model load when the actual problem is WSL2's default RAM allocation (50% of Windows host) being insufficient for the 17.69 GiB checkpoint. The smoking-gun log line is `[weight_utils.py:934] Auto-prefetch is disabled because ... checkpoint size (17.69 GiB) exceeds 90% of available RAM` — surfaced automatically by report.sh now, but not otherwise visible in standard triage output. Adds a troubleshooting entry pointing at the .wslconfig fix (memory=24GB minimum, swap=8GB), with the diagnostic log line so users can self-identify if they suspect this. Also cross-links the original RossNE99 repro thread for context. Was invisible until report.sh captured the boot log highlights — good worked example of why the standardized rig dump is more useful than ad-hoc nvidia-smi pastes. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
f55b0a734f |
docs: surface triage ladder at issue-filing time + add at-a-glance table
Two related improvements after [@noonghunna]'s observation that the
ladder we just added (commit
|
||
|
noonghunnaandClaude Opus 4.7
|
9560efd1f7 |
docs(FAQ): add 5-step triage ladder before symptom-matching
When users hit boot OOMs, MTP weirdness, or TQ3/long-context budget issues, our previous troubleshooting flow jumped straight to symptom-matching. That misses the systematic narrowing pass: validate the simplest stack works first, then add one variable per step until the failing layer is isolated. New leading section in Troubleshooting: - Step 1 — `vllm/minimal` (32K + fp8, no Genesis, no spec-decode): validates hardware, driver, Docker, NVIDIA Container Toolkit, model files, base vLLM. Strips out every layer that could be the cause. - Step 2 — `vllm/tools-text` (75K + fp8 + MTP + Genesis): adds Genesis + MTP K=3. Still fp8 KV (no TQ3 yet). Most common failure here is GENESIS_PIN-vs-tree mismatch — re-run setup.sh. - Step 3 — `vllm/long-text` (180K + TQ3 + MTP + full Genesis): adds TurboQuant 3-bit KV + long-context. The production-target single-card config. Failure here narrows to TQ3 setup, GDN >60K hardware wall, or Cliff 1 mech B (closed since v7.69 PN25). - Step 4 — `vllm/dual` (262K + fp8 + TP=2 + 2 streams, Genesis-less): adds TP=2 NCCL + multi-GPU memory split. Crucially removes Genesis, since dual.yml is intentionally Genesis-less. Failure here despite step 3 working narrows to TP=2 NCCL specifically. WSL2 is the most common trigger (its vGPU layer adds memory accounting wrinkles that bare-metal Linux doesn't have). - Step 5 — `vllm/dual-turbo` (262K + TQ3 + TP=2 + Genesis): the full multi-card stack. Failure here despite step 4 narrows to the TQ3-on-TP=2-with-Genesis intersection. The ladder works for both single and dual-card users because steps 1-3 isolate stack layers regardless of GPU count, and steps 4-5 add TP=2 surface separately. A dual-card user hitting issues should still run steps 1-3 on a single card first — it's the only way to tell apart "single-card stack issue that also breaks dual" vs "TP=2 NCCL specific." Existing symptom-pattern-matching list demoted to "Quick recognition guide" sub-section, kept verbatim — still useful for users who already know the failure surface and want to skip the ladder. Going forward we'll point to this ladder explicitly in triage replies on issues / discussions, replacing the ad-hoc "have you tried X?" pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
f8c9c365e0 |
docs: full sync to v7.69 + Cliff 2 60K closure recipes
Sweep all stale v7.66 / fc89395 substrate references to v7.69 (commit 2db18df) + local vllm#35975 inputs_embeds backport. Ship the Balanced MTP (long-text.yml, 180K + 0.93) and Max-context (long-text-no-mtp.yml, 200K + 0.95, no MTP) variants as the Cliff 2 closure recipes — both PASS the 60K single-prompt envelope (623s and 537s wall respectively). Updates: - CHANGELOGs (root + model) — new v7.69 PM entry above v7.66 - README + SINGLE_CARD + HARDWARE + EXAMPLES + FAQ + INTERNALS + VLLM engine doc — Cliff 2 status, substrate pins, mem-util defaults, variant table, sidecar list - vllm/README.md compose menu refreshed for the new ctx envelopes - model README patch surface table — added PN30 part3, PN32, P103, PN34 rows; collapsed P98 reference to PN34 env-gate - tools/charts/gen-perf.py + gen-vram.py — substrate label bumped to v7.69 + #35975, panel labels for the long-text variants updated, long-text-no-mtp 200K Max-context noted as bench-pending in chart - All performance + VRAM charts (svg + png) regenerated Cliff 2 60K closure: Genesis v7.69 (PN32 GDN chunked-prefill + P103 worker self-install + PN30 part3 + PN34 workspace_lock relax) plus local backport of vllm#35975 (~444 MiB freed on text-only paths). 3 sidecars dropped on long-text variants; 2 sidecars retained on master (patch_inputs_embeds_optional.py, patch_tolist_cudagraph.py). >60K single-prompt still hits the 24 GB hardware-physical wall on single-card. For those: dual-card TP=2 (verified at 237K) or llama.cpp single-card (262K, different engine). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
5aa97a25d9 |
v0.20 migration + Genesis v7.65 dev tip + cold-start cache + env-var alignment
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
|
||
|
noonghunnaandClaude Opus 4.7
|
ebacba1efd |
fix: address open issues #1, #4, #7
Closes / addresses 3 reported issues + adds requested feature:
#7 vid (PORT not honored, MODEL_DIR vs MODELS_DIR confusion):
- All 8 vLLM compose files now use "${PORT:-XXXX}:8000" so .env PORT
flows through. Defaults preserved per-variant (8020 single, 8010-8013
dual). llama.cpp composes already had this pattern.
- scripts/switch.sh: load .env early; per-variant default-port table;
new resolve_ready_url() picks PORT > variant default for the readiness
probe.
- scripts/launch.sh: same default-port table; final endpoint URL printed
to user reflects actual mapped port.
- .env.example: ⚠ box callout that variable names are CASE-SENSITIVE
(MODEL_DIR singular, NOT MODELS_DIR plural — silently ignored).
New PORT section documenting per-variant defaults.
#4 timxx (tools-text.yml fails "Free memory ... less than desired"):
- docs/FAQ.md: new entry "Container fails to start: Free memory..."
explaining the vLLM startup check, the two workarounds (free VRAM /
lower mem-util), and which configs hit it most often (0.97+ mem-util).
- Compose defaults unchanged (0.97 stays the right design target on
headless rigs); the FAQ documents the workaround for users with X11.
#1 fabriciomalta (per-config VRAM column):
- docs/SINGLE_CARD.md: TL;DR table now has VRAM column with mem-util.
- docs/DUAL_CARD.md: TL;DR table same + footnote explaining per-card
semantics and which dual configs would/wouldn't fit on 2× 20 GB cards
(relevant to fabriciomalta's 2× 3080-20GB use case).
#2 tenitram (empty responses) — fixed in master via
|
||
|
noonghunnaandClaude Opus 4.7
|
427d2f8aa9 |
docs+scripts+charts: propagate new ceilings (long-vision 198K, long-text 218K)
Sweep across all user-facing docs reflecting the post-PN12-anchor-fix ceilings established in |
||
|
noonghunnaandClaude Opus 4.7
|
b0eed46ff7 |
Add docs/CLIFFS.md — comprehensive prefill-cliff synopsis
Single canonical reference for everything we know about Cliff 1 (FA2 softmax_lse cap-leak) and Cliff 2 (fla.ops GDN forward intermediate buffer): TL;DR table, empirical bisection with stack traces, root-cause walk-through, why earlier "FFN intermediate buffer" framing was wrong, why mem-util doesn't help, why PN8 closes Cliff 1 on tools-text but not on TQ3 paths, why llama.cpp dodges both structurally, alternative attention backends with feasibility, who-can-fix-it landscape (Sandermage, Tri Dao, fla-org, QwenLM, us at any difficulty), recommended path forward, and re-test triggers. Cross-linked from FAQ.md and README.md. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
2d6b69dd50 |
Cliff 1 root cause revised: FA2 softmax_lse sized by max_seqlen
After bisecting long-vision config space (192K/128K/96K/86K at 0.98 and 0.92 mem-util) and second-opinion synthesis from ChatGPT + DeepSeek + vLLM source review, the actual root cause is: softmax_lse in flash_attn_varlen_func is allocated as [num_seqs, num_heads, max_seqlen] — sized by the max_seqlen parameter, NOT the actual cu_seqlens. vLLM passes attn_metadata.max_seq_len; during cudagraph capture that's set to max_model_len. So a 25K-token tool prefill at max-model-len=192K allocates softmax_lse for 192K, eating the activation headroom. The 50-138 MiB OOMs we'd been observing are downstream of this leak. Empirical OOM site (verified in our docker logs): _vllm_fa2_C.varlen_fwd in flash_attn_varlen_func. Upstream root cause: Dao-AILab/flash- attention#1011 (open since 2024). vLLM cap-leak path: vllm#40961. Earlier "FFN intermediate buffer" characterization was wrong. Updates: - UPSTREAM.md: new FA2 section (Dao-AILab/flash-attention#1011); added vllm#40961 (cudagraph capture max_seq_len pattern), vllm#40069 (TurboQuant follow-ups tracker), and vllm#25543 (V0 deprecation removed max_seq_len_to_capture, so commonly-suggested mitigation doesn't apply on V1 nightly) - FAQ.md: corrected Cliff 1 explanation - SINGLE_CARD.md: corrected "Cliff 1 still fires" caveat - CHANGELOG: documented bisection + revision - memory/qwen36_27b_prefill_cliffs.md: revised Cliff 1 mechanism; noted Cliff 2 likely shares the same architectural pattern Practical implication: no new variant ships. Default 48K + 0.92 + TQ3 + vision stays the prefill-safe ceiling — pushing higher requires upstream fix at FA repo, not config tuning. tools-text.yml (75K + FP8 + PN8 closes Cliff 1) remains the IDE-agent path. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
37a4895f6d |
Remove fast-chat.yml; extend P68/P69 disable to default
fast-chat (20K, fp8, vision) and default docker-compose.yml (48K, TQ3, vision) had effectively the same TPS post-PN8. fast-chat's only remaining differentiator was "smaller context = ~3s faster boot," and 20K is actively bad for IDE-agent users (Copilot tool-schema preamble alone hits 20K). Net negative — removed. Default compose was missed in the previous P68/P69 fix — it had the same env vars enabled and the same silent-stop bug above 8000 chars. Both now disabled with the same explanatory comment. Updated: - scripts/switch.sh, scripts/launch.sh — drop the variant - docs/SINGLE_CARD.md, FAQ.md, engines/VLLM.md, model + vllm + patches READMEs — references removed or pointed to default/tools-text - All sibling compose YAML "see also" tables — fast-chat row removed, tools-text row repurposed for IDE-agent guidance - CHANGELOG entry; old historical entries kept as-is (append-only) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
aab8ff4a0e |
Disable Genesis P68/P69 in shipped composes (silent-stop bugfix)
P68 (auto force tool_choice=required) and P69 (inject "must use a
tool" reminder) silently fired on prompts > 8000 chars — every IDE
agent (Cline, Cursor, OpenCode, Copilot Gateway) blew past that
threshold instantly and got silent finish_reason=stop with no
content + no tool_calls on greetings or clarifying questions.
Bisection on club-3090#2 (HoodOG1 + tenitram):
state A (P64+P68+P69+PN8): broken
state B (P64+P69+PN8, P68 off): still broken — model loops on
"I cannot respond with plain text" then stops mid-reasoning
state D (P64+PN8, P68+P69 off): clean — greeting → plain-text
reply; tool request → clean read_file call
P64 (qwen3coder MTP streaming early-return fix) and PN8 (FP8+MTP
draft online-quant memory savings) stay enabled — real bugfixes,
no user-intent override.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
noonghunnaandClaude Opus 4.7
|
26ac8118de |
Restructure docs around hardware axis: SINGLE_CARD.md + DUAL_CARD.md
User feedback: navigating to relevant docs was cumbersome. The natural first decision is "1 GPU or 2 GPUs?" and the existing docs mixed model- specific reference with deployment guidance. New navigation: - README.md adds a "Pick your path" pivot pointing at hardware-axis pages - docs/SINGLE_CARD.md — 1× 3090 deployment menu (workload → compose → TPS) with all single-card configs (vLLM + llama.cpp), VRAM budget, prefill cliffs explained operationally, what single-card can't do - docs/DUAL_CARD.md — 2× 3090 mirror (4 dual variants + TP=2 explainer + what dual unlocks vs single + Marlin pad fork dependency) Slimming: - models/qwen3.6-27b/README.md: dropped duplicated variant tables (now in GPU-count pages); kept model-specific content (quants, Genesis patch surface table, what's working / not, VRAM diagram) - models/qwen3.6-27b/USE_CASES.md: deleted. Per-workload content absorbed into the GPU-count pages (deduplicated). Troubleshooting list moved to docs/FAQ.md as a new "Troubleshooting" subsection. Image-token cost / vision specifics absorbed into SINGLE_CARD.md. Reference updates: 8 files updated (engines/VLLM.md, engines/README.md, COMPARISONS.md, ARCHITECTURE.md, EXAMPLES.md, FAQ.md, INTERNALS.md, top-level README) — all USE_CASES.md links re-pointed to SINGLE_CARD/ DUAL_CARD where appropriate. Net delta: -167 lines (was 197 in USE_CASES + duplicated tables in model README; now 342 lines split between SINGLE_CARD + DUAL_CARD with content deduplicated against each other). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
f275bf502a |
FAQ: add VS Code Copilot LLM Gateway entry
Documents the two compatibility issues users will hit with VS Code's Copilot LLM Gateway: 1. Tool-schema preamble is ~20K tokens — fast-chat.yml's 20K cap is too small. Recommended pick is tools-text.yml (75K + fp8 + PN8, Cliff 1 closed since Genesis v7.62.x). 2. Copilot probe-style requests with max_tokens=64 truncate tool-call JSON mid-string. With tool_choice: required + minItems: 1 in their structured-outputs schema, the model must emit a tool call that takes real arguments — won't fit in 64 tokens. Manifests as "empty response" client-side. Server-side correct. Background + debug-log analysis from tenitram on club-3090 #2. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
51a4001af7 |
Genesis v7.62.x + PN8 on FP8 paths (closes Cliff 1 on tools-text)
scripts/setup.sh — GENESIS_PIN bumped from bf667c7 (v7.54) to 917519b (v7.62.x release, 2026-04-29). New patches: PN8 (MTP draft online-quant propagation, backport of vllm#40849), PN11 (Quentin-M streaming tool-call IndexError fix vllm#41142), per-GPU profile auto-rec, k8v4 unlock on hybrid GDN via P4+P98. PN8 enabled on FP8 paths only: - tools-text.yml: -900 MiB at boot, Cliff 1 25K tool prefill closes, -7% code TPS. Net win — production-safe for tool-using agents. - fast-chat.yml: -800 MiB at boot, no cliff to test at 20K, -4.7% code TPS. Free VRAM is useful for tighter mem-util configs. PN8 not enabled on TQ3 paths (default 48K, long-vision, long-text) or dual configs: - default 48K: PN8 is no-op on TQ3 + 0.92 (plenty of headroom already) - long-vision: PN8 grows KV pool 230 MiB and lifts engine ceiling 192K → 198K, but does NOT close Cliff 1 — the 138 MiB allocate is an FFN intermediate-buffer activation peak (intermediate_size × max-num- batched-tokens), not a draft-model footprint - long-text: engine ceiling at 206K is gated by attention-block-size divisor, not KV; PN8 has nothing to give - dual.yml: deliberately Genesis-less by design; not worth restructuring Verify-full passes on default 48K + v7.62.x without PN8 (8/8). Verify- stress on tools-text + PN8 passes all checks including the 25K tool prefill that was the launch-tweet headline caveat. Cross-rig data shared with Sandermage: https://github.com/noonghunna/qwen36-27b-single-3090/issues/1#issuecomment-4343317153 Docs updated: cross-cutting CHANGELOG, per-model CHANGELOG, USE_CASES.md (Cliff 1 closure note on tools-text), FAQ.md (Cliff 1 entry + new PN8 entry). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
1b9374b8b5 |
Add docs/FAQ.md — common questions answered for tweet click-throughs
Topics: hardware (4090 / 5090 / NVLink / non-NVIDIA / Windows-WSL2), engine choice (vLLM vs llama.cpp / why not Ollama-LMStudio / MTP not EAGLE / why not GGUF on vLLM / why AutoRound), performance (TPS expectations, ctx-load decode drop, prefill cliffs explained, vllm#40914), setup (model paths, GPU index override, multi-variant ports, Open WebUI), community (bench contributions, bug reports, Genesis bumping). Linked from top-level README. Designed to absorb repeat issue-tracker questions; each answer is 2-4 sentences with links to deeper docs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |