b0eeb21ff53a2c0d89677be8d0a3af4eef8a2f4f
54
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 noonghunnaandClaude Opus 4.8
|
d09a063317 |
Phase 2b: add --json/CLI contracts to 7 stack scripts (additive)
Data-layer contracts the cockpit (and any jq user) consumes — all strictly
additive (existing human output byte-identical), full guard suite green (54/54):
- registry-emit.sh --json : {variants,defaults,profiles{engines,models,hardware,drafters}}
- tools/kv-calc.py --fit <slug|model> --card <gpu> --json : structured fit verdict
- gpu-mode.sh --list-modes [--json] : scene catalog (serving/studio/ops)
- estate_cli.py report-state/diagnose --json : structured estate read
- pull.sh --profile-like --dry-run --json : structured swap_path (not a message blob)
- health.sh CONTAINER= : Doctor probes any engine container (was qwen36-27b-hardcoded)
- switch.sh --explain <slug> [--json] : joined registry/engine/model/hw/drafter + fit + bench
Built + adversarially reviewed via workflow. The review caught a real
switch<->kv-calc seam defect (switch fed hyphenated 'rtx-3090', kv-calc matched
only 'rtx3090' -> fit silently 'unavailable' on the 3090 rig); fixed: kv-calc
accepts hyphenated hardware-profile ids + hyphen-strip fallback; switch surfaces
kv-calc's structured verdict regardless of RC and renders its real keys; both
tests now exercise the seam. No shared-module edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
|
3531fd3551 |
Add opt-in LMCache KV-offload compose (vllm/qwen-27b-dual-lmcache, incubating) (#421)
Layers an LMCache tiered persistent prefix-KV cache (MP/HMA connector) onto the dual-max fidelity profile (FP8 + int8-PTH KV + MTP n=3 @262K). Controlled A/B shows ZERO decode penalty (74 narr / 94 code TPS == without LMCache, MTP intact ~83% accept); reuses cached prefix KV across long / multi-session workloads (cold->warm TTFT ~7-8x) instead of re-prefilling. - New vllm-lmcache engine profile: lmcache/vllm-openai DIGEST-pinned (the tag is mutable — bundled vLLM 0.22.1-dev then 0.23.1-dev under the same tag). - preflight_lmcache_ram guard: rejects --l1-size-gb over-allocation even under --force (the l1=100-on-94GB host-OOM that forced a reboot, #133). - env-gated L2 disk tier hook (LMCACHE_L2_ADAPTER, off by default). - INTERNALS.md RAM/disk sizing section; registry entry + count bumps; full guard suite green. Incubating (hidden, --force to launch): runs LMCache's third-party image with a newer vLLM than our v0.22.0 pin; L2 latency unmeasured on-rig; 38 GB on-demand. Refs #133. Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> |
||
|
noonghunnaandClaude Opus 4.8
|
35a69650fe |
feat(catalog): add 🐣 Incubating status tier + VibeThinker-3B (incubating)
Introduce a new pre-experimental status tier, `incubating`, for composes that work but aren't ready for the actionable list — niche specialists or models that fail the standard functional gate by design. Incubating composes are HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and launch-gated behind `--force` (non-functional), so a half-validated model is catalogued and discoverable without cluttering the recommended set. Tier wiring (reuses the existing status plumbing — no new emit column): - compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI (stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted) - switch.sh: both --list filters skip incubating unless --all, with a `(+N incubating hidden — --all)` header note - AGENTS.md: Status enum row + Caveats-required references - docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune): - `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090, full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized), --reasoning-parser qwen3, no tool-calling - Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS), qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive 3B: non-terminating empty output). Always-reasoning + no-tools → fails verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing. Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.8
|
111c72b0ea |
switch.sh --owui: auto-register a launched model in Open WebUI
New opt-in flag: after a catalog model is up + ready, switch.sh --owui upserts an OpenAI connection in Open WebUI (host.docker.internal:<port>) so it appears in the chat picker — no manual Admin->Connections step. scripts/lib/owui-register.sh is conditional (no-op if OWUI not running), idempotent (skips if already present), and drives OWUI's admin config API via a token forged from the container secret. Validated end-to-end on Deckard-40B (:8199). Guard suite 41/42; the 1 failure (test-measurement-record) is pre-existing bench-output-format drift, unrelated to this change (no switch.sh/owui reference). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.8
|
bb9f9f23a0 |
Repoint stale single-card launcher hints off deprecated vLLM composes
When the Genesis/nightly single-card vLLM family was deprecated 2026-05-31 (vllm/default, long-text, long-vision, long-text-no-mtp, bounded-thinking, tools-text), the hand-written user-facing hints weren't repointed (the deprecation-checklist step 4 was missed) — surfaced by a #35 commenter who read them as live. The DEFAULTS resolver was fine (vllm/default → vllm/minimal), but the docs still advertised the dead composes. - switch.sh header: the "Single-card vLLM" subset listed 6 deprecated composes as recommended options AND omitted beellama/dflash (the actual single-card default). Rewrote to lead with beellama/dflash + the functional vllm/minimal, with a dated deprecation note (mirrors the dual-section note). - preflight.sh: the "start with: switch.sh vllm/default" hint now points at beellama/dflash (the single-card default). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
noonghunna
|
27b9fe5e45 |
feat(beellama): release v0.3.0 Q8_K_XL dual composes (experimental) + centralize image pin (#296)
Ships 3 dual-card beellama v0.3.0 Q8_K_XL composes 🧪 experimental for community testing (#288): qwen-mtp-dual :8064, qwen-dflash-dual :8065 (262K), gemma-q8-dflash-dual :8066 (192K); flips the q4ks gemma dual ⏸️→🧪. Centralizes BEELLAMA_IMAGE injection (engine install.spec, one-line bump) at Anbeeld's official server-cuda-v0.3.0 tag + adds scripts/beellama-pin-bump.sh (manual, no auto-push). Catalog: q8kxl weights variants, 3 registry entries, beellama-local +mtp_gguf, unsloth-mtp-gguf engine_type fix. Also a stale test-model-default-resolver fix + UPSTREAM.md row 38. Gate 41/41. |
||
|
noonghunna
|
b327f0311a |
refactor(gemma): rename dual slugs + ladder gemma-bf16-mtp to 131K (#286)
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41. |
||
|
noonghunnaandtekgnosis-net
|
6cafaf80b6 |
Operational robustness (#281): orphan-safe switch.sh · reboot-surviving vLLM · multi-GPU power sweep (#285)
Re-bases tekgnosis-net's #281/#282 onto master: (1) switch.sh registry-derived VARIANT_CONTAINER closed-world teardown (+--remove-orphans) — fixes beellama/ik-llama/sglang VRAM leak; (2) 29 vLLM composes restart: ${CLUB3090_RESTART:-unless-stopped} (reboot survival, opt-out knob); (3) power-cap-sweep.sh multi-GPU (board-power sum + cap restore). 3 new tests; suite 41/41. Closes #281, supersedes #282.
Co-Authored-By: tekgnosis-net <[email protected]>
|
||
|
noonghunna
|
611c430f0a |
beellama Gemma-4 ctx: single 128K (caveats) + dual 262K parked/upstream-gated (#284)
Single beellama/gemma-dflash 128K (caveats; 140K OOMs under load, CTX_SIZE=102400 agent-safe). New dual beellama/gemma-dflash-dual parked upstream-gated (262K works but DFlash multi-GPU broken in 07ac3ce; fixes on v0.3.0 dev branch, no release). switch.sh --list hides upstream-gated (parked) like deprecated with its own note. UPSTREAM row tracks Anbeeld #39. Suite 38/38. |
||
|
noonghunna
|
6274a341c2 |
switch.sh --list: show max context per slug (+ registry<->compose drift) (#283)
Adds the max context size to switch.sh --list. registry-emit threads a ctx label through the variant TSV (before status_note); switch.sh renders it rightmost. Production -> bare ctx; caveats/NA -> folded into the health paren (comma-separated). Rounded to nearest K (163840->164K; 32768 reads 33K). Shows registry vs compose default as a single value when they match, 'validated/compose' (e.g. 164K/200K) when they drift -- the only 3 drifts are experimental lanes; all production/caveats match. Suite 38/38. |
||
|
noonghunna
|
4d47d77fce |
Prune dual vLLM composes: qwen-27b -> one config; gemma-31b default -> gemma-int8 (#279)
Settle the per-model dual vLLM set. qwen3.6-27b dual -> ONE (vllm/dual fp8 262K vision MTP); deprecate dual-dflash/dflash-noviz/tq3-nomtp/bf16/int8. gemma-4-31b dual -> keep TWO: DEFAULTS moved gemma-mtp -> gemma-int8 (full 262K + vision + 4 streams; rides v0.21.0+#40391 overlay), gemma-mtp kept as the stable v0.22.0 32K fallback. Registry entries kept (deprecated, not deleted) so patches.yml + hardware-gating tests stay valid. Out of scope: carnice/qwopus fine-tunes, gemma-26b-a4b, multi4. Suite 38/38. |
||
|
|
bca55e54f2 |
Deprecate Genesis vLLM composes; vLLM single default → vllm/minimal (#276)
* Deprecate Genesis vLLM composes; repoint vLLM single default to vllm/minimal Mark all 8 Genesis-loading qwen3.6-27b vLLM composes deprecated (registry status + compose-header Status): vllm/default, vllm/long-text, vllm/long-text-no-mtp, vllm/long-vision, vllm/bounded-thinking, vllm/tools-text, vllm/dual-turbo, vllm/dual-tq3-mtp-genesis. Genesis is on hold pending Sandermage's next stable release; the stack is moving to stable vLLM + beellama single-card. Repoint DEFAULTS[(qwen3.6-27b, vllm, single)]: vllm/default -> vllm/minimal (the Genesis-free fp8-KV config). The `vllm/default` token now resolves single -> vllm/minimal, dual -> vllm/dual. The model single-card default stays beellama/dflash. Resolver guard tests updated (vllm/default single -> vllm/minimal); SINGLE_CARD.md example commands redirected off the deprecated slugs. Suite 38/38. Follow-ups: (1) bump vllm/minimal + vllm/dual to v0.22.0 (Genesis-free -> stable); (2) SINGLE_CARD.md narrative reframe (single-card vLLM deprecated, beellama default). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * switch.sh --list: hide deprecated by default, reveal with --all With 9 Genesis composes now deprecated, --list was cluttered. Hide deprecated variants from the default --list (tally + display loops); --all reveals them alongside other-topology variants. Footer shows "(+N deprecated hidden --all)" so they stay discoverable, never silently dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
|
39c7cc8dc8 |
feat(switch): hardware-filter --list by GPU count; add --all (#267)
`switch.sh --list` showed every slug regardless of topology, so a single-GPU box listed dual/multi configs it can't actually run — noise. Filter `--list` to the topologies the detected GPU count supports: 1 GPU -> single only, 2 -> single+dual, 4+ -> everything. Detection reuses switch_topology_from_gpus (CUDA/NVIDIA_VISIBLE_DEVICES, then nvidia-smi) mapped to a topology rank. - `--all` (and the `--list-all` alias) bypass the filter for discoverability; --list is deferred until args are parsed so `--list --all` works in either order. - Don't silently hide: print a one-line note naming the detected GPU count and exactly which topologies were hidden, plus a `(+N … hidden — --all)` tally in the header. No note under --all / when nothing is hidden. - Fail-open: with no detection signal (no selector, no nvidia-smi) show ALL — never hide off a guessed "single" fallback. - Counts (header + per-model) reflect the visible set; PR-A health markers/grouping and PR-B Defaults view unchanged. New test scripts/tests/test-list-topology-filter.sh covers 1/2-GPU filtering, --all, --list-all, order independence, the --all-without-list guard, and fail-open. Docs: --help usage, FAQ model-switch entry, README. Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
|
26eac83f76 |
feat: model-default resolver + user-pinnable defaults (PR-B) (#266)
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
|
1182d6b2c3 |
feat(registry): slug health/availability flag (#265)
* feat(registry): add slug health/availability flag Add a lifecycle `status` to every registry slug so `switch.sh --list`, launch, and switch are no longer blind to a compose's health. Previously status lived only in compose-header comments, which drifted: the Genesis dual compose declared "Working (with Genesis)" while its pin is parked and it won't boot clean — a user could boot a broken slug unknowingly. - compose_registry.py: `_entry()` gains keyword-only `status` (default "production") + `status_note`, validated against the enum (production/caveats/experimental/preview/upstream-gated/deprecated). Add `compose_header_status()` mapping a compose's profile-schema `Status:` emoji to that enum. - Sweep every compose `Status:` header to a canonical enum value and re-flag the non-functional slugs: all *genesis* + gemma-4-31b single fp8 -> upstream-gated; carnice -> caveats; qwopus + qwen-a3b-preview -> preview; bf16/int8 A/Bs, llamacpp bounded-thinking, PRISM/APEX eval lanes, gemma-4-26b-a4b onboarding -> experimental; tq3-mtp -> deprecated. - registry-emit.sh emits `status` + `status_note` as the last two VARIANT fields; both loaders + the parity tests read the extended field list. - switch.sh --list: status marker (caveats -> "(caveats)", the NA set -> "(NA: <word>)"); model/topology grouping preserved. Launch/switch gate: production launches, caveats launches with a notice, NA warns + requires --force. launch.sh surfaces the flag before delegating to switch.sh. - New drift-guard test test-compose-status-drift.sh: registry status in enum, compose header maps to enum, and the two agree. 45 entries unchanged; kv-calc calibration 17/17; full suite green (only the pre-existing test-submit-bench fixture failure remains). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * feat(switch): add model/variant counts to --list Header line shows supported-model count + total variants with the health split (N production · N caveats · N NA); each model group shows its variant count. Widen the marker column so the (NA: …)/(caveats) markers align. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
|
5a43d4c475 |
chore(vllm): remove redundant nvlink-* dual composes (NVLink auto-detected) (#257)
The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp, nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) and their vllm/dual-nvlink* launch slugs are removed. They were thin Docker Compose `extends:` stubs whose ONLY override was NVLINK_MODE=force_on — fully redundant since every base dual compose now auto-detects NVLink at boot via detect_nvlink.sh (NVLINK_MODE=auto): it probes nvidia-smi and flips on NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. NVLink rigs get the identical fast path from the base dual compose with no separate slug; force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual` if auto-detect ever misses. Scope (prune only; no engine-image change — the v0.22.0 stable-engine consolidation is a separate follow-up): - delete 4 nvlink-*.yml composes - compose_registry.py: drop the 4 _entry blocks (50 -> 46 entries) - test-compose-registry-disk.sh: count guards 50/51 -> 46/47 - profile_runtime.yml + patches.yml: drop the 4 nvlink blocks/slug refs - test-profiles-compat.sh: retire the C13/E3 NVLink-required-compose scenarios (the estate NVLink-gating code is now dormant; cleanup tracked as a follow-up, code retained) - docs: switch.sh help, AGENTS.md, HARDWARE.md, UPSTREAM.md, patches/README.md, fp8-mtp/dflash/dflash-noviz compose headers, per-model CHANGELOG removal entry; add a docs/README.md pointer to `switch.sh --list` as the authoritative registry-derived compose x slug matrix Historical NVLink bench rows (JusefPol #31, danbedford #74/#92/#96) are preserved in BENCHMARKS.md. Full test suite green (the lone test-submit-bench failure is a worktree-isolation artifact — it needs gitignored results/rebench/ fixtures absent from a fresh worktree; passes on the working tree). Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
|
ced19578a3 |
feat(switch): group --list by model · topology (#264)
`switch.sh --list` printed a flat `slug → full/path` list, so a reader couldn't tell which model/topology a slug like `vllm/default` or `vllm/minimal` was bound to without parsing the path. Group the output by model, then topology, with the serving artifact (<quant>/<serving>.yml) in the right column — so every slug's binding is visible at a glance. Display-only: no slug renames, no registry/emit-contract change. model is parsed from the variant's compose dir (field 2); topology/quant/serving from the file field. Backward-compatible — slugs and their behavior are untouched. Verified: switch/launch registry-parity + compose-disk/mounts tests pass; full suite green except the pre-existing fixture-only test-submit-bench. No test parses --list output. Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.8
|
45686808cb |
Prune Gemma vLLM variants (9->3) and pin survivors to v0.21.0
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly (e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class). Prune the 9-variant set to 3 and repoint survivors onto the immutable stable release tag vllm/vllm-openai:v0.21.0 (never :latest). Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash, gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path folds into gemma-int8 via a CTX env override. Gemma is decoupled from the shared Qwen nightly profiles via a new vllm-gemma-stable engine profile; Qwen nightly profiles are untouched. Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the diagnose tool's cross-model disk-source proxies for two Qwen overlays (vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities are image-baked in the pinned nightly, not mountable files, so mark them image_baked and have diagnose skip the disk-source check for image-baked overlays. Runtime-neutral (no Qwen compose mounts them). switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin loop in launch.sh had it; switch.sh did not -> "unexpected engine pin export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable profile (was unbound under set -u). Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite 35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output, qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton "fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant; beellama remains the single-card gemma path on Ampere. Refs #451, #250, #167. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
a0520e2060 |
refactor(launch): derive launcher tables from the registry + <engine>/default resolver
PR-B of the compose-quant-hierarchy refactor (follows PR-A #231). launch.sh no longer hardcodes its variant/port/container/kvcalc maps — it derives them from compose_registry.py via a shared emitter, the same single source of truth switch.sh already uses. Adds a topology-aware <engine>/default resolver. - scripts/lib/registry-emit.sh (new): shared emitter — emits switch/launch tables from COMPOSE_REGISTRY, parses container_name from each compose, and exposes registry_default_target() for <engine>[/<topology>]/default. - launch.sh: drop the hardcoded `declare -A LAUNCH_*` maps; populate from the emitter. PRIMARY_MODEL constant (default qwen3.6-27b) backs bare <engine>/default. - switch.sh: drop inline derivation; resolve vllm/default, vllm/dual/default, vllm/multi4/default via the shared emitter. - compose_registry.py: add kvcalc_key to entries; tools/kv-calc.py aliases. - bench-row-formatter.sh: DEFAULTS-aware compose_display (drops the now-dead docker-compose.yml branch). - tests: test-launch-registry-parity.sh + test-default-resolver.sh (new); test-switch-registry-parity refreshed onto the shared emitter. Default resolution is an alias layer over existing registry keys — no keys renamed, no compose content changed. Side effect: the launcher<->registry drift that left vllm/gemma-mtp pointing at a non-existent dual/.../fp8-mtp.yml is gone (now derives the registry's bf16-mtp.yml). Implemented via Codex per the PR-B brief; independently re-validated before commit: 7/7 gates PASS (switch+launch parity, default-resolver, launch-compat, registry-disk, mounts-resolve, kv-calc 22/22); live vllm/default->dual + vllm/dual/default->dual both verify-full 8/8; leak-clean. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
2135230f8f |
docs(switch): correct ik-llama/iq4ks-mtp usage comment 262K -> 200K
The help line showed the 262K native max while the registry max_ctx, the compose default, and BENCHMARKS all use the #201 safe default of 200K. Sync the comment. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
b116750ef1 |
fix(registry+bench): sync vision defaults to the 2026-05-25 re-tune (#438)
Follow-up to PR #227 (#437). The vision re-tune changed the compose defaults but left the registry / switch.sh / BENCHMARKS describing stale values. - compose_registry: llamacpp/mtp-vision max_ctx 49152 -> 150000 (matches the re-tuned compose default; ik-llama/iq4ks-mtp-vision already correct at 163840). switch.sh derives variants from the registry, so this also fixes its dynamic list. - switch.sh usage comments: both vision variants now show the 1M-px default + "full-res = override, lower ctx" note (were "49K" / bare "160K"). - BENCHMARKS: marked the superseded 2026-05-20 49K row; added two re-tuned vision rows (llama 150K@1M-px fills 138K/561MB; ik 160K@1M-px fills 147K/503MB, graduated EVAL #402 -> Production). Decode TPS carried from siblings (ctx-independent, not re-benched); rows focus on the measured ctx-ceiling + VRAM + verify-stress [8/8]. ik vision was ALREADY registry-wired (parity test green); this is the data sync. Refs #438. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
9a27de83d7 |
fix(switch): shell env wins over .env (MODEL_DIR etc.) + CRLF-tolerant (#425)
switch.sh sourced .env unconditionally (`set -a; source`), so a stale .env `MODEL_DIR` clobbered an exported one — the opposite precedence to launch.sh (which guards on `-z MODEL_DIR`) and to docker compose (shell env > .env file). A user who `export MODEL_DIR=…` then ran switch/launch got the .env value instead (reported on #187 by rhossack + camelCase). Parse .env line-by-line: a variable already set in the environment wins; values are taken literally (matches docker compose's own .env semantics, no shell expansion); a trailing CR is stripped so Windows-editor CRLF .env files work (#187). Also print `[switch] MODEL_DIR=…` so the resolved value + its source is unambiguous. Closes #425. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
|
0b4e694ad3 |
feat(llama.cpp): Structured-CoT bounded-thinking compose + grammar-dialect fix (#214)
* Add llama.cpp bounded-thinking compose * fix(structured-cot/llamacpp): ship no-underscore grammar variant + record on-rig validation On-rig validation found the xgrammar deepseek-scratchpad.gbnf does NOT parse under llama.cpp's GBNF — underscored rule names (line_char/line_chars) trip "parse: error parsing grammar: expecting newline or end at _char", and llama-server silently falls back to UNCONSTRAINED generation (FREE==FSM token counts — silent no-fire). Fix: deepseek-scratchpad.llamacpp.gbnf — identical language, rule names with no underscores (llama.cpp GBNF is [a-zA-Z0-9-] only). Validated 2026-05-24 on 1x 3090: parses, fires (reasoning_content = PLAN/NOTE/VERDICT), coexists with --spec-type draft-mtp (draft accept 0.66-0.88). FREE->FSM 2113->1009 tok (2.1x) and 934->331 (2.8x) @ temp 0.6. Repointed the llama.cpp compose + READMEs + STRUCTURED_COT.md llama.cpp examples to the variant; left the vLLM grammar + refs untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * docs(structured-cot): redact pre-existing /home/wasif absolute paths Pre-commit leak scan caught host-local /home/wasif/structured-cot/runs/... paths in the 'What's saved on disk' section — pre-existing (2026-04-30 doc), not introduced by this branch. Redacted to host-local relative form per no-internal-paths-in-public-docs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
c7fc9ca638 |
docs: correct single-card llama.cpp/ik_llama ctx 262K -> 200K (shipped default)
The single-card MTP defaults were bumped 262144 -> 200000 (PR #201 + the ik commit) after the "boots != fills" finding: 262K boots and passes verify-stress but only fills ~125K before the FA-scratch-at-fill OOM; 200K fills ~183K with ~1.1 GB margin (see docs/CLIFFS.md). The docs lagged. Corrected to 200K (the shipped, fillable default) across the Q4_K_M-MTP + ik IQ4_KS single-card paths: - README.md (quick-start + TL;DR robustness pitch) - docs/SINGLE_CARD.md (cliff-warning row, config table, llamacpp/default + ik narratives, fallback table, Cliff 2 explainer) - docs/engines/IK_LLAMA.md (defaults + max-context override) - models/qwen3.6-27b/llama-cpp/README.md (single/mtp.yml heading + workhorse line) - scripts/switch.sh (--list/--help comment) Also reconciled an older drift: llamacpp/default = llamacpp/mtp alias (collapsed 2026-05-22), Q4_K_M MTP, no vision, ~52/61 TPS -- several spots still described it as the retired Q3_K_XL @ ~21 TPS + vision. Left for a separate fill-test (smaller-quant / no-MTP headroom may genuinely exceed 200K): the "going-to-262K" max-ctx sections in docs/engines/LLAMA_CPP.md + VLLM.md, and the Q3_K_XL legacy rows in models/qwen3.6-27b/llama-cpp/README.md. Dual-card vLLM 262K, the model's 262144 max_position_embeddings, and the CLIFFS "boots!=fills" explainer are correct as-is and untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
|
eba4870893 |
feat(ik-llama): wire iq4ks-mtp + iq4ks-mtp-vision into launch.sh + switch.sh (#189)
The ik_llama composes shipped (#180) but were never registered, so they were raw-`docker compose`-only — awkward now that ik IQ4_KS is the path we point VRAM-tight / WSL single-card users to (its ~0.5-0.8 GB leaner footprint is the one place ik's edge pays rent). Register the two stable variants in compose_registry.py (→ switch.sh derives them) and add them to launch.sh's variant maps (compose/model/engine/kvcalc/order/port/container) so the wizard offers them and `switch.sh ik-llama/iq4ks-mtp` works like the others. The experimental two-stage compose stays raw-compose-only until benched. Verified: registry imports (45 entries), ik present in all 7 launch maps, bash -n clean, composes config-clean. Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
|
28cff3f559 |
refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias) (#181)
* refactor(llamacpp): collapse single-card composes 3→2 (default = mtp alias) Mainline Q4_K_M reaches the full 262K with -ub 512 (validated: boot + serve + verify-stress 7/7 + soak PASS), so the old vanilla Q3_K_XL docker-compose.yml no longer earns a separate profile. Collapse the llama.cpp single-card tree to two compose files: - `llamacpp/default` is now an ALIAS for `llamacpp/mtp` (mtp.yml) — launch.sh + compose_registry repointed. All references (estate-CLI default, tests, docs) keep resolving; the variant name survives. - Retired models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml. - mtp.yml: document the -ub 512 -> 262K recipe in the header. - mtp-vision.yml: align with the ik vision profile — default 160K (was 49K) via -ub 512, and pin --image-min-tokens 1024 / --image-max-tokens 4096 (mainline previously left image bounds at the model default). Validated: full-res 2048^2 image @ ~22.2 GB / 24 (~2.4 GB headroom). - switch.sh / README / SINGLE_CARD.md: descriptions + the dead docker-compose.yml link repointed. Note: on mainline llama.cpp the VRAM lever is -ub (not -b — mainline honors -ub independently, unlike ik_llama which forces n_ubatch=n_batch). Docs follow-up (deferred): SINGLE_CARD.md prose still describes `default` as the old Q3_K_XL path in places; mtp.yml's "froggeric suppresses --reasoning off" note is now disproven (froggeric v19 + --reasoning off works on mainline b9246). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * docs(engines): ik_llama coverage + MTP-merged-on-mainline corrections - Correct stale "llama.cpp MTP via community PR / not merged" across INFERENCE_ENGINES.md, LLAMA_CPP.md, engines/README.md — PR #22673 merged on mainline 2026-05-16 (the b9246 image we bench is post-merge). - Add ik_llama.cpp coverage that was missing: engine list / repo tree / supported-models table (README), workload picker (SINGLE_CARD), choose table (engines/README), model engines line (qwen3.6-27b/README). - Document new ik_llama features: two-stage ngram+MTP (PR #1789), -vhad, --recurrent-ckpt-mode, -ctk-first (IK_LLAMA.md, INFERENCE_ENGINES.md). - IK_LLAMA gotcha: native template won the 8-pack A/B (103 vs 99, toolcall tied) so the ik composes default native; froggeric stays vLLM-only. - New compose iq4ks-two-stage.yml (ngram-mod + MTP via --spec-stage). Boots + two-stage spec-dec engages on cu13-server (validated 2026-05-22); perf bench pending. Review fixes on top of the drafted changes: repo-tree └→├, a 131K↔262K self-contradiction in llama-cpp/README, a now-stale "mainline MTP still open PR" bullet, and softened/attributed the unverified IQ5_KS PPL figures. External PR links verified resolving; no internal paths leaked. Docs drafted via the Qwen coding agent; reviewed, corrected, and the new compose boot-validated by Claude. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * refactor(composes): promote native template default + bump llama-cpp context ceilings llama-cpp/mtp.yml: default ctx 131K → 262K (full native), -ub 512 llama-cpp/mtp-vision.yml: default ctx 49K → 160K, -b/-ub 1024 ik-llama/mtp{,-vision}.yml: native template (froggeric v19 A/B'd, native won 8-pack 103 vs 99), +'-vhad' (V-cache Hadamard), +'--recurrent-ckpt- mode auto' (hybrid-aware MTP rejection), removed ipc:host (not needed) Retire stale artifacts: ik-llama/patches/froggeric-chat-template/ — native won, no longer mounted llama-cpp/recipes/single-card-*.sh — host-binary scripts superseded by compose files (canonical path since v0.8.0) 8-pack ship-gate: 99/150 (vs prior native 103/150) — within noise, 6/8 packs within ±1. Delta is cli-40 variance, not a flag effect (-vhad and --recurrent-ckpt-mode are not output-quality levers). * docs(composes): guard -np 1 with hardware-conditional rationale Add inline ⚠ comment to all 5 single-card composes explaining why -np 1 is intentional on a single 24 GB card (compute-bound, not memory-bound — extra slots divide throughput, don't multiply it). Also expose NP as an env var on the two mainline composes (was hardcoded; ik composes already had ${NP:-1}). The 5090/multi-GPU carve-out is explicit: 'on a higher-throughput card or multi-GPU the trade may flip — re-validate before raising.' Prevents future agents from blindly parallelizing slots on Ampere. * fix(composes): move -np guard comment out of folded scalar The ⚠ -np 1 rationale comment was inside 'command: >-' in all 5 single-card composes. YAML folded scalars treat # lines as content, not comments — docker compose config failed on all 5 files. Move the comment block to YAML-level (between volumes: and command:). All 5 now pass 'docker compose -f <file> config'. Lesson learned: always validate compose edits with 'docker compose -f <file> config >/dev/null' before committing. * docs(FAQ): expand WSL2 section with GPU overhead guidance The existing FAQ entry only mentioned TDR and expandable_segments gotchas. Expanded to explain the ~1.3 GiB invisible WSL2 overhead, how it affects each engine path differently (dual-card: noise, single-card vLLM: one env var, single-card llama.cpp: lower ctx), and provide a concrete VRAM budget table showing which composes OOM on WSL2 vs which fit at defaults. Key finding: mainline llama.cpp mtp.yml at 262K leaves only ~1.5 GB headroom on Linux — WSL2 eats that to ~0.2 GB (instant OOM). Fix is CTX_SIZE=131072 (drops to 20 GB, ~2.7 GB WSL2 headroom). ik_llama composes (IQ4_KS, smaller weights) fit at defaults on WSL2. No compose file changes — this is FAQ docs only. The llama.cpp composes already expose CTX_SIZE as an env var override. --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
ed1507122c |
llama.cpp single: thinking-off policy alignment + MTP profile family
Closes the dataextract 0/15 quality regression caused by mounting the vLLM-only froggeric Jinja template on the llama.cpp engine, which silently suppressed --reasoning off. Native template + --reasoning off restores it. Stack-wide Qwen3.6 thinking-off-by-default policy now shipped on the llama.cpp path (matches 22/24 vLLM composes). Adds three named single-card profiles: - llamacpp/default (262K vanilla, unchanged) — cliff-immune fallback - llamacpp/mtp (NEW: 131K + MTP n=2) — single-card workhorse, ~60 TPS code - llamacpp/mtp-vision (NEW: 49K + MTP + vision) — first stack profile combining MTP + vision on build 9235 (the older "strip mmproj when MTP" rule was obsolete; sweep-verified MTP + vision coexist) Retires single/concurrent.yml — single-card concurrency is anti-value (per-slot ctx ~48K, worse than every other profile). Concurrency belongs on dual. Fixes a latent user-facing bug: engines/llama-cpp-mainline.yml had supported_drafters: [draft-mtp] (the llama.cpp CLI flag value) while the compat-layer C7 gate compares against drafter spec_method (mtp). Net effect pre-fix: llamacpp/mtp would have been silently filtered out of launch.sh candidate lists. Fixed [draft-mtp] -> [mtp]. Lowers -ub default 2048 -> 1024 on the llama.cpp single composes. The per-pass activation peak halves; verify-stress goes 5/7 -> 7/7 including the 60K + 91K needle rungs previously treated as architectural Cliff 2 territory. Cliff 2 single-prompt at 50-60K was config-driven on llama.cpp, not architectural; CLIFFS.md note added (vLLM Cliff 2 narrative unchanged — different kernel-level failure). Migrates 6 stale call sites for the engine-id rename (llama-cpp-mainline -> llama-cpp-local) that earlier work lagged. Measured impact, Config A (llamacpp/mtp, 131K, MTP n=2, ub=1024): - bench: 51.28 narr / 59.72 code decode TPS (n=3, CV 1.9%/0.5%) - verify-stress: 7/7 PASS (incl. 60K + 91K needle recall) - quality 8-pack: 102/150 (68%) — beats every Qwen vLLM dual config in discussion #119 by 6-16 pp - aider-polyglot-30: 17/30 (56.7%) — matches Qwen vLLM bf16 dual exactly (17/30) on half the hardware - per-GPU code TPS (59.72) ~equal to vLLM dual configs (60-63), confirming the engine-side per-card rate is identical and vLLM dual's aggregate advantage is purely from the second card Config B (llamacpp/mtp-vision, 49K, MTP + vision): - multimodal vision probe passed - bench: 56.52 narr / 66.17 code decode TPS (n=5) - verify-stress: 7/7 PASS Tests green: test-launch-compat, test-profiles-compat, test-switch-registry-parity, test-pullgate-gates, test-patch-attribution. Leak-grep clean. YAML lint clean. Performance charts regenerated to include the two new entries. Doc alignment: SINGLE_CARD, CLIFFS, FAQ, EXAMPLES, README, llama-cpp README, COMPOSE_GENERATOR, PULL_GATE all reflect the new family. CHANGELOG.md + UPSTREAM.md left intact per append-only history rule. Followups (queued, not blocking): #400 MTP retune A/B (n=4 + p-min=0 on the fixed config), #403 soak-test container glob (unlocks Cliff 2b validation on llama.cpp), #405 benchlocal-cli port-offset for parallel rebench, #401 + #404 ik_llama + Tom's TurboQuant trackers. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
e6503bc046 |
feat(switch): v0.8.2 STEP V3 — switch.sh ↔ compose_registry parity (CONTRACT-2b-ii)
switch.sh now DERIVES its VARIANTS + VARIANT_DEFAULT_PORT tables from compose_registry.py (the single source of truth) instead of a hardcoded `declare -A` map that had drifted: 20 registered composes (incl. vllm/dual-int8 — shipped as dual/int8.yml but unlaunchable, which cost a real A/B a config pivot) were not launchable. All 42 registered composes are now launchable; zero launcher-only ghosts. New deterministic test-switch-registry-parity.sh (no docker/GPU/network) fails CI on ANY registry↔launcher mismatch in EITHER direction: registry ⊆ launcher (zero registered-but-unlaunchable), launcher ⊆ registry (zero ghosts) — driven through the FULL shipped `switch.sh --list` path so a manual post-derivation ghost is caught too — plus spec parity, port parity, and every resolved compose file exists on disk. Negative-case verified: a synthetic registry/launcher mismatch makes the test exit 1. Additive: no [C0]/decision-logic change; no shipped compose touched. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunna
|
c3063838bf | feat(launch): export profile vllm pins | ||
|
John Karabudak
|
e00626a50e |
feat: unify dual-card composes with NVLink auto-detection
As discussed in #98, collapse 8 Qwen dual-card compose files into 4. NVLink presence is now detected at startup instead of requiring a separate compose per interconnect. The 4 nvlink-*.yml files become deprecated stubs that extend the unified compose with NVLINK_MODE=force_on. Add entrypoint conditionals and env-var substitution so the same compose selects the correct NCCL settings and --disable-custom-all-reduce flag based on detected topology. Extend the same pattern to all 7 Gemma dual composes. |
||
|
noonghunna
|
26985527f7 |
Add hardware-aware compose preflight
Release / release (push) Failing after 50s
|
||
|
noonghunnaandClaude Opus 4.7
|
4866913a10 |
fix(switch): GPU memory pre-flight + widen RUNNING_PATTERN
Release / release (push) Failing after 49s
Catches the failure mode reported by alexpolo1 on Discord 2026-05-11:
switch.sh reports "no club-3090 container running" but the GPU is still
pinned at ~22 GiB from a non-managed process, and the new container
OOMs at boot with a cryptic vLLM ValueError.
Two changes:
1. Widen RUNNING_PATTERN from a hard-coded variant list to `^(vllm-|llama-cpp-)`
so down_running() also catches locally-built and one-off `docker run`
instances under the same image families.
2. Add gpu_preflight() between down_running() and up_variant():
- Queries nvidia-smi for free memory per GPU.
- If any card has <80% free (insufficient for the typical 0.92
gpu-memory-utilization), abort with a diagnostic listing the
holding PIDs from nvidia-smi --query-compute-apps and suggesting
specific cleanup commands.
- FORCE=1 env bypasses the check for users who know what they're doing.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
noonghunnaandClaude Opus 4.7
|
acd7ffb67c |
restructure: promote topology to a directory level (single/dual/multi4)
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`, with topology as a folder rather than a filename prefix. Solves all 7 inconsistencies surfaced in the post-rename audit (single-card composes without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity, fine-tunes encoding model name in filename, etc.) by making the directory hierarchy enforce the convention. Layout: models/<model>/<engine>/compose/<topology>/<feature>.yml Where: - <model>: qwen3.6-27b, gemma-4-31b - <engine>: vllm, llama-cpp, sglang - <topology>: single, dual, multi3, multi4, multi8 - <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc. Each topology subdir has a `docker-compose.yml` for the recommended starter — bare `cd <topology> && docker compose up` works because docker compose finds that filename automatically. Variants drop the `docker-compose.` prefix since they're invoked via `-f` flag. 27 compose file moves total: - 18 Qwen vLLM composes redistributed across single/dual/multi4 - 2 Qwen llama-cpp composes into single/ - 6 Gemma vLLM composes redistributed across single/dual - 1 untracked qwopus-bf16mtp moved to dual/ Inside each compose: relative paths to `../patches/` and `../cache/` bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache` to `../../../../../models-cache` (one extra `..` for the new depth). Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs, sibling-table cross-references in compose headers, scripts, patch READMEs, .github issue templates, tools/residency-instrument). scripts/switch.sh VARIANTS map updated; tags themselves unchanged (`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`, `vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.). AGENTS.md "Compose layout" section rewritten to describe the new hierarchy, with concrete examples and the fine-tune exception (`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename prefix until the fine-tune graduates to its own model directory). All switch.sh paths verified to resolve to actual files post-move. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
d33e6f82da |
composes: rename dual4 → multi4 to align topology prefix with MULTI_CARD.md framing
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` / `multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md, MULTI_CARD.md). Compose filename topology prefix should match. `dual4.yml` was an awkward outlier — it had `dual` in the prefix but actually meant TP=4 (4-card config). The clean shape: GPU count is implicit when there's no ambiguity (`single` always 1, `dual` always 2), and explicit when there is (`multi3` / `multi4` / `multi8`). Renames: - docker-compose.dual4.yml → docker-compose.multi4.yml - docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh keep their existing names (backward compat for users running `bash scripts/switch.sh vllm/dual4`); only the file paths in the VARIANTS map are updated. References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md, sibling-table cross-references in 7 other compose headers. AGENTS.md "Topology prefix" row tightened: examples now read `single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink` which is actually an interconnect feature suffix; dropped `quad` which doesn't exist as a name in our convention). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
fe86b48c21 |
docs+composes: align Gemma 4 compose names to Qwen's <topology>-<feature>.yml convention
Rename the 4 shipped Gemma composes to drop the redundant `gemma-` prefix
(model is already named by the parent directory `models/gemma-4-31b/`):
- gemma-mtp.yml → dual.yml (recommended default, MTP implicit
— matches Qwen's `dual.yml` shape)
- gemma-mtp-int8.yml → dual-int8.yml (MTP + INT8 PTH KV variant)
- gemma-mtp-tp1.yml → single.yml (TP=1 single-card variant — boot
OOMs on Ampere 24 GB; for 32 GB+ only)
- gemma-dflash.yml → dual-dflash.yml (DFlash drafter variant — same shape
as Qwen's `dual-dflash.yml`)
Convention going forward (matches what Qwen has done since launch):
- File name: `<topology>-<feature>.yml`, model implied by parent directory
- Topology: `single` | `dual` | `dual-nvlink` | `dual4` | etc.
- Feature suffix: `-turbo` | `-dflash` | `-int8` | `-awq` | etc.
Registry tags in `scripts/switch.sh` keep their existing names
(`vllm/gemma-mtp`, `vllm/gemma-mtp-tp1`, `vllm/gemma-dflash`) — only the
file paths in the VARIANTS map are updated. This preserves backward
compat for users running `bash scripts/switch.sh vllm/gemma-mtp` etc.
Container names (`vllm-gemma-4-31b-mtp` etc.) are unchanged — they already
include the model name in the `vllm-<model>-<feature>` form, so no rename
needed there.
Plus continuing the at-a-glance profile schema rollout from
|
||
|
Dan
|
63ab224c57 | Add dual-nvlink-dflash-noviz compose variant (NVLink + DFlash N=5, 200K ctx, no vision) | ||
|
Dan Bedford
|
0d199a1f50 |
Add docker-compose.dual-nvlink-dflash.yml (#92)
Adds NVLink-enabled DFlash compose for 2× 3090 with NVLink bridge. Mirrors `docker-compose.dual-dflash.yml` but enables NCCL P2P over NVLink (`NCCL_P2P_LEVEL=NVL`) and re-enables vLLM's custom all-reduce kernel. Drops `expandable_segments=True` per JusefPol's NVLink startup-crash report (#31). Validation (rig: 2× 3090 + NVLink, 230W cap, EDT — see PR body): - verify-full: 8/8 PASS - verify-stress: 7/7 PASS (incl. 91K Cliff 2 needle) - soak-continuous: PASS (0 errors, 0 silent-empty, 0 MiB growth, 100% TPS retention) - Canonical bench: narr 101.55 / code 163.33 wall TPS (CV 1.8% / 1.9%) NVLink lift vs his own PCIe baseline (`dual-dflash.yml` 86.62 / 141.02): +17% narr / +16% code — matches the +15-19% NVLink lift the controlled A/B in BENCHMARKS shows on DFlash paths (K+1 verify is heavily cross-card matmul). Port: 8018. Status: community-contributed, experimental. |
||
|
noonghunnaandnoonghunna
|
89b65dd57c |
Add Gemma 4 + DFlash compose (vLLM PR #41703 Codex-rebased overlay) (#81)
Cross-rig data on z-lab/gemma-4-31B-it-DFlash block-diffusion drafter — first Ampere consumer benchmark of DFlash on Gemma 4. PR #41703 was needs-rebase against pre-SpecDecodeBaseProposer-refactor main; ChatGPT/Codex cherry-picked the 6 PR commits onto upstream/main 5d0fd87038b cleanly with one manual fix on top (_warn_if_multimodal → _raise_if_multimodal rename, otherwise multimodal inputs throw NotImplementedError). Bench at shipped n=7 (TP=2, 2× 3090 PCIe, 230W cap): narrative: 95 wall TPS (1.56× over no-spec-decode baseline) code: 168 wall TPS (2.74× over baseline) Avg accept code: ~60%, AL 5.23 n-sweep summary (n=4..15): code TPS saturates at n=7; n=8 strictly dominated by n=7 (worse on both narr+code); n=15 past the knee. Narrative monotonically degrades with bigger n — n=5 is best for prose at 109/141, override hint documented in compose comment for chat workloads. Soak PASS: 100 turns, 0 errors, 0 silent-empty, 0 MiB growth, 98.6% TPS retention, p50 decode 55.78 TPS (vs 52.71 at n=5 — n=7 is strictly better under soak conditions too, with 2.2 GB lower peak VRAM). DFlash vs MTP on Gemma 4: DFlash wins code (+18%), MTP wins narrative (+15%). Different operating regimes — block-diffusion's larger draft horizon helps deterministic code more than prose. Adds: - models/gemma-4-31b/vllm/compose/docker-compose.gemma-dflash.yml - models/gemma-4-31b/vllm/patches/vllm-gemma4-dflash/ (12 RO-mounted Python files + README documenting provenance + drop conditions) - scripts/switch.sh entry: vllm/gemma-dflash → port 8032 - BENCHMARKS.md row under Gemma 4 31B section Drop the entire patches dir + overlay block when PR #41703 merges and a vLLM :nightly tag rebuilds against it. Co-authored-by: noonghunna <[email protected]> |
||
|
|
e6e6c827ba |
add Gemma 4 31B + Google MTP drafter (first Ampere data) (#68)
* add Gemma 4 31B + Google MTP drafter — first Ampere consumer cross-rig data Adds models/gemma-4-31b/ tree with two compose variants (TP=2 + TP=1), vendored vLLM PR #41745 overlay, switch.sh / launch.sh wiring, plus the script-level extensions needed to make `bash scripts/{bench,verify-*, soak-test}.sh` auto-detect Gemma containers. Validated 2026-05-05 on 2× RTX 3090 (Ampere sm_86, PCIe-only): - canonical bench (3 warm + 5 measured per prompt): narrative wall TPS = 108.87 (CV 3.2%) code wall TPS = 142.25 (CV 2.3%) - soak-continuous (5 sessions × 5 turns × 4 prompts = 100 turns): verdict PASS, 0 errors, 0 silent-empty, 0 MiB VRAM growth, 98.3% TPS retention, p50 decode 111.79 - VRAM 22.5 GB/card. Speedup vs no-spec-decode baseline: 1.79× narr / 2.31× code. First published Ampere consumer numbers on Google's Gemma 4 MTP "assistant" drafters (released 2026-05-05). Discussion #67 has the announcement + upstream context. BENCHMARKS.md gets a new "Gemma 4 31B" section with both TP=2 (working) and TP=1 (upstream-blocked on Ampere) rows. Components landed: - models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml (TP=2) - models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml (TP=1 — preserved for re-test when upstream Ampere fp8 path lands) - models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/ — vendored overlay of vllm-project/vllm#41745 (lucianommartins/gemma4-mtp). 7 modified Python files RO-mounted over the stock nightly image. Same shape as vllm-marlin-pad. Drop the entire tree when PR merges + propagates. - scripts/switch.sh + launch.sh: vllm/gemma-mtp + vllm/gemma-mtp-tp1 variants registered. RUNNING_PATTERN extended for vllm-gemma-4-31b*. - scripts/preflight.sh: autodetect_endpoint extended for gemma containers. - scripts/soak-test.sh: container grep extended. - scripts/soak-helper.py: SOAK_NO_CHAT_TEMPLATE_KWARGS=1 env knob to skip the Qwen3-specific `chat_template_kwargs.enable_thinking` body field for non-Qwen models that reject it (Gemma 4, etc.). - BENCHMARKS.md: new Gemma 4 31B section with both TP rows. - docs/UPSTREAM.md: PR #41745 row + the Ampere fp8-blocked finding + transformers 5.8.0 dependency entry. Pre-merge dependencies (drop when both land): 1. vllm-project/vllm#41745 → drop the patches/ tree + the volume block 2. transformers ≥ 5.8.0 → drop the entrypoint pip install line Out of scope: - TP=1 single-card. Tested + upstream-blocked on Ampere consumer: fp8 KV needed for fit; fp8_e4m3 hits Triton "fp8e4nv not supported" on sm_86; fp8_e5m2 rejected by gemma4_mm.py:1336 allowlist. Compose is preserved with the failing config baked in for future re-test. - Gemma 4 26B-A4B MoE single-card. Active params ~4B → should fit cleanly without fp8. Queued as the obvious follow-up. - vllm/vllm-openai:gemma4-0505-cu129 image swap. The recipe-image is Hopper-tagged and may not include sm_86 kernels; testing it would collapse this whole 4-layer wrangle to a one-line image bump but needs explicit Ampere validation first. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * models/gemma-4-31b/vllm/cache: gitignore + README (mirror qwen3.6 pattern) Replaces the .gitkeep stubs with the standard .gitignore (ignore everything except .gitignore + README.md) + README documenting the cache lifecycle. Mirrors models/qwen3.6-27b/vllm/cache/ exactly. Boot times this enables (validated tonight): - cold first boot: ~3-7 min (TP=1 / TP=2) - warm subsequent boot: ~2-3 min (cache hit) Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
|
d6e7c1f805 |
add dual-nvlink-turbo variant (rebased on v7.72.2 master, sibling-table edits dropped) (#65)
Adds docker-compose.dual-nvlink-turbo.yml: NVLink + TurboQuant KV (TQ3) + MTP n=3 + 4-stream + 262K. Mirrors current dual-turbo.yml (post-#59) with the three NVLink-specific deltas applied: - NCCL_P2P_LEVEL=NVL (vs NCCL_P2P_DISABLE=1 on PCIe) - PYTORCH_CUDA_ALLOC_CONF without expandable_segments (JusefPol crash repro) - --disable-custom-all-reduce removed (NVLink P2P → custom kernel wins) Image pin matches master (nightly-01d4d1ad3); retired sidecars (patch_workspace_lock_disable, patch_tolist_cudagraph) excluded — superseded by Genesis v7.72.2 PN34 + P78 natives. Bench (danbedford rig): 101.49 narr / 133.20 code wall TPS, 20.4 GB/card. +12.6% narr / +10.7% code over own PCIe-only dual-turbo baseline. A/B tested on the same rig. Bench was on v7.69 — re-bench welcomed on the v7.72.2 pin this compose ships with. Wired into scripts/switch.sh + scripts/launch.sh as `vllm/dual-nvlink-turbo` on port 8017. Added BENCHMARKS row + updated UPSTREAM tracker (Marlin pad PR row) to list the new compose. Sibling-table standardization across the 7 other compose files dropped per the original author's comment — that's better filed as a separate issue scoped to the table format. Co-authored-by: noonghunna <[email protected]> Co-authored-by: Dan <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
b6c8708209 |
feat(preflight): compose-dependency + HF_TOKEN + KV-format checks (#37, #47, #219)
Closes the UX gap raised by @snoby on #37 — previously, launching a compose that needs the DFlash draft model without first running WITH_DFLASH_DRAFT=1 setup.sh would fail with vLLM's generic "Invalid repository ID or local directory" pydantic error. Adds three new preflight functions in scripts/preflight.sh: 1. preflight_hf_token (soft warn) — wired into setup.sh before the model download. Catches missing HF_TOKEN early and tells the user exactly what to do (visit hf.co/settings/tokens, accept T&C, export the token). Skip via PREFLIGHT_NO_HF_TOKEN=1. 2. preflight_compose_deps (hard error) — wired into switch.sh just before docker compose up. Parses the target compose YAML for known model-path patterns (qwen3.6-27b-dflash, qwen3.6-27b-mtp-head, qwen3.6-27b-autoround-int4) and verifies each exists on host. If missing, emits a clear "Fix:" line with the exact setup.sh invocation needed (e.g. WITH_DFLASH_DRAFT=1). Refuses to proceed with exit 1. Skip via PREFLIGHT_NO_COMPOSE_DEPS=1. 3. preflight_kv_format_hint (soft warn) — wired into switch.sh as the last pre-up check. Detects smallest VRAM via nvidia-smi; if <24 GB and the target compose uses turboquant_3bit_nc KV, emits the @efschu finding from #47 (TQ3 → fp8_e5m2 swap rule) with cross-link to docs/HARDWARE.md + a tools/kv-calc.py one-liner to predict the user's specific config. Skip via PREFLIGHT_NO_KV_HINT=1. Tested on this rig: - HF_TOKEN unset → warns - Missing DFlash dir → hard errors with WITH_DFLASH_DRAFT=1 hint - Missing main model → hard errors with generic setup.sh hint - 24 GB rig + TQ3 compose → silent (correct, TQ3 is right pick) - Simulated 20 GB rig + TQ3 compose → fires the fp8_e5m2 hint All preflights are individually skippable via PREFLIGHT_NO_<NAME>=1 env vars (matches existing PREFLIGHT_NO_FETCH=1 / PREFLIGHT_NO_GENESIS_PIN=1 patterns). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
b956c85477 |
feat(bounded-thinking): Phase 3 grammar A/B complete; DeepSeek scratchpad is the new recommended grammar
After the full Phase 3 5-grammar A/B (HE+ 164 + LCB v6 50, n=214 problems
× 5 conditions = 1070 generations), bounded-thinking.yml is updated to
recommend the DeepSeek scratchpad grammar (PLAN/NOTE×0-15/VERDICT FSM at
tools/grammar-eval/deepseek-scratchpad.gbnf) as the default.
Phase 3 results (combined HE+ + LCB v6, all FSM-enforced):
- DeepSeek scratchpad: 87.4% Pass@1 (+1 net vs andthattoo, +4pp on LCB)
- andthattoo G/A/E: 86.9% (the originally-published technique)
- Holiday tagline: 86.4%
- PROMPT_TERSE (no FSM): 82.2% (Phase 2's n=30 win was subset-selection bias)
- FREE (no constraint): 78.0% (baseline)
Phase 1 reproducibility is exact: HE+ FSM Δ +4.3pp / LCB v6 Δ +24.0pp,
both match Phase 1's published numbers — validates the bench harness.
Compose ships unchanged engine-side (same vLLM image, same Genesis stack,
same TQ3 KV, same MTP n=3, same enable_in_reasoning flag). Only the
docstring's recommended-grammar pointer + the on-disk grammar files in
tools/grammar-eval/ change. Three grammars are now validated and available
client-side via extra_body={"structured_outputs": {"grammar": ...}}:
- DeepSeek scratchpad (default, best LCB)
- andthattoo G/A/E (originally-published, ~4× tighter think budget)
- Holiday tagline (extreme 24-token compression, wins LCB by 4pp too)
Combined-accuracy spread is within noise (0.5pp at n=214), so we ship one
compose rather than three siblings — choice is at the client, not at the
compose level.
Bug fix bundled: tools/grammar-eval/subset-bench.py --full --include-lcb
mode now correctly threads dataset kind through run_condition so LCB
problems use mod.run_tests_livecodebench instead of HE+ assertion-based
testing. Without this, the Phase 3 LCB shard crashed with KeyError:
'prompt' on the HE→LCB transition.
This wraps active research on bounded-thinking. Reopen if upstream FSM-
regress cluster behavior changes (Genesis pin bump, vLLM grammar engine
swap, model-family change), or if user demand surfaces for a sibling
compose pinning a non-default grammar.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
||
|
Will HampsonandWhamp
|
e3f8f02047 |
feat(qwen3.6-27b/vllm): add dual4 + dual4-dflash composes (TP=4, 4×3090, #44)
First 4-card variants for Qwen3.6-27B vLLM. Two new composes for 4× RTX 3090 PCIe rigs: - `docker-compose.dual4.yml` — TP=4 fp8/MTP baseline. 63 narr / 76 code TPS, 6.77× concurrency at 262K, ~23.5 GB/card peak - `docker-compose.dual4-dflash.yml` — TP=4 + DFlash spec-decode. 64 narr / 104 code TPS, 2.27× concurrency at 262K, ~22 GB/card peak Both pass verify-full + verify-stress 7/7 (incl. Cliff 2 needle recall at 58K + 91K) on @Whamp's 4× RTX 3090 PCIe rig. Both pass v2 continuous soak (dual4: 20 sessions, 0 MiB growth, 90.8% TPS retention; dual4-dflash: 5 sessions, 0 MiB growth, 100% TPS retention) — first cross-rig confirmation that TP=4 escapes Cliff 2b. Closes #26 (4×3090 wishlist). Co-authored-by: Whamp |
||
|
noonghunna
|
8e9cf70d99 |
fix(soak-test, switch): calibration + boot-progress UX from first cross-rig runs
Three calibration fixes to soak-test + two boot-progress improvements to switch.sh, all from the first cross-rig soak runs on this rig today. soak-test.sh — baseline timing Brief said "capture warm baseline after first turn" but turn 3 ships a 12K-char tool-result paste that fills prefix cache by ~1000 MiB on the first hit. With baseline at turn 1, every healthy config false-positives with "growth > 200 MiB threshold" on session 1 alone. Fixed: capture baseline at end of session 1 (after all 5 turn shapes run once). Sessions 2-N then measure real accretion from steady state. soak-helper.py — decode_tps guard Thinking-mode requests where vLLM bundles all reasoning into the terminal streaming chunk produce wall ≈ ttft (no separate content delta visible). Old code computed decode_tps = completion_tokens / (wall - ttft) with wall - ttft ≈ 0, yielding ~2 billion TPS. Fixed: when ttft is None OR wall - ttft < 100ms, report decode_tps = 0 (caller filterable). soak-helper.py — TPS outlier filter in cmd_summary Defensive — even if a future helper bug regresses, we filter unrealistic decode_tps values (>500 t/s) from all summary computations. tps_retention metric was previously being inflated by 2e9 outliers in early sessions before the helper-bug landed mid-run. Cross-rig validation results: Config Baseline Max VRAM Growth Verdict vllm/long-text 180K + 0.93 23316 23316 0 MiB PASS vllm/long-vision 145K + 0.95 22778 22778 0 MiB PASS Both configs are soak-clean under v1 synthetic shapes. Long-vision was expected to FAIL based on issue #41 — but our 5-turn fixtures reset the conversation each turn while hermes accumulates context across turns. v1 catches a class of issue (raw VRAM accretion across requests); v2 fixtures will need session = continuous conversation to catch the hermes-shaped class. Methodology limitation documented in CHANGELOG. switch.sh — wait_ready crash detection + boot-stage progress Closes the "container crashed silently in 2s, you wait 600s for nothing" UX gap that surfaced today when long-text's MODEL_DIR was misconfigured. - Crash detection: docker inspect -f '{{.State.Running}}' between polls; if false, dump last 30 log lines + exit 1 immediately (5s vs 600s) - Boot-stage markers: grep docker logs for "Resolved architecture", "Loading weights", "Compilation finished", "Capturing CUDA graphs", "Application startup complete" — surface one line per phase transition so the wait isn't silent. Validated end-to-end on long-vision boot today: visible progress at 60s / 68s / 80s / 120s / 176s / 196s = ready. |
||
|
noonghunna
|
43fe2a4e20 |
feat: detect repo drift in preflight + add scripts/update.sh
Two-part addition for the most common stale-setup pattern: user cloned
weeks ago, master has moved (Genesis pin bumps, compose changes, vendored
patch updates), they re-run their compose, hit a stale config, and file
an issue we already solved on master. Wispborne's _register_op_once and
GuiPerPT's pre-pull boot OOM both surfaced through this loop.
scripts/preflight.sh — preflight_repo_drift:
- Skips silently if not a git repo, on a non-master branch, or if
PREFLIGHT_NO_FETCH=1 (offline rigs / CI / forks tracking elsewhere).
- Verifies origin remote is noonghunna/club-3090 (avoids false positives
on forks pointing elsewhere).
- timeout 5 git fetch --quiet origin master — bounded so flaky networks
don't block boot.
- On behind > 0: WARN with commit count, last-fetch age (h/d), and the
one-line fix command. Soft-warning, never blocks. Tells user about
PREFLIGHT_NO_FETCH=1 for opting out.
Wired into both launch.sh and switch.sh, alongside the existing
preflight_genesis_pin so users get one consolidated stale-setup signal.
scripts/update.sh — the easy upgrade path:
- Refuses on dirty tree (git status --porcelain) — surfaces the local
edits and tells the user to commit or stash first. We don't clobber
the rare user who's been editing a compose locally.
- Refuses on non-master branch — feature branches and fork-trackers
should pull manually; this script is the master-from-origin path.
- git pull --ff-only — no merge commits, no rebase ambiguity. Diverged
branches get an explicit error pointing at git pull --rebase.
- Re-runs setup.sh (idempotent — re-pins Genesis, re-vendors Marlin).
- Tells the user to restart their variant via switch.sh — doesn't auto-
restart, so they can A/B old-vs-new if they want.
- --dry-run shows the plan without changing anything.
- --force re-runs setup.sh even when up-to-date (for "I edited Genesis
by hand and want it re-pinned" cases).
Why detection-then-explicit-command instead of "press y to auto-update":
the user will rarely have local commits (they're consumers of the recipes,
not vLLM contributors), but we still want consent — they should see what
they're committing to. The dirty-tree guard handles the rare custom-edit
case without nagging the common path.
|
||
|
noonghunna
|
75de7c95db |
feat(launch/switch): register vllm/dual-nvlink as a known variant
Wires JusefPol's NVLink compose into the same machinery the rest of the
variants use:
- switch.sh — adds vllm/dual-nvlink to the usage doc, VARIANT_DEFAULT_PORT
(8014, matches the compose's PORT fallback), and the VARIANTS map. Now
'bash scripts/switch.sh vllm/dual-nvlink' works the same way 'vllm/dual'
does.
- launch.sh — adds the variant to LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER
so 'bash scripts/launch.sh --variant vllm/dual-nvlink' resolves the
endpoint URL + container name correctly for post-launch verify.
Intentionally NOT added to the launch.sh interactive wizard's dual-card menu
(line 169). The wizard runs on every fresh setup, and offering an NVLink
option to a PCIe-only user would silently boot a config that crashes on
their topology. Users with NVLink can invoke explicitly via --variant.
|
||
|
noonghunnaandClaude Opus 4.7
|
d552ed9216 |
feat(preflight): warn when Genesis tree out of sync with setup.sh's declared pin
Catches the failure mode @wispborne hit in #32: user pulled latest club-3090 (which bumped GENESIS_PIN in scripts/setup.sh from old → new), but didn't re-run setup.sh. The on-disk Genesis tree at models/qwen3.6-27b/vllm/patches/genesis/ stays at the OLD pin while setup.sh's declared pin advances. vLLM boots against the outdated tree and hits mysterious patch failures (e.g. v7.66 PN25's infer_schema crash that v7.69 PN25 worker-spawn registration fixed). Adds new `preflight_genesis_pin` function to scripts/preflight.sh: - Parses `GENESIS_PIN="${GENESIS_PIN:-<default>}"` from setup.sh to extract the declared default value - Reads on-disk HEAD via `git -C <genesis-dir> rev-parse --short HEAD` - Compares (declared short-form vs on-disk short-form match) - On mismatch: emits a [preflight] WARN block with both pins + a "Fix: bash scripts/setup.sh qwen3.6-27b" hint - Soft-warning only — does not block boot Wired into both entry points: - scripts/launch.sh (the wizard-driven path) — adds the check after preflight_running, before the variant pick - scripts/switch.sh (the direct stateless switcher) — adds the check inside up_variant() right before the docker compose up call Skips silently if the Genesis tree hasn't been cloned yet (caller should run setup.sh first; preflight isn't the place to handle that case). Also skips if setup.sh isn't present at the expected path (weird state — silent skip rather than false warnings). Tested locally: - Current state (declared 2db18df = on-disk 2db18df): silent ✓ - bash -n passes on all three modified scripts Coverage gap: users who bypass switch.sh + launch.sh and run `docker compose -f docker-compose.X.yml up -d` directly won't get the warning. That's the canonical-path-vs-bypass tradeoff; we lead with switch.sh in the docs. Closes the procedural follow-up flagged in [#32 comment-4364895764](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364895764) ("setup.sh should be re-run after every git pull that bumps GENESIS_PIN. We could add a check for this — compare setup.sh's declared pin vs the on-disk tree's HEAD, warn if they differ."). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
1f09a059d5 |
fix(scripts): register vllm/long-text-no-mtp in switch.sh + launch.sh
[@wispborne](https://github.com/wispborne) caught: docker-compose.long-text-no-mtp.yml ships in the v7.69 cutover but its corresponding variant key wasn't added to scripts/switch.sh's VARIANTS map or scripts/launch.sh's wizard list. Result: \`bash scripts/switch.sh vllm/long-text-no-mtp\` failed and the wizard never offered it. Adds: - scripts/switch.sh: * VARIANT_DEFAULT_PORT[vllm/long-text-no-mtp]=8021 (matches compose) * VARIANTS[vllm/long-text-no-mtp]= compose path * RUNNING_PATTERN extended to match its container name * Header docs updated: long-text 218K → 180K (Balanced MTP), new long-text-no-mtp 200K (Max-context) line - scripts/launch.sh: * VLLM_OPTS wizard list — both Balanced MTP + Max-context surfaced explicitly, with correct ctx ceilings (180K + 200K, not stale 218K) * LAUNCH_DEFAULT_PORT + LAUNCH_DEFAULT_CONTAINER updated to match Stale 218K references in the wizard description corrected to current shipped values: long-text is 180K (after the 0.93 mem-util backoff + v7.69 cutover), long-text-no-mtp is 200K + 0.95. Reported in [club-3090 #32 comment-4364904521](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364904521). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunna
|
3d151b9edc |
feat(vllm): structured-CoT bounded-thinking compose (cross-rig port)
Port andthattoo/structured-cot to our stack — Qwen3.6-27B AutoRound INT4
dense / 1× RTX 3090 / vLLM nightly + MTP n=3 + TQ3 KV. Re-benched on
full HumanEval+ 164 + LiveCodeBench v6 50.
Headline (max_tokens=4096, greedy):
- HumanEval+ 164: FSM 92.7% vs FREE 88.4% (+4.3pp), 30.7× compression
- LiveCodeBench v6 50: FSM 66.0% vs FREE 42.0% (+24pp), 26.2× compression
The +Δpp partly reflects FSM dodging the max_tokens=4096 truncation trap
rather than pure reasoning gain — see docs/STRUCTURED_COT.md "Honest
caveats" for the full picture.
Three port surprises worth keeping (all in docs):
1. vLLM dev205+ defaults StructuredOutputsConfig.enable_in_reasoning=False;
grammar mask only fires post-</think> unless overridden.
2. Legacy extra_body={"guided_grammar": ...} is silently dropped on
dev205+ (tip-off: identical FREE/FSM token counts). Use the new
structured_outputs.grammar field.
3. Qwen3.6 chat template auto-prefixes <think>\n; drop the leading
literal from upstream grammars when porting.
Files added:
- models/qwen3.6-27b/vllm/compose/docker-compose.bounded-thinking.yml
- docs/STRUCTURED_COT.md (public writeup)
- models/qwen3.6-27b/vllm/diagnostics/structured-cot-bench.md (internal)
Files updated:
- scripts/launch.sh wizard + scripts/switch.sh variant map
- models/qwen3.6-27b/README.md (recommended single-card list, patch surface)
- models/qwen3.6-27b/vllm/README.md (compose menu)
- docs/SINGLE_CARD.md (TL;DR table now four rows)
- CHANGELOG.md (new top entry)
Also reverts the long-text.yml experimental flag added during smoke
testing (the flag now lives only in bounded-thinking.yml) and adds the
Genesis pre-flight check we previously skipped on long-text.
Credit: andthattoo for the technique, the grammar files, and the eval
harness.
|
||
|
noonghunnaandClaude Opus 4.7
|
ebacba1efd |
fix: address open issues #1, #4, #7
Closes / addresses 3 reported issues + adds requested feature:
#7 vid (PORT not honored, MODEL_DIR vs MODELS_DIR confusion):
- All 8 vLLM compose files now use "${PORT:-XXXX}:8000" so .env PORT
flows through. Defaults preserved per-variant (8020 single, 8010-8013
dual). llama.cpp composes already had this pattern.
- scripts/switch.sh: load .env early; per-variant default-port table;
new resolve_ready_url() picks PORT > variant default for the readiness
probe.
- scripts/launch.sh: same default-port table; final endpoint URL printed
to user reflects actual mapped port.
- .env.example: ⚠ box callout that variable names are CASE-SENSITIVE
(MODEL_DIR singular, NOT MODELS_DIR plural — silently ignored).
New PORT section documenting per-variant defaults.
#4 timxx (tools-text.yml fails "Free memory ... less than desired"):
- docs/FAQ.md: new entry "Container fails to start: Free memory..."
explaining the vLLM startup check, the two workarounds (free VRAM /
lower mem-util), and which configs hit it most often (0.97+ mem-util).
- Compose defaults unchanged (0.97 stays the right design target on
headless rigs); the FAQ documents the workaround for users with X11.
#1 fabriciomalta (per-config VRAM column):
- docs/SINGLE_CARD.md: TL;DR table now has VRAM column with mem-util.
- docs/DUAL_CARD.md: TL;DR table same + footnote explaining per-card
semantics and which dual configs would/wouldn't fit on 2× 20 GB cards
(relevant to fabriciomalta's 2× 3080-20GB use case).
#2 tenitram (empty responses) — fixed in master via
|