master
25 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JesseB-Kit
|
6eaeb2bd44 | feat: switch dual-max KV from int8_per_token_head to fp8 | ||
|
noonghunna
|
d2e63b06b6 |
feat(beellama): add beellama.cpp DFlash as first-class compose engine (#268)
Onboards Anbeeld's llama.cpp fork (beellama.cpp) as a registry engine
with two single-card DFlash composes, both experimental pending a
published Docker image:
- beellama/dflash Qwen3.6-27B Q5_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8060
- beellama/gemma-dflash Gemma-4-31B Q4_K_S + Anbeeld DFlash-IQ4_XS,
q5_0/q4_1 KV, 102K ctx, port 8061
Composes reference the locally-built ${BEELLAMA_IMAGE:-beellama-cpp:local}
(entrypoint /app/llama-server) — upstream ships no pullable image, so
they ship as Status: Experimental and surface as (NA: experimental) in
--list (launch is --force-gated). Configs follow the fork's quickstart
docs (Qwen "Precision combo", Gemma "Balanced combo"), deviating only on
the stack-wide --reasoning off default.
Catalog wiring: engine profile beellama-local.yml; two DFlash GGUF
drafters (anbeeld-qwen-dflash, anbeeld-gemma-dflash); compose_registry.py
entries with kvcalc_key="SKIP" (llama.cpp-family, like ik-llama);
weights-map entries (target + draft GGUFs) on both models; q4_1 added to
rtx-3090 supported KV; INFERENCE_ENGINES + UPSTREAM notes; test counts
bumped (engines 8->9, drafters 6->8, registry 45->47, disk 46->48).
Default-resolver invariant preserved: beellama is #1 in
ENGINE_PREFERENCE[single], but its (NA) status + no DEFAULTS row means
the resolver skips it, so qwen3.6-27b single still resolves to
ik-llama/iq4ks-mtp (asserted by test-model-default-resolver.sh). It
auto-promotes only when a published image lands and the composes flip to
production. Full catalog suite green (only the pre-existing
test-submit-bench.sh fixture failure remains).
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
noonghunna
|
26eac83f76 |
feat: model-default resolver + user-pinnable defaults (PR-B) (#266)
* feat(switch): model-default resolver + user-pinnable defaults
Add a two-layer default scheme on top of the existing <engine>/default map.
<engine>/default stays the maintainer's recommendation (read-only to users);
the new <model>/default is the user's preference — their .env pin if set,
else a curated pick for the detected topology.
compose_registry.py: two maintainer knobs next to DEFAULTS —
RECOMMENDED_DEFAULT_MODELS (short opt-in shortlist, not an exhaustive
ranking; new models are NOT auto-added) and ENGINE_PREFERENCE (per-topology
engine order). Plus resolver helpers (curated_default_target,
community_default_target stub, model_default_pin_key, engine_set/model_set,
slug_topology, model_of_slug).
registry-emit.sh: shared resolver model_default_target(root, model, topology)
— the single injection point for both launchers. Precedence ladder: --variant
(caller) -> .env pin -> community seam (None today) -> curated ENGINE_PREFERENCE
walk (skips non-functional (NA) slugs) -> degradation (notice + nearest-lower
topology, else a clear "pick explicitly" message; never crashes). Plus
x_default_dispatch: X/default with X in engine-set -> engine rec; X in
model-set -> model default; else error (engines + model-ids are disjoint).
switch.sh: <model>/default token; --set-default <slug> / --clear-default
<model> (round-trip the .env pin CLUB3090_DEFAULT_<MODELID>); a Defaults view
appended to --list (also standalone via --defaults) marking user-pin vs
curated. PR-A's grouping/markers/counts and the --force gate preserved.
launch.sh: bare invocation -> first installed shortlist model -> its
<model>/default (no full wizard); a pinned fast-path ("Launch your default
<slug>? [Y/n]"); a post-boot offer ("Make <slug> your default? [y/N]"). Any
narrowing flag keeps the explicit wizard path. Also load CLUB3090_DEFAULT_*
pin keys from .env even when MODEL_DIR is exported in the shell (the existing
.env loader is MODEL_DIR-gated, which would otherwise hide the pin).
Pin validation is warn + fall back, never blocking: unknown slug / wrong
model / topology-mismatch / (NA) status -> notice + curated default.
Tests: new test-model-default-resolver.sh (curated walk, (NA) skip,
degradation, X/default dispatch, pin override + all validation paths,
community seam skipped, .env round-trip); test-default-resolver.sh extended
with <model>/default launch.sh dispatch. Full suite green (only the
pre-existing test-submit-bench.sh fixture failure remains); 45 entries
unchanged; kv-calc calibration 17/17.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: document the model-default resolver + user pins
Ship the user/contributor docs alongside the resolver code (repo convention —
update existing docs, no new top-level doc).
- README: single-card realign (ik-llama = fastest blessed single default,
llama.cpp = cliff-immune alternative — matches ENGINE_PREFERENCE single
order); add "pin your default" + <model>/default to Quick start.
- CLAUDE.md (= AGENTS.md): document RECOMMENDED_DEFAULT_MODELS +
ENGINE_PREFERENCE + the shared resolver as maintainer knobs next to DEFAULTS.
- FAQ: extend "switch to a different model" with <model>/default; new "How do
I set my own default config?" Q (two-layer model, --set-default/--clear-
default/--defaults, .env key, warn+fallback validation).
- SINGLE_CARD / DUAL_CARD: the resolver + the per-topology engine order; pin
hint.
- GETTING_STARTED: first-run uses <model>/default + the in-flow "set as
default?" prompt.
- ADDING_MODELS: a new model resolves via ENGINE_PREFERENCE; add a DEFAULTS row
per engine; RECOMMENDED_DEFAULT_MODELS is not auto-grown.
- UPSTREAM: beellama Docker-image row (gates beellama onboarding; the resolver
skips it today and it auto-promotes to single default on catalog).
- models/qwen3.6-27b/CHANGELOG: dated PR-B entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
noonghunna
|
1182d6b2c3 |
feat(registry): slug health/availability flag (#265)
* feat(registry): add slug health/availability flag Add a lifecycle `status` to every registry slug so `switch.sh --list`, launch, and switch are no longer blind to a compose's health. Previously status lived only in compose-header comments, which drifted: the Genesis dual compose declared "Working (with Genesis)" while its pin is parked and it won't boot clean — a user could boot a broken slug unknowingly. - compose_registry.py: `_entry()` gains keyword-only `status` (default "production") + `status_note`, validated against the enum (production/caveats/experimental/preview/upstream-gated/deprecated). Add `compose_header_status()` mapping a compose's profile-schema `Status:` emoji to that enum. - Sweep every compose `Status:` header to a canonical enum value and re-flag the non-functional slugs: all *genesis* + gemma-4-31b single fp8 -> upstream-gated; carnice -> caveats; qwopus + qwen-a3b-preview -> preview; bf16/int8 A/Bs, llamacpp bounded-thinking, PRISM/APEX eval lanes, gemma-4-26b-a4b onboarding -> experimental; tq3-mtp -> deprecated. - registry-emit.sh emits `status` + `status_note` as the last two VARIANT fields; both loaders + the parity tests read the extended field list. - switch.sh --list: status marker (caveats -> "(caveats)", the NA set -> "(NA: <word>)"); model/topology grouping preserved. Launch/switch gate: production launches, caveats launches with a notice, NA warns + requires --force. launch.sh surfaces the flag before delegating to switch.sh. - New drift-guard test test-compose-status-drift.sh: registry status in enum, compose header maps to enum, and the two agree. 45 entries unchanged; kv-calc calibration 17/17; full suite green (only the pre-existing test-submit-bench fixture failure remains). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(switch): add model/variant counts to --list Header line shows supported-model count + total variants with the health split (N production · N caveats · N NA); each model group shows its variant count. Widen the marker column so the (NA: …)/(caveats) markers align. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
5a43d4c475 |
chore(vllm): remove redundant nvlink-* dual composes (NVLink auto-detected) (#257)
The four dual/autoround-int4/nvlink-*.yml composes (nvlink-fp8-mtp, nvlink-turbo, nvlink-dflash, nvlink-dflash-noviz) and their vllm/dual-nvlink* launch slugs are removed. They were thin Docker Compose `extends:` stubs whose ONLY override was NVLINK_MODE=force_on — fully redundant since every base dual compose now auto-detects NVLink at boot via detect_nvlink.sh (NVLINK_MODE=auto): it probes nvidia-smi and flips on NCCL_P2P_LEVEL=NVL + custom-all-reduce when a bridge is present, else NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. NVLink rigs get the identical fast path from the base dual compose with no separate slug; force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual` if auto-detect ever misses. Scope (prune only; no engine-image change — the v0.22.0 stable-engine consolidation is a separate follow-up): - delete 4 nvlink-*.yml composes - compose_registry.py: drop the 4 _entry blocks (50 -> 46 entries) - test-compose-registry-disk.sh: count guards 50/51 -> 46/47 - profile_runtime.yml + patches.yml: drop the 4 nvlink blocks/slug refs - test-profiles-compat.sh: retire the C13/E3 NVLink-required-compose scenarios (the estate NVLink-gating code is now dormant; cleanup tracked as a follow-up, code retained) - docs: switch.sh help, AGENTS.md, HARDWARE.md, UPSTREAM.md, patches/README.md, fp8-mtp/dflash/dflash-noviz compose headers, per-model CHANGELOG removal entry; add a docs/README.md pointer to `switch.sh --list` as the authoritative registry-derived compose x slug matrix Historical NVLink bench rows (JusefPol #31, danbedford #74/#92/#96) are preserved in BENCHMARKS.md. Full test suite green (the lone test-submit-bench failure is a worktree-isolation artifact — it needs gitignored results/rebench/ fixtures absent from a fresh worktree; passes on the working tree). Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
9821c94efb |
refactor(compose): insert <quant> layer + make the registry the single source of truth
Restructure all 47 composes to models/<model>/<engine>/compose/<topology>/<quant-slug>/<serving>.yml and centralize defaults as registry pointers. Move + path-rewire only — no compose runtime-config changes (plus the +1 ../ depth bump each moved file requires, and descriptive names for the former docker-compose.yml defaults). - <quant-slug> = 1:1 with a weights artifact == compose_registry weights_variant == weights.py key (provider-quant form). Fixes the prior coarse `gguf` and the bf16/int8-files-mislabeled-as-autoround_int4 weights_variant. - default = DEFAULTS[(model,engine,topology)] pointer in compose_registry.py; no default.yml / docker-compose.yml. KV-dtype variants (bf16/int8) stay as files under autoround-int4/. - +1 ../ on every relative mount in moved composes (cache/patches/scripts/models-cache). - Rewired: compose_registry (47 compose_path + weights_variant + DEFAULTS), launch.sh, gpu-mode.sh, bench-row-formatter, profile *.yml, generate_compose.py, weights.py (+ back-compat ALIASES), AGENTS.md, ADDING_MODELS.md, BENCHMARKS + doc path refs. - New guard tests: test-compose-registry-disk.sh, test-compose-mounts-resolve.sh. - Variant keys (vllm/dual, ik-llama/iq4ks-mtp, ...) unchanged — launch/switch UX preserved. Validation: docker compose config 47/47 · parity/launch-compat/registry-disk/ mounts-resolve/profiles-compat/model-weights-registry PASS · kv-calc 22/22 · leak-clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
00366a58d7 |
reorg: services/ consolidation + gpu-mode under git + ComfyUI + pin tracker + path updates
Some checks are pending
Release / release (push) Waiting to run
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
noonghunna
|
acd7ffb67c |
restructure: promote topology to a directory level (single/dual/multi4)
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`, with topology as a folder rather than a filename prefix. Solves all 7 inconsistencies surfaced in the post-rename audit (single-card composes without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity, fine-tunes encoding model name in filename, etc.) by making the directory hierarchy enforce the convention. Layout: models/<model>/<engine>/compose/<topology>/<feature>.yml Where: - <model>: qwen3.6-27b, gemma-4-31b - <engine>: vllm, llama-cpp, sglang - <topology>: single, dual, multi3, multi4, multi8 - <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc. Each topology subdir has a `docker-compose.yml` for the recommended starter — bare `cd <topology> && docker compose up` works because docker compose finds that filename automatically. Variants drop the `docker-compose.` prefix since they're invoked via `-f` flag. 27 compose file moves total: - 18 Qwen vLLM composes redistributed across single/dual/multi4 - 2 Qwen llama-cpp composes into single/ - 6 Gemma vLLM composes redistributed across single/dual - 1 untracked qwopus-bf16mtp moved to dual/ Inside each compose: relative paths to `../patches/` and `../cache/` bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache` to `../../../../../models-cache` (one extra `..` for the new depth). Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs, sibling-table cross-references in compose headers, scripts, patch READMEs, .github issue templates, tools/residency-instrument). scripts/switch.sh VARIANTS map updated; tags themselves unchanged (`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`, `vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.). AGENTS.md "Compose layout" section rewritten to describe the new hierarchy, with concrete examples and the fine-tune exception (`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename prefix until the fine-tune graduates to its own model directory). All switch.sh paths verified to resolve to actual files post-move. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
d33e6f82da |
composes: rename dual4 → multi4 to align topology prefix with MULTI_CARD.md framing
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` / `multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md, MULTI_CARD.md). Compose filename topology prefix should match. `dual4.yml` was an awkward outlier — it had `dual` in the prefix but actually meant TP=4 (4-card config). The clean shape: GPU count is implicit when there's no ambiguity (`single` always 1, `dual` always 2), and explicit when there is (`multi3` / `multi4` / `multi8`). Renames: - docker-compose.dual4.yml → docker-compose.multi4.yml - docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh keep their existing names (backward compat for users running `bash scripts/switch.sh vllm/dual4`); only the file paths in the VARIANTS map are updated. References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md, sibling-table cross-references in 7 other compose headers. AGENTS.md "Topology prefix" row tightened: examples now read `single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink` which is actually an interconnect feature suffix; dropped `quad` which doesn't exist as a name in our convention). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
af00ab7bef |
llama.cpp: --reasoning-format none default (opencode unblock, #97)
Qwen3.6's thinking mode emits <think>...</think> blocks that llama.cpp's
peg-native parser routes to OpenAI's reasoning_content field by default.
opencode (and most simple OpenAI-compat clients) ignore reasoning_content
and wait indefinitely for content deltas that never arrive — even though
the server returns 200 cleanly with all tokens decoded.
Diagnosed via @syangsao's curl SSE capture: every delta was reasoning_content,
never content. Verified bug + workaround:
- @syangsao Fix 2 (chat_template_kwargs.enable_thinking: false in client):
confirmed unblocked, 28.88 TPS decode / 741 TPS prompt at 45K accumulated ctx
- This commit ships Fix 1 (--reasoning-format none server-side) so every
contributor doesn't have to discover the per-request workaround
Changes:
- docker-compose.yml: --reasoning-format ${REASONING_FORMAT:-none} default
- docker-compose.concurrent.yml: same
- Header docs: explain the opencode interaction + REASONING_FORMAT=auto
override path for clients that DO render reasoning_content
- INFERENCE_ENGINES.md: cross-link the issue + fix in llama.cpp's
Reasoning-channel separation row
- CHANGELOG: dated entry with diagnosis + bench numbers + companion
observation about DeltaNet preventing prefix-cache reuse across turns
Power users wanting reasoning_content separation: set REASONING_FORMAT=auto
in .env or shell. Default `none` is the right pick for the common
opencode/IDE-agent flow that the README positions as the daily-driver path.
|
||
|
noonghunna
|
b893d60f43 |
docs: BENCHMARKS rows + CHANGELOG entry for danbedford NVLink+DFlash variants
Both composes added to master in |
||
|
Erik LaBianca
|
276ab89291 |
composes: PYTORCH_CUDA_ALLOC_CONF env-override knob + WSL2 boot-crash docs (#84)
A single-card RTX 3090 Ti rig on WSL2 (driver 596.36) hits RuntimeError: CUDA driver error: device not ready from gptq_marlin_repack immediately after weight load on the v7.72.2-uplift nightly pin. Bisect ruled out Genesis, spec-decode, TQ3 KV, async-residual error, and TDR (registry already extended + Windows rebooted). PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False resolves it. Same env-var workaround as JusefPol's NVLink boot-crash (PR #31), already hardcoded in the dual-nvlink*.yml composes. Replaces the hardcoded PYTORCH_CUDA_ALLOC_CONF line in 14 single-card and PCIe dual-card composes with a ${PYTORCH_CUDA_ALLOC_CONF:-...} override (defaults preserved). Pattern matches existing MAX_MODEL_LEN / GPU_MEMORY_UTILIZATION overrides from #79. The two dual-nvlink*.yml composes are unchanged — their existing JusefPol-driven default already has expandable_segments off. Documentation: - docs/HARDWARE.md: new "disable PyTorch expandable_segments" subsection alongside the TDR fix, with stack trace, what was ruled out, override recipe, and a single uncontrolled observation about weight-load time (32 sec → 13 sec). - docs/FAQ.md: WSL2 question now cross-links both the TDR and expandable_segments fix subsections. - .env.example: documents the override under "vLLM tuning knobs". - CHANGELOG.md (top-level + per-model): dated 2026-05-06 entries. The exact failing call hasn't been isolated. The cuMemMap virtual-memory API used by expandable_segments:True is the suspected culprit since both known occurrences respond to the same workaround, but no specific call has been proven to return cudaErrorNotReady. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
c6e6163901 |
release(v7.72.2-uplift): Genesis pin + vLLM pin + sidecar consolidation (#59)
Aligns club-3090 with Sandermage's Genesis v7.72.2 release (2026-05-05) which shipped 7 new patches (PN59-PN67), v7.72.1 P68 xgrammar-incompat auto-skip, and v7.72.2 PN70 schema-subset filter. Pin bumps: - scripts/setup.sh GENESIS_PIN: 2db18df (v7.69) → 7b9fd319 (v7.72.2) - All 16 composes vLLM image: nightly-7a1eb8ac2 → nightly-01d4d1ad3 (Sander's PROD-validated pin, Genesis allowlist entry #2) 6 local sidecars deleted (Genesis natives supersede): - patch_inputs_embeds_optional.py → PN35 (vllm#35975 backport) - patch_pn30_dst_shaped_temp_fix.py → PN30 v7.68 - patch_pn25_genesis_register_fix.py → PN25 - patch_tolist_cudagraph.py → P78 (Sander's v7.72 CHANGELOG retires this) - patch_workspace_lock_disable.py → PN34 - patch_pr40798_workspace.py → research artifact, no compose mounted it 7 Genesis-loaded composes (yml, dual-turbo, long-text, long-text-no-mtp, long-vision, bounded-thinking, tools-text) had volume mounts and entry- point shell invocations cleaned up; GENESIS_ENABLE_PN59_STREAMING_GDN=1 added to all 7 for consistency. dual.yml left intentionally Genesis-free as a debugging fallback for cross-engine bisect — useful when isolating "is this Genesis or upstream vLLM" during silent-empty / OOM triage. Bench (dual-turbo, 2× 3090, single-stream, 5 measured runs each): - Narrative wall TPS: 81.21 (CV 2.3%), AL 3.46 - Code wall TPS: 108.20 (CV 0.9%) - VRAM/card: 20.0 GB (-2.1 GB vs v7.69 baseline of 22.1) - All 8/8 verify-full checks pass - verify-stress 6/7 (probe 7 = known TQ3 borderline at 60K, container alive throughout — not a regression) Cross-rig finding filed as Sandermage/genesis-vllm-patches#22 — PN59 streaming-GDN doesn't engage on chunked-prefill on Ampere consumer: its eligibility check rejects calls with chunk_indices/chunk_offsets populated, which vLLM's mandatory --max-num-batched-tokens 4128 always sets on 24 GB single-card configs. PN59 falls back to _vanilla_path which OOMs at the same chunk_o.py:161 site PN59 was meant to eliminate. Single-card 24 GB Cliff 2b is therefore unchanged — workaround is dual.yml/dual-turbo.yml (TP=2) or llamacpp/default. Docs warnings placed across README, docs/CLIFFS, docs/HARDWARE, docs/SINGLE_CARD, docs/FAQ, docs/UPSTREAM, BENCHMARKS, and the 3 affected single-card compose YAMLs. v7.72.1 closes #57 (lex's xgrammar-patternProperties fire on long-prompt agentic IDE traffic). Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
fbd3531960 | docs: add Carnice BF16MTP to DUAL_CARD, vllm README, and CHANGELOG | ||
|
Will Hampson
|
e3f8f02047 |
feat(qwen3.6-27b/vllm): add dual4 + dual4-dflash composes (TP=4, 4×3090, #44)
First 4-card variants for Qwen3.6-27B vLLM. Two new composes for 4× RTX 3090 PCIe rigs: - `docker-compose.dual4.yml` — TP=4 fp8/MTP baseline. 63 narr / 76 code TPS, 6.77× concurrency at 262K, ~23.5 GB/card peak - `docker-compose.dual4-dflash.yml` — TP=4 + DFlash spec-decode. 64 narr / 104 code TPS, 2.27× concurrency at 262K, ~22 GB/card peak Both pass verify-full + verify-stress 7/7 (incl. Cliff 2 needle recall at 58K + 91K) on @Whamp's 4× RTX 3090 PCIe rig. Both pass v2 continuous soak (dual4: 20 sessions, 0 MiB growth, 90.8% TPS retention; dual4-dflash: 5 sessions, 0 MiB growth, 100% TPS retention) — first cross-rig confirmation that TP=4 escapes Cliff 2b. Closes #26 (4×3090 wishlist). Co-authored-by: Whamp |
||
|
noonghunna
|
f8c9c365e0 |
docs: full sync to v7.69 + Cliff 2 60K closure recipes
Sweep all stale v7.66 / fc89395 substrate references to v7.69 (commit 2db18df) + local vllm#35975 inputs_embeds backport. Ship the Balanced MTP (long-text.yml, 180K + 0.93) and Max-context (long-text-no-mtp.yml, 200K + 0.95, no MTP) variants as the Cliff 2 closure recipes — both PASS the 60K single-prompt envelope (623s and 537s wall respectively). Updates: - CHANGELOGs (root + model) — new v7.69 PM entry above v7.66 - README + SINGLE_CARD + HARDWARE + EXAMPLES + FAQ + INTERNALS + VLLM engine doc — Cliff 2 status, substrate pins, mem-util defaults, variant table, sidecar list - vllm/README.md compose menu refreshed for the new ctx envelopes - model README patch surface table — added PN30 part3, PN32, P103, PN34 rows; collapsed P98 reference to PN34 env-gate - tools/charts/gen-perf.py + gen-vram.py — substrate label bumped to v7.69 + #35975, panel labels for the long-text variants updated, long-text-no-mtp 200K Max-context noted as bench-pending in chart - All performance + VRAM charts (svg + png) regenerated Cliff 2 60K closure: Genesis v7.69 (PN32 GDN chunked-prefill + P103 worker self-install + PN30 part3 + PN34 workspace_lock relax) plus local backport of vllm#35975 (~444 MiB freed on text-only paths). 3 sidecars dropped on long-text variants; 2 sidecars retained on master (patch_inputs_embeds_optional.py, patch_tolist_cudagraph.py). >60K single-prompt still hits the 24 GB hardware-physical wall on single-card. For those: dual-card TP=2 (verified at 237K) or llama.cpp single-card (262K, different engine). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
ae4846fd63 |
docs + charts: v7.66 + Cliff 1 mech B closed across all 4 TQ3 composes
Phase 3 of the v7.66 migration: documents the new state, regenerates performance/VRAM charts, posts cross-rig data to Sander on discussion #19 + issues #15/#16/#17. What changed ------------ **docs/SINGLE_CARD.md** - Updated TL;DR table: long-text 180K + 0.95, long-vision 145K + 0.95, bounded-thinking 180K + 0.95. - Removed Cliff 1 mech B "limitation to know" — now closed. - Added "What was Cliff 1 mech B (now closed) ✅" historical note. - Updated activation budget rationale to reflect PN12+PN25 pool residence. **docs/DUAL_CARD.md** - Bench protocol substrate: Genesis v7.65 → v7.66 dev tip. **docs/CLIFFS.md** - "vLLM pin compatibility status" rewritten for v7.66 + Cliff 1 mech B closure. - Replaced "v0.20 unblock" prose with "Cliff 1 mech B — what closed it" section explaining the two compounding fixes (PN25 v3 + PN30 dst-shaped). - Updated Genesis patches table with PN30 + PN33 + Sander v7.66 PN25. - Added "Local sidecars retained on master" table — 4 sidecars, why each one is still needed. - Updated "Validation across all 4 TQ3 variants" to 2026-05-02 numbers (180K / 145K / 180K / 262K — all 6/7 probes pass). **docs/UPSTREAM.md** - Genesis issue tracker updated with v7.66 cross-rig findings: - #16 PN25: both v7.65 and v7.66 mechanisms fail on TP=1 - #17 PN30: layout-correctness diagnosis + our corrected fix - #15 PN31: doesn't fit on 24 GB (lower mem-util sufficient) - PN33 partial (boot-time closes, runtime decode still fires) **docs/engines/VLLM.md, README.md, model README** - Genesis pin references bumped d89a089 → fc89395. **models/qwen3.6-27b/CHANGELOG.md** - New entry: "2026-05-02 — Genesis v7.66 + Cliff 1 mech B closed ⭐" with full validation matrix, sidecar inventory, and links to per-config result summaries. **tools/charts/gen-perf.py + gen-vram.py** - Substrate footnote: v7.65 dev (d89a089) → v7.66 dev (fc89395) - Compose ctx labels: long-text 214K → 180K, long-vision 198K → 145K, bounded-thinking 214K → 180K, mem-util 0.985 → 0.95 - Regenerated all 14 chart files (performance + vram, single + dual + combined). Cross-rig data posted to Sander ------------------------------- - [discussion #19 reply](https://github.com/noonghunna/club-3090/discussions/19#discussioncomment-16785590) — comprehensive update covering all 4 patches (PN33 partial, PN25 still TP=1-broken, PN30 layout diagnosis + corrected fix, PN31 still 24 GB-incompatible) - [genesis-vllm-patches#16 update](https://github.com/Sandermage/genesis-vllm-patches/issues/16#issuecomment-4362835267) — v7.66 mechanism still TP=1-broken - [genesis-vllm-patches#17 update](https://github.com/Sandermage/genesis-vllm-patches/issues/17#issuecomment-4362836196) — PN30 layout-correctness diagnosis + corrected fix offered - [genesis-vllm-patches#15 update](https://github.com/Sandermage/genesis-vllm-patches/issues/15#issuecomment-4362836881) — PN31 cross-rig confirmation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Codex CLI (ChatGPT) <noreply@openai.com> |
||
|
noonghunna
|
5aa97a25d9 |
v0.20 migration + Genesis v7.65 dev tip + cold-start cache + env-var alignment
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.
What changes
------------
Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)
Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
P98 covers same surface but auto-skips on v0.20 due to drift-marker false
positive — pending Sandermage marker fix)
Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
- PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
- PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
- PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
- PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
- PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
- P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
- P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
- P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
(Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
downgrade — pending v0.20 verification that #40880 closure makes it
redundant)
Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
`models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
drops to ~3.2 min (47% faster). Per-stage savings on long-text:
- Dynamo bytecode transform: 18s → 5s (-73%)
- torch.compile: 57s → 9s (-85%)
- Initial profiling/warmup: 51s → 7s (-87%)
Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.
Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
vllm#39226 strict assertion. Caught during validation, fixed.
Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text: 185K → 214K (+16%)
- long-vision: 140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)
Bench results (n=5, results/v0.20-migration/):
- long-text 214K narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8) narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K narr 48.82 / code 65.98 (n=3)
Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.
Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
+ FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved ✅ closed (PR #12, #13, #14, #15, P104
superseded by PN17 + P15B)
Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
worker-fork registration fix. Workarounds documented (tools-text fp8 path
/ --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
|
||
|
noonghunna
|
427d2f8aa9 |
docs+scripts+charts: propagate new ceilings (long-vision 198K, long-text 218K)
Sweep across all user-facing docs reflecting the post-PN12-anchor-fix ceilings established in |
||
|
noonghunna
|
51a4001af7 |
Genesis v7.62.x + PN8 on FP8 paths (closes Cliff 1 on tools-text)
scripts/setup.sh — GENESIS_PIN bumped from bf667c7 (v7.54) to 917519b (v7.62.x release, 2026-04-29). New patches: PN8 (MTP draft online-quant propagation, backport of vllm#40849), PN11 (Quentin-M streaming tool-call IndexError fix vllm#41142), per-GPU profile auto-rec, k8v4 unlock on hybrid GDN via P4+P98. PN8 enabled on FP8 paths only: - tools-text.yml: -900 MiB at boot, Cliff 1 25K tool prefill closes, -7% code TPS. Net win — production-safe for tool-using agents. - fast-chat.yml: -800 MiB at boot, no cliff to test at 20K, -4.7% code TPS. Free VRAM is useful for tighter mem-util configs. PN8 not enabled on TQ3 paths (default 48K, long-vision, long-text) or dual configs: - default 48K: PN8 is no-op on TQ3 + 0.92 (plenty of headroom already) - long-vision: PN8 grows KV pool 230 MiB and lifts engine ceiling 192K → 198K, but does NOT close Cliff 1 — the 138 MiB allocate is an FFN intermediate-buffer activation peak (intermediate_size × max-num- batched-tokens), not a draft-model footprint - long-text: engine ceiling at 206K is gated by attention-block-size divisor, not KV; PN8 has nothing to give - dual.yml: deliberately Genesis-less by design; not worth restructuring Verify-full passes on default 48K + v7.62.x without PN8 (8/8). Verify- stress on tools-text + PN8 passes all checks including the 25K tool prefill that was the launch-tweet headline caveat. Cross-rig data shared with Sandermage: https://github.com/noonghunna/qwen36-27b-single-3090/issues/1#issuecomment-4343317153 Docs updated: cross-cutting CHANGELOG, per-model CHANGELOG, USE_CASES.md (Cliff 1 closure note on tools-text), FAQ.md (Cliff 1 entry + new PN8 entry). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
1cc6ee6e24 |
CHANGELOG: capture post-launch polish day in cross + per-model logs
Cross-cutting CHANGELOG.md gets an entry for: launch.sh / switch.sh / health.sh / EXAMPLES.md / FAQ.md / VRAM diagram / Kaitchup citation / README two-routes polish. Per-model models/qwen3.6-27b/CHANGELOG.md gets the model-scoped slice: Q3_K_XL first-bench (21 TPS, mainline regression flagged), llama.cpp Docker compose addition (default + concurrent), stress-test sweep finding (no Cliff 1 / no Cliff 2 on llama.cpp — reframes launch story), VRAM diagram, Kaitchup quant validation. No code changes — just dated history capture. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
noonghunna
|
5060e22a6c |
Split verify-full.sh → verify-full.sh (fast functional) + verify-stress.sh (boundary)
Recent additions to verify-full.sh (#8 tool-prefill OOM, #9 cascade detection, #10 MTP AL) made the script slow — the longctx needle ladder (#7) alone could run 5+ min, and the full 10-check suite was approaching 10 min. Awkward for "is the stack functional" iteration during dev work. verify-full.sh (8 fast checks, ~1-2 min) 1. Server reachable 2. Genesis patches applied 3. Basic completion (Paris) 4. Tool calling 5. Streaming (SSE) 6. Thinking / reasoning mode 7. Output quality / cascade detection (was #9) 8. MTP acceptance length threshold (was #10) Run: after every config change to confirm the stack still serves cleanly. verify-stress.sh (2 boundary checks, ~5-10 min) 1. Long-context needle ladder (4 depths, 10K / 30K / 60K / 90K) — was #7 2. Tool response prefill OOM (~25K-token mock tool message) — was #8 Run: before publishing or when investigating prefill-OOM regressions specifically. Smoke-tested against dual.yml on dual-card: verify-full.sh: 8/8 green in 65 seconds verify-stress.sh: 2/2 green (skipped longctx for this smoke), 15s Same env-var conventions (URL, MODEL, CONTAINER, SKIP_LONGCTX, SKIP_TOOL_PREFILL, PREFILL_TARGET_CHARS). Doc updates: - top-level README repo layout: lists both scripts with timing/scope - docs/ARCHITECTURE.md: scripts/ section + design rules updated - models/qwen3.6-27b/USE_CASES.md: tool-prefill reference points at verify-stress.sh now - models/qwen3.6-27b/CHANGELOG.md: dated entry documenting the split |
||
|
noonghunna
|
c70147426d |
Dual-card re-bench on club-3090 substrate + fix dual-turbo mount path
The published dual-card TPS numbers (T1, DT1, D5, D7 in BENCHMARKS.md)
were measured pre-v714-formalization on a different nightly + Genesis
tree. Re-benched all 4 dual composes today on the unified club-3090
substrate (dev205 + Genesis v7.51-stable + Marlin pad fork mounted).
Also caught + fixed a stale mount path in dual-turbo.yml — predecessor
referenced patch_tolist_cudagraph.py at `../patches/genesis/` (old
qwen36-dual-3090 layout); club-3090 has it at `../patches/` (top-level).
Container died at boot with 'cant find __main__ module' before fix.
Audited all 9 composes — only dual-turbo had the bug.
Re-bench numbers (3 warm + 5 measured per prompt arm):
Compose Narr (CV) Code (CV) AL VRAM/card vs claimed
dual.yml 69.05 (2.3%) 88.58 (3.4%) 3.4 23.6 GB -3% / -1%
dual-turbo.yml (now TQ3) 53.65 (2.7%) 72.93 (2.7%) 3.4 24.1 GB -8% / +6% vs k8v4
dual-dflash.yml 81.94 (4.3%) 124.93 (5.8%) 4.1-4.4 23.6 GB +5% / -2%
dual-dflash-noviz.yml 78.19 (2.5%) 126.99 (2.2%) 4.2-4.4 23.8 GB +2% / +2%
Net: most numbers within run-to-run variance. dual-turbo's TQ3 swap (from
k8v4) cost ~8% narrative but recovered ~6% code — net trade for ~9× the
KV pool capacity (which is what the compose exists for).
verify-full.sh: dual.yml passes 10/10 incl. 90K needle. dual-turbo passes
10/10 too. DFlash variants passed all checks except longctx (skipped for
time; recall path validated previously).
Doc updates:
- models/qwen3.6-27b/README.md: dual-card variant table updated with
measured numbers (was 71/89, 58/69, 78/128, 77/124 → 69/89, 54/73,
82/125, 78/127). Also corrected DFlash variants to FP16 KV (was
written as fp8 in the table but file uses default FP16).
- models/qwen3.6-27b/USE_CASES.md: quick map updated with measured
dual TPS for each workload.
- dual.yml header: measured-numbers line replaces predecessor's claim;
variant matrix in dual.yml updated.
- dual-turbo.yml header: TPS regression vs fp8 noted as ~22% (was
"~25% trade" claim).
- CHANGELOG.md: dated entry documenting the re-bench + path fix.
|
||
|
noonghunna
|
b641719eb8 |
Add long-vision + long-text composes (formalize R3' / R3''' bench rows)
The v714 formalization round (2026-04-27) measured these as opt-in tiers edit-able into docker-compose.yml. That made reproducibility fragile: users who wanted to boot the exact 192K-vision or 205K-text-only configs had to hand-edit max-model-len, gpu-memory-utilization, and --language-model-only. Promoting both to dedicated compose files so each published bench row boots with a single -f flag. New composes: docker-compose.long-vision.yml 192K + 0.98 + vision (R3', 51/68 TPS) docker-compose.long-text.yml 205K + 0.98 + no vision (R3''', 50/66 TPS) Both validate via `docker compose config`. Both carry the same prefill cliff caveats as the edit-the-default approach did: - Cliff 1: ≥25K-token tool-message prefills OOM (ampersandru #1 class) - Cliff 2: ≥50-60K single prompts OOM (DeltaNet GDN forward, hardware-bound) The full 192K/205K is for steady-state context accumulation across many small turns, NOT for stuffing 192K of fresh tokens in one request. Header / docs alignment: - models/qwen3.6-27b/README.md: variant table now lists long-vision + long-text rows; Quick map updated. - models/qwen3.6-27b/USE_CASES.md: Frontier 192K-262K section now references both composes directly instead of "edit the default". - vllm/README.md: "Pick a compose" code block adds two boot lines. - default's header variant matrix lists the new files. - CHANGELOG.md: dated entry documenting the addition + rationale. Did NOT add separate composes for 64K / 96K / 128K opt-in tiers. Those are interpolation points between the safe default (48K) and the frontier (192K/205K); users can edit if they want a midpoint, but it's not worth a separate file per benchmarked combination. |
||
|
noonghunna
|
2511a98110 |
Fix .gitignore + add the entire models/ tree (initial commit was incomplete)
The old repos used models/ as the model-weights download directory, so their .gitignore excluded it. We copied that .gitignore into club-3090 without updating, which silently dropped the entire models/ subtree from the initial commit on GitHub (everything still on disk locally, just not tracked). Fixes: - .gitignore: remove models/, add models-cache/ (the new default for weights download). Update genesis patches path to the new location models/<model>/vllm/patches/genesis/. Update compose-state ignores to use **/compose/ glob since composes are now nested. Add vllm-src/ for the dual-card Marlin pad fork mount. - Add the missing models/qwen3.6-27b/ tree: - README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md - vllm/README.md + 9 docker-compose.yml variants + patches dir - llama-cpp/README.md + 2 launch recipes - sglang/README.md (currently blocked status) Also fixes: stale comment header in fast-chat.yml that referenced the deleted longctx-experimental.yml and self-described as "default" when it's the chat-only variant. |