Filter out 'chore(changelog): regenerate for vX.Y.Z [skip ci]' commits
that the release.yml workflow auto-creates. They're pure machine
traffic and only added noise to release notes.
CHANGELOG.md and GitHub Release notes now render commit subjects only.
Full commit message body (why/how/validation data) stays in 'git log',
one click away via the SHA link. Keeps both surfaces skim-readable.
v0.3.2 release pages were 100+ lines per commit; this brings them down
to ~1 line per commit.
The v0.3.2 tag was originally pushed at commit 64b0474 but GitHub didn't
emit a CreateEvent (likely dedup after delete + re-push to same SHA), so
the release.yml workflow never fired. Empty commit gives the tag a fresh
SHA that GitHub will process cleanly.
CHANGELOG.md is now auto-generated from commit messages by git-cliff in
the release workflow. Hand-edits below the static header will be wiped on
the next tag.
Workflow (`.github/workflows/release.yml`):
- On tag push (`v[0-9]+.[0-9]+.[0-9]+`):
1. Render GitHub Release body: `git-cliff --latest --strip header`
→ just the per-version section, no SemVer preamble repeat
2. Regenerate full CHANGELOG.md: `git-cliff` (default = all tags)
→ preserves header + all historical sections
3. Commit CHANGELOG.md back to master with `[skip ci]` marker
4. Publish GitHub Release with the latest-only body
Template (`cliff.toml`):
- `[changelog].header` now holds the SemVer preamble + CalVer→SemVer
mapping table (preserved across regens; stripped from GitHub Release
bodies via `--strip header`).
- `body` template now renders the **full commit message** (subject as
bold bullet, body indented below) instead of just the first line.
Rich narrative I write in commit message bodies (tables, validation
numbers, before/after diffs) now flows into both CHANGELOG.md and the
GitHub Release page from the same source.
- Per-release Pin/Diff footer guarded with `{% if version %}` so the
Unreleased section doesn't emit empty links.
CHANGELOG.md replaced with the auto-gen output. Past hand-written tables
and phase breakdowns are replaced by the corresponding commit messages
(those were already rich for commits that mattered — v0.3.1 soak-helper
fix has its Before/After table in the commit body and renders fine).
Going forward: just write rich commit messages and tag. Both surfaces
update automatically. No hand-edit of CHANGELOG.md required.
When the user runs quality-test.sh with a localhost-style URL
(default `http://localhost:8020`, or any `localhost`/`127.x`/`[::1]`
variant), auto-export `BENCHLOCAL_HERMES_RESOLVE_LOCALHOST=1` so
benchlocal-cli rewrites the hermes-agent's outbound model endpoint
from `localhost:<port>` to `host.docker.internal:<port>` inside the
Docker sandbox container.
Without this, the hermes-agent inside the sandbox can't reach the
host's vLLM (localhost resolves to the container itself) and every
scenario fails with `"API call failed after 3 retries: Connection
error."` — produced spurious 0/20 grades on this rig prior to the
benchlocal-cli runner.py 9c1566f fix.
Skips the auto-set when:
- User already set the env var (explicit override)
- URL points at a non-loopback host (real LAN IP, k8s service name,
host.docker.internal already) — no rewrite needed
Emits a stderr breadcrumb when the auto-set fires so users can see
what changed.
vLLM nightly (0.20.2rc1.dev9+) emits the qwen3 reasoning parser's output
under `delta.reasoning` (legacy field name), not `delta.reasoning_content`
that soak-helper.py was watching. Result: for any thinking-on response
whose `<think>` block doesn't close within `max_tokens`, soak-helper saw
zero deltas → fell back to the "couldn't measure" path → reported
`ttft_ms == t_ms` and `decode_tps = 0.0`. The model was generating
correctly; the harness just couldn't see the wire output.
Repro request (JDWarner's #107 turn 5): math problem with
`max_tokens=2000` + `chat_template_kwargs.enable_thinking=true`.
Before patch:
status=200 t_ms=22709 ttft_ms=22709 decode_tps=0.0
completion_tokens=2000 content="" reasoning_content=""
After patch (same request, same compose, same model):
status=200 t_ms=22709 ttft_ms=234 decode_tps=88.985
completion_tokens=2000 content="" reasoning_content="Here's a thinking
process:\n\n1. **Understand the User's Problem:**\n..." (3959 chars)
Validation soak (fresh-mode, 20 sessions × 5 turns = 100 turns, qwen3.6-27b
dual.yml):
verdict PASS
silent_empty 0 / 100 (0.0%) ← was ~3-5/40 baseline
p50_decode_tps 90.22
p95_ttft_ms 1389
errors 0
max_growth 0 MiB / 200
Closes the cross-rig "silent-empty turn-5" pattern parked behind the
Cliff 2b investigation — it was a harness measurement bug, not a model
or rig issue.
club-3090 is software (docker composes + system scripts + patch bundles
that downstream rigs run as-is), not just rolling recipes. Switch from
CalVer to SemVer from v0.3.0 onward; past CalVer tags (v2026.05.09,
v2026.05.10) preserved for history.
CHANGELOG.md: convention note + retroactive CalVer→SemVer mapping +
new v0.3.0 (2026-05-10) entry covering 8 commits since v2026.05.10:
- Qwen 3.6 27B thinking OFF default across all 21 composes
- MODEL_DIR UX overhaul (closes#116) — interactive setup prompt,
$MODEL_DIR placeholder everywhere
- power-cap-sweep --include-commit (closes#112)
- BENCHMARKS aider-polyglot-30 row
cliff.toml: drop "snapshot of the rolling stack — not a versioned
API" framing; add SemVer note. Tag pattern v[0-9]+.[0-9]+.[0-9]+
already matches both CalVer and SemVer, so the cliff release
workflow needs no changes.
New "Quality benches — Aider Polyglot 30" section captures pass-rate /
agentic-coding signal alongside the existing TPS rows. First two rows:
- Qwen 3.6 27B (AutoRound INT4) on dual.yml: 20/30 = 66.7%, 19 min wall
- Gemma 4 31B (Intel AutoRound INT4) on dual.yml: 17/30 = 56.7%, 19 min wall
Both run on 2× 3090 PCIe, 230 W cap, threads=2. Qwen edges Gemma by +10pp
despite being smaller; java is the biggest swing (Qwen 4/5 vs Gemma 1/5).
Critical caveat documented: Qwen with thinking ON is unusable for agentic
benches on this hardware — hits the 1500s subprocess cap before any
exercise completes. The new --default-chat-template-kwargs flag in our
vLLM Qwen composes (commit 534d29f) sets enable_thinking=false by default.
Aider-polyglot run via benchlocal-cli's aider-polyglot-30 pack. Cross-rig
re-run path documented in the section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@laurimyllari noted in disc #62 that with the project moving fast,
including the club-3090 git commit in sweep output helps correlate
cross-rig sweeps to the script revision they were run against. He
stamped `aa99173` manually; the script should do it for us.
New flag:
--include-commit Stamp the club-3090 git commit (short SHA) in the
report header next to the date. Off by default.
Implementation:
- Captures `git -C "$REPO_ROOT" rev-parse --short HEAD` once at header-build
time (REPO_ROOT was already known to the script).
- Injects into the report header next to **Date:**, e.g.:
**Date:** 2026-05-10T17:55:00Z **club-3090 commit:** `534d29f`
- Suppress (don't stamp "n/a") when run from a non-clone or git is
unreachable. Closes the curl-pipe-from-docs UX hole — `curl ... | bash`
users don't have a clone, so the field just disappears rather than
showing a confusing "n/a".
Off by default per the issue rationale: surprise stamping confuses
contributors running from documentation snippets.
Closes#112.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 29d17ed which used the wrong vLLM flag name (`--chat-template-kwargs`),
causing boot failure: "vllm: error: unrecognized arguments: --chat-template-kwargs
{"enable_thinking": false}".
vLLM's actual flag for setting server-side default chat template kwargs is
`--default-chat-template-kwargs` (with the `default-` prefix). Confirmed by
- vLLM nightly source: vllm/engine/arg_utils.py defines
`default_chat_template_kwargs: dict[str, Any] | None = None` with
json.loads parsing.
- vLLM PR #37739 ("Fix default_chat_template_kwargs handling in Responses API")
references it as already available in the shared render stack.
Behavior unchanged: thinking OFF by default for all 17 vLLM Qwen 27B composes
(plus bounded-thinking unaffected — it intentionally keeps thinking ON).
Per-request override still works via OpenAI extra_body:
{"chat_template_kwargs": {"enable_thinking": true}}
Verified: vllm-qwen36-27b-dual now boots cleanly with the corrected flag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, all 19 vLLM Qwen composes used --reasoning-parser qwen3 (parses
<think>...</think> output blocks) but did NOT explicitly disable the
thinking template. That meant Qwen3 default — thinking ON — applied
across the board. This burns hidden token budget on internal CoT for
every request, hurts latency, and creates a "Qwen looks much slower
than Gemma" gap on agentic benchmarks (which is exactly what we just
hit on aider-polyglot — Qwen exceeded the 1500s timeout, Gemma
finished in 19 min).
Change:
- All vLLM composes (18 of them) now pass `--chat-template-kwargs
'{"enable_thinking": false}'` after `--reasoning-parser qwen3`.
- llama.cpp single/docker-compose.yml: DISABLE_THINKING default flipped
0 → 1 (thinking now OFF by default; opt back in via DISABLE_THINKING=0).
- llama.cpp single/concurrent.yml: gained `--chat-template-kwargs` flag
with default `{"enable_thinking":false}` (overridable via
CHAT_TEMPLATE_KWARGS env).
NOT changed:
- bounded-thinking.yml — that's the structured-CoT compose where thinking
IS the feature. Reverted my initial blanket change for that one.
- qwopus-bf16mtp.yml — already had enable_thinking=false (preview compose).
Users who want thinking ON can:
- For vLLM: pass `chat_template_kwargs: {enable_thinking: true}` in the
per-request body (works fine).
- For llama.cpp: set `DISABLE_THINKING=0` in compose/.env.
Aligns with Gemma 4's "thinking off by default" (it ships that way upstream)
and removes the Qwen vs Gemma framework-bench skew.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously setup.sh silently defaulted MODEL_DIR to <repo>/models-cache,
which meant fresh users got ~14-21 GB of model weights downloaded INTO
their git tree without realizing it. The relative path is also wrong
when run from anywhere except the repo root.
New 4-step resolution order in setup.sh:
1. MODEL_DIR exported in calling shell → use as-is (unchanged)
2. .env at repo root sets MODEL_DIR → source it (NEW)
3. Interactive prompt (only on TTY) → ask user (NEW)
4. Silent fallback to <repo>/models-cache (unchanged for non-TTY)
The interactive prompt only fires when:
- MODEL_DIR is not in the calling env, AND
- .env doesn't already set it, AND
- both stdin AND stdout are TTYs (CI / scripted runs unaffected)
Three options offered:
1. <repo>/models-cache (the old silent default — kept as option)
2. $HOME/models (sensible cross-rig default)
3. custom absolute path
After picking, optionally persists the choice to .env (gitignored) so
re-runs skip the prompt. Existing .env files are updated in-place if
they already set MODEL_DIR; appended-to otherwise.
Closes the UX hole RobH589 hit in club-3090#116 — the relative
../../../../../models-cache default that was resolving wrong when
not run from the compose dir.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two recipe scripts (single-card-default.sh, single-card-max-ctx.sh) had
the dev rig path /mnt/models/huggingface/... baked into the MODEL_PATH
default + the comment-block instructions for downloading the GGUF.
Updated:
- Comments now show \${MODEL_DIR}/qwen3.6-27b-gguf/... as the placeholder
(matches what the just-fixed README + LLAMA_CPP.md docs say).
- MODEL_PATH default changed from /mnt/models/huggingface/... to
\${MODEL_DIR:-\$HOME/models}/qwen3.6-27b-gguf/... — falls back to
~/models/ if MODEL_DIR isn't set, which is a more reasonable default
for cross-rig users than our /mnt/models/huggingface/ path.
- file-exists check at line 25 still fails loudly with the resolved path
if neither MODEL_DIR nor MODEL_PATH is set correctly.
Follow-up to fbf3431 (de-bind \$MODEL_DIR from rig path in docs).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User docs were hardcoding the dev rig path (/mnt/models/huggingface/...) as
if it was canonical. It's not — cross-rig users have models at /data/models,
~/models, /mnt/nvme/llms, etc. Setting MODEL_DIR per their setup is the
intended UX (the compose already supports it via env-var default).
Updates:
- models/qwen3.6-27b/llama-cpp/README.md: download examples now use
\$MODEL_DIR/qwen3.6-27b-gguf/ instead of /mnt/models/huggingface/...
- models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml: header
comment uses \$MODEL_DIR/qwen3.6-27b-gguf/ for download examples + says
"MODEL_DIR=/your/models/dir docker compose up -d" instead of our path.
- docs/engines/LLAMA_CPP.md: same treatment + cleaned up Qwen3.5 + DFlash
draft path examples to also use \$MODEL_DIR.
- scripts/preflight.sh: hf download hint shows literal \${MODEL_DIR} so user
knows what to set, plus explicit "set MODEL_DIR first" line. Previously
echoed the resolved relative path (../../../../models-cache) which lands
outside the repo if pwd isn't the compose dir.
Caught by RobH589 in club-3090#116 — they hit the path-resolved-to-root-of-drive
case from the relative-path default. Closes the doc UX side; the compose's
env-var override mechanism was already correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the GGUF dir move (/mnt/models/gguf/qwen3.6-27b/ → /mnt/models/
huggingface/qwen3.6-27b-gguf/), four refs were not updated and led to
a path-mismatch loop reported in club-3090#116:
- scripts/preflight.sh `hf download` hint pointed at qwen3.6-27b/, but
the compose default expects qwen3.6-27b-gguf/. Same for the mv hint
for mmproj relocation, and the in-container mmproj default at line 329.
- models/qwen3.6-27b/llama-cpp/README.md example command still said
`MODEL_DIR=/mnt/models/gguf` (now /mnt/models/huggingface).
- models/qwen3.6-27b/llama-cpp/compose/single/{docker-compose,concurrent}.yml
header comment said "Q5_K_XL" but the actual default has been Q3_K_XL
for a while (this one predates the reorg — just stale doc).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dev rig had grown across three home dirs (`/opt/ai/compose/`, `/opt/ai/github/`,
`/home/wasif/`) and three repos (single-3090, dual-3090, club-3090). Disk-out
on `/` (97% used) forced a cleanup; rather than just prune, we consolidated
the layout across the whole stack while we were at it. This commit captures
what landed inside this repo.
Services consolidation:
- services/{ollama,openwebui,litellm,qdrant,searxng}/ migrated in from
/opt/ai/compose/<svc>/ (zero functional change — same docker-compose.yml).
- services/litellm/config.yaml rewritten: explicit routes for current
primaries (qwen3.6-27b-autoround → :8010, gemma-4-31b-autoround → :8030).
Removed `* → ollama/*` wildcard.
- services/comfyui/ migrated in (was /opt/ai/compose/comfyui/) — wired into
gpu-mode with full mutex against vLLM/SGLang.
scripts/gpu-mode.sh under git:
- Was a loose /opt/ai/gpu-mode.sh outside any repo. Symlinked at
/usr/local/bin/gpu-mode.
- Five Gemma 4 31B modes added: gemma, gemma-dflash, gemma-int8,
gemma-dflash-int8, gemma-awq.
- One ComfyUI mode (mutex with all LLM serving).
- prune / prune-all subcommands (safe image prune; aggressive variant adds
build cache --keep-storage 5GB + dangling networks).
- gpu-mode status now shows Docker disk + /var/lib/docker + /tmp sizes.
- compose_at() passes --env-file <repo>/.env so MODEL_DIR resolves
regardless of which compose dir gpu-mode cd's into. Fixes the recurring
"MODEL_DIR not set, defaulting to ../../../../../models-cache" warning.
- stderr no longer swallowed by compose_at() (real errors surface).
- Cross-model VRAM mutex: every Qwen mode stop_all_gemma + stop_comfyui
and vice-versa.
scripts/maintenance/ — new hygiene-tools subdir:
- list-image-pins.sh: engine-agnostic pin auditor. Scans every compose's
`image:` line, groups by `<repo>:<tag>`, flags pin-drift (multiple tags
per repo), ranks composes by patch surface.
Pin tracking:
- docs/UPSTREAM.md gains a "Pinned images" section: table of every pinned
image, why each pin exists, retirement candidate criteria.
- docs/NIGHTLY_BUMP_RUNBOOK.md (new): 7-step procedure for bumping pinned
engine images (scope → branch → patch survival → boot → verify-full +
verify-stress → bench delta → land → retire). Engine-specific notes
for vLLM nightly hashes, llama.cpp digest pinning, SGLang variants.
Path updates from the engine + model dir consolidation:
- /opt/ai/vllm-src/ → /opt/ai/engines/vllm/primary/
(in setup.sh, INTERNALS.md, several patch READMEs, docs/HARDWARE.md,
docs/FAQ.md, docs/DUAL_CARD.md, docs/UPSTREAM.md, models/qwen3.6-27b/
CHANGELOG.md)
- /mnt/models/gguf/qwen3.6-27b/ → /mnt/models/huggingface/qwen3.6-27b-gguf/
(in models/qwen3.6-27b/llama-cpp/{compose/single/*.yml, recipes/*.sh,
README.md}, docs/engines/LLAMA_CPP.md)
CHANGELOG.md narrative gap fill:
- 2026-05-10 entry for this reorg.
- 2026-05-09 entry for compose convention formalization (topology
promoted to dir level, profile schema, Status enum + Caveats, cliff
CI swap, Discord launch).
- 2026-05-08 entry for Gemma 4 INT8 PTH unblock + 262K validation.
- 2026-05-07 entry for power-cap-sweep campaign + HARDWARE.md cross-rig
charts + cross-rig benchmark rows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands the Gemma 4 31B work that was in flight last week + Qwopus3.6-27B
preview compose. Captures four distinct serving paths for Gemma 4:
- dual/awq.yml — AWQ-4bit (text-only, simplest path)
- dual/dflash-int8.yml — DFlash + INT8 PTH KV (262K ctx, full pipeline)
- vllm-gemma4-dflash-int8/ — vendored patches stacking DFlash spec-decode
+ INT8 PTH KV across model_executor + v1/spec_decode + v1/attention +
v1/worker (~13 patched files; PR #42102 + #40391-rebased + tool-parser
fixes#42006 + #41991 stacked)
- vllm-gemma4-fp8-ampere/ — earlier Phase 2 attempt before INT8 PTH
reframe (kept for forensics); Ampere has no native FP8 tensor cores
- vllm-perheadkv-hybridpage-fix/ — hybrid-page bug fix surfaced during
Phase 3
- vllm-pr40391-perheadkv/ — PR #40391 vendored at the tree level
(separate from the rebased variant under refs/jianc99-dflash-gemma4)
Plus:
- models/qwen3.6-27b/vllm/compose/dual/qwopus-bf16mtp.yml — preview compose
for Carnice AutoRound Recipe D output (port 8071, NOT production —
see club-3090-todo.md for known gaps + cheap A/Bs).
- AGENTS.md — codify compose naming + profile-schema + experimental-compose
conventions that the new files follow.
- docs/QUALITY_TEST.md — runbook for `quality-test.sh` + the benchlocal-cli
packs it wraps.
CHANGELOG.md narrative entries for these are added separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two findings from @easel's deep cross-rig writeup on issue #102 (5090
Laptop WSL2):
1. **HARDWARE.md** — new "GPU memory budget on WSL2" subsection. WSL2
container CUDA-context consumes ~1.31 GiB before vLLM profiler runs.
`gpu_memory_utilization=0.95` crashes at boot; 0.944 works. Validated
on 2 machines. Formula: `(vram_total - 1.31) / vram_total`.
Recommendation: `GPU_MEMORY_UTILIZATION=0.94` in `.env` on WSL2.
2. **CLIFFS.md** — new section "Cliff 3 — DeltaNet SSM state is not
prefix-cacheable (the prefill cliff)". This is a structural finding
that explains a class of failures we'd been describing without
naming. @easel's warm-cache run (68.7% KV-block hit, turn 10 at
35.6K tokens took 577s — 2.3× the cold-start 254s) is the smoking
gun: prefix cache helps attention but DeltaNet's recurrent state
`h_t = f(h_{t-1}, x_t)` must be recomputed from scratch every turn.
PN32 fixes OOM stability; nothing fixes the O(n) prefill scaling
because the architecture itself is sequential.
Practical ceiling on single-card vLLM (any 24 GB Qwen3-Next config):
sub-30s TTFT only below 5K accumulated tokens. 22-35K = 3-4 min/turn.
~74K = 10+ min client timeout.
Elevates "for single-card agentic Qwen3-Next, use llama.cpp" from
implied to explicit in the docs. Dual-card extends the envelope to
25-30K accumulated; deep sessions (50K+) still need llama.cpp on
either topology.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three related fixes for running benchmarks without IDE agent interference:
1. All 18 vLLM compose files: port binding is now
${BIND_HOST:-0.0.0.0}:${PORT:-<n>}:8000
Setting BIND_HOST=127.0.0.1 in .env restricts the API to localhost,
preventing IDE agents (Cline, Cursor) from competing for the
max-num-seqs=1 slot and causing verify-stress HTTP 000 failures.
2. scripts/preflight.sh: port auto-detection regex now matches
127.0.0.1:<port>->8000/tcp in addition to 0.0.0.0: and [::]:
Previously all verify-*/bench scripts silently produced no output
when BIND_HOST=127.0.0.1 was set.
3. scripts/report.sh: SOAK_TIMEOUT_S is now forwarded to soak-test.sh
from the shell environment. Previously the variable was read from
compose .env (docker-compose only) and silently ignored by the
script, always using the 1800s default regardless of what was set.
Docs: .env.example gains BIND_HOST and SOAK_TIMEOUT_S entries.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Four small fixes addressing low soak-test compliance in cross-rig bench
contributions. Audit of 5 recent #113/#107/#102/#104/#93 showed soak data
IS being run but it's hidden in the main report and the dedicated
template field comes out empty (template said "leave blank if you ran
--full"). Older BENCHMARKS rows often omit soak verdict entirely.
1. **`.github/ISSUE_TEMPLATE/numbers-from-your-rig.yml`**: replace optional
"soak summary" textarea with a required dropdown listing PASS / borderline /
FAIL / Skipped+reason / Not-yet-run. Verdict is now grep-able even when
the data is buried in the main report textarea.
2. **`scripts/soak-test.sh`**: add `--continuous` / `--quick` / `--fresh`
flags + `--help` + cleaner usage docs. Was 5 env vars to invoke
(`SOAK_MODE=continuous SOAK_SESSIONS=5 SOAK_TURNS=5 CONTAINER=... ENDPOINT=...`);
now `bash scripts/soak-test.sh --continuous` does the same with
auto-detect (existing logic preserved + exposed). Env vars still work
for back-compat.
3. **`scripts/report.sh`**: when `--bench` (or partial) ran without
`--soak`/`--full`, append a "⚠ Soak: not included" reminder block to
the report so contributors know what's missing before pasting into
the issue template.
4. **`BENCHMARKS.md`**: Notes-column convention — every row should start
with explicit `Soak: ✓ PASS` / `⚠ borderline` / `✗ FAIL` / `—` so
readers can grep at a glance. Updated 2 recent rows (ygafarov #113,
JDWarner #107) to use the convention. Older rows backfill as the
convention spreads.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First Strix-Halo-miniPC + oculink-eGPU class on the matrix. Single 3090
over PCIe x4 (oculink) on AMD Ryzen AI MAX+ 395 / 124 GB RAM / CachyOS /
290W cap. Result: 68.86 narr / 91.70 code TPS via vllm/default + TQ3 at
48K — clean MTP AL 3.31 (77% accept), CV 1.6%/2.7%.
Soak FAIL is borderline (240 MiB > 200 MiB threshold, 3 turns >30s) but
100% TPS retention + 0 errors + 0 silent-empty suggests x4-PCIe accretion
+ bus-latency under prefill, not Cliff 2b. Worth flagging as a possible
"eGPU bus class" threshold allowance for soak-test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-ran --full --enable-sandboxed-packs on both duals to validate v0.4
sandbox infrastructure end-to-end with real model output. Both sandbox
containers (BugFind/CLI/HermesAgent) bring up cleanly; SandboxClient
lifecycle works as designed.
Headline: Qwen3.6-27B 129/150 (86%), Gemma 4 31B 126/150 (84%). Both
score 56/75 (75%) on the deterministic suite; sandbox numbers (97%/93%)
are partly inflated by v0.4 shape-check verifiers — BugFind matches
<solution> blocks, HermesAgent passes any non-empty response, only CLI
applies real shell-parseability + safety checks. Full upstream fixture
parity is queued for v0.5.
Surprising: BugFind 14/15 (Qwen) and 15/15 (Gemma) — both models do
emit solution-block-shaped output without explicit prompting. Suggests
the v0.4 shape-check is more meaningful than I'd expected.
Run on: vLLM nightly 01d4d1ad, 2× RTX 3090 TP=2, benchlocal-cli v0.4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both score 56/75 (75%) on the 5-pack deterministic suite (toolcall,
instructfollow, structoutput, dataextract, reasonmath). Per-pack
breakdown is appended to the Quality: profile field.
Notable: ReasonMath is the weakness on both (Qwen 33%, Gemma 40%) —
verifier requires strict `key=value` final-answer format that neither
model emits without explicit prompting. Other 4 packs unchanged from
the earlier --medium baseline.
Sandboxed packs (bugfind-15, cli-40, hermesagent-20) skipped — gated
on --enable-sandboxed-packs and the v0.4 verifiers are still
shape-checks pending v0.5 fixture parity.
Run on: vLLM nightly 01d4d1ad (2026-05-04 image), 2× RTX 3090 TP=2,
benchlocal-cli v0.4. Endpoints: 8010 (Qwen) / 8030 (Gemma).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Release Drafter only catches PRs; this repo's workflow is mostly
direct-to-master commits per the auto-mode pattern, so 80%+ of
substantive changes were invisible to it.
git-cliff is commit-based: parses every commit since the last tag,
categorizes by conventional-commit prefix (`docs:`, `scripts:`,
`composes:`, `models:`, `fix:`, `chore:`) with keyword-based fallback
for un-prefixed commits (`Document Cliff 1`, `Add Gemma 4 compose`,
`power-cap-sweep:`, `verify-full.sh:`). Squash-merged PRs flow
through the same parsers since their squashed title becomes the
commit message.
Workflow triggers on `v[0-9]+.[0-9]+.[0-9]+` tag push, runs git-cliff
with `--latest`, creates a GitHub Release with the categorized body.
Tested locally on 274 commits since repo init: 49 land in catch-all
"Other" (genuinely unconventional one-off commits); rest distribute
across 9 categories. Existing v2026.05.09 release stays as-is
(hand-written); next CalVer tag onwards uses this pipeline.
Future cadence: `git tag v$(date +%Y.%m.%d) && git push origin v...`
— workflow does the rest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Maintains a draft GitHub release continuously, categorizing changes by
PR label or conventional-commit prefix in the title. Categories:
- 🎯 New models + serving paths
- 🔧 Pin bumps + upstream
- 📊 Benchmarks + cross-rig data
- ⚠️ Cliffs, gotchas, regressions
- 🛠️ Scripts + tooling
- 🐛 Bug fixes
- 📝 Documentation
- 🧹 Maintenance
Versioning is CalVer (year.month.day) — this repo is a rolling config
stack, not a versioned API, so date tags are honest about that. Tag
manually via `git tag v2026.05.09 && git push origin v2026.05.09`;
Release Drafter populates the body, you click "Publish" in the UI.
Autolabeler maps existing commit-prefix conventions (docs:, scripts:,
composes:, models:, fix:, chore:) to categories without per-PR label
discipline. Files-based fallback catches direct-to-master doc edits
that touch BENCHMARKS.md / HARDWARE.md / CLIFFS.md.
Closes the loop on @laurimyllari's "include git commit in output"
suggestion (issue #112) by giving cross-rig contributors a stable
version tag to cite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
His earlier 15-cap sweep (260-400W, 16832066) is superseded by a
38-cap sweep at 10W resolution from 230-600W (16854218) — strictly
richer envelope. Two findings worth surfacing:
- Firmware boost-clock plateau at SM 2610 MHz: caps 400-600W produce
identical 51.96 TPS at 393W actual draw. Auto-plateau detector in
power-cap-sweep.sh caught both sub-plateaus cleanly (400-470W and
480-600W).
- decode-concurrent N=4 plateau lower than decode-single (46 vs 52
TPS) — on a 4090 + 27B Q3_K_XL the GPU is under-load at c=4 even
at full TDP, so single-stream wins. Useful cross-rig signal.
HARDWARE.md table + chart caption updated; chart regenerated with
both load-mode curves overlaid + plateau zone shaded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous "Status optional, only when not production" convention
left readers guessing whether absence-of-status meant "validated
production" or "author forgot to fill it in." Making Status required +
enumerated removes that ambiguity.
Schema enhancement (codified in AGENTS.md):
- Status: now REQUIRED, exactly one of:
✅ Production — verify-full + stress + bench + soak PASS
⚠️ Production w/ caveats — works under documented constraints
🧪 Experimental — under active validation
👁️ Preview — known quality issues, not production
⏸️ Upstream-gated — blocked by external action (PR/driver)
🗑️ Deprecated — kept for historical reference
- Caveats: REQUIRED if Status is ⚠️/👁️/⏸️/🗑️, OMITTED for ✅/🧪.
Single-line summary or short bullet list with issue/PR links.
Status applied to all 27 composes (vLLM + llama-cpp, all topologies):
✅ Production: 20 (canonical Qwen + Gemma + llama.cpp + gemma-awq)
⚠️ Production w/ caveats: 4 (long-text*, long-vision, carnice-bf16mtp)
⏸️ Upstream-gated: 2 (gemma single boot-OOM, gemma dflash-int8 needs #42102)
👁️ Preview: 1 (qwopus-bf16mtp)
🧪 Experimental: 0
🗑️ Deprecated: 0
llama-cpp composes also gained `Genesis: N/A — llama.cpp engine` for
parity with the vLLM schema (Genesis is Qwen3-Next-specific and
vLLM-only; explicit N/A prevents future readers from looking for
patches that don't exist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three rationale points for why models/<m>/<engine>/patches/ and cache/
sit parallel to compose/ rather than nested under each topology:
1. Patches are reused across topologies (e.g. vllm-marlin-pad/ is
mounted by dual/, multi4/, and every dual/nvlink-*.yml). Topology
subdir would force symlinks or duplication.
2. Patches are scoped by (model, engine), not topology — a vLLM source
override doesn't change based on TP value; it's engine-internal.
3. Caches (torch_compile/, triton/) warm-start across composes — sharing
at engine level means switching from single/default to single/long-text
reuses JIT'd kernels.
Documents the relative-path convention (../../patches/, ../../cache/
from compose/<topology>/<file>.yml) and the rule for genuinely
topology-specific patches if any ever land: keep at engine level,
document the constraint in the patch's README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documentation distinguishes 1 / 2 / 3+ GPUs as `single` / `dual` /
`multi`, with separate doc pages (SINGLE_CARD.md, DUAL_CARD.md,
MULTI_CARD.md). Compose filename topology prefix should match.
`dual4.yml` was an awkward outlier — it had `dual` in the prefix
but actually meant TP=4 (4-card config). The clean shape: GPU
count is implicit when there's no ambiguity (`single` always 1,
`dual` always 2), and explicit when there is (`multi3` / `multi4`
/ `multi8`).
Renames:
- docker-compose.dual4.yml → docker-compose.multi4.yml
- docker-compose.dual4-dflash.yml → docker-compose.multi4-dflash.yml
Registry tags `vllm/dual4` and `vllm/dual4-dflash` in scripts/switch.sh
keep their existing names (backward compat for users running
`bash scripts/switch.sh vllm/dual4`); only the file paths in the
VARIANTS map are updated.
References updated: BENCHMARKS.md, docs/MULTI_CARD.md, docs/UPSTREAM.md,
models/qwen3.6-27b/CHANGELOG.md, models/qwen3.6-27b/vllm/patches/README.md,
sibling-table cross-references in 7 other compose headers.
AGENTS.md "Topology prefix" row tightened: examples now read
`single · dual · multi3 · multi4 · multi8` (dropped `dual-nvlink`
which is actually an interconnect feature suffix; dropped `quad`
which doesn't exist as a name in our convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three subsections under "Compose variants" documenting the convention
established by the 2026-05-09 Gemma 4 alignment pass:
1. Compose filename convention — <topology>-<feature>.yml, model
implied by parent directory. Examples: dual.yml, dual-turbo.yml,
dual-int8.yml, single.yml. Filename collisions across model
directories are fine (path disambiguates). Registry tags in
scripts/switch.sh decouple from filenames; rename file paths
while keeping tags backward-compatible.
2. Profile schema header — every compose declares (Model, Topology,
Drafter, KV, Vision, Max-ctx, Genesis, Best-for) in a structured
block at the top, before any free-form description. Schema
forces explicit declaration; catches drift between header
description and actual config.
3. Where experimental / unvalidated composes live — same directory
as shipped composes, untracked until verify-full + verify-stress
+ bench + soak validation passes. Mark with `Status: ⚠️
EXPERIMENTAL` or `⚠️ PREVIEW` so readers know. Don't create a
separate experimental/ subdirectory — relative paths to
../patches and ../cache would need re-pathing on promotion.
Future Claude sessions and any human contributor cloning this repo
now get this convention loaded by default via AGENTS.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the at-a-glance "Profile" block to the remaining composes for
full coverage across all 27 shipped + experimental compose files:
Qwen vLLM (5):
- docker-compose.yml (single-card default)
- docker-compose.long-text-no-mtp.yml
- docker-compose.long-vision.yml
- docker-compose.minimal.yml
- docker-compose.tools-text.yml
Qwen llama.cpp (2):
- docker-compose.yml (262K cliff-immune fallback)
- docker-compose.concurrent.yml (4-slot multi-tenant)
Each schema declares: Model / Topology / Drafter / KV / Vision /
Max ctx / Genesis / Best-for. The qwopus-bf16mtp.yml schema (still
untracked, preview-status) is also updated locally for consistency.
This completes the Option A naming/documentation pass:
Total composes: 27
With profile schemas: 27 (100%)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename the 4 shipped Gemma composes to drop the redundant `gemma-` prefix
(model is already named by the parent directory `models/gemma-4-31b/`):
- gemma-mtp.yml → dual.yml (recommended default, MTP implicit
— matches Qwen's `dual.yml` shape)
- gemma-mtp-int8.yml → dual-int8.yml (MTP + INT8 PTH KV variant)
- gemma-mtp-tp1.yml → single.yml (TP=1 single-card variant — boot
OOMs on Ampere 24 GB; for 32 GB+ only)
- gemma-dflash.yml → dual-dflash.yml (DFlash drafter variant — same shape
as Qwen's `dual-dflash.yml`)
Convention going forward (matches what Qwen has done since launch):
- File name: `<topology>-<feature>.yml`, model implied by parent directory
- Topology: `single` | `dual` | `dual-nvlink` | `dual4` | etc.
- Feature suffix: `-turbo` | `-dflash` | `-int8` | `-awq` | etc.
Registry tags in `scripts/switch.sh` keep their existing names
(`vllm/gemma-mtp`, `vllm/gemma-mtp-tp1`, `vllm/gemma-dflash`) — only the
file paths in the VARIANTS map are updated. This preserves backward
compat for users running `bash scripts/switch.sh vllm/gemma-mtp` etc.
Container names (`vllm-gemma-4-31b-mtp` etc.) are unchanged — they already
include the model name in the `vllm-<model>-<feature>` form, so no rename
needed there.
Plus continuing the at-a-glance profile schema rollout from 4d7356a:
profile blocks added to 9 more Qwen composes (bounded-thinking,
carnice-bf16mtp, dual4, dual4-dflash, dual-nvlink, dual-nvlink-turbo,
dual-nvlink-dflash, dual-nvlink-dflash-noviz, long-text). Each declares
Model / Topology / Drafter / KV / Vision / Max ctx / Genesis / Best-for.
References updated across BENCHMARKS.md, README.md, DUAL_CARD.md,
UPSTREAM.md, scripts/setup.sh, scripts/switch.sh, all 6 Gemma compose
cross-references, model patch READMEs, and codex-brief-dflash-int8.md.
Refs: noonghunna/club-3090#67
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First TB3 dual-eGPU + mixed-arch (A5000+3090) cross-rig data point. Each
card on a separate Razer Core X over Thunderbolt 3 (PCIe x4 Gen 3, ~3.94
GB/s effective vs ~32 GB/s on PCIe x16 Gen 4 — an ~8× link cut) on an
Intel NUC11TNH host with 16 GB system RAM total.
Headline: dual.yml hits 56.83 narr / 72.47 code wall TPS + soak p50 93.09
with 100% retention, 0 silent-empty, 0 MiB VRAM growth, verify-full/stress
all PASS. Within run-to-run noise of dual.yml PCIe x16 baseline — confirms
decode is per-card-bandwidth bound and cross-card NCCL allreduce on the
fp8 path doesn't dominate even at 8× reduced inter-card bandwidth.
Extends @aaronlockhartdev's #91/#95 patched-P2P finding (only +2%/+9% on
dual.yml from peer-bandwidth uplift) in the opposite direction: even with
8× *less* peer bandwidth, decode holds.
Refs: noonghunna/club-3090#107
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Listed as a community pointer per disc #108 announcement. Repo is AGPL-3.0,
provides browser admin panel + OpenAI-compatible proxy + multi-instance GPU
orchestration on top of club-3090. Marked "not yet officially adopted" — the
intent is a non-binding pointer until VykosX's project converges on a
stable surface area. Open invitation in the section text for other community
projects to be linked similarly.
Refs: noonghunna/club-3090#108
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three documentation/script follow-ups from @easel's #102 re-bench
on RTX 5090 Laptop:
- HARDWARE.md: new "Laptop GPUs — EC-managed power" subsection.
nvidia-smi -pl returns N/A on laptop-class GPUs (EC owns the
envelope, not the OS). Documents clock-lock as the only
software characterization path on laptops.
- CLIFFS.md: new "naming trap" callout in the KV-format section.
fp8_e5m2 is 8 bits/token; turboquant_3bit_nc packs 3 bits.
At 180K on 24GB, TQ3 fits where fp8 OOMs (4.36 GiB available
vs 6.64 GiB needed for fp8). Pin: TQ3 = long-context KV;
fp8 = short-context throughput.
- verify-stress.sh: auto-detect VLLM_ENFORCE_EAGER=1 in the
running container's env via docker inspect; when set, bump
STRESS_LONGCTX_TIMEOUT_S 300→600s and STRESS_TOOL_PREFILL_-
TIMEOUT_S 240→480s. Eager-mode prefill at 60K-140K runs
200-290s and was false-positiving as HTTP 000 (curl timeout)
in @easel's run. Both env vars also exposed for manual override.
Refs: noonghunna/club-3090#102
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bench parser was reading delta.content + delta.reasoning_content per
the laurimyllari fix in 71e5954. @alexpolo1's vLLM 0.20.1 dual.yml
sweep on issue #104 surfaced a third convention: delta.reasoning
(no _content suffix) — used by some vLLM versions / DeepSeek-style
streaming. Same symptom as #62: all 18 caps reading narr=0.00 / code=0.00
while the sampler captured correct power, SM clock, mem clock, throttle %,
pstate, p-state.
Confirmed by inspecting his attached SSE file (power-cap-N180-narrative.sse,
976 lines): 487 of 488 streamed chunks were in delta.reasoning, zero in
content or reasoning_content.
Fix: add delta.reasoning as a third addend in both decode-single and
decode-concurrent parser sites (lines 422, 550). Validated against the
attached SSE — would have produced 487 chunks / ~48.7 TPS at 180W cap,
consistent with steady-state throttled performance at 735 MHz SM clock.
Backwards-compatible: when the field is empty (most servers), text
contribution is "" so behavior is unchanged.
Refs: #104 (alexpolo1 dual 3090 sweep), #62 (laurimyllari 4090 sweep)
Phase 1 (commit 595be8f) bumped gemma-mtp.yml + gemma-mtp-tp1.yml to the
post-merge nightly that contains PR #41745 natively, but kept the
overlay tree as fallback. Phase 2 (gemma-mtp-int8.yml) and Phase 1
re-bench (105.91/141.11 TPS) both validated the post-merge nightly is
parity-clean → fallback no longer needed.
Removes 8 files at models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/.
Also cleaned up gemma-dflash.yml header which still referenced the
dropped overlay path. The DFlash compose itself is unchanged — it
still vendors PR #41703 (z-lab Gemma 4 DFlash drafter, still
unmerged) at vllm-gemma4-dflash/. That's a separate concern and
stays put until #41703 lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose now defaults to int8_per_token_head KV (the working
Ampere-compatible per-token-head dtype) rather than any fp8 variant.
Filename now matches what it actually does. fp8 PTH is still available
on Ada/Blackwell via KV_DTYPE=fp8_per_token_head env override.
Renamed via git mv (preserves history). Updates:
- container_name: vllm-gemma-4-31b-mtp-fp8 → vllm-gemma-4-31b-mtp-int8
- cache dirs: torch_compile_fp8 / triton_fp8 → torch_compile_int8 / triton_int8
- header comment: file's purpose now described as INT8 per-token-head
with hardware compatibility table showing FP8 PTH available on Ada+
- patches/ READMEs: updated companion-overlay references to new filename
No code/overlay changes — purely the rename and naming-consistency.
The vendored PR #40391 + #42006 + #41991 trees are unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. llama-cpp compose entrypoint regression (introduced in 8f103f3):
`exec llama-server` assumed PATH includes the binary, but the
ggml-org/llama.cpp:server-cuda image stores it at /app/llama-server.
Container hit a restart loop with `exec: llama-server: not found`
when we tried to recreate after Gemma 4 validation window. Fixed
to `exec /app/llama-server`. Affects ALL composes that mount this
pattern; verified production Qwen3.6-27B llama-cpp container now
restarts cleanly.
2. Gemma 4 fp8 PTH was the wrong default for Ampere — should be INT8 PTH:
Phase 2 boot crashed with:
ValueError: type fp8e4nv not supported in this architecture.
The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')
This is a hardware capability gap on sm_86. Per-token-head fp8 KV
uses Triton kernels with `fp8e4nv` storage; Ampere doesn't implement
that fp8 variant (only fp8e4b15 / fp8e5). Ada/Blackwell users have
fp8e4nv support; Ampere users don't.
The right Ampere unblock for Gemma 4 long-context is `int8_per_token_head`,
which dispatches to standard PyTorch torch.int8 ops (not Triton fp8).
PR #40391's whole purpose was unblocking the per-token-head KV family
regardless of underlying dtype — the page-size mismatch fix applies to
INT8 PTH exactly the same way it applies to FP8 PTH.
Same memory savings (1 byte/element vs bf16's 2 bytes) → same ~120K
ctx target. Different precision profile (INT8 has better near-zero
precision, narrower dynamic range than e5m2).
Updated:
- Default `--kv-cache-dtype` from `fp8_per_token_head` to `int8_per_token_head`
- Header comment with full hardware compatibility table (sm_86/89/90/120)
- Filename retained as `gemma-mtp-fp8.yml` for git history; could rename
to `gemma-mtp-pth.yml` (per-token-head, hardware-agnostic) in a
follow-up if the misnomer becomes confusing
Live validation pending — needs a fresh Phase 2 boot test with INT8 PTH
to confirm it actually runs cleanly on this rig + benchmark at extended
context. Will batch with another Qwen-down validation window.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New compose docker-compose.gemma-mtp-fp8.yml unlocks per-token-head fp8 KV
on Gemma 4 + 2× 3090 Ampere by vendoring three upstream-open vLLM PRs.
Target: 4× context lift over gemma-mtp.yml's 32K bf16 KV ceiling
(~120K with fp8 KV at the same TP=2 / mem-util headroom).
Phase 2 of the Gemma 4 unblock cycle (Phase 1 = #41745 overlay drop +
nightly bump shipped in commit 595be8f).
What's vendored:
1. vllm-pr40391-rebased/ — vLLM PR #40391 by @lisp19 (Gemma 4 KV cache
page-size alignment for per-token-head quantization). Rebased onto
post-Mamba-hybrid-support main; conflict in
vllm/v1/worker/gpu/attn_utils.py:_reshape_kv_cache resolved by
combining main's hybrid attn/mamba dispatch with PR #40391's
MLA-vs-standard-attention split for page_size_padded handling.
7 source files vendored; test files not vendored.
2. vllm-gemma4-tool-parser-fixes/ — PR #42006 (whytem) MTP streaming
multi-tool calls + PR #41991 (the-david-oy) parser bounds, stacked
on a single tool_parsers/gemma4_tool_parser.py file. Both target
non-overlapping line regions; clean stack with no manual conflict
resolution. Same family of bugs as the Qwen3 tool-parser SSE-silence
we shipped in commit 8f103f3 for issue #72.
Compose docker-compose.gemma-mtp-fp8.yml on port 8032 (separate from
gemma-mtp.yml's 8030 to allow A/B-comparison runs without recreate
churn). Mounts both overlays + sets --kv-cache-dtype fp8_e5m2 +
--max-model-len 98304 (initial test target; bump to 120K after first
validation passes). Separate torch_compile + triton cache dirs from
gemma-mtp.yml because cudagraph capture would re-key on the patched
kernel paths anyway.
Validation status: file-level only (Python parses, YAML config parses,
all overlay paths exist). Live boot validation pending — needs Qwen
container down on dual 3090. Expect verify-stress 91K-needle to be
the critical gate; Codex's earlier overlay attempts on this code area
produced decode-TPS decay turn-1 33 → turn-5 10 (30% retention) at
long context.
Drop triggers (when each upstream PR merges):
gh api repos/vllm-project/vllm/pulls/40391 --jq '.state, .merged_at'
gh api repos/vllm-project/vllm/pulls/42006 --jq '.state, .merged_at'
gh api repos/vllm-project/vllm/pulls/41991 --jq '.state, .merged_at'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: lisp19 (PR #40391 author) <noreply@github.com>
Co-Authored-By: whytem (PR #42006 author) <noreply@github.com>
Co-Authored-By: the-david-oy (PR #41991 author) <noreply@github.com>
