club-3090/models/qwen3.6-27b/CHANGELOG.md

# Qwen3.6-27B — Changelog

Dated history for Qwen3.6-27B configs in this repo. Combines the single-card and dual-card timelines (both were previously separate repos; consolidated here 2026-04-28).

## 2026-07-06 — dual-max KV switch: int8_per_token_head → fp8

Switches `vllm/qwen-27b-dual-max` (compose: `dual/fp8/mtp.yml`) from `int8_per_token_head` KV cache to `fp8`.

## 2026-05-30 — beellama.cpp as a first-class compose engine (DFlash, single-card)

Onboards [beellama.cpp](https://github.com/Anbeeld/beellama.cpp) (Anbeeld's llama.cpp fork — DFlash cross-attention spec-dec + SWA windowed KV) as a registry engine, with one single-card DFlash compose per model:

- **Qwen3.6-27B** → `beellama/dflash` (port 8060): Unsloth `Q5_K_S` target + Anbeeld `DFlash-IQ4_XS` draft (`--spec-type dflash`), `q5_0`(K)/`q4_1`(V) KV, 102K ctx, `-np 1`. Anbeeld's "Precision combo". DFlash is **tool-grammar-neutral** here — the external drafter does not amplify the CodeAct attractor the built-in MTP head does ([#237](https://github.com/noonghunna/club-3090/discussions/237)).
- **Gemma-4-31B** → `beellama/gemma-dflash` (port 8061): Unsloth `Q4_K_S` target + Anbeeld `DFlash-IQ4_XS` draft, same `q5_0`/`q4_1` KV + 102K ctx. The only single-card engine that does Gemma-4 **windowed KV** (big ctx) *and* spec-dec in one GGUF (ik-llama walls ~24K; mainline ~10 TPS).

**Status: 🧪 Experimental** — both reference a **locally-built** image `${BEELLAMA_IMAGE:-beellama-cpp:local}` (upstream ships no pullable image; build via the fork's `.devops/cuda.Dockerfile` with `CUDA_DOCKER_ARCH=86` + `-DGGML_CUDA_FA_ALL_QUANTS=ON`). Surfaced as `(NA: experimental)` in `--list`; launch is `--force`-gated.

**Catalog wiring:** new engine profile `engines/beellama-local.yml`; two DFlash GGUF drafters (`anbeeld-qwen-dflash`, `anbeeld-gemma-dflash`); `compose_registry.py` entries (`kvcalc_key="SKIP"` — llama.cpp-family); weights-map + hardware (`q4_1`) updates; INFERENCE_ENGINES + UPSTREAM notes.

**Default-resolver invariant preserved:** beellama is `#1` in `ENGINE_PREFERENCE[single]`, but its `(NA)` status + the absence of a `DEFAULTS` row means the resolver skips it → `qwen3.6-27b single` still resolves to `ik-llama/iq4ks-mtp` (asserted by `test-model-default-resolver.sh`). Auto-promotes to the single default only when a published image lands and the composes flip to production. Full suite green (only the pre-existing `test-submit-bench.sh` fixture failure remains).

## 2026-05-30 — Model-default resolver + user-pinnable defaults (`<model>/default`, `--set-default`)

Adds a two-layer default scheme on top of the existing `<engine>/default` map. `<engine>/default` stays the maintainer's recommendation (read-only to users); the new `<model>/default` is the *user's* preference — their `.env` pin if set, else a curated pick for the detected topology.

**What changed:**
- `compose_registry.py`: two maintainer knobs next to `DEFAULTS` — `RECOMMENDED_DEFAULT_MODELS` (a short opt-in shortlist, **not** an exhaustive ranking; new models are not auto-added) and `ENGINE_PREFERENCE` (per-topology engine order: single = `[beellama, ik-llama, llamacpp, vllm]`, dual/multi = `[vllm, ik-llama, llamacpp, beellama]`). Plus resolver helpers (`curated_default_target`, `community_default_target` stub, `model_default_pin_key`, `engine_set`/`model_set`, etc.).
- `registry-emit.sh`: shared resolver `model_default_target(root, model, topology)` — the single injection point for both launchers. Precedence ladder: `--variant` (caller) → `.env` pin → community seam (returns `None` today) → curated `ENGINE_PREFERENCE` walk (skips non-functional `(NA)` slugs) → degradation (notice + nearest-lower topology, else a clear "pick explicitly" message — never crashes). Plus `x_default_dispatch`: `X/default` with `X ∈ engine-set` → engine rec; `X ∈ model-set` → model default; else error (engines and model-ids are disjoint).
- `switch.sh`: `<model>/default` token; `--set-default <slug>` / `--clear-default <model>` (round-trip the `.env` pin `CLUB3090_DEFAULT_<MODELID>`); a "Defaults" view appended to `--list` (also standalone via `--defaults`) showing each model's resolved default + whether it's a user pin or curated.
- `launch.sh`: bare invocation → first installed shortlist model → its `<model>/default` (no full wizard); a pinned fast-path (*"Launch your default `<slug>`? [Y/n]"*); a post-boot offer (*"Make `<slug>` your default for `<model>`? [y/N]"*). Any narrowing flag keeps the explicit wizard path.
- Pin validation is **warn + fall back, never blocking**: unknown slug / wrong model / topology-mismatch / `(NA)` status → notice + curated default.
- Docs: README single-card realign (`ik-llama` = fastest blessed single default, `llama.cpp` = cliff-immune alternative) + "pin your default" in Quick start; FAQ (extended switch-model entry + new "set my own default" Q); SINGLE_CARD / DUAL_CARD / GETTING_STARTED / ADDING_MODELS resolver notes; UPSTREAM beellama Docker-image row (gates beellama onboarding; auto-promotes to single default on catalog). New test `scripts/tests/test-model-default-resolver.sh`.

Builds on the slug health flag below (the resolver skips non-functional defaults). Verified: 45 entries unchanged, kv-calc calibration 17/17, full suite green (only the pre-existing `test-submit-bench.sh` fixture failure remains).

## 2026-05-30 — Slug health/availability flag (registry `status` + `--list` markers + launch gate)

Added a lifecycle/health flag to every registry slug so `switch.sh --list` and the launch/switch path are no longer blind to a compose's lifecycle stage. Previously status lived only in compose-header comments, which drifted — `dual/autoround-int4/tq3-mtp-genesis.yml` declared `✅ Working (with Genesis)` even though the Genesis pin is parked/drifted and the compose won't boot clean.

**What changed:**
- `compose_registry.py`: `_entry()` gains a keyword-only `status` (default `production`) + `status_note`, validated against the canonical enum (`production · caveats · experimental · preview · upstream-gated · deprecated`). Added a `compose_header_status()` helper that maps a compose's profile-schema `Status:` emoji to the enum.
- Swept **every** compose `Status:` header to a canonical enum value (decision §12.5/§13): non-conforming strings normalized — e.g. genesis `✅ Working (with Genesis)` → `⏸️ Upstream-gated`, `⛔ TOMBSTONED` → `🗑️ Deprecated`, `🔵 v0.7.3 ONBOARDING` → `🧪 Experimental`, `🔵 PREVIEW` → `👁️ Preview`, `⭐ Code-optimized` → `✅ Production`.
- Re-flagged the non-functional slugs: all `*genesis*` + gemma-4-31b single fp8 → `upstream-gated`; carnice → `caveats`; qwopus + qwen-a3b-preview → `preview`; the bf16/int8 A/B composes, llamacpp bounded-thinking, all PRISM/APEX eval lanes, and the gemma-4-26b-a4b onboarding composes → `experimental`; `tq3-mtp` → `deprecated`.
- `registry-emit.sh` now emits `status` + `status_note` on the VARIANT row (last two fields); both loaders + the parity tests read the extended field list.
- `switch.sh --list` shows a health marker: `production` unmarked · `caveats` → `(caveats)` · the (NA) set → `(NA: <word>)`. Model·topology grouping (#264) preserved.
- Launch/switch gate: `production` launches silently, `caveats` launches with a one-line notice, the (NA) set warns + requires `--force`. `launch.sh` surfaces the same flag before delegating to `switch.sh`.
- New drift-guard test `scripts/tests/test-compose-status-drift.sh`: asserts every registry status ∈ enum, every compose header maps to the enum, and the two agree.

Foundational for the upcoming model-default resolver (it must skip non-functional slugs when picking a curated default) and a safety fix on its own — users could previously boot the pin-drifted Genesis composes blind. Verified: 45 entries unchanged, kv-calc calibration 17/17, full test suite green (only the pre-existing `test-submit-bench.sh` fixture failure remains).

## 2026-05-29 — vLLM `nvlink-*` dual composes removed (NVLink auto-detected)

The four `dual/autoround-int4/nvlink-*.yml` composes (`nvlink-fp8-mtp`, `nvlink-turbo`, `nvlink-dflash`, `nvlink-dflash-noviz`) are removed, along with their `vllm/dual-nvlink*` launch slugs. They were thin `extends:` stubs whose only override was `NVLINK_MODE=force_on` — redundant since every dual compose now auto-detects NVLink at boot via `detect_nvlink.sh` (`NVLINK_MODE=auto` → flips on `NCCL_P2P_LEVEL=NVL` + custom-all-reduce when a bridge is present, else `NCCL_P2P_DISABLE=1` + `--disable-custom-all-reduce`). NVLink rigs now get the same path from the base dual compose with no separate slug; force it explicitly with `NVLINK_MODE=force_on scripts/switch.sh vllm/dual` if auto-detect misses. The historical NVLink bench rows (JusefPol PR #31, danbedford #74/#92/#96) are preserved in `BENCHMARKS.md`. The earlier dated entries below that describe adding these composes are kept as append-only history.

## 2026-05-07 — `llamacpp/default` adds `--reasoning-format none` (opencode unblock)

@syangsao reported opencode hangs indefinitely against `llamacpp/default` despite the server returning 200 with content tokens generated successfully ([#97](https://github.com/noonghunna/club-3090/issues/97)). Diagnosis via curl SSE capture: every delta was in the `reasoning_content` field, never `content` — Qwen3.6's thinking mode emits `<think>` blocks that llama.cpp's peg-native parser routes to `reasoning_content` by default. opencode (and most simple OpenAI-compat clients) ignore `reasoning_content` and wait indefinitely for `content` deltas that never arrive.

**Fix**: added `--reasoning-format ${REASONING_FORMAT:-none}` to `models/qwen3.6-27b/llama-cpp/compose/docker-compose.yml` and `single/concurrent.yml`. Default `none` collapses thinking into the content stream — opencode and other simple clients work out-of-box. Power users wanting `reasoning_content` separation set `REASONING_FORMAT=auto` in `.env` or shell.

Cross-rig validated by @syangsao (1× 3090 water, 330W cap, b9014 image): Fix 2 path (`chat_template_kwargs.enable_thinking: false` in opencode config) confirmed unblocked. Fix 1 (server-side flag) is the same root-cause solution applied at the compose layer so every contributor doesn't hit this. Bench numbers from his unblocked session: 28.88 TPS decode / 741 TPS prompt at 45K accumulated context — within Q3_K_XL Qwen3.6 + DeltaNet hybrid expectations.

Companion observation: DeltaNet hybrid prevents prefix-cache reuse across turns ("forcing full prompt re-processing due to lack of cache data ... SWA or hybrid/recurrent memory"). Each multi-turn opencode interaction does full prefill — known characteristic, not a regression.

## 2026-05-07 — NVLink + DFlash compose variants added (#92, #96)

Two new community-contributed composes from @danbedford for 2× 3090 with NVLink bridge:

- **`dual/nvlink-dflash.yml`** (port 8018) — 185K ctx + DFlash N=5 + vision. NCCL P2P over NVLink + custom_all_reduce ENABLED. Drops `expandable_segments=True` (NVLink startup-crash fix from JusefPol/PR #31). Bench (2× 3090 NVLink, 230W cap): **101.55 / 163.33 narr/code wall TPS** (CV 1.8%/1.9%), **+17% narr / +16% code over his PCIe `dual-dflash` baseline** of 86.62 / 141.02. PASSES verify-full 8/8 + verify-stress 7/7 + continuous soak (0 err, 100% retention).

- **`dual/nvlink-dflash-noviz.yml`** (port 8019) — text-only variant of the above. Drops MoonViT to free ~0.78 GB/card → max_model_len pushed from 185K to **188K**. Empirically determined: 189K had 1/3 success rate (flaky on fresh reboot), 188K is the stable ceiling. Bench: **103.24 / 167.45 narr/code wall TPS** (CV 2.2%/3.6%), **+17% narr / +17% code over PCIe `dual-dflash-noviz` baseline** (88.31 / 142.79). PASSES same validation chain.

Both variants registered in `scripts/launch.sh` and `scripts/switch.sh`. Sibling-list headers updated across `dual.yml`, `dual-nvlink.yml`, `dual-dflash.yml`, `dual-dflash-noviz.yml` for cross-reference. Marked **community-contributed, experimental** in headers.

Note: both composes use `--tool-call-parser qwen3_coder` but are direct-cmd (no entrypoint script), so they don't currently receive the `qwen3coder_tool_parser_deferred_commit.py` sidecar shipped 2026-05-07 for issue #72. Consistent with the existing direct-cmd pattern (`dual.yml`, `dual-dflash.yml` also lack the sidecar). If the SSE-silence bug fires on these variants, follow-up PR can add an entrypoint script.

## 2026-05-06 — `PYTORCH_CUDA_ALLOC_CONF` override knob added to 14 composes

Follow-up to the v7.72.2-uplift pin bump: a single-card RTX 3090 Ti rig on WSL2 (driver 596.36) hit `gptq_marlin_repack` boot crashes (`CUDA driver error: device not ready`) on the new nightly. The minimal compose (no Genesis, no spec-decode, no TQ3 KV) reproduced cleanly with just `--quantization auto_round`, and `CUDA_LAUNCH_BLOCKING=1` did not move the failure site (rules out async-residual error from a prior kernel).

`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False` resolves the crash. This is the same workaround that already addresses JusefPol's NVLink boot-crash report (PR #31, hardcoded in the `dual-nvlink*.yml` composes). The exact failing call hasn't been isolated.

All 14 single-card and PCIe dual-card composes now expose `PYTORCH_CUDA_ALLOC_CONF` as a `${...}:-` override knob (defaults preserved); affected users drop `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False` into a `.env`. The two `dual-nvlink*.yml` composes are unchanged.

Single-observation note: weight-load on a fresh boot (caches cleared) was 32 sec with `expandable_segments:True` and 13 sec with `expandable_segments:False` on this rig. Not a controlled benchmark — cache state and other factors weren't held constant. Suggestive only.

See [docs/HARDWARE.md](../../docs/HARDWARE.md#fix--disable-pytorch-expandable_segments-if-boot-crashes-at-weight-repack) for the full failure signature and override recipe.

## 2026-05-05 — v7.72.2-uplift: Genesis pin bump + sidecar consolidation ⭐

Aligns Qwen3.6-27B configs with Genesis [v7.72.2](https://github.com/Sandermage/genesis-vllm-patches/blob/main/CHANGELOG.md) (pin SHA `7b9fd319`) and Sander's PROD-validated vLLM pin (`nightly-01d4d1ad375...`, allowlist entry #2).

**Pin bumps:**
- `scripts/setup.sh` `GENESIS_PIN`: `2db18df` (v7.69) → `7b9fd319` (v7.72.2)
- All 16 composes' vLLM image: `nightly-7a1eb8ac2ec...` → `nightly-01d4d1ad375...`

**Sidecars retired** — 6 local `.py` patches deleted from `vllm/patches/`, all confirmed redundant on v7.72.2: `patch_inputs_embeds_optional.py` (PN35 supersedes), `patch_pn30_dst_shaped_temp_fix.py` (PN30 v7.68), `patch_pn25_genesis_register_fix.py` (PN25), `patch_tolist_cudagraph.py` (P78), `patch_workspace_lock_disable.py` (PN34), `patch_pr40798_workspace.py` (research artifact).

**PN59 added to 7 Genesis-loaded composes** (`docker-compose.yml`, `dual-turbo.yml`, `long-text.yml`, `long-text-no-mtp.yml`, `long-vision.yml`, `bounded-thinking.yml`, `tools-text.yml`) for consistency.

`dual/docker-compose.yml` left intentionally Genesis-free as a debugging fallback for cross-engine bisect.

**dual-turbo bench (2× 3090, single-stream)**: 81.21 narr / 108.20 code wall TPS (5 measured runs each, CV 2.3%/0.9%), AL 3.46. **VRAM dropped from 22.1 GB/card → 20.0 GB/card** (PN35 native fold + Sander's audit-pass cleanups).

**Cross-rig PN59 finding**: single-card 24 GB Cliff 2b unchanged on `long-text.yml` despite v7.72.2's PN59 streaming-GDN orchestrator. Filed [Sandermage/genesis-vllm-patches#22](https://github.com/Sandermage/genesis-vllm-patches/issues/22) with reproducer + 4 fix proposals.

**v7.72.1 closes [#57](https://github.com/noonghunna/club-3090/issues/57)** (lex's xgrammar-patternProperties fire on long-prompt agentic IDE traffic).

See cross-cutting [CHANGELOG.md](../../CHANGELOG.md) entry for the full narrative + bench delta table; [vllm/patches/README.md](vllm/patches/README.md) for what's load-bearing now.

---

## 2026-05-04 — Carnice-V2-27B + BF16 MTP overlay — new compose variant ⭐

Adds `dual/carnice-bf16mtp.yml`: [kai-os/Carnice-V2-27b](https://huggingface.co/kai-os/Carnice-V2-27b) (Hermes-style agentic fine-tune of Qwen3.6-27B) quantized to INT4 via delta-merge of Lorbus's AutoRound grid, with a BF16 MTP overlay for clean spec-decode acceptance.

**Key findings from the diagnostic push:**
- Hypothesis B (MTP quant-grid mismatch) accounted for ~70% of the AL gap. Un-quantizing 7 mtp.layers.0.* projections (BF16 overlay) recovered AL from 2.0 → 3.0.
- Tool-call format: Carnice's Hermes-style template used XML, but vLLM's `--tool-call-parser hermes` expects JSON. Patched chat template instructs JSON output inside `<tool_call>` tags. Vendored at `patches/carnice-chat-template.jinja`.
- Full 262K context confirmed (22,246 MiB/card), 2 streams, same fp8 KV + MTP n=3 as dual.yml.

**Validation:**
- `verify-full.sh`: 7/8 PASS (thinking test lenient — Carnice is concise, not verbose)
- `verify-stress.sh`: 6/7 PASS (needle recall at ≥60K — model-level GDN attention ceiling)
- `bench.sh` (n=5): **71.75 narr / 80.35 code wall TPS**, MTP AL 3.02-3.14, TTFT 141ms
- `soak-test.sh` (8×3 turns): PASS — 0 MiB growth, 0 errors, 101.6% TPS retention

**Compose:** `dual/carnice-bf16mtp.yml`

## 2026-05-03 late PM — `multi4-dflash.yml` TP=4 DFlash validated on 4× RTX 3090 PCIe ⭐

Adds `multi4/dflash.yml`, a 4-card full-context DFlash variant validated on Whamp's 4× RTX 3090 PCIe rig for [club-3090 discussion #26](https://github.com/noonghunna/club-3090/discussions/26). This is a capacity / 262K-code variant, not a replacement for the faster 2-card DFlash short-prompt path.

**Config accepted by vLLM pre-check:**
- `tensor_parallel_size=4`
- `max_model_len=262144`
- `max_num_seqs=2`
- `max_num_batched_tokens=8192`
- `dtype=bfloat16`, FP16/default KV (required by DFlash on Ampere)
- `speculative_config={"method":"dflash","num_speculative_tokens":5}`
- reported GPU KV cache size: **207,264 tokens**
- reported max concurrency at 262K/request: **2.27×**

**Validation:**
- Boot: clean, ready after 375s on a warm image/model cache.
- `verify-full.sh`: PASS.
- `verify-stress.sh`: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: **58,570 tokens** and **91,070 tokens**.
- `bench.sh`: **64.00 narrative / 104.40 code wall TPS** (CV 2.8% / 3.0%), TTFT 143ms / 164ms.
- DFlash AL during code bench: last three log samples **4.43 / 4.37 / 4.35**.
- Peak VRAM during bench: **21,960 MiB/card**.

**Interpretation:** TP=4 DFlash gives a useful code-speed uplift over `multi4.yml` (104 vs 76 code TPS) while retaining full 262K admission, but PCIe TP=4 allreduce keeps it below the 2-card DFlash variants' raw single-stream TPS. Use it for 4-card, full-context, code-heavy work with two admitted streams.

## 2026-05-03 PM — `multi4.yml` TP=4 baseline validated on 4× RTX 3090 PCIe ⭐

Adds `multi4/docker-compose.yml`, a measured 4-card fp8/MTP baseline derived from `dual.yml` by scaling tensor parallelism and streams from 2 → 4. Validation came from Whamp's 4× RTX 3090 PCIe rig in [club-3090 discussion #26](https://github.com/noonghunna/club-3090/discussions/26).

**Config accepted by vLLM pre-check:**
- `tensor_parallel_size=4`
- `max_model_len=262144`
- `max_num_seqs=4`
- `max_num_batched_tokens=8192`
- `kv_cache_dtype=fp8_e5m2`
- reported GPU KV cache size: **483,200 tokens**
- reported max concurrency at 262K/request: **6.77×**

**Validation:**
- Boot: clean, ready after 355s on a warm image/model cache.
- `verify-full.sh`: PASS after warm retry (first Paris request hit a cold-path 30s script timeout; direct retry returned HTTP 200 in 0.2s and full rerun passed).
- `verify-stress.sh`: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: **58,569 tokens** and **91,070 tokens**.
- `bench.sh`: **63.01 narrative / 76.25 code wall TPS** (CV 2.1% / 4.0%), TTFT 111ms / 132ms.
- MTP AL during code bench: last three log samples **3.42 / 3.53 / 3.62**.
- Peak VRAM during bench: **23,494 MiB/card**.

**Interpretation:** TP=4 gives the first published 4×3090 Cliff 2 boundary data and higher full-context concurrency headroom, but single-stream TPS is lower than TP=2 on PCIe-only allreduce (published TP=2 fp8/MTP baseline is ~69 / 89 TPS). Use `multi4.yml` for 4-card capacity / Cliff 2 margin, not for fastest single-user short-prompt decode.

## 2026-05-02 PM — Genesis v7.69 + vllm#35975 backport — Cliff 2 60K CLOSED ⭐⭐

Genesis pin bump `fc89395` (v7.66) → `2db18df` (v7.69 dev tip). All three v7.66/v7.68 regressions we surfaced upstream landed in v7.69, plus a local backport of [vllm#35975](https://github.com/vllm-project/vllm/pull/35975) brings the Cliff 2 single-prompt envelope to **60K cleanly on TQ3 + MTP K=3 at 24 GB**. Two shippable single-card recipes ship at this pin.

**v7.69 closes upstream:**

| Patch | What | Status before v7.69 | Status in v7.69 |
|---|---|---|---|
| PN30 v7.68 part3 | DS conv state row-stride fix (replaces our dst-shaped sidecar) | drift-markers too generic — silent re-fail | landed clean, our sidecar drops |
| P103 worker self-install | FLA Cliff 2 chunked fwd_h+fwd_o orchestrator survives `exec vllm serve` | setattr lost on worker spawn → "rebound at 0 caller sites" | self-install hook in `chunk.py`, fires on TP=1 |
| PN32 v1 | GDN chunked-prefill threshold + chunk size env-vars | not yet shipped | landed (`PN32_GDN_CHUNKED_PREFILL=1`, `PN32_GDN_CHUNK_SIZE=8192`, `PN32_GDN_CHUNK_THRESHOLD=16384`) |

**Codex P103 cu_seqlens gate fix queued for v7.70:** filed [Genesis #18](https://github.com/Sandermage/genesis-vllm-patches/issues/18) — `_single_seq_cu` detection lets `cu_seqlens.shape[0] == 2` enter the chunked path. Diagnostic logging on dev205 showed `q.shape[1]` always 4128 (capped by vLLM `max_num_batched_tokens` for spec-decode K=3) so MAX_T=16384 never engages on real serving — gate-redirect works but doesn't fire under our admission cap. Cliff 2 in practice is **residency**, not gate logic: 50 MiB OOM after 394 successful T=4128 chunks → cumulative state filling 22.96 GiB of 23.56 GiB total. Lower mem-util buys activation headroom by spending KV capacity. Three distinct ceilings clarified per Codex round 2: declared `max_model_len` (admission), safe single-prompt prefill length (Cliff 2), concurrency capacity.

**Local backport added:** [vllm#35975](https://github.com/vllm-project/vllm/pull/35975) (skip `inputs_embeds` GPU buffer for text-only models). Two-site regex text-patch via `patch_inputs_embeds_optional.py`:
- `gpu_model_runner.py`: wraps `self.inputs_embeds = self._make_buffer(...)` in `if self.supports_mm_inputs or self.enable_prompt_embeds:`
- `llm_base_proposer.py`: wraps `self.inputs_embeds = torch.zeros(...)` in `if self.supports_mm_inputs:`

Measured ~444 MiB freed on text-only paths (claimed ~64 MiB upstream — claim assumes a smaller config; our 180K + MTP K=3 path benefits more).

**Mem-util matrix on long-text 180K + MTP K=3 (60K stress):**

| mem-util | #35975 | 60K result |
|---|---|---|
| 0.95 | no | OOM (Cliff 2) |
| 0.95 | yes | OOM (Cliff 2) |
| 0.92 | yes | PASS — 643s wall |
| **0.93** | **yes** | **PASS — 623s wall** ⭐ shipped |

**Two shippable Cliff-2-closed variants:**

| Variant | max-model-len | mem-util | MTP K | TPS regime | Single-prompt envelope | Use when |
|---|---|---|---|---|---|---|
| `long-text.yml` (Balanced MTP) ⭐ | 180000 | 0.93 | 3 | 50 narr / 67 code (cold short prompt); decode wins from MTP at high accept | **60K** PASS @ 623s; 90K indeterminate (curl >25 min) | default for steady-state agent + chat |
| `long-text-no-mtp.yml` (Max-context) | 200000 | 0.95 | off | ~33 narr / ~40 code (no spec-decode) | **60K** PASS @ 537s | one-shot >50K input where you can wait + don't need MTP |

Both top out at the **60K hardware-physical wall on 24 GB single-card.** 90K MTP-off attempt didn't complete in 25 min (HTTP 000 at 1500s curl); not shipped, not failed — indeterminate.

**Sidecars dropped (3, all closed natively in v7.69):**
- `patch_pn25_genesis_register_fix.py` → covered by v7.66 PN25 + v7.69 worker-spawn registration
- `patch_pn30_dst_shaped_temp_fix.py` → covered by v7.69 PN30 part3
- `patch_workspace_lock_disable.py` → covered by v7.69 PN34_WORKSPACE_LOCK_RELAX env-gate

**Sidecars retained on master (2):**
- `patch_inputs_embeds_optional.py` (NEW) — backport of vllm#35975. Drop when upstream merges.
- `patch_tolist_cudagraph.py` — unchanged.

**Genesis env bundle (long-text.yml):**
```
GENESIS_ENABLE_P103=1
GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
GENESIS_PN32_GDN_CHUNK_SIZE=8192
GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
GENESIS_FLA_FWD_H_MAX_T=16384
GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
GENESIS_VLLM_SSM_CONV_STATE_LAYOUT=DS
```

Branch `v7.69-cliff2-test` merged to master at commit `15b84df`. Per-round bisect log + mem-util sweep at `results/v0.20-migration/v769-codex-r1-test.summary` (404 lines, six bisect rounds).

**Cross-rig contributions filed:**
- [Genesis discussion #19](https://github.com/Sandermage/genesis-vllm-patches/discussions/19) — three v7.66/v7.68 regression reports + acknowledgement of all closures in v7.69.
- [Genesis issue #18](https://github.com/Sandermage/genesis-vllm-patches/issues/18) — P103 cu_seqlens gate fix proposal for v7.70.

## 2026-05-02 — Genesis v7.66 + Cliff 1 mech B closed ⭐

Genesis pin bump `753344b` → `fc89395` (v7.66 dev tip). Cliff 1 mech B closed across all 4 TQ3 composes via two local backports:

**PN25 v3 import-time backport** (`patch_pn25_genesis_register_fix.py`):
Sander's PN25 mechanisms — both v7.65 `@torch.library.custom_op` (which crashes on `infer_schema` inside dynamo trace) AND v7.66 `direct_register_custom_op` + `Library("genesis", "FRAGMENT")` (which crashes on `instantiate_user_defined_class_object` inside dynamo trace) — fail on TP=1 spawn. Our v3 text-patches `vllm/model_executor/layers/activation.py` to register the op at module-import time, BEFORE any trace context exists. Survives both Sander mechanisms because it sidesteps registration-during-trace entirely.

**PN30 dst-shaped temp fix** (`patch_pn30_dst_shaped_temp_fix.py`):
Sander's PN30 `a9977d8` corrupts DS conv state row strides by raw-memcpying a compact `.contiguous()` source-tail into a strided destination. Our fix builds a destination-shaped temp inside `collect_mamba_copy_meta` (where both source AND destination block IDs are known) and does the strided copy correctly. Diagnosis credit: ChatGPT/Codex CLI cross-check.

**Validation matrix (v7.66 + local sidecars, verify-stress.sh 7-probe ladder):**

| Compose            | Ctx | mem-util | Probes | Failure          |
|--------------------|-----|----------|--------|------------------|
| long-text          | 180K | 0.95 | 6/7 | Cliff 2 architectural |
| long-vision        | 145K | 0.95 | 6/7 | Cliff 2 architectural |
| bounded-thinking   | 180K | 0.95 | 6/7 | Cliff 2 architectural |
| dual-turbo (TP=2)  | 262K | 0.85 | 6/7 | Cliff 2 architectural |

Backoff from 214K + 0.985 → 180K + 0.95 (long-text/bounded-thinking) and 198K + 0.98 → 145K + 0.95 (long-vision) was needed to give activation headroom for the PN12+PN25 FFN pool residence + PN30 dst-shaped temp lifecycle. Vision tower's persistent ~1 GB tightens long-vision further.

**Sander's v7.66 PN33 partial:** PN33 (default ON) closes BOOT-time profile_run workspace_lock issue, but the runtime decode workspace_lock at `turboquant_attn.py:1350:_decode_attention` still fires on TP=1. Cross-rig data sent to Sander via [discussion #19 reply](https://github.com/noonghunna/club-3090/discussions/19#discussioncomment-16785590).

**Sander's v7.66 PN31:** still doesn't fit on 24 GB. Per-shape persistent buffers + PN12+PN25 pool residence outpace activation budget at `chunk_fwd_o`. Lower mem-util (0.95) is sufficient to close the 25K tool-RETURN path PN31 was designed to fix without needing PN31 itself.

**Sidecars retained on master (4):**
- `patch_pn25_genesis_register_fix.py` (PN25 v3 import-time, TP=1 only)
- `patch_pn30_dst_shaped_temp_fix.py` (replaces Sander's compact `.contiguous()`)
- `patch_workspace_lock_disable.py` (PN33 narrowed but didn't close runtime decode path)
- `patch_tolist_cudagraph.py` (cudagraph capture fix, unchanged)

Per-config + cross-rig results in `results/v0.20-migration/v766-pin-results.summary` and the per-compose `*-pn30.summary` files.

## 2026-05-01 PM — vLLM v0.20 + Genesis v7.65 dev tip migration ⭐

Master pin migration from `vllm-openai:nightly-07351e088...` (`0.19.2rc1.dev205`) + Genesis v7.64 (`64dd18b`) to `vllm-openai:nightly-7a1eb8ac2ec...` (`0.20.1rc1.dev16+g7a1eb8ac2`) + Genesis v7.65 dev tip (commit `d89a089`). v0.20's revised TQ FA prefill paths ([vllm#40092](https://github.com/vllm-project/vllm/pull/40092)) and Genesis v7.65's PN26b sparse-V kernel + PN17 FA2 lse-clamp + P38B/P15B in-source hooks together close Cliff 1 mech B sub-mechanisms that forced the dev205 backoffs. Three of our local sidecars (`patch_pn12_ffn_pool_anchor.py`, `patch_pn12_compile_safe_custom_op.py`, `patch_fa_max_seqlen_clamp.py`) replaced by Genesis-native equivalents.

**Sidecars retained:**
- `patch_workspace_lock_disable.py` (NEW) — relaxes vllm#39226 strict assertion to one-shot WARNING. Sandermage's P98 covers the same surface but auto-skips on v0.20 (drift-marker false-positive). Drop when Sandermage ships marker fix.
- `patch_tolist_cudagraph.py` — unchanged.

**Sidecars dropped:**
- `patch_pn12_ffn_pool_anchor.py` → covered natively by PN12 on v0.20
- `patch_pn12_compile_safe_custom_op.py` → covered by Genesis PN25
- `patch_fa_max_seqlen_clamp.py` → covered by PN17 + P15B

**Mamba block_size cap fix:** v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid Mamba+TQ3 the engine forces `block_size=4128`. Bumped `GENESIS_PROFILE_RUN_CAP_M` and `GENESIS_PREALLOC_TOKEN_BUDGET` from 4096 → 4128 across all 5 main composes.

**Restored ceilings (vs dev205 backoff):**

| Variant | Before (dev205+v7.64) | After (v0.20+v7.65 dev) | Δ |
|---|---|---|---|
| `long-text.yml` | 185K + 0.975 | **214K + 0.985** | +29K (+16%) |
| `long-vision.yml` | 140K + 0.95 | **198K + 0.98** | +58K (+41%) |
| `bounded-thinking.yml` | 185K + 0.975 | **214K + 0.985** | +29K (+16%) |
| `tools-text.yml` | 75K + 0.97 (fp8) | **75K + 0.97** (unchanged) | flat |
| `dual-turbo.yml` | 262K + 0.85 | **262K + 0.85** (full v7.65 PROD env-vars) | flat ctx, +9% TPS |

**Bench results (n=5, 3 warmups + 5 measured, canonical narr+code prompts):**

| Variant | Narr wall_TPS (CV) | Code wall_TPS (CV) | TTFT | AL | Avg accept | VRAM | KV pool tokens |
|---|---|---|---|---|---|---|---|
| `long-text.yml` 214K | 49.74 (2.6%) | 67.39 (2.7%) | 154/155 ms | 3.34-3.51 | 78-84% | 23.4 GB | 284,832 (1.03×) |
| `long-vision.yml` 198K | 50.32 (2.3%) | 66.12 (4.1%) | 159/158 ms | 3.40-3.56 | 79-85% | 22.3 GB | 264,192 (1.02×) |
| `bounded-thinking.yml` 214K | 49.77 (1.4%) | 65.80 (2.3%) | 155/154 ms | 3.25-3.61 | 75-87% | 21.7 GB | 284,832 (1.03×) |
| `tools-text.yml` 75K (fp8) | 53.32 (2.3%) | 69.66 (1.4%) | 150/153 ms | 3.53-3.59 | 84-87% | 22.2 GB | 104,000 (1.05×) |
| `dual-turbo.yml` 262K (TP=2) | 58.33 (2.9%) | 76.01 (4.5%) | 112/110 ms | 3.39-3.51 | 79-84% | 19.8 GB/card | **1,523,232 (4.67×)** |

**Concurrent throughput on dual-turbo** (canonical code prompt, 2 runs per stream):

| Streams | Total TPS | Per-stream mean | Per-stream CV | Speedup |
|---|---|---|---|---|
| 1 | 74.03 | 73.99 | 3.7% | 1.00× |
| 2 | 128.74 | 65.57 | 14.1% | 1.74× |
| 3 | 126.52 | 55.41 | 31.9% | 1.71× |
| **4** | **269.03** | **74.05** | **3.1%** | **3.63×** |

n=4 lands at near-single-stream per-stream TPS — true parallel decoding of full-context streams, not interleaved. The n=2/n=3 dips are scheduler artifacts on small bench sizes (high CV at n=3 confirms interleave behavior). Practically: dual-turbo serves either 1 stream at 76 TPS or 4 streams at 269 TPS aggregate.

**Validation:** verify-full ✅ 8/8 on every variant. verify-stress 33K AND 50K tool-prefill ✅ PASS on every variant (the 50K cliff that fired on EVERY dev205 config no longer reproduces). Branch `v0.20-migration`; bench captures at `results/v0.20-migration/`.

## 2026-04-30 PM — Cliff 1 closes; long-text 218K + long-vision 198K

PN12 was silently no-op'd on dev205+ via anchor drift (same bug class as P101). Genesis `apply_all` reported "PN12 applied" while live `vllm/model_executor/layers/activation.py` retained the vanilla `SiluAndMul.forward_cuda`. Local sidecar `patch_pn12_ffn_pool_anchor.py` repairs it; bundled Genesis tree carries the fix via [PR #13](https://github.com/Sandermage/genesis-vllm-patches/pull/13). Combined with local `patch_fa_max_seqlen_clamp.py` (P104 FA softmax_lse clamp), Cliff 1 closes on TQ3 paths.

**New shipped ceilings:**
- `long-text.yml`: 205K → **218K** at 0.985 mem-util (no vision, no override). Engine ceiling vLLM-reported 218K. Verify-stress + verify-full pass; MTP AL 2.66; VRAM 23.7/24 GB.
- `long-vision.yml`: 192K → **198K** at 0.98 mem-util (vision on). Engine ceiling vLLM-reported 198K. 0.985 + vision reopens Cliff 1 (more goes to KV at the cost of activation budget; vision tower's persistent ~1 GB makes 0.98 the right balance).
- `--num-gpu-blocks-override 50` no longer needed at 0.985 — anchor-fixed PN12 cuts allocator churn enough that natural activation budget at higher mem-util is sufficient on text-only path.
- 0.99 mem-util ruled out — driver/system reserves ~440 MiB; vLLM startup check fails at 0.99.

**Cliff 2 unchanged.** Single-prompt >50–60K still OOMs in DeltaNet GDN. Both long-* variants stay "steady-state accumulation across many turns, not single-shot big prompts."

**Variants stay distinct:** `docker-compose.yml` (48K, below both cliffs, fast boot) and `tools-text.yml` (FP8 path for IDE agents) remain valuable for their respective use cases. Four-variant menu kept; the long-* options now ship at higher ceilings.

Branch `cliff1-fa-clamp` carries the changeset; commits `41eabac` (PN12 sidecar) → `f3e5b52` (218K bisection) → `26e5f65` (docs).

## 2026-04-29 — Genesis v7.62.x + PN8 enabled on FP8 paths

Sandermage shipped Genesis v7.62.x (commit `917519b`) on 2026-04-29 with PN8 (MTP draft online-quant propagation — backport of vllm#40849) targeting the FP8+MTP memory-headroom problem. We benched the patch across all 5 single-card composes that use Genesis:

| Compose | KV | mem-util | PN8 effect | TPS Δ | Verdict |
|---|---|---|---|---|---|
| `tools-text.yml` (75K, fp8) | fp8 | 0.97 | **−900 MiB at boot · Cliff 1 closes** ⭐ | −7% code | **PN8 enabled** |
| `fast-chat.yml` (20K, fp8) | fp8 | 0.95 | **−800 MiB at boot** | −4.7% code | **PN8 enabled** |
| `docker-compose.yml` (48K, TQ3) | TQ3 | 0.92 | no-op (already plenty of headroom) | −3% / −5% | PN8 not enabled |
| `long-vision.yml` (192K, TQ3) | TQ3 | 0.98 | KV pool +230 MiB, engine ceiling 192K → 198K, but Cliff 1 still fires | −5% | PN8 not enabled (commented in env, opt-in) |
| `long-text.yml` (205K, TQ3) | TQ3 | 0.98 | no effect (engine ceiling capped by block-size divisor at 206K) | not benched | PN8 not enabled |

Why split-decision: the **Cliff 1 OOM that ampersandru hit on `long-vision.yml`** is an FFN intermediate-buffer activation peak (138 MiB allocate at `intermediate_size=17408 × max-num-batched-tokens=4128`), not a draft-model footprint. PN8's quant-config propagation doesn't reach that buffer on TQ3 paths. On FP8 paths the draft head's own footprint shrinks meaningfully — that's where the win is.

**Cross-rig data + analysis posted to Sandermage**: [single-3090 #1 comment 4343317153](https://github.com/noonghunna/qwen36-27b-single-3090/issues/1#issuecomment-4343317153).

Other v7.62.x items relevant to us (not yet benched here):
- **PN11** (Quentin-M, vllm#41142 streaming tool-call IndexError fix) — applies cleanly via the auto-detected REC; planned to enable in tools-text + fast-chat next pass.
- **TurboQuant k8v4 unlocked on hybrid GDN via P4 + P98** — Sandermage's A5000 measurement +1.9%; we'll bench on dual.
- **Per-GPU recommendation system** (`vllm/_genesis/gpu_profile.py`) — boot log now lists `[REC]/[OFF]` per patch on this card. Nice ergonomics.

## 2026-04-28 (post-launch) — llama.cpp Q3_K_XL + Docker compose + stress-test findings + VRAM diagram

- **First measured TPS for UD-Q3_K_XL on this stack:** 21.22 narr / 20.79 code @ 262K context + vision (single 3090, q4_0 KV). VRAM 20.17 GB / 24 GB at boot. Lower than memory's 28.5 baseline (Q4_K_M, 2026-04-23 on llama.cpp commit `9ab47e7d8`) — investigating mainline regression vs current `0d0764dfd`. ngram-mod path measured at 22.04 / 26.11 (+25% on code, draftless via `--spec-type ngram-mod`).
- **llama.cpp Docker compose** at `models/qwen3.6-27b/llama-cpp/compose/`:
  - `docker-compose.yml` — single slot, 262K ctx, q4_0 KV, vision via mmproj. Uses `ghcr.io/ggml-org/llama.cpp:server-cuda`.
  - `single/concurrent.yml` — 4 parallel slots, 192K ctx pool, vision. Multi-tenant variant.
- **All three llama.cpp configs pass verify-full + verify-stress** on this stack. Crucial finding: llama.cpp R1 (Q4_K_M @ 262K + q4_0 KV), Q3_K_XL @ 262K + vision, and Q4_K_M + ngram-mod @ 32K all clear the 90K needle ladder + 25K tool-prefill checks. **No Cliff 1, no Cliff 2** — the prefill OOMs that bite vLLM single-card 192K configs don't fire in llama.cpp on this model. Trade is the ~2-3× lower TPS (21 vs 51-55 vLLM). Reframes our launch positioning around "vLLM dual = max throughput, llama.cpp single = max robustness." Single feature gap: llama.cpp doesn't peel `<think>` into `reasoning_content` (parser issue, not model). Tool calling, streaming, vision, output quality all clean on `--jinja`.
- **`models/qwen3.6-27b/README.md`** — added "VRAM allocation across configs" section with embedded `docs/img/vram-budget-dual.svg`. Per-card stacked bars across 7 configs (3 single, 4 dual) showing weights / KV / vision / DFlash draft / activations / free headroom on the 24 GB budget. Visualizes the TP=2 unlock concretely.
- **`models/qwen3.6-27b/llama-cpp/README.md`** — quant table updated. UD-Q3_K_XL marked ⭐ as our default with citation to Benjamin Marie's [Kaitchup Q3.6-27B GGUF eval](https://kaitchup.substack.com/p/summary-of-qwen36-gguf-evals-updating) — independent H100-validated pick of Q3_K_XL as the optimal accuracy/efficiency/footprint balance, complementary to our 3090 speed measurements.

## 2026-04-28 — Split verify-full.sh → verify-full.sh (fast) + verify-stress.sh (boundary)

Recent additions to `verify-full.sh` (#8 tool-prefill OOM, #9 cascade detection, #10 MTP AL) made the script slow — the longctx needle ladder (#7) alone could run 5+ min, and the full 10-check suite was approaching 10 min. Awkward for "is the stack functional" iteration during dev work.

Split into two scripts:

- **`verify-full.sh`** — fast functional smoke, 8 checks, ~1-2 min. Contains: server reachability, Genesis patches applied, basic completion (Paris), tool calling, streaming, thinking mode, output quality / cascade detection, MTP acceptance length. **Run after every config change** to confirm the stack still serves cleanly.

- **`verify-stress.sh`** — boundary-case stress test, 2 checks, ~5-10 min. Contains: long-context needle ladder (4 depths up to 90K tokens) + tool-response prefill OOM (~25K-token mock tool message). **Run before publishing or when investigating prefill-OOM regressions** specifically.

Same env-var conventions (URL, MODEL, CONTAINER, SKIP_LONGCTX, SKIP_TOOL_PREFILL, PREFILL_TARGET_CHARS). Both pass on the new club-3090 default + dual.yml + dual-turbo.

## 2026-04-28 — Dual-card re-bench on club-3090 substrate (revised TPS numbers)

The published dual-card TPS numbers were measured pre-v714 formalization (April 24-25 timeframe), on a different vLLM nightly + Genesis tree. Re-benched all 4 dual composes on the club-3090 unified substrate (dev205 + Genesis v7.51-stable + Marlin pad fork mounted) to reconcile.

Also: caught a stale mount path in `dual-turbo.yml` — predecessor mounted `patch_tolist_cudagraph.py` from `../patches/genesis/` (where it lived in the old qwen36-dual-3090 layout); club-3090 has it at `../patches/` (top-level). Fixed before measurement; container booted clean. All other composes already had correct paths.

**Measured numbers (3 warmup + 5 measured per prompt arm, narr 1000 tok + code 800 tok):**

| Compose | Narr TPS (CV) | Code TPS (CV) | TTFT | MTP/DFlash AL | VRAM/card | Was claimed | Δ% |
|---|---|---|---|---|---|---|---|
| dual.yml | 69.05 (2.3%) | 88.58 (3.4%) | 145ms | 3.38-3.48 | 23.6 GB | 71/89 | -3% / -1% |
| dual-turbo.yml (now TQ3) | 53.65 (2.7%) | 72.93 (2.7%) | 113ms | 3.41-3.42 | 24.1 GB | 58/69 (k8v4) | -8% / +6% |
| dual-dflash.yml | 81.94 (4.3%) | 124.93 (5.8%) | 138ms | 4.10-4.35 | 23.6 GB | 78/128 | +5% / -2% |
| dual-dflash-noviz.yml | 78.19 (2.5%) | 126.99 (2.2%) | 143ms | 4.24-4.37 | 23.8 GB | 77/124 | +2% / +2% |

Net: most numbers within run-to-run variance. The dual.yml fp8 path is essentially unchanged. dual-turbo's TQ3 swap (from k8v4) cost ~8% narrative but recovered ~6% code — net trade for ~9× the KV pool capacity.

All 4 composes pass `verify-full.sh` functional checks (skipped longctx ladder on the DFlash variants for time; fp8 + MTP variants pass full 10/10 including the 90K-token needle). Updated all docs (README compose table, USE_CASES.md, dual.yml header, dual-turbo.yml header) with the measured numbers.

## 2026-04-28 — Add long-vision + long-text composes (R3' / R3''' from formal v714 round)

Previously the 192K and 205K opt-in tiers were documented as "edit max-model-len + mem-util in docker-compose.yml" — fragile for reproducibility against published bench numbers. Promoted both to dedicated compose files:

- **`single/long-vision.yml`** — TQ3 + Genesis P65 + MTP n=3 + 192K + 0.98 mem-util + vision tower active. Matches R3' bench row (50.93 narr / 67.69 code TPS, AL 3.40-3.58 80-86% accept). Container name: `vllm-qwen36-27b-long-vision`. Same prefill caveats as edit-the-default did.
- **`single/long-text.yml`** — Same config + `--language-model-only` + max-model-len 205K. Matches R3''' (50.11 narr / 65.84 code TPS). Container name: `vllm-qwen36-27b-long-text`.

Trade-off: 2 more compose files (now 11 vs 9). Net: every published bench row from the v714 formalization round (R2, R3, R3', R3''', R4, R6, R7) now boots cleanly with one `-f` flag — no error-prone editing for users who want to reproduce. R1 (eager) and R5 (longctx) stay deleted (obsolete, not niche).

Header references updated: model README compose table, USE_CASES.md frontier-context section, default's variant matrix, vllm/README.md "Pick a compose" code block.

## 2026-04-28 — Repo migration to club-3090

Configs migrated from the predecessor repos (`qwen36-27b-single-3090`, `qwen36-dual-3090`) into this repo's `models/qwen3.6-27b/vllm/compose/` directory. File renames:

| Old path | New path |
|---|---|
| `qwen36-27b-single-3090/compose/docker-compose.yml` | `models/qwen3.6-27b/vllm/compose/docker-compose.yml` |
| `qwen36-27b-single-3090/compose/docker-compose.fast-chat.yml` | `models/qwen3.6-27b/vllm/compose/docker-compose.fast-chat.yml` |
| `qwen36-27b-single-3090/compose/single/tools-text.yml` | `models/qwen3.6-27b/vllm/compose/single/autoround-int4/tools-text.yml` |
| `qwen36-27b-single-3090/compose/docker-compose.no-genesis-mtp.yml` | `models/qwen3.6-27b/vllm/compose/docker-compose.no-genesis-mtp.yml` |
| `qwen36-27b-single-3090/compose/single/minimal.yml` | `models/qwen3.6-27b/vllm/compose/single/autoround-int4/minimal.yml` |
| `qwen36-dual-3090/compose/docker-compose.yml` | `models/qwen3.6-27b/vllm/compose/dual/autoround-int4/fp8-mtp.yml` |
| `qwen36-dual-3090/compose/docker-compose.turbo.yml` | `models/qwen3.6-27b/vllm/compose/dual/autoround-int4/turbo.yml` |
| `qwen36-dual-3090/compose/docker-compose.dflash.yml` | `models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash.yml` |
| `qwen36-dual-3090/compose/docker-compose.dflash-noviz.yml` | `models/qwen3.6-27b/vllm/compose/dual/autoround-int4/dflash-noviz.yml` |
| `qwen36-27b-single-3090/patches/patch_tolist_cudagraph.py` | `models/qwen3.6-27b/vllm/patches/patch_tolist_cudagraph.py` |

Functional content identical — only paths changed. Anyone with scripts referencing the old paths needs to update; the old repos still serve the old paths read-only.

## 2026-04-28 — Compose rename: v7.14 is the zero-arg vLLM default (single-card)

**Breaking change** at the time (mitigated by being on a small-audience repo).

- `docker-compose.v714.yml` → **`docker-compose.yml`**. Running `docker compose up -d` (with no `-f` flag) now boots the production-safe TQ3 + Genesis v7.14 + MTP n=3 + 48K + 0.92 config.
- The previous zero-arg default (fp8 + MTP n=3 + 20K) → **`docker-compose.fast-chat.yml`**. Pick this one when you only need ≤20K context and want the maximum-TPS chat path (~5-7% faster than the new default).
- `docker-compose.longctx-experimental.yml` → **deleted**. Superseded by the default's opt-in 128K + 0.95 tier.

## 2026-04-28 — Prefill-OOM tests + safe v714 default

Triggered by ampersandru's production OOM report ([noonghunna/qwen36-27b-single-3090#1](https://github.com/noonghunna/qwen36-27b-single-3090/issues/1)) — a Hermes-class agent fetching ~25K tokens of news as a tool reply at 192K context crashed the engine.

**Discovered two distinct activation-memory cliffs** on this hardware:
- **Cliff 1** — TurboQuant attention scratch + tool-response prefill, fires on ≥25K-token tool messages at high `--gpu-memory-utilization`. OOM site: TurboQuant forward (dequant scratch + mid_o/output buffers), ~138 MiB allocate.
- **Cliff 2** — DeltaNet/GLA recurrent state buffer, fires on any single prompt above ~50-60K tokens regardless of mem-util. OOM site: `fla.ops.chunk.chunk_gated_delta_rule_fwd_h.h.new_empty(...)`. NT grows linearly with prompt length; chunked-prefill doesn't help.

**Shipped:**
- `verify-full.sh` extended from 7 → 10 checks: #8 tool-response prefill OOM, #9 output quality / cascade detection, #10 MTP acceptance length threshold.
- `verify-full.sh #7` long-context needle ladder treats engine HTTP 400 (oversize ctx rejection) as a clean "skipped at this depth" rather than a failure.
- vLLM single-card default lowered to **48K + 0.92** — below both cliffs. All 10 checks pass.
- README/docs document the full opt-in matrix (64K → 205K) with safe single-prompt + tool-prefill envelopes per tier.
- Three-layer defense documented: vLLM `--max-model-len` HTTP 400 rejection + agent-framework truncation + system-prompt limits.

TPS unchanged at the new default: 51 narr / 68 code TPS (CV ~2.3%). Hardware-bound.

## 2026-04-28 — Inherited prefill-OOM tests on dual-card

The dual-card stack adopted the new `verify-full.sh` checks for safety even though TP=2 + fp8 KV (the dual-card default) gives much wider safety margins than single-card TQ3 KV — the cliffs are not active failure modes on dual hardware.

All compose variants pinned to `vllm/vllm-openai:nightly-07351e0883470724dd5a7e9730ed10e01fc99d08` (= vLLM `dev205+g07351e088`). Previously some tracked `:nightly` and drifted with upstream.

## 2026-04-27 — Full-matrix re-bench + substrate unification (single-card)

Discovered and fixed four real compose drift bugs during a complete re-bench cycle:

- **Image split**: composes had drifted across two different vLLM image pins. All six unified to `vllm/vllm-openai:nightly-07351e0883470724dd5a7e9730ed10e01fc99d08`.
- **`eager.yml` config drift**: shipped with `gpu-memory-utilization=0.92` and `max-model-len=131072` while [@ampersandru](https://github.com/ampersandru)'s actual measurement was `0.97` + `125000`. As-shipped failed to boot. Compose deleted entirely.
- **`v714.yml` mount path**: `patch_tolist_cudagraph.py` was mounted from a wrong path. Fixed.
- **Bench harness regression**: `scripts/bench.sh` had silently dropped the code-prompt arm. Restored.
- **Genesis exoneration**: A/B between default + Genesis vs no-Genesis confirmed Genesis is performance-neutral on the fp8+MTP path. Cross-rig confirmed by [u/sudeposutemizligi](https://www.reddit.com/r/LocalLLaMA/) on TP=2 + dev45 + no Genesis (55 narrative / 68 code, same hardware class).

## 2026-04-27 — Removed: docker-compose.eager.yml (single-card)

Originally proposed by [@ampersandru](https://github.com/ampersandru) as a 125K path that bypasses the cudagraph bug class via `--enforce-eager`. Re-bench cycle measured 25.5 narr / 32.3 code — strictly dominated by `longctx-experimental.yml` at the same 125K context (38/50 TPS). Compose removed.

## 2026-04-27 — Patch hardening (single-card)

- `patches/patch_tolist_cudagraph.py` was silently failing on (a) any non-docker setup (hardcoded `dist-packages` path) and (b) any vLLM nightly past the one we initially tested against. Fixed: patcher auto-discovers vLLM via `import vllm` and uses single-line regex anchors. Bug reported by [@3dluvr](https://github.com/3dluvr) in single-3090 #1.

## 2026-04-25 — Genesis v7.14 (Sandermage upstream)

Genesis v7.14 shipped with the **P65** patch root-causing [vllm#40880](https://github.com/vllm-project/vllm/issues/40880) — the silent tool-call cascade bug under MTP × TurboQuant × cudagraph. P65 forces `cudagraph_mode=PIECEWISE` for spec-decode → eager continuation runs the correct branch.

This shipped as a workaround. The proper fix is a custom multi-query Triton kernel (P67) that handles K+1 query against compressed cached KV under cudagraph capture — designed-but-not-implemented as of v7.14.

The dual-card **Turbo variant** (`dual/turbo.yml`) loads Genesis v7.14 with P64/P65/P66/P68/P69 enabled via env vars. ~25% per-stream TPS regression vs fp8 default but **4.59× concurrency at full 262K vs fp8's 2.36×** — aggregate throughput exceeds fp8 above ~3 concurrent streams.

We adjusted two consumer-Ampere knobs vs Sandermage's A5000-class defaults: `gpu-memory-utilization 0.92 → 0.85` and `max-num-batched-tokens 8192 → 4128`. Without these, deep-prefill (60K+) requests OOM on 24 GB cards.

## 2026-04-22 — DFlash N=5 + Qwen3.6-27B (Luce z-lab fork)

[Luce z-lab](https://github.com/luce-spec)'s DFlash spec-decode draft model for Qwen3.6-27B clears verify-full.sh on dual-3090. Single-stream **78 / 128 TPS narr/code** — substantially faster than MTP n=3's 71 / 89.

Two DFlash variants ship in the dual-card path:
- `dual/dflash.yml` — vision + DFlash N=5 + 185K context
- `dual/dflash-noviz.yml` — text-only + DFlash N=5 + 200K context

Required workaround: vllm#40334 (DFlash `combine_hidden_states` dtype mismatch) is open. Compose sets `--dtype bfloat16` to match the draft's training dtype.

## 2026-04-15 — Marlin pad-sub-tile-n (PR #40361 — our patch)

Filed [vllm#40361](https://github.com/vllm-project/vllm/pull/40361) — fixes a crash in vLLM's Marlin INT4 kernel where output features < 64 cause `GPTQ_MARLIN_MIN_THREAD_N (64) > out_features` on TP=2.

Status: **OPEN, MERGEABLE**, awaiting maintainer review. Until it lands, dual-card composes volume-mount our [patched fork](https://github.com/noonghunna/vllm) at `/opt/ai/engines/vllm/primary/`.

## 2026-04-08 — Initial dual-card release

vLLM-based dual-3090 recipe shipping at TP=2 with fp8 KV + MTP n=3, full feature parity with the single-card project plus the Marlin pad workaround. Was its own repo at the time; now lives here.

## Earlier — Initial single-card release

Initial single-card release shipped a `docker-compose.longctx-experimental.yml` at 125K with `cudagraph_mode=NONE` as the long-context option. v7.14 superseded this; deprecated and removed during 2026-04-27 cleanup.