club-3090/docs/UPSTREAM.md

# Upstream tracker

Issues and PRs in upstream repos that affect this stack — what we depend on, what we've filed, what unblocks for us when each lands.

This file is the **single source of truth** for upstream status. When you file or notice an upstream issue / PR / commit relevant to club-3090, add a row here. When status changes (closed, merged, propagated), update it. Don't scatter the same link across multiple docs without coming back here first.

If you're adding a new compose that depends on an unmerged upstream patch (volume-mount of a fork, monkey-patch script), it MUST link to a row in this file so future readers know when the workaround can drop.

---

## How rows work

Each row covers one upstream link with: **title • status • our dependency / impact • workaround (if any)**.

**Status vocabulary:**
- 🟢 **Landed** — merged upstream + propagated to our pinned versions (pin-bump done)
- 🔵 **Merged, awaiting propagation** — merged upstream but our nightly / commit pin hasn't picked it up yet
- 🟡 **Open / in review** — PR open, no merge yet; we depend on it landing
- 🟠 **Open / blocked or stalled** — PR exists but progress stalled
- 🔴 **Open, no PR yet** — issue acknowledged but no fix in progress (us or upstream)
- ⚫ **Workaround locally, no plan to merge** — fixed in our patches, upstream not pursuing
- ✅ **Resolved** — closed and resolved (kept for historical context)
- ❌ **Closed without fix** — closed, won't fix, kept for context

---

## Active follow-ups (next-week revisit queue) 🗓️

Items deferred for review next week (week of 2026-05-10). Audit at the start of that week — most should be either ready to action or have new upstream signal worth re-evaluating.

| Item | Why deferred | Trigger to revisit |
|---|---|---|
| **🟡 [Sandermage/genesis-vllm-patches#22](https://github.com/Sandermage/genesis-vllm-patches/issues/22)** — PN59 streaming-GDN doesn't engage on chunked-prefill (single-card 24 GB Cliff 2b stays open under v7.72.2). Filed 2026-05-05 with reproducer + 4 fix proposals. | Awaiting Sander's review. The cleanest of our 4 proposals is making `has_no_chunk_metadata` rejection optional (env-gated), letting single-seq chunked-prefill take the streaming path. | Sander posts a candidate fix or comment on the issue; or we run a one-line A/B if he requests it. **Until resolved, single-card 24 GB long-context users should run `dual.yml` / `dual-turbo.yml` / `llamacpp/default`.** |
| ~~**Genesis pin bump `2db18df` → `f2147ad`**~~ | ✅ **Done 2026-05-05** — bumped to `7b9fd319` (v7.72.2) on branch `v7.72.2-uplift`. Drops `patch_inputs_embeds_optional.py` (PN35 native), `patch_pn30_dst_shaped_temp_fix.py` (PN30 v7.68), `patch_pn25_genesis_register_fix.py` (PN25), `patch_tolist_cudagraph.py` (P78), `patch_workspace_lock_disable.py` (PN34), `patch_pr40798_workspace.py` (research artifact). | — |
| ~~**Enable P68/P69 across composes**~~ | Superseded by v7.72.1 P68 auto-skip + v7.72.2 PN70 schema-subset filter — both ship default-aware behavior. Closed [#57](https://github.com/noonghunna/club-3090/issues/57) along the way. | — |
| ~~**Rebase + ping vllm#40361**~~ (our Marlin pad-sub-tile-n PR) | ✅ **Closed-superseded 2026-06-12 — NOT rebased.** Upstream merged [#45295](https://github.com/vllm-project/vllm/pull/45295) (mgoin) "Consolidate Marlin thread-tile padding across all dense Marlin paths" — one shared `marlin_padded_nk` mechanism covering WNA16/AWQ/GPTQ + FP4/FP8, which subsumes our per-case W4A16 sub-tile-n pad (issue #40354). Closed our PR in its favor (it also explicitly replaces the per-dtype #43806/#43910 — maintainers chose one general fix). | ✅ **DONE 2026-07-02** — #45295 is native in **v0.24.0**, which `vllm-stable` now pins, so the vendored `vllm-marlin-pad` overlay is retired. De-registered in `patches.yml` (`upstream.status: merged`, `load_bearing_when: []`, `delivery: none`); no live compose mounts it (archive-only). See the #40361 row below. |
| **✅ vLLM stable pin: v0.22.0 → v0.24.0 — BUMPED 2026-06-30** (`vllm-stable` engine) | **Done.** `vllm-stable` `install.spec` bumped `v0.22.0 → vllm/vllm-openai:v0.24.0` (the first release that includes [#45295](https://github.com/vllm-project/vllm/pull/45295)), so the bump *also* retires the `vllm-marlin-pad` overlay (native now). 10 `engine=="vllm-stable"` compose `:-` defaults bumped; `arch_patches.yml` carries `vllm-stable@v0.24.0 loads:true` rows (v0.22.0 kept as history); pin-asserting fixtures updated (gemma asserts left on v0.22.0). **Live-validated on 2× 3090 (caps 370/420 W):** TP=2 device-selection intact (v0.24.0 dropped internal `CUDA_VISIBLE_DEVICES`), MTP accept 3.51, KV 622K/2.37×, verify-full 8/8, verify-stress NIAH→240K, soak-continuous PASS, bench 70/93 ≈ par, quality toolcall 11/15 · instructfollow 15/15. Full suite green (59/60; the 1 = worktree-fixture-absent `test-submit-bench`, green on master). | **Done — follow-ups:** `vllm-gemma-stable` (#40391/#42006) consolidation onto v0.24.0 = separate PR (gated on the gemma KV + parser re-validation); the `vllm-marlin-pad` patches.yml entry is now **de-registered** (✅ 2026-07-02 — `upstream.status: merged`, `load_bearing_when: []`, `delivery: none`; archive-only, no live compose mounts it); streaming tool-calls validated clean on v0.24.0 → the `qwen3coder` deferred-commit sidecar ([#72](https://github.com/noonghunna/club-3090/issues/72)) is droppable (verify on its own). |
| **🟡 beellama — validate sm_89/sm_120; drop our unofficial image when Anbeeld's official lands** | beellama is a **registry engine** (`beellama-local` profile + two composes). **`beellama/dflash` (Qwen3.6-27B) was promoted to the single-card DEFAULT on 2026-05-30** (⚠️ `caveats` + `DEFAULTS[(qwen3.6-27b, beellama, single)]`; it is #1 in `ENGINE_PREFERENCE[single]`, so the resolver now picks it over ik-llama). Basis: code-throughput leader (~100 TPS vs ik 69), slight 8-pack quality edge (107 vs 99 think-off), output-lossless DFlash, 160K validated ceiling. **`beellama/gemma-dflash` (Gemma-4-31B) was likewise promoted to the single-card DEFAULT** (⚠️ `caveats` + `DEFAULTS[(gemma-4-31b, beellama, single)]`) — it is the *only* viable fast single-card Gemma-4 path (vLLM FA-walled at head_dim=512, ik-llama walls ~24K), so Gemma single had no default before; re-point to the no-fork mainline path when [llama.cpp#23398](https://github.com/ggml-org/llama.cpp/pull/23398) (Gemma-4 MTP) merges. Both default to our **unofficial multi-arch image `ghcr.io/noonghunna/beellama-cpp:multiarch-b9459-07ac3ce`** — a club-3090 build of `Anbeeld/beellama.cpp` (MIT; `.devops/cuda.Dockerfile` with `CUDA_DOCKER_ARCH="86;89;120"` + `-DGGML_CUDA_FA_ALL_QUANTS=ON`) covering **sm_86 / sm_89 / sm_120 = RTX 3090 / 4090 / 5090**. **sm_89 / sm_120 are COMPILED but UNVALIDATED** — only sm_86 / 3090 is verified on our rig. ~~Upstream still ships **no official image**~~ (historical — see #239; official CI images ship since PR [Anbeeld#48](https://github.com/Anbeeld/beellama.cpp/pull/48) and the launchers inject them since our #296). **2026-07-04 UPDATE:** a Discord cross-rig report (WSL, 3060 Ti + 4090) failed verify on the pinned v0.3.2-preview digest → reproduced NOT-broken on our 3090s (boot 20s, verify + verify-full all-pass; their case pending failure text — leading suspect is a `CUDA_VISIBLE_DEVICES`/WSL device-selection interaction, the compose pins GPU 0). Root-caused the 50-series gap while investigating: **the CI builds on the Dockerfile-default CUDA 12.4, whose nvcc cannot target sm_120** (no cubin, max PTX `compute_90`) → every official tag (preview AND stable) lacks Blackwell. Filed **[Anbeeld#85](https://github.com/Anbeeld/beellama.cpp/issues/85)** (CUDA_VERSION 12.8.1 + explicit arch list incl `120`). Our multiarch snapshot is **v0.3.0-feature-level and unmaintained** (predates KVarN + the v0.3.1 fixes) — interim 50-series path only; self-build recipe (2 build-args, FA_ALL_QUANTS now hardcoded upstream via #48) verified + documented in `engines/beellama-local.yml` notes. | (1) A 4090 / 5090 user confirms boots + verify (numbers-from-your-rig) → drop the "unvalidated" caveat. (2) ~~Official images land~~ ✅ done (#48 → repointed in #296). (3) **[Anbeeld#85](https://github.com/Anbeeld/beellama.cpp/issues/85) lands (sm_120 via CUDA 12.8)** → retire the noonghunna snapshot + drop the 5090 override notes from the registry/engine docs; the v0.3.1 stable re-pin (#455) inherits it. |
| **🟢 beellama multi-GPU DFlash — dual composes RELEASED `🧪 experimental` on v0.3.0 (Anbeeld's official image, PR #296); promote → `⚠️`/`✅` on a tagged stable release + prose recovery** | Evaluated dual-card (layer-split, 262K) Gemma-4 compose 2026-05-31 (broken on our pinned `07ac3ce`: decode `-1`, accept 0.357, ~24/38 TPS, `--device-draft` `ggml_abort`). **2026-06-01: built the v0.3.0 dev branch (`efe856397`) + validated on 2× 3090 — multi-GPU DFlash is FIXED.** No `decode failed -1` (boot logs `drafter=1 devices; enabling GPU cross ring`); `--spec-draft-device CUDA0` no longer `ggml_abort`s. **Code DFlash up to ~157 TPS dual (≈4.2× the ~37 no-spec baseline), accept 0.33–0.64.** Two residual issues, **neither multi-GPU**: (a) ~~prose acceptance regressed to ~0.07 / net-negative~~ **RETRACTED 2026-06-03** — a careful tok/s re-test (bench.sh narrative, measured no-spec controls, 3 images: `efe856397`/`e0663be`/`63abcd3`) shows DFlash prose **net-positive everywhere**: qwen single **+27%** / dual **+52%**, gemma single **+28–31%**. The same `efe856397` image we logged at ~0.07 AR now reads ~0.32 at the same tok/s → it was an **AR over-read** (adaptive-DM AR is noisy/prompt-dependent — Anbeeld's #288 caution) **+ a wrong no-spec baseline** (real dual-Q8 no-spec = 23.4, not the ~37 we used). New adaptive-DM HEAD `63abcd3` is neutral. See `learnings/qwen3.6-27b.md` + `gemma-4-31b.md` 2026-06-03; (b) **MTP unsupported** — `--spec-type mtp` + `radamanthys-assistant` GGUF fails: `unknown model architecture: gemma4_mtp` (distinct from CLOSED [#36](https://github.com/Anbeeld/beellama.cpp/issues/36) = garbage MTP *output* on Qwen, not arch-load). Built + **published multi-arch image** `ghcr.io/noonghunna/beellama-cpp:multiarch-v0.3.0-efe856397` (same public package as stable `b9459`) + filed **PR [Anbeeld#48](https://github.com/Anbeeld/beellama.cpp/pull/48)** (FA_ALL_QUANTS Dockerfile fix + CUDA→GHCR nightly workflow). **2026-06-01 RELEASE (PR [#296](https://github.com/noonghunna/club-3090/pull/296)):** flipped the dual gemma compose `⏸️`→`🧪 experimental` + shipped 3 new Q8_K_XL duals (`beellama/qwen-mtp-dual` :8064 embedded-MTP, `beellama/qwen-dflash-dual` :8065 @ 262K, `beellama/gemma-q8-dflash-dual` :8066 @ 192K) for community v0.3.0 testing (#288). `BEELLAMA_IMAGE` is now injected centrally from `engines/beellama-local.yml` `install.spec`, repointed at **Anbeeld's OFFICIAL** `ghcr.io/anbeeld/beellama.cpp:server-cuda-v0.3.0-*` commit tag (was our self-hosted snapshot); bump via `scripts/beellama-pin-bump.sh`. **2026-06-12 PIN BUMP: install.spec → `server-cuda-preview-v0.3.2-317c65e27e1e`** (v0.3.2 PREVIEW — commit-suffixed for an immutable pin; adds experimental KVarN KV-compression). Maintainer chose the preview over the v0.3.1 stable for the newer build; validated single-card (q5ks-dflash, GPU0) — boots on the preview image, `verify-full` all-pass (Paris/tool_calls/streaming/thinking), prose coherent, DFlash spec-dec active (acceptance ~0.12–0.17 on short tasks, not collapsed). Composes STAY `🧪` (preview = rolling pre-release, not stable). Findings: [#288](https://github.com/noonghunna/club-3090/discussions/288), [beellama #39](https://github.com/Anbeeld/beellama.cpp/issues/39). | **Anbeeld tags a STABLE release** → promote `🧪`→`⚠️`/`✅`, keep `BEELLAMA_IMAGE` on upstream's image (now done), retire our snapshot fallback. **The first stable tag now exists: `v0.3.1` (`server-cuda-v0.3.1`, non-prerelease — Qwen3 MTP post-norm + CUDA KV-quant fixes).** Repoint install.spec v0.3.2-preview → **v0.3.1 stable** to UN-PARK (#455) once it passes the full gate (verify-stress + bench + soak + quality). (The "prose-recovery" half of this gate was DROPPED 2026-06-03 — DFlash prose is net-positive on tok/s; the regression was a measurement artifact, not real.) |

See the platform-specific tables below for the rows these reference.

---

## Pinned images

What container image each compose pins, why each pin exists, and which pins
are candidates for retirement when their reason resolves. This section answers
"where the engine pins live" and drives the work in
[`NIGHTLY_BUMP_RUNBOOK.md`](./NIGHTLY_BUMP_RUNBOOK.md).

Run `bash scripts/maintenance/list-image-pins.sh` for a live snapshot.

vLLM compose files do not hardcode full nightly image tags. They use:

```yaml
image: ${VLLM_IMAGE:-vllm/vllm-openai:nightly-${VLLM_NIGHTLY_SHA}}
```

`scripts/launch.sh`, `scripts/switch.sh`, and estate boot resolve
`VLLM_NIGHTLY_SHA` from `scripts/lib/profiles/engines/<engine-id>.yml →
install.spec`. `VLLM_IMAGE` is a full-image override for users who want to pin a
specific upstream nightly, or run a current image when a pinned nightly has been
purged (e.g. `VLLM_IMAGE=vllm/vllm-openai:latest`).

| Pin source | Composes using it | Reason for pin | Retirement candidate? |
|---|---|---|---|
| `scripts/lib/profiles/engines/vllm-nightly-mtp.yml` → `vllm/vllm-openai:nightly-01d4d1ad...` (PURGED) | 🗑️ **DEPRECATED 2026-06-05 (#254)** — Genesis/TQ3 composes archived to `compose/_archive/`, zero registry users | Genesis MTP nightly; image 404 on Docker Hub + Genesis path on hold upstream. Retained as the `genesis_equipped` test anchor (`required_genesis:true`). | Do not bump; revive only if Genesis re-anchors on a post-#42521 nightly. |
| `scripts/lib/profiles/engines/vllm-nightly-dflash.yml` → `vllm/vllm-openai:nightly-e47c98ef...` | DFlash vLLM composes | DFlash overlay baseline. | Bump this YAML after DFlash overlay drift is revalidated. |
| `scripts/lib/profiles/engines/vllm-nightly-full.yml` → `vllm/vllm-openai:nightly-e47c98ef...` (PURGED) | 🗑️ **DEPRECATED 2026-06-05 (#254)** — `vllm/dual-int8` archived, zero registry users | Experimental full-overlay nightly; image 404 on Docker Hub. | Do not bump; retained for revival. |
| `scripts/lib/profiles/engines/vllm-stable.yml` → `vllm/vllm-openai:v0.22.0` (overlay-free) | Qwen3-Next vLLM: `vllm/minimal`, `vllm/dual` (27B), `vllm/qwen-35b-a3b-dual` (35B-A3B); generic `dense`; uncurated derived-emission bases | Stock immutable image, **no vendored overlays** — the load-bearing distinction (CONTRACT-5) that lets it be a derived-emission base. Successor to the purged `vllm-nightly-clean` (#254). | Bump `install.spec` only; never the compose literals (drift-guarded by `test-compose-image-drift`). |
| `scripts/lib/profiles/engines/vllm-gemma-stable.yml` → `vllm/vllm-openai:v0.22.0` (overlay-carrying; split off `vllm-stable` in #324) | Gemma 4 31B vLLM: `vllm/gemma-bf16-mtp` (**v0.22.0**, #42006 tool-parser overlay only, 131K) + `vllm/gemma-int8-mtp` (**v0.22.0** as of 2026-05-31, #40391 + #42006 overlays, 262K); `vllm/gemma-mtp-tp1` **DEPRECATED 2026-05-31** (fp8 dead on Ampere) | Both gemma duals now on immutable `v0.22.0`, both carry the #42006 streaming-multi-tool fix (lean diff-apply); `gemma-int8-mtp` ALSO carries the #40391 per-head-KV overlay. **#40391 DID need a real rebase onto v0.22.0** — the old v0.21.0-era full-module copies `ImportError`'d on v0.22.0 (`cannot import get_kv_cache_spec_kind` — v0.22.0 added that symbol; the stale copies lacked it). Rebased = v0.22.0-stock + the ~240-line #40391 delta (2 club-3090 resolutions: hand-merged `attn_utils` `is_mla()` branch onto v0.22.0's refactored `_reshape_kv_cache`; kept `kv_cache_utils`'s `replace` import that the PR removed but v0.22.0 still uses). Re-delivered LEAN (boot-time diff-apply `install_script`, not 7 full-module mounts). Live-validated 2026-05-31: boots, `int8_per_token_head` KV inits, KV pool 447K @ 262K, bench 95.7/125.8, coherent generation. The 2 other overlays int8 used to carry (#41800, #41991) are in stock v0.22.0; #42006 is vendored on both duals (see its row). | Drop the #40391 overlay entirely when [PR #40391](https://github.com/vllm-project/vllm/pull/40391) merges + lands in a release. **UPDATE 2026-07-02 (v0.24.0 consolidation):** the 31b DEFAULT moved to overlay-free **bf16** (`vllm/gemma-31b-dual`, stock v0.24.0, ~224K). On v0.24.0 int8-PTH allocates 262K but SILENTLY craters recall past ~32K — #40391 is NOT native there (both cyankiwi + w4a16 crater identically; the SAME cyankiwi weights recall clean on v0.22.0+#40391). The #40391 + #42006 overlays and `vllm-gemma-stable` v0.22.0 **STAY** — still used by the 26b-a4b-single (`gemma-a4b` copy) + the now-**deprecated** 31b composes (`gemma-int8-mtp` / `gemma-bf16-mtp` / `qat-w4a16`). The 262K int8-PTH 31b path returns overlay-free when #40391 merges. |
| `ghcr.io/ggml-org/llama.cpp:server-cuda` | 2 (Qwen 3.6-27B llama-cpp) | Stable tag, no hash drift on upstream side. No patches mounted. | Not a retirement candidate — drift-free. Capture digest if reproducibility matters. |

**Retirement workflow:** see [`NIGHTLY_BUMP_RUNBOOK.md`](./NIGHTLY_BUMP_RUNBOOK.md).

### Retired pins

| Pin | Former use | Notes |
|---|---|---|
| `vllm/vllm-openai:nightly-01d4d1ad` | Qwen 3.6-27B Genesis baseline | Retired from default profile pins; retained as historical validation context. |

---

## vLLM (`vllm-project/vllm`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| **[#43562](https://github.com/vllm-project/vllm/issues/43562)** (+ [TRT-LLM #10241](https://github.com/NVIDIA/TensorRT-LLM/issues/10241)) — `--kv-cache-dtype nvfp4` unsupported on consumer Blackwell (sm_120/121) | 🟡 Open upstream / **local gate added** | nvfp4 KV forces vLLM's **trtllm-gen FP4 FMHA**, built ONLY for **datacenter Blackwell sm_100/sm_103** (B100/B200/GB200). Consumer Blackwell (sm_120/121 — RTX 5090 / PRO 6000 Blackwell) is a higher cc number but a **different family with no FMHA build** → crashes at boot (`requires sm100f`). NVFP4 *weights* work on consumer; only the KV/attention path doesn't. Empirically hit on two 5090s via the #246 A/B ([disc #571](https://github.com/noonghunna/club-3090/discussions/571), 2026-07-05); our #246 gate had wrongly used a numeric `≥10.0` floor that passed sm_120. | Local: `gates.py` `_ARCH_KERNEL_SM_FAMILY` now allowlists nvfp4 to `{sm_100, sm_103}` (excludes sm_120/121); `arch-ab.sh` nvfp4 arm refuses on consumer Blackwell with the fp8_e4m3 fallback; nvfp4 dropped from the 5090 / PRO-6000-Blackwell hardware-profile KV lists. **Consumer-Blackwell FP4-era KV = `fp8_e4m3`.** Re-test trigger: TRT-LLM #10241 ships the sm_120 FP4 FMHA and a vLLM release picks it up → re-add nvfp4 to the consumer profiles + arch-ab arms. |
| **(to file)** — torch_aot_compile cache key ignores GPU arch / resolved quant kernel → cross-arch cache load crashes (`AttributeError: 'QKVParallelLinear' object has no attribute 'workspace'`) | 🔴 **To file upstream** (mechanism pinned 2026-07-04; draft pending maintainer review) | **Root cause pinned by the stack trace in club-3090 [#548](https://github.com/noonghunna/club-3090/issues/548):** the crash fires inside `torch/_dynamo/aot_compile.py __call__` immediately after `Directly load AOT compilation from path …/torch_aot_compile/61bda05f…` — an AOT graph **compiled on our sm_86 rig** (Marlin-processed FP8 layers → graph bakes `layer.workspace` reads) loading onto **sm_120** (Cutlass-processed layers → no `workspace` attr). vLLM's AOT cache key (`61bda05f…`) is IDENTICAL across the two arches despite different per-layer kernel processing — cross-arch cache hits should be impossible. The trigger on user rigs was **us**: A1's promotion accidentally git-shipped the whole compiled cache (3,667 files), and the compose warm-start mount fed it to every fresh pull (fixed — cache un-tracked, house `.gitignore` pattern applied). | Two independent unblocks: **(a)** delete the mounted `torch_compile` cache contents → fresh AOT compile against the rig's own kernel selection (native Cutlass FP8 on sm_89+ — the preferred path, likely faster than #567's numbers); **(b)** `VLLM_TEST_FORCE_FP8_MARLIN=1` forces layer processing to match the sm_86-baked graph (validated in [#567](https://github.com/noonghunna/club-3090/issues/567), but costs native FP8 GEMMs). Upstream fix = include compute capability / resolved kernel config in the AOT cache key. |
| **[#42609](https://github.com/vllm-project/vllm/issues/42609)** — custom_all_reduce IPC handle fails with `expandable_segments` | 🟡 Open / **local fix active** | `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` crashes vLLM custom all-reduce at graph-buffer registration — `custom_all_reduce.cuh:455 'invalid argument'` (`cudaIpcGetMemHandle` on a `cuMemMap` VA range is not a valid IPC source). Upstream's title scopes the repro to **DP>1 AND TP>1**, but the same crash class fires on **TP-only** startup paths ([#42583](https://github.com/vllm-project/vllm/issues/42583) Qwen3.5+LoRA TP=4, [#45604](https://github.com/vllm-project/vllm/issues/45604) MiniMax TP=4). On our **2-card NVLink / patched-PCIe-P2P** rigs custom AR is ON at TP=2, so the `expandable_segments:True` that dual composes inject **for the PCIe (custom-AR-OFF) path** crashes boot when P2P is enabled. Earlier signal: JusefPol's NVLink boot crash (club-3090 PR #31). | [`scripts/detect_nvlink.sh`](../scripts/detect_nvlink.sh) strips **only** the `expandable_segments` token from `PYTORCH_CUDA_ALLOC_CONF` on the P2P / custom-AR path (other knobs like `max_split_size_mb` preserved); the PCIe path keeps it. Guard: `scripts/tests/test-detect-nvlink-alloc-conf.sh`. Drop when [#43923](https://github.com/vllm-project/vllm/pull/43923) (the open in-engine fix targeting `custom_allreduce` + `expandable_segments`) merges and lands in our pin. NB: [#40812](https://github.com/vllm-project/vllm/pull/40812) (merged v0.20.1, already in our v0.22.0 pin) only auto-disables `expandable_segments` around the **cumem sleep-mode pool**, not the `custom_all_reduce` path — it does **not** cover this crash. |
| **[#45163](https://github.com/vllm-project/vllm/pull/45163)** — DiffusionGemma (vLLM's first discrete-diffusion LM) | 🟢 **#45163 MERGED (in v0.24.0) — on stock v0.24.0 + 3 vendored Ampere/TP fixes** | The DiffusionGemma 26B-A4B arch (`DiffusionGemmaForBlockDiffusion`, V2 model runner) is in **no** released/nightly vLLM and only in this unmerged draft — BUT vLLM publishes an official **`vllm/vllm-openai:gemma`** image (a stock build of the dgemma branch commit `74b5964f`) that has it baked in. Needed to serve `vllm/diffusiongemma-dual` (the dLLM studio model). | Engine `vllm-diffusion-gemma` **pins `vllm/vllm-openai:gemma` BY DIGEST** (`sha256:9c719fc0…`, purge-resistant) + bind-mounts **3 fix files** that are NOT upstream (vLLM tests H100/B200 + TP=1): [`models/diffusiongemma-26b-a4b/vllm/patches/gemma-image-fixes/`](../models/diffusiongemma-26b-a4b/vllm/patches/gemma-image-fixes/README.md) — `marlin.py`+`marlin_utils_fp8.py` (sm_86 fp8 Marlin sub-tile-K pad; `:gemma` clean dies in warmup on the `K=352/1056` wall) + `diffusion_gemma.py` (TP-vocab soft-embed + dtype fix). Status 🧪 experimental (no DEFAULTS row; launch `--force`) — soak-continuous PASS (0-growth, 0/25 silent-empty) + 8-pack 100/150, but held at experimental because the arch is in *no release* (unmerged + mutable `:gemma` tag) and the dLLM SSE/streaming-NIAH checks can't cleanly gate. **Supersedes the former 123-file sideload-on-nightly** (PR #358 → simplified here: same perf, 120 fewer files). **Re-pin trigger**: vLLM re-pushes `:gemma` → re-pin the digest + rebase the 3 fixes. **Promote trigger (decision 2026-06-11)**: when #45163 merges into a pinnable release → re-pin to it, re-run the gate, flip 🧪 → ⚠️ caveats (documented caveats: long-ctx exact-recall degrades past ~30K, block-streaming SSE, dual-only) — see club-3090 task. **Drop trigger**: the K-pad lands upstream (our #40361 / an Ampere Marlin fix) AND the TP-vocab fix merges into `:gemma` → mount nothing. **(2026-06-12: the K-pad half landed on vLLM main via [#45295](https://github.com/vllm-project/vllm/pull/45295) — general `marlin_padded_nk`; our #40361 closed-superseded. Drop still gated on a stock image that includes #45295 + the TP-vocab merge.)** **UPDATE 2026-07-02:** #45163 **MERGED** (shipped in v0.24.0) → engine `vllm-diffusion-gemma` bumped off the `:gemma` branch digest to stock **`vllm/vllm-openai:v0.24.0`** (arch native). The 3 Ampere/TP fixes STAY vendored + re-validated on stock v0.24.0 (boot + coherent TP=2). #45295's marlin-pad (native in v0.24.0) is the *dense/linear* path — it does NOT cover DiffusionGemma's `marlin_moe_wna16` sm_86 fp8 K-pad (native-no-mounts test still crashed there), so `marlin.py`+`marlin_utils_fp8.py` remain required; native `diffusion_gemma.py` is still TP=1 → the TP-vocab fix remains required. Compose also gained the `--model` flag fix (v0.24.0 positional-arg change). Stays 🧪 (still 3 vendored fixes + no soak on v0.24.0). |
| **club-3090 #451 / #250 / #167 — Gemma 4 31B vLLM nightly prune + stable repoint** | 🟢 Local cleanup in progress | Gemma 31B vLLM composes referenced purged Docker Hub nightlies (`bf610c2f`, `e47c98ef`), recreating the same fresh-user failure class as #250/#167. | Active set pruned from 9 to 3 and repointed to immutable `vllm/vllm-openai:v0.21.0` via `vllm-gemma-stable`; Qwen remains on its separate nightly/Genesis profiles. **Live-check DONE 2026-05-31 on stock v0.22.0: `vllm/gemma-mtp-tp1` DEPRECATED** — fp8 KV is hardware-impossible for Gemma 4 on Ampere sm_86 (gemma4 attention asserts `kv_cache_dtype ∈ {fp8, fp8_e4m3, nvfp4}` → rejects `fp8_e5m2`; `fp8`/`fp8_e4m3` need the `fp8e4nv` kernel sm_86 lacks; `nvfp4` Blackwell-only). bf16 boots but is dominated by the `beellama/gemma-dflash` single-card default. Single-card Gemma → beellama; `vllm/gemma-bf16-mtp` (dual) bumped to v0.22.0, validated. |
| [#35936](https://github.com/vllm-project/vllm/pull/35936) — `tool_choice="required"` falls back to configured tool parser | 🟡 Open / **local overlay active** | Qwen3-Coder with `--tool-call-parser qwen3_coder` emits XML-style tool calls. On pinned nightly `1acd67a79`, non-streaming `tool_choice="required"` validates JSON only, bypasses the configured parser, and returns `tool_calls=[]`. MLS-Bench hits this when `thinking.enabled=false`. | Vendored overlay: [`models/qwen3.6-27b/vllm/patches/vllm-pr35936-required-fallback/README.md`](../models/qwen3.6-27b/vllm/patches/vllm-pr35936-required-fallback/README.md). Drop when #35936 or equivalent lands in our pinned image. |
| **[#41800](https://github.com/vllm-project/vllm/pull/41800)** — `truncate_prompt_tokens` kwarg on `get_max_tokens()` | ✅ **Merged upstream 2026-05-06 at `d5b31c95`** / **local overlay active on pre-fix engine pins** | opencode (and other agentic clients sending `truncate_prompt_tokens`) fail with HTTP 400 `get_max_tokens() got an unexpected keyword argument` on engines pinned to `01d4d1ad` (Genesis MTP), `e47c98ef` (DFlash, full). All three SHAs predate `d5b31c95`. `vllm-nightly-clean` (`bf610c2f`, post-fix) doesn't need the overlay. Tracking issue: club-3090 #139. Triggered by club-3090 #138 (SEVENID's opencode failure). | Vendored overlay: [`models/qwen3.6-27b/vllm/patches/vllm-pr41800-truncate-prompt-tokens/README.md`](../models/qwen3.6-27b/vllm/patches/vllm-pr41800-truncate-prompt-tokens/README.md). Wired into 18 affected composes (every compose routing through `vllm-nightly-(mtp\|dflash\|full)`). Install script has upstream-fix detection: no-ops cleanly when run against a post-`d5b31c95` nightly. **Drop trigger per engine**: bump each affected engine's pin past `d5b31c95`. For `vllm-nightly-mtp` that requires Genesis v7.73.x; for `vllm-nightly-dflash` and `vllm-nightly-full` it requires re-validating PR #41703 and PR #42102 overlays on a newer base. |
| **[#42006](https://github.com/vllm-project/vllm/pull/42006)** — Gemma 4 MTP streaming multi-tool calls | 🟡 **Open upstream / VENDORED on BOTH gemma duals (v0.22.0, lean diff-apply)** | Briefly dropped 2026-05-31 on a "minimal surface" call, then **RE-INSTATED the same day once the bug was reproduced live** on stock v0.22.0: a *streamed* response with 2+ tool calls drops the `arguments` of every call except the last (`get_weather(Tokyo)`+`get_weather(London)`+`get_time(Paris)` → `get_weather()`/`get_weather()`/`get_time({"city":"Paris"})` — only the last kept args). **Non-streaming returns all three correctly** → confirmed streaming-parser bug. #41991 (formerly stacked) is now in stock v0.22.0 → dropped; only #42006 remains. | Vendored as a single-file boot-time diff-apply: [`models/gemma-4-31b/vllm/patches/vllm-pr42006-v0.22.0/`](../models/gemma-4-31b/vllm/patches/vllm-pr42006-v0.22.0/README.md), carried by **both** `gemma-bf16-mtp` + `gemma-int8-mtp` (re-introduces the only overlay on bf16-mtp). Drop when #42006 merges + lands in the pin: `gh api repos/vllm-project/vllm/pulls/42006 --jq '.state, .merged_at'`. |
| **[#45413](https://github.com/vllm-project/vllm/pull/45413)** — ParserEngine core (declarative streaming parser) | ✅ **Merged 2026-06-15 / NOT in v0.23.0** (tagged before merge) | Declarative, config-driven streaming parser engine replacing hand-rolled per-model logic with a shared state machine. Handles reasoning/content/tool-call transitions, incremental JSON argument extraction, and detokenizer hold-back recovery via a single `ParserEngineConfig`. **Potential to eliminate our vendored qwen3coder deferred-commit sidecar** ([#72](https://github.com/noonghunna/club-3090/issues/72)) and the Gemma4 parser overlays — but likely does NOT cover the MTP spec-decode-streaming path ([#39598](https://github.com/vllm-project/vllm/pull/39598)). | **Wait for v0.24.0** (first release to include it). **Adoption plan**: boot new image on a test branch → run `stream-toolcall-probe.py` for Qwen3.6-27B (check #39598 residual) → run DiffusionGemma streaming tool replay (check #443 residual) → if clean, drop vendored overlays and update composes + patches.yml; if broken, keep overlays and file upstream issue. Flagged by @troymroberts in [discussion #451](https://github.com/noonghunna/club-3090/discussions/451). |
| **[#45588](https://github.com/vllm-project/vllm/pull/45588)** — Gemma4 → ParserEngine migration | ✅ **Merged 2026-06-15 / NOT in v0.23.0** (tagged before merge) | Migrates Gemma4 (incl. DiffusionGemma) reasoning + tool parsers to the new ParserEngine framework. Replaces the hand-coded `gemma4_reasoning_parser.py` + `gemma4_tool_parser.py` with a single declarative `gemma4.py` state machine. **steamEngineer tested on pinned `:gemma`** (PR #443 ablation): non-streaming tool_calls OK; **streaming leaks raw `<\|tool_call>…<tool_call\|>` tokens into content** under Hermes (block-canvas SSE chunking). Hybrid (engine reasoning + legacy tool mount) also broken. | **Keep legacy `gemma4_*_parser.py` overlays** until upstream fixes block-canvas streaming tool extraction. See [`models/diffusiongemma-26b-a4b/vllm/patches/gemma-image-fixes/patch_docs/diffusionGemma_streaming_tool_regression.md`](../models/diffusiongemma-26b-a4b/vllm/patches/gemma-image-fixes/patch_docs/diffusionGemma_streaming_tool_regression.md) for full ablation. Experiment preserved on fork branch `experiment/dgemma-parser-engine-45588`. **Adopt trigger**: lands in a release + streaming tool extraction clean on DiffusionGemma block-canvas. |
| [#40361](https://github.com/vllm-project/vllm/pull/40361) — Marlin pad-sub-tile-n | 🟣 **Closed 2026-06-12 — superseded by [#45295](https://github.com/vllm-project/vllm/pull/45295)** (mgoin, merged 2026-06-12: general `marlin_padded_nk` tile-pad across all dense Marlin paths incl. WNA16/AWQ/GPTQ — subsumes our per-case W4A16 sub-tile-n pad; explicitly the consolidated alternative to per-dtype #43806/#43910). | ✅ **#45295 native in v0.24.0** — `vllm-stable` pins v0.24.0, so the tile-pad is stock. **No live compose mounts the vendored patch** (archive-only; the earlier `dual/awq-bf16-int4/int8` / `multi4/fp8/mtp` / `nex-n2-mini` are all under `_archive/`). Patch **de-registered** 2026-07-02 (`patches.yml`: `upstream.status: merged`, `load_bearing_when: []`, `delivery: none`). | Vendored mount: see [`models/qwen3.6-27b/vllm/patches/vllm-marlin-pad/README.md`](../models/qwen3.6-27b/vllm/patches/vllm-marlin-pad/README.md). |
| [#40807](https://github.com/vllm-project/vllm/issues/40807) — `.tolist()` cudagraph crash on continuation-prefill | ✅ **Retired locally** (2026-05-05 Genesis v7.72.2 bump) — Genesis ships [P78 `TOLIST_CAPTURE_GUARD`](../models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-mtp-genesis.yml) as the equivalent fix. Currently disabled (`=0`) on `tq3-mtp-genesis.yml` after rebench-full leg 6 (2026-05-11) passed clean with it off — apparent root cause is now covered by Genesis PN34 (workspace-lock relax) + post-#41434 attention rework. Non-Genesis composes on `1acd67a79` pin run without any guard for this bug; unvalidated at long-context TurboQuant chunked-prefill (worth testing per cferra's vllm#41403 validation pass — see vllm#40798 row below). | None active. Drop the Genesis env var permanently if a future v7.73.x rebench leaves it OFF without regression. |
| [#40798](https://github.com/vllm-project/vllm/pull/40798) + [#42215](https://github.com/vllm-project/vllm/pull/42215) — share decode scratch workspace pre-CUDA-graph + decode-kernel warmup | 🟡 Open, validated cross-rig | Pair closes the `AssertionError: Workspace is locked but allocation requires NMB` crash that fires at `turboquant_attn.py:_continuation_prefill` for ≥48K-token chunked-prefill with TurboQuant KV. Independently validated on 2× 3090 sm_86 by cferra (vllm#41403 [comment](https://github.com/vllm-project/vllm/issues/41403#issuecomment-4435164709), 2026-05-12). | Genesis [PN34 `WORKSPACE_LOCK_RELAX`](../models/qwen3.6-27b/vllm/compose/dual/autoround-int4/tq3-mtp-genesis.yml) addresses the same symptom via a different mechanism (relax-lock vs reserve-before-capture). On non-Genesis composes (`1acd67a79` pin) we currently have no guard — re-validate against this PR pair once they propagate to a nightly we pin to, then A/B PN34 vs upstream. |
| [#40849](https://github.com/vllm-project/vllm/pull/40849) — MTP draft online-quant propagation | 🟡 Open / Genesis backport active | Closes Cliff 1 on FP8+MTP path (`tools-text.yml`). | Genesis PN8 backport: `GENESIS_ENABLE_PN8_MTP_DRAFT_ONLINE_QUANT=1`. |
| [#40914](https://github.com/vllm-project/vllm/pull/40914) — Sandermage K+1 verify routing | 🟡 Open, ❌ negative on our Qwen3.6-27B stack | **Reframed 2026-05-11:** the synthetic `seq_lens` K+1 route is not the P67-equivalent we need here. Local rebase on post-#41434 nightly made MTP acceptance look perfect (AL=4.0 / ~100%) but produced `!`-flood needle corruption plus tool/multi-turn timeouts. Dropping it improved verify-stress from 3/7 to 5/7, but TQ3/TQ4/k8v4 + MTP still fail long-context needles. | Do not ship Genesis-free TQ+MTP on #40914 alone. Use `dual/autoround-int4/tq3-nomtp.yml` without Genesis, or `dual/autoround-int4/tq3-mtp-genesis.yml` with Genesis P67/P67b. |
| [#40334](https://github.com/vllm-project/vllm/pull/40334) — DFlash `combine_hidden_states` dtype mismatch | 🟡 Open | All `dual-dflash*.yml` need `--dtype bfloat16` flag to work around. | Composes set `--dtype bfloat16`. Drop when this lands. |
| [#40382](https://github.com/vllm-project/vllm/issues/40382) — Gemma-4 + DFlash unservable on Ampere | 🟠 Open, no fix in progress | Blocks DFlash on Gemma-4 family. Not directly our problem (we serve Qwen3.6) but tracked because future model adds may hit it. | None — different attention backend selection. |
| **[#41559](https://github.com/vllm-project/vllm/issues/41559) — DFlash spec-decode incompatible with all KV cache quantization** (seantechco, filed 2026-05-03) | ❌ **OUR FIX PR #42102 CLOSED AS SLOP** by @benchislett on 2026-05-15 (no comment, just `closed-as-slop` label). Issue #41559 still OPEN upstream. | Local fix preserved: three-layer patch (4 files, +333/-35) on branch `dflash-noncausal-kv-quant` (commits `cfb8f711` + `5cb61c60`). (1) `vllm/v1/core/kv_cache_utils.py` partitions DFlash drafter specs into independent KV groups before unify, allocator extended to size isolated tensors by their own page_size; (2) `vllm/model_executor/models/qwen3_dflash.py` overrides drafter cache_dtype to "auto" when engine global is quantized; (3) `vllm/v1/attention/backends/flash_attn.py` FA metadata scheduler uses per-spec dtype when spec's kv_quant_mode is NONE. **Validated locally on dual 3090 Ampere**: Gemma 4 + z-lab DFlash drafter + INT8 PTH KV target boots HEALTHY at 65K, narrative 95.89 / code 168.09 TPS, AL 5.0-5.3 preserved, NIAH PASS at 32K, KV pool 149,345 tokens (4× lift). | **Vendor permanently.** Patch lives at `retired Gemma DFlash INT8 overlay (removed from the active tree in #451; recover from git history if revisited)` and is baked into `vllm-nightly-full` + `vllm-nightly-dflash` EngineProfiles. Re-engagement with upstream NOT recommended (vLLM has hardened anti-AI-PR policy). Watch issue #41559 for any newer maintainer-blessed PR; drop our overlay then. |
| [#40354](https://github.com/vllm-project/vllm/issues/40354) — Marlin TP=2 W4A16 < 64 | 🟢 **Resolved upstream by [#45295](https://github.com/vllm-project/vllm/pull/45295)** (general tile-pad; the <64 WNA16 sub-tile-n case is now padded). Our #40361 closed-superseded. | ✅ Effective in **v0.24.0** (`vllm-stable` pinned 2026-06-30); vendored patch de-registered 2026-07-02. | See #40361 row. |
| [#39931](https://github.com/vllm-project/vllm/issues/39931) — DeltaNet rollback support | 🔴 Open, architectural | Blocks **all** spec-decode (EAGLE / DFlash) on Qwen3-Next family across engines. The reason "speculative decoding doesn't work" on this stack. | Use MTP (no rollback needed) until this lands. |
| [#40124](https://github.com/vllm-project/vllm/issues/40124) — related architectural | 🔴 Open | Pairs with #39931 for DeltaNet rollback. | Same as above. |
| [#40880](https://github.com/vllm-project/vllm/issues/40880) — MTP × TQ × cudagraph cascade | ✅ Closed upstream issue, but not solved by direct upstream vLLM | Genesis P65 removed the CUDA-graph-specific failure mode; P67/P67b is the correctness path for K+1 multi-query TurboQuant attention. Round-4 testing showed `--enforce-eager` alone does not close TQ+MTP needles without P67-equivalent behavior. | Use Genesis P67/P67b or disable MTP on TurboQuant. |
| [#40831](https://github.com/vllm-project/vllm/issues/40831) — TQ × spec-decode corruption | ✅ Closed issue, open upstream gap | Our new matrix reproduces the same class across TQ3, TQ4, and k8v4 under MTP. TQ3 no-MTP passes 7/7, so the bug is the MTP x TurboQuant multi-query path, not precision. | Same as #40880. |
| [#40798](https://github.com/vllm-project/vllm/pull/40798) — workspace-manager refactor | ❌ Negative result | Hypothesized fix for #40831 / #40880; backporting it (Probe 8) didn't resolve the bug. Kept for context — saved future time on the same dead end. | n/a |
| [#40875](https://github.com/vllm-project/vllm/issues/40875) — ngram spec-dec default `prompt_lookup_min=2` corrupts tool-call output (Qwen3-class + structured output) | 🟡 Open / **mitigated** | Upstream-confirmed bug: the ngram (prompt-lookup) default of `2` mangles tool-call args; config-only fix is `prompt_lookup_min=8`. mgabor3141 ([#178](https://github.com/noonghunna/club-3090/issues/178)) independently reports a **likely-distinct** corruption mode under MTP-3 + `--tool-call-parser qwen3_coder` on chained agent calls — that mode, plus whether to flip the vLLM agent-facing parser default to `qwen3_xml`, is tracked separately (vLLM-only; llama.cpp/ik use the native template + `--jinja`, unaffected). | `prompt_lookup_min=8` set in composes that use ngram. |
| [#41142](https://github.com/vllm-project/vllm/pull/41142) — Quentin-M streaming tool-call IndexError | 🟡 Open / Genesis backport active | Closes a streaming tool-call crash on Hermes / similar templates. | Genesis PN11 backport (auto-enabled where REC). |
| [#39056](https://github.com/vllm-project/vllm/issues/39056) — Qwen3-Next tool-calls inside `<think>` reasoning region lost | 🟡 Open / **auto fixed on v0.22.0; required residual = #39598** | Tool-call XML inside `<think>…</think>` swallowed by the reasoning-parser before the tool-parser sees it. | **v0.22.0 status (verified 2026-06-13, club-3090#145):** `tool_choice=auto` + streaming + thinking is **clean** on stock v0.22.0 — and the `qwen3_coder`→`qwen3_xml` swap is a **no-op** (A/B byte-identical, both parsers; club-3090 PR #400 closed on this). The earlier "`qwen3_xml` pivot, less `<think>`-sensitive" workaround is **disproven**. The only residual is **`tool_choice=required`+thinking+streaming, which is MTP-gated → tracked at [#39598](https://github.com/vllm-project/vllm/pull/39598)**. |
| [#39598](https://github.com/vllm-project/vllm/pull/39598) — kotori-yan qwen3coder MTP streaming early-return | 🟡 Open / **UN-MITIGATED on v0.22.0 (P64 retired)** | MTP spec-decode drops **streamed** tool-calls (empty `tool_calls[]`, `finish_reason=stop`) when MTP bundles the call in one delta. Was fixed by Genesis P64 (`GENESIS_ENABLE_P64_QWEN3CODER_MTP_STREAMING=1`); **Genesis retired ([#182](https://github.com/noonghunna/club-3090/issues/182) / #254) → P64 gone → the drop is back on stock v0.22.0** (shipped silently into the dual composes — invisible because benchlocal is non-streaming + verify-full's streaming check has no tools). **Confirmed 2026-06-13** (`scripts/stream-toolcall-probe.py`): MTP n=3 dropped **13/20** `tool_choice=required`+thinking+streaming requests; **no-MTP clean 0/20**; **parser-independent** (qwen3_coder ≡ qwen3_xml, byte-identical drop set). `auto`+streaming is **clean even with MTP**. | **Mitigation:** `tool_choice=auto` (clean w/ MTP) or disable MTP for required+streaming workloads. **Durable fix:** upstream — [#45413](https://github.com/vllm-project/vllm/pull/45413) rewrites the *parser* and likely does NOT cover this *spec-decode-streaming* path; needs a P64-equivalent / MTP-streaming fix. |
| **qwen3coder tool-parser SSE-silence on prose `<tool_call>`** ([upstream #22975 closed-as-stale](https://github.com/vllm-project/vllm/issues/22975); reported on club-3090 as [#72](https://github.com/noonghunna/club-3090/issues/72)) | ⚫ Local workaround / **upstream PR deferred until cross-rig validation lands** | When the model's prose mentions the literal `<tool_call>` text (e.g. agent reasoning that describes the markup), `extract_tool_calls_streaming` flips `is_tool_call_started=True` permanently on either the special-token-id or the string match. Subsequent deltas return `None`; the serving layer skips them; SSE wire goes silent for 30-120s while tokens decode server-side and never reach the client. Verified bug still present in vLLM main as of 2026-05-07 (no deferred-commit guard in current source). Upstream issue #22975 reports a related symptom (`<tool_call>` markup remains as plain content) but was closed-as-stale 90+ days ago without a fix — different observed surface, likely shared root cause. | `models/qwen3.6-27b/vllm/patches/local/qwen3coder_tool_parser_deferred_commit.py` runs after `apply_all` in the entrypoint of all 8 Genesis-equipped composes. Defers `is_tool_call_started=True` until `<function=` confirms within a 64-char slack window past the `<tool_call>` tag. **Direct-cmd composes (`dual.yml`, `dual-dflash*.yml`, `minimal.yml`, `multi4*.yml`, `carnice-bf16mtp.yml`, `qwopus-bf16mtp.yml`) don't currently receive the sidecar** — they have no entrypoint script. Plan: ship local sidecar → validate cross-rig → file upstream PR (with cross-rig evidence and the V2 deferred-commit logic) once the local fix has held up under multi-rig real-world traffic. |
| [#40961](https://github.com/vllm-project/vllm/pull/40961) — Preserve max_seq_len in ubatch metadata during CUDA graph capture | 🟡 Open PR | Confirms the cap-leak pattern: cudagraph capture passes `max_model_len` as `max_seq_len` through ubatch metadata. PR is *fixing a missing pass-through* for SWA models (where seqlen=1 at capture broke kernel selection) — by establishing that `max_model_len` is what gets carried through capture metadata, it cements the source of Cliff 1's max-ctx-dependent FA2 workspace sizing. | Stay at `default` 48K — see FA2 #1011 row + INTERNALS.md Cliff 1 mechanism. |
| [#40069](https://github.com/vllm-project/vllm/issues/40069) — [Tracking] TurboQuant / HIGGS Attention follow-ups | 🟡 Open tracker | Umbrella tracking for TurboQuant + attention backend issues on our stack class. | Watch for cross-references when Cliff 1/2 work lands upstream. |
| [#25543](https://github.com/vllm-project/vllm/pull/25543) — [V0 Deprecation] Remove `max_seq_len_to_capture` | ✅ Merged 2025-09-24 | Important to know: the `--max-seq-len-to-capture` flag (commonly suggested as a Cliff 1 mitigation) **does not exist in V1**. Don't recommend it. | n/a — flag removed. |
| [#39226](https://github.com/vllm-project/vllm/pull/39226) — workspace-resize GPU memory leak fix | 🔵 Merged into v0.20.0; covered by sidecar | Strict `WorkspaceManager.lock()` semantics. After our 2026-05-01 v0.20 + Genesis v7.65 dev tip migration, the surfaces that locked at 0 MB on our config are largely covered by v0.20's revised TQ FA paths ([#40092](https://github.com/vllm-project/vllm/pull/40092)). For the residual cases, our local `patch_workspace_lock_disable.py` sidecar (mounted on every TQ3 compose) downgrades the strict assertion to a one-shot WARNING. P98 covers the same surface but auto-skips on v0.20 due to a drift-marker false-positive (filed as side-note, awaiting Sandermage marker fix). | Drop the sidecar when Sandermage ships the marker fix that re-enables P98 on v0.20. |
| [#40092](https://github.com/vllm-project/vllm/pull/40092) — TurboQuant FA3/FA4 prefill paths | 🔵 Merged into v0.20.0 | TQ + flash-attention 3/4 prefill support. Relevant if/when v0.20 unblocks for us — the FA varlen workspace allocator behavior may change under FA3/FA4 vs the FA2 path we currently hit. | Track. Re-evaluate the `flash_attn_interface.py:300` cliff (Genesis #15) once v0.20 unblocks since FA3/FA4 may have different workspace semantics. FA3/FA4 not enabled on Ampere SM 8.6 anyway (Hopper+ only). |
| [#40941](https://github.com/vllm-project/vllm/pull/40941) — TurboQuant share buffers | 🔵 Merged into v0.20.0 | Sandermage's bare_metal_27b_int4_TQ_k8v4.sh comments call out P98 as the workaround for "WorkspaceManager fix vs vllm#40941". Same WorkspaceManager class that vllm#39226 made strict. | Sandermage's **P98** is the workaround — required for TQ k8v4 on hybrid. Worth a focused investigation: enable P98 on the v0.20-experimental compose to see if it also unblocks vllm#39226's path. |
| [#35975](https://github.com/vllm-project/vllm/pull/35975) — Skip `inputs_embeds` GPU buffer for text-only models ⭐ | 🟡 Open upstream / **local backport active** / **Genesis PN35 lands same fix on dev `f2147ad`** (2026-05-03) | Frees ~444 MiB at boot on Qwen3.6-27B (both `gpu_model_runner.py` + `llm_base_proposer.py` call sites compound; PR claims ~64 MiB but our config has multiple residency points that benefit). **Critical for Cliff 2 closure** at 60K on TP=1 + 24GB — combined with mem-util 0.93, closes the late-stage 50 MiB activation peak. Diagnosed by ChatGPT/Codex as the missing margin; cross-rig validated 2026-05-02 PM. Upstream PR last updated 2026-03-13 (51d stale); Sandermage's PN35 is the practical replacement. | `patch_inputs_embeds_optional.py` ships at compose-entrypoint time (mounted on `long-text.yml` and `long-text-no-mtp.yml`). **Drops out when we bump GENESIS_PIN to dev tip — queued for next-week revisit.** |
| [#37429](https://github.com/vllm-project/vllm/pull/37429) — Hybrid Mamba/attention KV cache sizing | 🟡 Open | Could free more residency without trading mem-util. Larger/riskier than #35975 (architectural Mamba allocation change). Untested on this stack. | Not currently backported. Test on a separate branch when CI signals stabilize. |
| [#37521](https://github.com/vllm-project/vllm/pull/37521) — Spec-decode warmup memory accounting | 🟡 Open | Profiling/KV sizing leaves less false headroom. Genesis PN33 already extends this beyond the original `use_eagle()` gate — so most of the surface is covered, but watch for upstream refinement. | n/a — Genesis PN33 covers the path. |
| [#36598](https://github.com/vllm-project/vllm/issues/36598) — Triton autotuner OOM on Qwen3.5/Qwen3-Next GDN layers (non-SM90 GPUs) | ✅ Closed 2026-03-12, fix shipped via #36599 | Original report of first-inference OOM during Triton autotuning on non-SM90 hardware. Closed because the warmup fix landed. Reading thread is useful context for understanding the GDN kernel autotuner pressure on our hardware class. | n/a — fix in our image. |
| [#36599](https://github.com/vllm-project/vllm/pull/36599) — Warm up Triton autotuner for GDN layers during V1 profiling | ✅ Merged 2026-03-12 (in image SHA `7a1eb8ac`) | Adds `_warmup_triton_kernels()` at V1 profile phase. Warms with B=1, T=64 dummy tensors. Closes the boot-time first-inference autotuner OOM that #36598 reported. **DOES NOT close Cliff 2b** (multi-turn accumulated context) — the warmup uses T=64 but FLA kernels use `do_not_specialize=["T"]` so production T=4128 is the same autotune key, meaning runtime fragmentation isn't from missed autotune; it's from the per-shape Triton kernel binaries staying resident in CUDA context. Confirmed by Codex memo 2026-05-03. | n/a — fix in image; doesn't help our remaining cliff. |
| [#36973](https://github.com/vllm-project/vllm/issues/36973) — `_warmup_prefill_kernels` leaks ~3.4 GiB despite empty_cache | 🟡 Open, RTX 5090-specific | jhsmith409's report — Triton autotuner cubin retention initially suspected but haosdent comment #18-19 traced the bulk to **TMA overhead** scaling with SM count (~22 MiB/SM × 170 SMs on 5090 = 3.7 GiB). Closed via #37700 (TMA-disable for SM12x). **Doesn't apply to Ampere SM86** — no TMA hardware. Useful context though: thread comment #5 explicitly notes Triton autotuner keeps all variants loaded; `empty_cache()` only releases PyTorch's caching allocator, not CUDA-context cubins. | n/a — RTX 3090 doesn't have TMA. |
| [#37700](https://github.com/vllm-project/vllm/pull/37700) — Fix FLA Hopper/TMA misclassification on SM12x desktop Blackwell | 🟡 Open / closes #36973 for SM12x | Uses shared-memory threshold instead of `major >= 9` checks for TMA path selection. SM12x desktop Blackwell only — RTX 5090, DGX Spark GB10. Doesn't apply to Ampere SM86 (no TMA hardware). | n/a — different hardware family. |
| **Cliff 2b — multi-turn accumulated-context OOM (we filed)** | 🟡 Open, [Sandermage genesis-vllm-patches#19](https://github.com/Sandermage/genesis-vllm-patches/issues/19) | DeltaNet `chunk_gated_delta_rule_fwd` holds ~500 MiB of simultaneous live tensors at T=4128. Under multi-turn agent traffic (hermes/openhands/etc.), accumulated KV + this kernel's working set + model + workspace exceeds 24 GiB on 1× 3090. Cliff fires at ~21-26K accumulated context. We tested mem-util tuning, MTP-off, max-num-batched-tokens reduction, TRITON_CACHE_AUTOTUNING, expandable_segments, empty_cache between turns — none close it. Validated 2026-05-03: 6 single-card vLLM variants FAIL v2 continuous soak; only TP=2 / dual.yml passes. Filed with Sandermage proposing streaming refactor of GDN forward intermediates. | **`bash scripts/switch.sh vllm/dual` (TP=2)** for 2× rigs, **`llamacpp/default`** for 1× rigs. See [club-3090#41](https://github.com/noonghunna/club-3090/issues/41) + [docs/CLIFFS.md](CLIFFS.md) "Why TP=2 escapes" / "Why llama.cpp escapes" sections. |
| [#41745](https://github.com/vllm-project/vllm/pull/41745) — Add Gemma4 MTP speculative decoding support (lucianommartins) | 🟢 **Merged 2026-05-06, overlay dropped 2026-05-08** (commit [`595be8f`](https://github.com/noonghunna/club-3090/commit/595be8f)). Today's nightly tag `1acd67a795...` (2026-05-08 06:10 UTC) contains the merge. `dual.yml` + `single.yml` bumped to post-merge nightly; overlay tree `models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/` retained as fallback (drop in follow-up commit once Phase 2 cycle settles). | First-party MTP for Google's Gemma 4 "assistant" drafter family. Validated on this stack 2026-05-05 (with overlay): 109/142 TPS soak PASS. Re-validated 2026-05-08 (overlay dropped, post-merge nightly): **105.91/141.11 TPS** — within CV of the prior baseline → cleanup is parity-clean. | n/a — closed |
| **Gemma 4 + per-token-head KV on Ampere** ([#40388](https://github.com/vllm-project/vllm/issues/40388), [PR #40391](https://github.com/vllm-project/vllm/pull/40391)) | 🟡 **VENDORED + VALIDATED 2026-05-08** ([commit `f93d312`](https://github.com/noonghunna/club-3090/commit/f93d312) + bench [`160e8fc`](https://github.com/noonghunna/club-3090/commit/160e8fc)). Local rebase of PR #40391 onto post-#41745 main resolved the conflict in `vllm/v1/worker/gpu/attn_utils.py` (combined main's hybrid attn/mamba dispatch with PR #40391's MLA-vs-standard-attention split for `page_size_padded`). Vendored as full 7-file overlay. Compose: `dual/autoround-int4/int8.yml`. **Validated dual 3090 Ampere**: 7/7 verify-stress at 98K AND at 262K, plus 137K NIAH recall PASS. Bench: 96/127 TPS at 98K, 95/126 at 262K (~10% TPS cost vs bf16 / 32K for **8.2× context lift**). | Earlier 2026-05-06 Codex investigation memo at [`perheadkv-overlay-comparison.md`](../models/gemma-4-31b/vllm/patches/perheadkv-overlay-comparison.md) erroneously concluded "NOT split-able as an overlay" — that was based on PARTIAL overlays (worker-only or spec-only). A FULL PR #40391 overlay (all 8 files + post-#41745 rebase) works cleanly. Key insight: **INT8 PTH (not FP8 PTH) is the Ampere-target dtype** because Triton `fp8e4nv` kernel is not supported on sm_86 (only `fp8e4b15`/`fp8e5`); FP8 PTH crashes at `_initialize_kv_caches` on Ampere. INT8 PTH dispatches to standard `torch.int8` ops which work on all consumer GPUs. PR #40391's page-size mismatch fix applies to ANY per-token-head KV format — the dtype choice is downstream. Cross-rig validators: cferra (sm_120 Blackwell, FP8 PTH), noonghunna (sm_86 Ampere, INT8 PTH). **Phase 3 (PR #40391 + PR #41703 DFlash drafter combined) BOOT-BLOCKED 2026-05-08** — see also [#41559](https://github.com/vllm-project/vllm/issues/41559) row below for the underlying upstream blocker. 17-file merged overlay parses + compiles, fails `_init_minimal_kv_cache_for_profiling` with `NotImplementedError: page size of the layer is not divisible by the maximum page size` at `kv_cache_utils.py:1068`. **Diagnostic-print at `unify_kv_cache_spec_page_size` (2026-05-08)** reproduced exactly the symptom seantechco described in #41559: drafter silently uses BF16 KV regardless of `--kv-cache-dtype int8_per_token_head`. Three page sizes seen: target Gemma 4 global INT8 PTH 33,280 (padded by PR #40391), target Gemma 4 local INT8 PTH 66,560, DFlash drafter 131,072 (= 16 × 8 × 1024 BF16 K+V at head_dim=256). 131072 / 66560 = 1.97, 131072 / 33280 = 3.94 → no integer ratios → unify rejects. **Phase 3b validation (RedHatAI Gemma-aligned drafter, head_dim=256, num_kv_heads=16)** failed identically — drafter weights/architecture irrelevant; the actual blocker is per #41559: DFlash mandates non-causal cross-attention and every KV-quant backend rejects KV-quant when causal=False. **Why MTP `gemma4_assistant` works but DFlash doesn't**: MTP doesn't require non-causal attention, AND `gemma4_assistant` shares Gemma 4 architecture (same `gemma4.py:438` code path) so PR #40391's INT8 PTH padding propagates uniformly to drafter layers. Phase 3 stacks preserved as forensic artifacts at `retired Gemma DFlash INT8 overlay (removed from the active tree in #451; recover from git history if revisited)`. | Drop overlay when PR #40391 merges to vLLM main + propagates to a nightly tag. Track: `gh api repos/vllm-project/vllm/pulls/40391 --jq '.state, .merged_at'`. Local exploratory artifacts at `models/gemma-4-31b/vllm/patches/{vllm-perheadkv-hybridpage-fix,vllm-pr40391-perheadkv,vllm-gemma4-fp8-ampere}/` (NOT committed; reference for future iterations). **Phase 3 unblock paths**: (a) patch `qwen3_dflash.py:DFlashAttention.get_kv_cache_spec()` to honor `cache_config.cache_dtype` and return `kv_quant_mode=INT8_PER_TOKEN_HEAD` with appropriate `page_size_padded` (single-file fix, most tractable upstream PR); (b) drafter-isolated KV groups extending DFlash's existing `_get_dflash_isolated_group_ids` to skip page-size unify for draft layers; (c) wait for PR #41703 to merge then re-attempt — fresher base may have unrelated KV-cache refactors that change the picture. **As of #451 cleanup**: long-context Gemma 4 on Ampere ships only via MTP (`vllm/gemma-int8-mtp`, 98K default with `CTX=262144 MAX_NUM_SEQS=1` override). DFlash/AWQ vLLM composes were pruned from the active registry and remain recoverable from git history if revisited. |
| **Gemma 4 + native TurboQuant KV on Ampere** ([bug #40094](https://github.com/vllm-project/vllm/issues/40094), [tracking #41403](https://github.com/vllm-project/vllm/issues/41403); fix PRs [#42637](https://github.com/vllm-project/vllm/pull/42637), [#40108](https://github.com/vllm-project/vllm/pull/40108), [#41123](https://github.com/vllm-project/vllm/pull/41123)) | 🟡 Blocked on our pin, **tracked upstream — NOT a permanent / architectural verdict** | **Exact mechanism (empirically captured 2026-05-16, full traceback):** vLLM logs at `config.py:101` — *"Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence."* → `cuda.py:312` uses TRITON_ATTN → `ValueError: Selected backend AttentionBackendEnum.TRITON_ATTN is not valid for this configuration. Reason: ['kv_cache_dtype not supported']`. The force is **unconditional and at config time, *before* backend selection** — that is why `FLASH_ATTN` / `FLASHINFER` / `XFORMERS` / forced-`TURBOQUANT` / boundary-skip-disable are **all moot** (vLLM overrides every backend hint for Gemma-4's heterogeneous head_dim by design, as a numerical-correctness safeguard). Fails *earlier* than #41403 Gate-2 (boundary-skip — N/A here: Gemma-4 is hybrid so `get_boundary_skip_layers` returns `[]` already) and earlier than Gate-5 (page-size unify). Verified on Gemma-4-31B (text AutoRound-int4, Ampere sm_86, Genesis-free, pin `bf610c2f`). Gemma-4-26B-A4B fails even *earlier* at the WNA16/Marlin weight kernel (Genesis-free base lacks the Marlin-pad overlay — see #40361 row); turboquant verdict untested but predicted-same. **No PR enables `kv_cache_dtype` *inside* TRITON_ATTN** — that is the wrong frame; the upstream fixes operate at the **backend-selection / TQ-spec layer** so Gemma-4 is *not* force-pinned to TRITON_ATTN. **Contrast:** Qwen3-Next dense + MoE native turboquant works Genesis-free on the same pin (#39931 merged) — uniform-enough attention ⇒ no forced TRITON_ATTN ⇒ a turboquant-capable backend is used. Qwen3-Next-only capability today. | Gemma 4 stays on Genesis / `fp8_e5m2` KV — no change, no local action; **no runtime kernel/backend/knob lever exists** (closing probe exhausted them). **Re-test trigger** (any of, on merge into a nightly we pin): [#42637](https://github.com/vllm-project/vllm/pull/42637) (open, freshest 2026-05-16 — most precisely targets *this* mechanism: adds `TQSlidingWindowSpec` + **changes backend selection so TurboQuant is selected for heterogeneous-head attention** instead of force-pinning TRITON_ATTN); [#40108](https://github.com/vllm-project/vllm/pull/40108) (open, "ready/verified" — sliding-window + YOCO TurboQuant, different approach); [#41123](https://github.com/vllm-project/vllm/pull/41123) (open — "allow TurboQuant on hybrid models"). Watch: `gh pr view 42637 --repo vllm-project/vllm --json state,mergedAt` (and 40108, 41123). PR #41497 (preserve sliding-window KV specs) closed without merge. Same unblock pattern as #39931 (merged → unblocked Qwen). |
| **Gemma-4-12B unified — advertised 256K p-RoPE OOB past 131072** ([#39914](https://github.com/vllm-project/vllm/issues/39914) p-RoPE large-prefill; arch [PR #44429](https://github.com/vllm-project/vllm/pull/44429) merged 2026-06-03) | ✅ **Resolved upstream 2026-06-04 by a `config.json` fix** — overlay DROPPED. `google/gemma-4-12B-it` (+ the `-assistant` draft) now ship `max_position_embeddings=262144`, so the stock `gemma4_unified` image sizes the RoPE cache correctly and serves the full 256K with no patch. lucianommartins (gemma upstream) confirmed it was a config bug ([comment](https://github.com/vllm-project/vllm/issues/39914#issuecomment-4623154757)); we'd shared the runtime-sizing fix + patch link earlier ([comment](https://github.com/vllm-project/vllm/issues/39914#issuecomment-4622760819)). The `unsloth/gemma-4-12b-it` mirror still lagged at 131072 as of 2026-06-04 → bf16 source repointed unsloth→google. | Root cause: the preview Gemma4 language model sizes the fixed RoPE cos/sin cache from `config.max_position_embeddings` (was 131072) even when serving with `--max-model-len 262144`; large prefill then indexed past the cache (`< 131072` device assertions). The proper fix is the corrected config (262144), not engine code — so the former local `vllm-gemma4-prope-longctx` overlay (which sized the cache from runtime `max_model_len`) is now redundant and was removed. Overlay-free re-validation on dual 3090 TP=2 bf16 (2026-06-04): base + MTP both boot 262144 on the stock image, NIAH exact-recall PASS at 140K/170K/200K/230K/241K, MTP spec-decode active. | Ship `vllm/gemma-12b-dual-bf16-mtp` (+ single-card `vllm/gemma-12b-single-int8-mtp`) at **262144** on the stock image, sourcing bf16 + assistant from `google/` (gated; carries the 262144 config). No overlay. |

---

## Genesis (`Sandermage/genesis-vllm-patches`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| [#22](https://github.com/Sandermage/genesis-vllm-patches/issues/22) — PN59 streaming-GDN never engages on chunked-prefill ⚠️ | 🟡 **Open, filed 2026-05-05 by us** | Genesis v7.72.2 advertises PN59 as the structural Cliff 2b fix on 24 GB single cards, but its eligibility check rejects calls with `chunk_indices`/`chunk_offsets` populated — which vLLM's mandatory `--max-num-batched-tokens 4128` always sets. PN59 falls back to vanilla, OOMs at the same `chunk_o.py:161` site. Single-card 24 GB long-context (`long-text.yml` / `long-text-no-mtp.yml` / `long-vision.yml`) regresses vs the prior workarounds. | **Use `dual.yml` / `dual-turbo.yml` (TP=2)** or `llamacpp/default` (different engine, no Cliff 2b). Reproducer + 4 fix proposals in the issue body; awaiting Sander review. |
| [#5](https://github.com/Sandermage/genesis-vllm-patches/issues/5) — P8 ImportError on vLLM v0.20.0 | ✅ Closed (now on v0.20 pin since 2026-05-01) | Originally about P8 ImportError on the v0.20.0 GA tag. We migrated master to v0.20.1rc1.dev16 + Genesis v7.65 dev tip — P8 path no longer fires on our configs. | n/a — pin already moved. |
| [#6](https://github.com/Sandermage/genesis-vllm-patches/issues/6) — P65 PIECEWISE cost quantified | ✅ Closed | We characterized the +22 TPS narrative cost of P65 on Qwen3.6-27B + MTP. Sandermage acknowledged. Will recover when vllm#40914 lands. | Accept the cost on substrate-current; ampersandru's pre-P65 stack avoids it. |
| [#7](https://github.com/Sandermage/genesis-vllm-patches/issues/7) — P67 Triton CompilationError on Qwen3.6-27B | ✅ Closed | Resolved in v7.64 — P67 generalized to non-power-of-2 GQA via `BLOCK_QH = triton.next_power_of_2(HEADS_PER_KV)` + lane_valid mask. Tool-call 0/5 → 7/7 on 2× A5000 validation. | Now safe to enable on 27B configs with v7.64+. |
| [#9](https://github.com/Sandermage/genesis-vllm-patches/issues/9) — P68/P69 8000-char threshold breaks IDE agents | ✅ Closed 2026-05-01 — fix shipped in v7.65+ (50K-char default); we're on v7.69 so the fix lives in our pin | P68 silently rewrites `tool_choice: auto → required`; P69 injects "must use a tool" hint. New 50K threshold clears typical IDE-agent contexts (Cline ~30K / Cursor ~25K / Copilot ~20-25K) while genuine long-history sessions still trigger the reminder. | Composes still have P68/P69 env vars commented out — **enabling them across composes is queued for next-week revisit** (see "Active follow-ups" table above). Until then, manual override: `GENESIS_ENABLE_P68_AUTO_FORCE_TOOL=1 GENESIS_ENABLE_P69_LONG_CTX_TOOL_REMINDER=1`. |
| [#11](https://github.com/Sandermage/genesis-vllm-patches/issues/11) — Cliff 1 mech A FA2 softmax_lse clamp request | ✅ Closed (PN17 in v7.64; default-on across all TQ3 composes since the v0.20 migration) | Sandermage's PN17 lands the clamp at `flash_attn.py`. Active on every TQ3 compose. | n/a — default-on. |
| **Local P104 FA max_seqlen_k runtime clamp** | ✅ Dropped during v0.20 migration | Built 2026-04-30 as `patch_fa_max_seqlen_clamp.py`. Sandermage's PN17 + P15B together cover both layers (FA wrapper + TQ wrapper). Sidecar removed from compose mounts on 2026-05-01. | n/a — Genesis-native. |
| [PR #12](https://github.com/Sandermage/genesis-vllm-patches/pull/12) — P101 anchor drift fix | ✅ Closed; on v0.20 pin since 2026-05-01 | P101 anchor matches on `0.20.1rc1.dev16+g7a1eb8ac2`. | n/a — pin matches. |
| [PR #13](https://github.com/Sandermage/genesis-vllm-patches/pull/13) — PN12 anchor drift fix | ✅ Closed; on v0.20 pin since 2026-05-01 | PN12 anchors match natively. Local `patch_pn12_ffn_pool_anchor.py` sidecar removed. | n/a — pin matches. |
| [#14](https://github.com/Sandermage/genesis-vllm-patches/issues/14) — P38 silently no-op'd on TurboQuant KV path (we filed 2026-05-01) | ✅ Closed via P38B in Genesis v7.65 dev tip | Sandermage shipped **P38B** — text-patches `turboquant_attn.py` source to inject a delegate hook at the start of `_continuation_prefill` body. Active via `GENESIS_ENABLE_P38B_COMPILE_SAFE=1` on every TQ3 compose since 2026-05-01. | n/a — Genesis-native. |
| [#15](https://github.com/Sandermage/genesis-vllm-patches/issues/15) — FA varlen kernel workspace cliff at flash_attn_interface.py:300 (we filed 2026-05-01) | ✅ Closed via P15B in Genesis v7.65 dev tip | Sandermage shipped **P15B** — direct backport of our suggestion. Active via `GENESIS_ENABLE_P15B_FA_VARLEN_CLAMP=1` on every TQ3 compose since 2026-05-01. Empirically the cliff also doesn't reproduce on v0.20 (vllm#40092 changed workspace allocator behavior) — covered from two directions. | n/a — Genesis-native. |
| [#16](https://github.com/Sandermage/genesis-vllm-patches/issues/16) — PN25 worker-spawn registration | 🟡 Sander shipped d92bcb3 (v7.65) + Library refactor (v7.66); both fail on TP=1 | v7.65 used `@torch.library.custom_op` (failed at `infer_schema` inside dynamo trace). v7.66 refactored to `direct_register_custom_op` + `Library("genesis", "FRAGMENT")` — fails at `instantiate_user_defined_class_object` inside dynamo trace. Same root cause: torch.library construction inside trace context disallowed on TP=1 spawn. Cross-rig data on Sander's [discussion #19 reply](https://github.com/noonghunna/club-3090/discussions/19#discussioncomment-16785590). | Local `patch_pn25_genesis_register_fix.py` v3 — text-patches `activation.py` to register at module-import time, BEFORE any trace. Survives both v7.65 and v7.66 mechanisms. PR-ready upstream. |
| [#17](https://github.com/Sandermage/genesis-vllm-patches/issues/17) — DS conv state spec-decode crash | 🟡 Sander shipped a9977d8 (PN30) but `.contiguous()` is layout-incorrect | Sander's PN30 materializes `state[src, :, offset:].contiguous()` (compact 10240×5) and raw-memcpys into `state[dest]` (strided 10240×6) → corrupts DS row strides → eventual TQ store CUDA assert several layers downstream. Diagnosis credit: ChatGPT/Codex CLI cross-check 2026-05-02. Sent corrected fix to Sander. | Local `patch_pn30_dst_shaped_temp_fix.py` — patches `collect_mamba_copy_meta` to build dst-shaped temp instead of compact. Reuses Sander's `_GENESIS_PN30_TEMP_TENSORS` lifecycle. Validated on all 4 TQ3 composes; probes 4 + 5 pass cleanly. PR-ready upstream. |
| [#15](https://github.com/Sandermage/genesis-vllm-patches/issues/15) — PN31 FA varlen persistent out | 🟡 Sander shipped 753344b (PN31, default OFF); doesn't fit on 24 GB | Per-shape persistent buffer growth + PN12+PN25 pool residence outpaces activation budget at DeltaNet `chunk_fwd_o` on 24 GB single GPU. Sander explicitly flagged he couldn't validate on 24 GB. | Lower mem-util to 0.95 — gives enough activation headroom to close the 25K tool-RETURN path PN31 was meant to fix, without needing PN31. Cross-rig data on Sander #15 comment. |
| **PN33 spec-decode warmup K-aware** (Sander v7.66 fc89395, default ON) | 🟡 Partial close on TP=1 | Backport of vllm#37521 EXTENDED to MTP/ngram. Sander claimed it closes both ampersandru's mid-stream OOM AND our workspace_lock AssertionError. Cross-rig 2026-05-02: closes BOOT-time profile_run workspace_lock ✅, but runtime decode `turboquant_attn.py:1350:_decode_attention` AssertionError still fires ❌. | Local `patch_workspace_lock_disable.py` sidecar still required for runtime decode. Drop when upstream covers the runtime path. |
| **P98 marker false-positive on v0.20** (we filed 2026-05-01 in [#9 thread](https://github.com/Sandermage/genesis-vllm-patches/issues/9#issuecomment-4359541875)) | 🟡 Side-noted to Sandermage; awaiting his call on fix | P98's drift detection auto-skips on v0.20 (`UNIFORM_SINGLE_TOKEN_DECODE` marker false-positive) but the strict workspace lock still fires rare paths P98 was supposed to revert. | Local `patch_workspace_lock_disable.py` sidecar (mounted on every TQ3 compose) relaxes the strict assertion to a one-shot WARNING. Drop when Sandermage ships either a marker fix or P98 with explicit env-override. |
| **PN30 v7.68 part3 drift-marker false-positive** (we filed in noonghunna/club-3090#19 cross-rig retest) | ✅ Closed in v7.69 (commit 2db18df) | Part3's `upstream_drift_markers=["[Genesis PN30"]` (generic prefix) matched markers parts 1+2 wrote on the same file. Part3 skipped as `upstream_merged` → apply_all FAILS → vLLM aborts. v7.69 tightened to `[Genesis PN30 v7.68 dst-shaped]` (specific). | n/a — fixed in v7.69. |
| **P103 setattr lost on `exec vllm serve`** (we filed in noonghunna/club-3090#19) | ✅ Closed in v7.69 | v7.68 P103's `setattr` ran in entrypoint shell but was lost on `exec vllm serve` worker spawn (process image replaced). v7.69 ships chunk.py self-install hook appended to end-of-file — survives any startup mechanism. | n/a — fixed in v7.69. |
| **PN32 v1 chunked at wrong level** (we filed in noonghunna/club-3090#19) | ✅ Closed in v7.69 (PN32 v2) | PN32 v1 chunked outer-level inputs but inner FLA call still got full-prompt cu_seqlens, allocating full h tensor regardless. v7.69 PN32 v2 patches `_forward_core` directly + threads `last_recurrent_state` between chunks. | n/a — fixed in v7.69. |
| [#18](https://github.com/Sandermage/genesis-vllm-patches/issues/18) — P103 cu_seqlens=[0,T] single-seq case is bypassed (we filed 2026-05-02 PM) | 🟡 Open / v7.70 proposal | P103's gate currently bypasses chunking for ANY non-None cu_seqlens, but `cu_seqlens.shape[0] == 2` (single sequence boundary) is semantically dense B=1, not multi-seq varlen. Fix admits the chunked path on real serving. Diagnosis: ChatGPT/Codex CLI. Cross-rig observation: P103 chunked path never engages on real config because vLLM's outer chunked-prefill caps T at `max_num_batched_tokens=4128` (well below `_MAX_T=16384`), so the gate-fix is semantically correct but doesn't independently close 60K Cliff 2 on TP=1+24GB. | n/a yet — gate fix queued for v7.70. Real Cliff 2 closure on this config comes from [vllm#35975 backport](https://github.com/vllm-project/vllm/pull/35975) + mem-util 0.93 (see vLLM section above + [`docs/CLIFFS.md`](CLIFFS.md)). |

---

## FlashAttention 2 (`Dao-AILab/flash-attention`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| [#1011](https://github.com/Dao-AILab/flash-attention/issues/1011) — Variable memory allocation with varlen kernels | 🔴 Open since 2024, no fix | **Cliff 1 root cause.** `softmax_lse` is allocated as `[num_seqs, num_heads, max_seqlen]` — sized by `max_seqlen` parameter, NOT actual `cu_seqlens`. So a 25K-token chunked-prefill at `max_model_len=86K` allocates softmax_lse for 86K, not 25K. This is why Cliff 1 fires harder at higher max-ctx even when the actual prompt is the same. | None. Stay at `default` 48K (or `tools-text` 75K with PN8 mitigation). FA2 redesign of softmax_lse format would be the upstream fix. |

## flash-linear-attention (`fla-org/flash-linear-attention`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| **Cliff 2 — DeltaNet GDN forward OOM at 50–60K single-prompt** | 🔴 Open, **no upstream issue filed yet**. **Confirmed cleared on dual TP=2** (this rig, 2026-04-29 — see DUAL_CARD.md "237K single-prompt verified"). | The `chunk_gated_delta_rule_fwd` kernel allocates intermediate buffers proportional to `seq_len`. Fires on single-card regardless of mem-util. On dual TP=2 the activation memory splits across cards and the cliff doesn't fire — verified at 237K single-prompt prefill on `dual.yml` (~830 tok/s prefill, matches Sandermage's 262K @ 311s on 2× A5000). Sandermage explicitly punted on the single-card fix (genesis-vllm-patches issue #1: *"can't fix this short of multi-GPU TP=2 or upstream fla.ops changes"*). Likely the same architectural pattern as FA#1011 — recurrent state buffer pre-allocated by max_seq_len. | Single-card: use `tools-text.yml` (75K cap) or `llamacpp/default` (262K, different engine). Dual: `dual.yml` clears at ≥237K. |

---

## FlashQLA (`QwenLM/FlashQLA`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| **Ampere SM 8.6 / Ada SM 8.9 port** | 🔴 No issue filed; tweet to @QwenLM drafted but not yet posted | FlashQLA is QwenLM's TileLang DeltaNet kernels — would fix Cliff 2 if it ran on Ampere. Currently SM90+ only. | None. Watch the repo for Ampere support; revisit when an issue is filed and a port is on the roadmap. |

---

## Luce DFlash (`Luce-Org/lucebox-hub`) — separate llama.cpp fork (NOT our vLLM dual-dflash)

**Heads-up — naming clarification:**
- This section tracks **`Luce-Org/lucebox-hub`** (a llama.cpp fork from Luce). **As of 2026-05-04 this is no longer single-card-only** — see "Dual-GPU split landed" below.
- Our **`dual/autoround-int4/dflash.yml` / `dual-dflash-noviz.yml`** (vLLM TP=2 dual-card) were the vLLM DFlash path but were **deprecated 2026-05-31** — superseded by `dual.yml` (stock v0.22.0, 262K + vision + MTP + 2 streams) and stranded on a now-purged nightly ([#297](https://github.com/noonghunna/club-3090/discussions/297)). **DFlash on dual now ships on beellama** (`beellama/qwen-dflash-dual`, v0.3.0 🧪). They consumed the **same draft model** (`z-lab/Qwen3.6-27B-DFlash`), but the engine + topology differ from Luce's fork. Don't confuse the two.

### 🆕 Dual-GPU split landed (2026-05-02 + 2026-05-04)

Two @weicj PRs shipped that change the lucebox-hub serving topology. **Target weights on one GPU + DFlash draft (or PFlash drafter) on a separate GPU** — heterogeneous spec-decode, not weight-sharded TP. Each model lives entirely on its own card; they communicate at spec-decode boundaries via peer copies.

- [**lucebox-hub PR #80** — `bench(dflash): add dual-GPU target/draft split harness`](https://github.com/Luce-Org/lucebox-hub/pull/80) (merged 2026-05-04). New flags `--target-gpu` / `--draft-gpu` (also `DFLASH_TARGET_GPU` / `DFLASH_DRAFT_GPU` env). Validation on dual RTX 2080 Ti 22 GB: HumanEval 10-prompt at **51.86 tok/s, AL 7.09, 44.3% accept** on Qwen3.5-27B Q4 target + z-lab DFlash draft.
- [**lucebox-hub PR #78** — `bench(pflash): add dual-GPU PFlash phase-split harness`](https://github.com/Luce-Org/lucebox-hub/pull/78) (merged 2026-05-02). New flag `--pflash-gpu` + persistent `pflash_daemon`. Validation on same hardware: **single-GPU co-resident passes NIAH at 24,573 source tokens; dual-GPU phase split passes at 262,125 source tokens (10.7×).** Compressed context reaches 13,229 tokens at 262K source.

**Implication for our 2× 3090 stack:** the single-card limitations we documented (65K max_ctx, draft VRAM competing with target activations) are addressed by dual-GPU split. Target Qwen3.5-27B Q4_K_M gets a full 24 GB on GPU 0; DFlash draft + PFlash drafter live on GPU 1. **No NCCL/allreduce overhead per token** since each model lives entirely on its own card — should be faster per-stream than SGLang TP=2 + DFlash for single-stream workloads. Bench tracked at task #229 (queued, not yet executed locally — PR #80 is hours old as of this entry). **Qwen3.6-27B draft remains under training** so the dual-GPU benefit applies primarily to the stable Qwen3.5-27B + DFlash pair today.

Re-benched 2026-04-30 PM on Qwen3.6-27B Q4_K_M + matched z-lab/Qwen3.6-27B-DFlash draft (under training). Open issues against single-card lucebox-hub follow:

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| [z-lab/Qwen3.6-27B-DFlash](https://huggingface.co/z-lab/Qwen3.6-27B-DFlash) — draft model still under training | 🟡 Snapshot 2026-04-26 | Narrative AL ~3.7, code AL ~7.0 on `Luce-Org/lucebox-hub` (single-card llama.cpp). When training finishes, expected to climb toward Qwen3.5 reference (8.31 HE, 7.04 Math). The same caveat applies to vLLM `dual-dflash.yml` — published 82/125 TPS in [`docs/DUAL_CARD.md`](DUAL_CARD.md) was measured against this 2026-04-26 snapshot at peak code-prompt conditions; AL on real agent traffic will be lower until z-lab tags training-complete. | Re-test when z-lab tags training-complete. The vLLM dual-dflash path remains shipping — see [DUAL_CARD.md](DUAL_CARD.md) — but treat its numbers as a snapshot. For autonomous coding agents on dual-3090 today, `dual.yml` (FP8 + MTP) is the recommended robust path. |
| **Build fragility on `dflash` main HEAD** | 🔴 Reproducible 2026-04-30 PM | `cmake --build` errors with `ggml_turbo_wht` and `GGML_TYPE_TQ3_0` undefined. Required submodule commit `b6ffab4a9` not auto-fetched. Cross-rig signal — fresh clone fails. | After clone: `cd dflash/deps/llama.cpp && git fetch origin && cd ../../.. && git submodule update --init`. |
| **Daemon-mode "empty prompt" regression** | 🔴 Reproducible 2026-04-30 PM | After streaming requests, subsequent requests return `"empty prompt"` from the test_dflash daemon. Server keeps accepting requests but generates 0 tokens. Forces restart. | Restart server between request flavors; avoid mixing streaming + non-streaming. |
| **`enable_thinking` chat_template_kwargs honored differently than vLLM** | 🟡 Behavioural difference | Test sends `enable_thinking=true` and expects `reasoning_content` populated. Luce returns `content` directly. Not a missing feature, but breaks our `verify-full.sh` check 6. | Don't treat the thinking-mode test as a Luce-correctness signal until the chat-template path is documented. |
| **Greedy only** | 🟡 Documented limitation | `temperature` / `top_p` accepted but ignored. Real downside for creative-writing workloads. | Use vLLM long-text/long-vision when sampling matters. |
| **Prefill OOM in `fattn-chunked.cu` on 25K+ prompts at Q8_0 KV** | 🟡 Open (configuration trade) | Chunked flash-attention CUDA OOMs on large prefill at default Q8_0. **TQ3 KV (`DFLASH27B_KV_TQ3=1`) closes it** at max_ctx=65K — verify-stress passes 791 chars / finish=stop. Higher max_ctx (131K) reopens it. | Always set `DFLASH27B_KV_TQ3=1` for stress-test-passing config. Cap max_ctx at ~65K. |
| [**PFlash — long-context prefill accelerator**](https://www.lucebox.com/blog/pflash) (sibling tech to DFlash, same Luce-Org/lucebox-hub repo) | 🟢 **Public release 2026-04 + dual-GPU split shipped 2026-05-02 (PR #78)** | **Speculative prefill + block-sparse attention.** Compresses 128K prompts to ~6.5K tokens (`keep_ratio=0.05`) before target prefill. Single-card claimed: TTFT 24.8s vs 257s vanilla llama.cpp at 128K (~10.4× speedup). **Dual-GPU phase split (PR #78) extends the passing source-context ceiling from ~24K (single-card co-resident) to 262K (~10.7×) on dual 22 GB cards** — NIAH key/answer retained at 262K. C++/CUDA only, lives inside the lucebox-hub server stack. PFlash sits *in front of* DFlash decode: PFlash accelerates prefill, DFlash accelerates generation. **For 2× 3090 deployments**: pin PFlash drafter to GPU 1 via `--pflash-gpu`, target on GPU 0. The single-card-coresident limit (was the binding blocker for our use) no longer applies. MIT license. **Open exploration**: bench PFlash + DFlash dual-GPU vs vLLM `dual-dflash.yml` (185K, 82/125 TPS on 2× 3090) on TTFT-bound workloads. Tracked at task #229. | Re-evaluate as a club-3090 shipping option once we (a) reproduce the 262K passing source-ctx claim on 2× 3090 with verify-stress + soak-continuous + bench, OR (b) an upstream-vLLM port lands. The dual-GPU split removes the single-card co-residency blocker; remaining blockers are daemon-mode bugs (greedy-only, no vision, "empty prompt" regression) carried over from the single-card history. |

---

## llama.cpp (`ggml-org/llama.cpp`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| [PR #21089](https://github.com/ggerganov/llama.cpp/pull/21089) — TurboQuant KV mainline | 🟡 Open (CPU first, CUDA follow-on) | When CUDA path lands, `turbo3` becomes a first-class option on llama.cpp. Naming will migrate from `turbo3` → `tbq3_0`. | Use Tom's fork for now: [llama-cpp-turboquant](https://github.com/tdraxl/llama-cpp-turboquant). |
| **Q3_K_XL TPS regression (28.5 TPS @ 262K → 21 TPS today)** | 🔴 Suspected, no upstream issue filed | Measured 2026-04-23 vs 2026-04-28: same model, same hardware, 28.5 TPS dropped to 21 TPS between commits `9ab47e7d8` and `0d0764dfd`. Bisect or file. | None — we're on the slower commit. Tracked in [club-3090 TODO](https://github.com/noonghunna/club-3090) (private). |
| [PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673) — MTP support (am17an, `mtp-clean`) | 🟡 Open, unmerged | First-party MTP for llama.cpp via an MTP head baked into the GGUF (RDson republished `Qwen3.6-27B-MTP-Q4_K_M-GGUF` with the head wired). Benched on 1× 3090 (2026-05-05): **+34% narrative TPS at `n-max=3` (22.83 → 30.69)**, ~57% accept. Code at `n-max=5` hit 31.9 TPS. **NOT a club-3090 recommendation yet.** Reasons: (1) unmerged → forces every cross-rig user to compile am17an's fork or maintain a custom image; (2) q8_0 KV ceiling caps context at ~64-80K — current `llamacpp/default` ships 262K, trading that for +34% TPS isn't worth it for the cliff-immune audience; (3) MTP forces `n_parallel=1` (kills `llamacpp/concurrent.yml`); (4) RDson GGUF doesn't bundle mmproj (vision regression). Audience for this is empty — vLLM dual-turbo already gives 170 TPS for users wanting max single-stream throughput. | None recommended. Re-evaluate when PR merges + q4_0 KV variant tests recover 128K+ context + cross-rig data lands. Detailed bench + reasoning is documented out-of-tree in this stack's `learnings/qwen3.6-35b-a3b.md` ("llama.cpp MTP — PR #22673 path" subsection). |

---

## transformers (`huggingface/transformers`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| [#45283](https://github.com/huggingface/transformers/issues/45283) — Qwen3.5 GGUF support | ❌ Closed without fix 2026-04-28 (no associated PR; closing event has `source: null`, last comment was just `cc @SunMarc` — looks won't-fix or stale-bot) | Was tracked as the missing piece (alongside vllm#38140 / vllm#37797) for Qwen3.5/3.6 GGUF on vLLM/SGLang. Won't be picked up via transformers — **llama.cpp remains the only GGUF path** for this model family. | llama.cpp path. Don't expect a vLLM/SGLang GGUF route for Qwen3-Next family. |
| **transformers ≥ 5.8.0 required for `gemma4_assistant`** | ✅ Resolved 2026-05-08 (ships in pinned images) | `transformers 5.8.0` (released 2026-05-05) was the first version with the native `gemma4_assistant` model class (Google's Gemma 4 MTP drafter). The historical break: vLLM nightly `:nightly-01d4d1ad3` shipped transformers 5.7.0 → AutoConfig rejected the drafter checkpoint at validation time. | **No longer needed — the dep now ships in the pinned stable images.** Verified 2026-06-20 live in the running container: `vllm/vllm-openai:v0.22.0` carries **transformers 5.9.0** (≥ 5.8.0); the single-card pin `vllm/vllm-openai:v0.21.0` is likewise post-Gemma4-merge (compose-header verification 2026-05-08). The former `pip install --upgrade transformers==5.8.0` entrypoint line was removed 2026-05-08 — see the Gemma dual compose [`models/gemma-4-31b/vllm/compose/dual/autoround-int4/bf16-mtp.yml`](../models/gemma-4-31b/vllm/compose/dual/autoround-int4/bf16-mtp.yml) ("entrypoint upgrade line dropped"). |

---

## SGLang (`sgl-project/sglang`)

| Issue / PR | Status | Why it matters | Workaround |
|---|---|---|---|
| **Same Marlin pad-sub-tile-n bug as vllm#40361** | 🔴 Not filed; same kernel-line fix applies | Blocks Lorbus INT4 + EAGLE on SGLang. We haven't filed an SGLang PR. | None on SGLang. Use vLLM (with our patched fork) or wait for SGLang to pick up the upstream Marlin fix. |
| **DeltaNet KV rollback (vllm#39931 cross-engine)** | 🔴 Same architectural issue | Blocks EAGLE on Qwen3-Next family in SGLang too. | None — see vllm#39931. |

---

## Community templates / model assets (Hugging Face)

External-but-load-bearing resources that aren't issue trackers (no PR / merge state to track). Watch list — re-check when upstream Qwen / Gemma official templates change, or when these resources update.

| Resource | Status | Why it matters | Drop trigger |
|---|---|---|---|
| **[froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates)** — community fork of the default Qwen 3.5 / 3.6 chat templates fixing seven documented bugs (empty `<think></think>` spam in past turns, `</thinking>` hallucination on Qwen 3.6, unclosed thinking before tool call, no-user-query crash in agentic loops, `developer` role rejection, `\|items` filter for C++ engines, type-aware `tojson`). Surfaced by @troymroberts in [discussion #121](https://github.com/noonghunna/club-3090/discussions/121). Vendored snapshot at [`models/qwen3.6-27b/vllm/patches/froggeric-chat-template/chat_template.jinja`](../models/qwen3.6-27b/vllm/patches/froggeric-chat-template/chat_template.jinja); mounted default-on across all 22 vanilla Qwen 3.6 composes via `--chat-template`. Carnice and Qwopus composes intentionally excluded (ship their own bespoke templates). **ADOPTED 2026-05-18** for issue #150: froggeric **v19**, HF `c31fd393e531dbacd92b6deb99a2037cc949f950` (2026-05-16) — re-eval PASSED maintainer rig A/B. | ✅ **Re-eval PASSED (#150, 2026-05-18) — v19 ADOPTED.** Today-anchored A/B on `vllm/dual` (Qwen3.6-27B INT4, identical engine, template-only delta): **`hermesagent-20` +10pp (50% → 60%, +2 scenarios)**; 7 other packs flat (toolcall/instructfollow/structoutput/dataextract/reasonmath/bugfind/cli-40 unchanged); streaming tool-calls OK both arms (#145 clear); soak-continuous PASS 0/25 silent-empty both arms. **TPS-neutral** — symmetric restart+settle, 3 bench/arm: CODE 89.06 → 87.62 (−1.6%, within run-to-run noise), NARRATIVE flat. The first-pass −7% did **not** reproduce — it was an asymmetric-protocol artifact (candidate benched ~110s post-restart vs baseline post-settled-boot), not a template effect. Reproduces the prior froggeric finding on a clean today-anchored baseline. (Superseded prior: 🟡 first-pass A/B 2026-05-12 — +15pp `hermesagent-20` 45%→60%; the 45% base was stale, today's is 50%, control-run confound now resolved by the symmetric re-eval.) | Replace with default model template if upstream Qwen pushes equivalent fixes. **Watch for**: froggeric updates the template (Qwen 4 support, additional bug fixes), or Qwen upstream lands their own version. |

---

## Filing conventions

When you file or learn of a new upstream issue:

1. **Add a row** to the appropriate section of this file. Include the link, status emoji, one-line "why it matters," and the local workaround (if any).
2. **Cross-link** from any code, compose comment, or doc that depends on the workaround back to the row in this file (e.g., `# See docs/UPSTREAM.md — vllm#40361`).
3. **Update the row** when status changes — closed, merged, propagated, replaced. Don't delete; if a row is no longer load-bearing, mark it ✅ Resolved or ❌ Closed without fix and leave it as historical context.
4. **Bump the relevant pin** when an upstream lands (Genesis commit, vLLM nightly, llama.cpp commit). Add a CHANGELOG entry citing the upstream PR.

When you file an issue against an upstream repo from this work, **link back to club-3090** in the body so the upstream maintainer can see the affected user surface and re-test if needed.

---

## Related reading

- [`models/qwen3.6-27b/INTERNALS.md`](../models/qwen3.6-27b/INTERNALS.md) — model-specific deep dives (DFlash forensics, MTP head, AutoRound rationale)
- [`models/qwen3.6-27b/vllm/patches/README.md`](../models/qwen3.6-27b/vllm/patches/README.md) — local patches (tolist, Marlin pad fork, Genesis env-var matrix)
- [`AGENTS.md`](../AGENTS.md) — repo-wide conventions, including the rule that this file is the upstream-tracking single source of truth