club-3090/models/qwen3.6-27b/CHANGELOG.md

Qwen3.6-27B — Changelog

Dated history for Qwen3.6-27B configs in this repo. Combines the single-card and dual-card timelines (both were previously separate repos; consolidated here 2026-04-28).

2026-05-04 — Carnice-V2-27B + BF16 MTP overlay — new compose variant ⭐

Adds docker-compose.carnice-bf16mtp.yml: kai-os/Carnice-V2-27b (Hermes-style agentic fine-tune of Qwen3.6-27B) quantized to INT4 via delta-merge of Lorbus's AutoRound grid, with a BF16 MTP overlay for clean spec-decode acceptance.

Key findings from the diagnostic push:

• Hypothesis B (MTP quant-grid mismatch) accounted for ~70% of the AL gap. Un-quantizing 7 mtp.layers.0.* projections (BF16 overlay) recovered AL from 2.0 → 3.0.
• Tool-call format: Carnice's Hermes-style template used XML, but vLLM's --tool-call-parser hermes expects JSON. Patched chat template instructs JSON output inside <tool_call> tags. Vendored at patches/carnice-chat-template.jinja.
• Full 262K context confirmed (22,246 MiB/card), 2 streams, same fp8 KV + MTP n=3 as dual.yml.

Validation:

• verify-full.sh: 7/8 PASS (thinking test lenient — Carnice is concise, not verbose)
• verify-stress.sh: 6/7 PASS (needle recall at ≥60K — model-level GDN attention ceiling)
• bench.sh (n=5): 71.75 narr / 80.35 code wall TPS, MTP AL 3.02-3.14, TTFT 141ms
• soak-test.sh (8×3 turns): PASS — 0 MiB growth, 0 errors, 101.6% TPS retention

Compose: docker-compose.carnice-bf16mtp.yml

2026-05-03 late PM — dual4-dflash.yml TP=4 DFlash validated on 4× RTX 3090 PCIe ⭐

Adds docker-compose.dual4-dflash.yml, a 4-card full-context DFlash variant validated on Whamp's 4× RTX 3090 PCIe rig for club-3090 discussion #26. This is a capacity / 262K-code variant, not a replacement for the faster 2-card DFlash short-prompt path.

Config accepted by vLLM pre-check:

• tensor_parallel_size=4
• max_model_len=262144
• max_num_seqs=2
• max_num_batched_tokens=8192
• dtype=bfloat16, FP16/default KV (required by DFlash on Ampere)
• speculative_config={"method":"dflash","num_speculative_tokens":5}
• reported GPU KV cache size: 207,264 tokens
• reported max concurrency at 262K/request: 2.27×

Validation:

• Boot: clean, ready after 375s on a warm image/model cache.
• verify-full.sh: PASS.
• verify-stress.sh: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: 58,570 tokens and 91,070 tokens.
• bench.sh: 64.00 narrative / 104.40 code wall TPS (CV 2.8% / 3.0%), TTFT 143ms / 164ms.
• DFlash AL during code bench: last three log samples 4.43 / 4.37 / 4.35.
• Peak VRAM during bench: 21,960 MiB/card.

Interpretation: TP=4 DFlash gives a useful code-speed uplift over dual4.yml (104 vs 76 code TPS) while retaining full 262K admission, but PCIe TP=4 allreduce keeps it below the 2-card DFlash variants' raw single-stream TPS. Use it for 4-card, full-context, code-heavy work with two admitted streams.

2026-05-03 PM — dual4.yml TP=4 baseline validated on 4× RTX 3090 PCIe ⭐

Adds docker-compose.dual4.yml, a measured 4-card fp8/MTP baseline derived from dual.yml by scaling tensor parallelism and streams from 2 → 4. Validation came from Whamp's 4× RTX 3090 PCIe rig in club-3090 discussion #26.

Config accepted by vLLM pre-check:

• tensor_parallel_size=4
• max_model_len=262144
• max_num_seqs=4
• max_num_batched_tokens=8192
• kv_cache_dtype=fp8_e5m2
• reported GPU KV cache size: 483,200 tokens
• reported max concurrency at 262K/request: 6.77×

Validation:

• Boot: clean, ready after 355s on a warm image/model cache.
• verify-full.sh: PASS after warm retry (first Paris request hit a cold-path 30s script timeout; direct retry returned HTTP 200 in 0.2s and full rerun passed).
• verify-stress.sh: PASS 7/7. Canonical Cliff 2 probe 7 recalled both large needles: 58,569 tokens and 91,070 tokens.
• bench.sh: 63.01 narrative / 76.25 code wall TPS (CV 2.1% / 4.0%), TTFT 111ms / 132ms.
• MTP AL during code bench: last three log samples 3.42 / 3.53 / 3.62.
• Peak VRAM during bench: 23,494 MiB/card.

Interpretation: TP=4 gives the first published 4×3090 Cliff 2 boundary data and higher full-context concurrency headroom, but single-stream TPS is lower than TP=2 on PCIe-only allreduce (published TP=2 fp8/MTP baseline is ~69 / 89 TPS). Use dual4.yml for 4-card capacity / Cliff 2 margin, not for fastest single-user short-prompt decode.

2026-05-02 PM — Genesis v7.69 + vllm#35975 backport — Cliff 2 60K CLOSED ⭐⭐

Genesis pin bump fc89395 (v7.66) → 2db18df (v7.69 dev tip). All three v7.66/v7.68 regressions we surfaced upstream landed in v7.69, plus a local backport of vllm#35975 brings the Cliff 2 single-prompt envelope to 60K cleanly on TQ3 + MTP K=3 at 24 GB. Two shippable single-card recipes ship at this pin.

v7.69 closes upstream:

Patch	What	Status before v7.69	Status in v7.69
PN30 v7.68 part3	DS conv state row-stride fix (replaces our dst-shaped sidecar)	drift-markers too generic — silent re-fail	landed clean, our sidecar drops
P103 worker self-install	FLA Cliff 2 chunked fwd_h+fwd_o orchestrator survives exec vllm serve	setattr lost on worker spawn → "rebound at 0 caller sites"	self-install hook in chunk.py, fires on TP=1
PN32 v1	GDN chunked-prefill threshold + chunk size env-vars	not yet shipped	landed (PN32_GDN_CHUNKED_PREFILL=1, PN32_GDN_CHUNK_SIZE=8192, PN32_GDN_CHUNK_THRESHOLD=16384)

Codex P103 cu_seqlens gate fix queued for v7.70: filed Genesis #18 — _single_seq_cu detection lets cu_seqlens.shape[0] == 2 enter the chunked path. Diagnostic logging on dev205 showed q.shape[1] always 4128 (capped by vLLM max_num_batched_tokens for spec-decode K=3) so MAX_T=16384 never engages on real serving — gate-redirect works but doesn't fire under our admission cap. Cliff 2 in practice is residency, not gate logic: 50 MiB OOM after 394 successful T=4128 chunks → cumulative state filling 22.96 GiB of 23.56 GiB total. Lower mem-util buys activation headroom by spending KV capacity. Three distinct ceilings clarified per Codex round 2: declared max_model_len (admission), safe single-prompt prefill length (Cliff 2), concurrency capacity.

Local backport added: vllm#35975 (skip inputs_embeds GPU buffer for text-only models). Two-site regex text-patch via patch_inputs_embeds_optional.py:

• gpu_model_runner.py: wraps self.inputs_embeds = self._make_buffer(...) in if self.supports_mm_inputs or self.enable_prompt_embeds:
• llm_base_proposer.py: wraps self.inputs_embeds = torch.zeros(...) in if self.supports_mm_inputs:

Measured ~444 MiB freed on text-only paths (claimed ~64 MiB upstream — claim assumes a smaller config; our 180K + MTP K=3 path benefits more).

Mem-util matrix on long-text 180K + MTP K=3 (60K stress):

mem-util	#35975	60K result
0.95	no	OOM (Cliff 2)
0.95	yes	OOM (Cliff 2)
0.92	yes	PASS — 643s wall
0.93	yes	PASS — 623s wall ⭐ shipped

Two shippable Cliff-2-closed variants:

Variant	max-model-len	mem-util	MTP K	TPS regime	Single-prompt envelope	Use when
long-text.yml (Balanced MTP) ⭐	180000	0.93	3	50 narr / 67 code (cold short prompt); decode wins from MTP at high accept	60K PASS @ 623s; 90K indeterminate (curl >25 min)	default for steady-state agent + chat
long-text-no-mtp.yml (Max-context)	200000	0.95	off	~33 narr / ~40 code (no spec-decode)	60K PASS @ 537s	one-shot >50K input where you can wait + don't need MTP

Both top out at the 60K hardware-physical wall on 24 GB single-card. 90K MTP-off attempt didn't complete in 25 min (HTTP 000 at 1500s curl); not shipped, not failed — indeterminate.

Sidecars dropped (3, all closed natively in v7.69):

• patch_pn25_genesis_register_fix.py → covered by v7.66 PN25 + v7.69 worker-spawn registration
• patch_pn30_dst_shaped_temp_fix.py → covered by v7.69 PN30 part3
• patch_workspace_lock_disable.py → covered by v7.69 PN34_WORKSPACE_LOCK_RELAX env-gate

Sidecars retained on master (2):

• patch_inputs_embeds_optional.py (NEW) — backport of vllm#35975. Drop when upstream merges.
• patch_tolist_cudagraph.py — unchanged.

Genesis env bundle (long-text.yml):

GENESIS_ENABLE_P103=1
GENESIS_ENABLE_PN32_GDN_CHUNKED_PREFILL=1
GENESIS_PN32_GDN_CHUNK_SIZE=8192
GENESIS_PN32_GDN_CHUNK_THRESHOLD=16384
GENESIS_FLA_FWD_H_MAX_T=16384
GENESIS_ENABLE_PN34_WORKSPACE_LOCK_RELAX=1
GENESIS_VLLM_SSM_CONV_STATE_LAYOUT=DS

Branch v7.69-cliff2-test merged to master at commit 15b84df. Per-round bisect log + mem-util sweep at results/v0.20-migration/v769-codex-r1-test.summary (404 lines, six bisect rounds).

Cross-rig contributions filed:

• Genesis discussion #19 — three v7.66/v7.68 regression reports + acknowledgement of all closures in v7.69.
• Genesis issue #18 — P103 cu_seqlens gate fix proposal for v7.70.

2026-05-02 — Genesis v7.66 + Cliff 1 mech B closed ⭐

Genesis pin bump 753344b → fc89395 (v7.66 dev tip). Cliff 1 mech B closed across all 4 TQ3 composes via two local backports:

PN25 v3 import-time backport (patch_pn25_genesis_register_fix.py): Sander's PN25 mechanisms — both v7.65 @torch.library.custom_op (which crashes on infer_schema inside dynamo trace) AND v7.66 direct_register_custom_op + Library("genesis", "FRAGMENT") (which crashes on instantiate_user_defined_class_object inside dynamo trace) — fail on TP=1 spawn. Our v3 text-patches vllm/model_executor/layers/activation.py to register the op at module-import time, BEFORE any trace context exists. Survives both Sander mechanisms because it sidesteps registration-during-trace entirely.

PN30 dst-shaped temp fix (patch_pn30_dst_shaped_temp_fix.py): Sander's PN30 a9977d8 corrupts DS conv state row strides by raw-memcpying a compact .contiguous() source-tail into a strided destination. Our fix builds a destination-shaped temp inside collect_mamba_copy_meta (where both source AND destination block IDs are known) and does the strided copy correctly. Diagnosis credit: ChatGPT/Codex CLI cross-check.

Validation matrix (v7.66 + local sidecars, verify-stress.sh 7-probe ladder):

Compose	Ctx	mem-util	Probes	Failure
long-text	180K	0.95	6/7	Cliff 2 architectural
long-vision	145K	0.95	6/7	Cliff 2 architectural
bounded-thinking	180K	0.95	6/7	Cliff 2 architectural
dual-turbo (TP=2)	262K	0.85	6/7	Cliff 2 architectural

Backoff from 214K + 0.985 → 180K + 0.95 (long-text/bounded-thinking) and 198K + 0.98 → 145K + 0.95 (long-vision) was needed to give activation headroom for the PN12+PN25 FFN pool residence + PN30 dst-shaped temp lifecycle. Vision tower's persistent ~1 GB tightens long-vision further.

Sander's v7.66 PN33 partial: PN33 (default ON) closes BOOT-time profile_run workspace_lock issue, but the runtime decode workspace_lock at turboquant_attn.py:1350:_decode_attention still fires on TP=1. Cross-rig data sent to Sander via discussion #19 reply.

Sander's v7.66 PN31: still doesn't fit on 24 GB. Per-shape persistent buffers + PN12+PN25 pool residence outpace activation budget at chunk_fwd_o. Lower mem-util (0.95) is sufficient to close the 25K tool-RETURN path PN31 was designed to fix without needing PN31 itself.

Sidecars retained on master (4):

• patch_pn25_genesis_register_fix.py (PN25 v3 import-time, TP=1 only)
• patch_pn30_dst_shaped_temp_fix.py (replaces Sander's compact .contiguous())
• patch_workspace_lock_disable.py (PN33 narrowed but didn't close runtime decode path)
• patch_tolist_cudagraph.py (cudagraph capture fix, unchanged)

Per-config + cross-rig results in results/v0.20-migration/v766-pin-results.summary and the per-compose *-pn30.summary files.

2026-05-01 PM — vLLM v0.20 + Genesis v7.65 dev tip migration ⭐

Master pin migration from vllm-openai:nightly-07351e088... (0.19.2rc1.dev205) + Genesis v7.64 (64dd18b) to vllm-openai:nightly-7a1eb8ac2ec... (0.20.1rc1.dev16+g7a1eb8ac2) + Genesis v7.65 dev tip (commit d89a089). v0.20's revised TQ FA prefill paths (vllm#40092) and Genesis v7.65's PN26b sparse-V kernel + PN17 FA2 lse-clamp + P38B/P15B in-source hooks together close Cliff 1 mech B sub-mechanisms that forced the dev205 backoffs. Three of our local sidecars (patch_pn12_ffn_pool_anchor.py, patch_pn12_compile_safe_custom_op.py, patch_fa_max_seqlen_clamp.py) replaced by Genesis-native equivalents.

Sidecars retained:

• patch_workspace_lock_disable.py (NEW) — relaxes vllm#39226 strict assertion to one-shot WARNING. Sandermage's P98 covers the same surface but auto-skips on v0.20 (drift-marker false-positive). Drop when Sandermage ships marker fix.
• patch_tolist_cudagraph.py — unchanged.

Sidecars dropped:

• patch_pn12_ffn_pool_anchor.py → covered natively by PN12 on v0.20
• patch_pn12_compile_safe_custom_op.py → covered by Genesis PN25
• patch_fa_max_seqlen_clamp.py → covered by PN17 + P15B

Mamba block_size cap fix: v0.20 enforces long_prefill_token_threshold >= block_size; on hybrid Mamba+TQ3 the engine forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M and GENESIS_PREALLOC_TOKEN_BUDGET from 4096 → 4128 across all 5 main composes.

Restored ceilings (vs dev205 backoff):

Variant	Before (dev205+v7.64)	After (v0.20+v7.65 dev)	Δ
long-text.yml	185K + 0.975	214K + 0.985	+29K (+16%)
long-vision.yml	140K + 0.95	198K + 0.98	+58K (+41%)
bounded-thinking.yml	185K + 0.975	214K + 0.985	+29K (+16%)
tools-text.yml	75K + 0.97 (fp8)	75K + 0.97 (unchanged)	flat
dual-turbo.yml	262K + 0.85	262K + 0.85 (full v7.65 PROD env-vars)	flat ctx, +9% TPS

Bench results (n=5, 3 warmups + 5 measured, canonical narr+code prompts):

Variant	Narr wall_TPS (CV)	Code wall_TPS (CV)	TTFT	AL	Avg accept	VRAM	KV pool tokens
long-text.yml 214K	49.74 (2.6%)	67.39 (2.7%)	154/155 ms	3.34-3.51	78-84%	23.4 GB	284,832 (1.03×)
long-vision.yml 198K	50.32 (2.3%)	66.12 (4.1%)	159/158 ms	3.40-3.56	79-85%	22.3 GB	264,192 (1.02×)
bounded-thinking.yml 214K	49.77 (1.4%)	65.80 (2.3%)	155/154 ms	3.25-3.61	75-87%	21.7 GB	284,832 (1.03×)
tools-text.yml 75K (fp8)	53.32 (2.3%)	69.66 (1.4%)	150/153 ms	3.53-3.59	84-87%	22.2 GB	104,000 (1.05×)
dual-turbo.yml 262K (TP=2)	58.33 (2.9%)	76.01 (4.5%)	112/110 ms	3.39-3.51	79-84%	19.8 GB/card	1,523,232 (4.67×)

Concurrent throughput on dual-turbo (canonical code prompt, 2 runs per stream):

Streams	Total TPS	Per-stream mean	Per-stream CV	Speedup
1	74.03	73.99	3.7%	1.00×
2	128.74	65.57	14.1%	1.74×
3	126.52	55.41	31.9%	1.71×
4	269.03	74.05	3.1%	3.63×

n=4 lands at near-single-stream per-stream TPS — true parallel decoding of full-context streams, not interleaved. The n=2/n=3 dips are scheduler artifacts on small bench sizes (high CV at n=3 confirms interleave behavior). Practically: dual-turbo serves either 1 stream at 76 TPS or 4 streams at 269 TPS aggregate.

Validation: verify-full ✅ 8/8 on every variant. verify-stress 33K AND 50K tool-prefill ✅ PASS on every variant (the 50K cliff that fired on EVERY dev205 config no longer reproduces). Branch v0.20-migration; bench captures at results/v0.20-migration/.

2026-04-30 PM — Cliff 1 closes; long-text 218K + long-vision 198K

PN12 was silently no-op'd on dev205+ via anchor drift (same bug class as P101). Genesis apply_all reported "PN12 applied" while live vllm/model_executor/layers/activation.py retained the vanilla SiluAndMul.forward_cuda. Local sidecar patch_pn12_ffn_pool_anchor.py repairs it; bundled Genesis tree carries the fix via PR #13. Combined with local patch_fa_max_seqlen_clamp.py (P104 FA softmax_lse clamp), Cliff 1 closes on TQ3 paths.

New shipped ceilings:

• long-text.yml: 205K → 218K at 0.985 mem-util (no vision, no override). Engine ceiling vLLM-reported 218K. Verify-stress + verify-full pass; MTP AL 2.66; VRAM 23.7/24 GB.
• long-vision.yml: 192K → 198K at 0.98 mem-util (vision on). Engine ceiling vLLM-reported 198K. 0.985 + vision reopens Cliff 1 (more goes to KV at the cost of activation budget; vision tower's persistent ~1 GB makes 0.98 the right balance).
• --num-gpu-blocks-override 50 no longer needed at 0.985 — anchor-fixed PN12 cuts allocator churn enough that natural activation budget at higher mem-util is sufficient on text-only path.
• 0.99 mem-util ruled out — driver/system reserves ~440 MiB; vLLM startup check fails at 0.99.

Cliff 2 unchanged. Single-prompt >50–60K still OOMs in DeltaNet GDN. Both long-* variants stay "steady-state accumulation across many turns, not single-shot big prompts."

Variants stay distinct: docker-compose.yml (48K, below both cliffs, fast boot) and tools-text.yml (FP8 path for IDE agents) remain valuable for their respective use cases. Four-variant menu kept; the long-* options now ship at higher ceilings.

Branch cliff1-fa-clamp carries the changeset; commits 41eabac (PN12 sidecar) → f3e5b52 (218K bisection) → 26e5f65 (docs).

2026-04-29 — Genesis v7.62.x + PN8 enabled on FP8 paths

Sandermage shipped Genesis v7.62.x (commit 917519b) on 2026-04-29 with PN8 (MTP draft online-quant propagation — backport of vllm#40849) targeting the FP8+MTP memory-headroom problem. We benched the patch across all 5 single-card composes that use Genesis:

Compose	KV	mem-util	PN8 effect	TPS Δ	Verdict
tools-text.yml (75K, fp8)	fp8	0.97	−900 MiB at boot · Cliff 1 closes ⭐	−7% code	PN8 enabled
fast-chat.yml (20K, fp8)	fp8	0.95	−800 MiB at boot	−4.7% code	PN8 enabled
docker-compose.yml (48K, TQ3)	TQ3	0.92	no-op (already plenty of headroom)	−3% / −5%	PN8 not enabled
long-vision.yml (192K, TQ3)	TQ3	0.98	KV pool +230 MiB, engine ceiling 192K → 198K, but Cliff 1 still fires	−5%	PN8 not enabled (commented in env, opt-in)
long-text.yml (205K, TQ3)	TQ3	0.98	no effect (engine ceiling capped by block-size divisor at 206K)	not benched	PN8 not enabled

Why split-decision: the Cliff 1 OOM that ampersandru hit on long-vision.yml is an FFN intermediate-buffer activation peak (138 MiB allocate at intermediate_size=17408 × max-num-batched-tokens=4128), not a draft-model footprint. PN8's quant-config propagation doesn't reach that buffer on TQ3 paths. On FP8 paths the draft head's own footprint shrinks meaningfully — that's where the win is.

Cross-rig data + analysis posted to Sandermage: single-3090 #1 comment 4343317153.

Other v7.62.x items relevant to us (not yet benched here):

• PN11 (Quentin-M, vllm#41142 streaming tool-call IndexError fix) — applies cleanly via the auto-detected REC; planned to enable in tools-text + fast-chat next pass.
• TurboQuant k8v4 unlocked on hybrid GDN via P4 + P98 — Sandermage's A5000 measurement +1.9%; we'll bench on dual.
• Per-GPU recommendation system (vllm/_genesis/gpu_profile.py) — boot log now lists [REC]/[OFF] per patch on this card. Nice ergonomics.

2026-04-28 (post-launch) — llama.cpp Q3_K_XL + Docker compose + stress-test findings + VRAM diagram

• First measured TPS for UD-Q3_K_XL on this stack: 21.22 narr / 20.79 code @ 262K context + vision (single 3090, q4_0 KV). VRAM 20.17 GB / 24 GB at boot. Lower than memory's 28.5 baseline (Q4_K_M, 2026-04-23 on llama.cpp commit 9ab47e7d8) — investigating mainline regression vs current 0d0764dfd. ngram-mod path measured at 22.04 / 26.11 (+25% on code, draftless via --spec-type ngram-mod).
• llama.cpp Docker compose at models/qwen3.6-27b/llama-cpp/compose/:
  • docker-compose.yml — single slot, 262K ctx, q4_0 KV, vision via mmproj. Uses ghcr.io/ggml-org/llama.cpp:server-cuda.
  • docker-compose.concurrent.yml — 4 parallel slots, 192K ctx pool, vision. Multi-tenant variant.
• All three llama.cpp configs pass verify-full + verify-stress on this stack. Crucial finding: llama.cpp R1 (Q4_K_M @ 262K + q4_0 KV), Q3_K_XL @ 262K + vision, and Q4_K_M + ngram-mod @ 32K all clear the 90K needle ladder + 25K tool-prefill checks. No Cliff 1, no Cliff 2 — the prefill OOMs that bite vLLM single-card 192K configs don't fire in llama.cpp on this model. Trade is the ~2-3× lower TPS (21 vs 51-55 vLLM). Reframes our launch positioning around "vLLM dual = max throughput, llama.cpp single = max robustness." Single feature gap: llama.cpp doesn't peel <think> into reasoning_content (parser issue, not model). Tool calling, streaming, vision, output quality all clean on --jinja.
• models/qwen3.6-27b/README.md — added "VRAM allocation across configs" section with embedded docs/img/vram-budget-dual.svg. Per-card stacked bars across 7 configs (3 single, 4 dual) showing weights / KV / vision / DFlash draft / activations / free headroom on the 24 GB budget. Visualizes the TP=2 unlock concretely.
• models/qwen3.6-27b/llama-cpp/README.md — quant table updated. UD-Q3_K_XL marked ⭐ as our default with citation to Benjamin Marie's Kaitchup Q3.6-27B GGUF eval — independent H100-validated pick of Q3_K_XL as the optimal accuracy/efficiency/footprint balance, complementary to our 3090 speed measurements.

2026-04-28 — Split verify-full.sh → verify-full.sh (fast) + verify-stress.sh (boundary)

Recent additions to verify-full.sh (#8 tool-prefill OOM, #9 cascade detection, #10 MTP AL) made the script slow — the longctx needle ladder (#7) alone could run 5+ min, and the full 10-check suite was approaching 10 min. Awkward for "is the stack functional" iteration during dev work.

Split into two scripts:

• verify-full.sh — fast functional smoke, 8 checks, ~1-2 min. Contains: server reachability, Genesis patches applied, basic completion (Paris), tool calling, streaming, thinking mode, output quality / cascade detection, MTP acceptance length. Run after every config change to confirm the stack still serves cleanly.

• verify-stress.sh — boundary-case stress test, 2 checks, ~5-10 min. Contains: long-context needle ladder (4 depths up to 90K tokens) + tool-response prefill OOM (~25K-token mock tool message). Run before publishing or when investigating prefill-OOM regressions specifically.

Same env-var conventions (URL, MODEL, CONTAINER, SKIP_LONGCTX, SKIP_TOOL_PREFILL, PREFILL_TARGET_CHARS). Both pass on the new club-3090 default + dual.yml + dual-turbo.

2026-04-28 — Dual-card re-bench on club-3090 substrate (revised TPS numbers)

The published dual-card TPS numbers were measured pre-v714 formalization (April 24-25 timeframe), on a different vLLM nightly + Genesis tree. Re-benched all 4 dual composes on the club-3090 unified substrate (dev205 + Genesis v7.51-stable + Marlin pad fork mounted) to reconcile.

Also: caught a stale mount path in dual-turbo.yml — predecessor mounted patch_tolist_cudagraph.py from ../patches/genesis/ (where it lived in the old qwen36-dual-3090 layout); club-3090 has it at ../patches/ (top-level). Fixed before measurement; container booted clean. All other composes already had correct paths.

Measured numbers (3 warmup + 5 measured per prompt arm, narr 1000 tok + code 800 tok):

Compose	Narr TPS (CV)	Code TPS (CV)	TTFT	MTP/DFlash AL	VRAM/card	Was claimed	Δ%
dual.yml	69.05 (2.3%)	88.58 (3.4%)	145ms	3.38-3.48	23.6 GB	71/89	-3% / -1%
dual-turbo.yml (now TQ3)	53.65 (2.7%)	72.93 (2.7%)	113ms	3.41-3.42	24.1 GB	58/69 (k8v4)	-8% / +6%
dual-dflash.yml	81.94 (4.3%)	124.93 (5.8%)	138ms	4.10-4.35	23.6 GB	78/128	+5% / -2%
dual-dflash-noviz.yml	78.19 (2.5%)	126.99 (2.2%)	143ms	4.24-4.37	23.8 GB	77/124	+2% / +2%

Net: most numbers within run-to-run variance. The dual.yml fp8 path is essentially unchanged. dual-turbo's TQ3 swap (from k8v4) cost ~8% narrative but recovered ~6% code — net trade for ~9× the KV pool capacity.

All 4 composes pass verify-full.sh functional checks (skipped longctx ladder on the DFlash variants for time; fp8 + MTP variants pass full 10/10 including the 90K-token needle). Updated all docs (README compose table, USE_CASES.md, dual.yml header, dual-turbo.yml header) with the measured numbers.

2026-04-28 — Add long-vision + long-text composes (R3' / R3''' from formal v714 round)

Previously the 192K and 205K opt-in tiers were documented as "edit max-model-len + mem-util in docker-compose.yml" — fragile for reproducibility against published bench numbers. Promoted both to dedicated compose files:

• docker-compose.long-vision.yml — TQ3 + Genesis P65 + MTP n=3 + 192K + 0.98 mem-util + vision tower active. Matches R3' bench row (50.93 narr / 67.69 code TPS, AL 3.40-3.58 80-86% accept). Container name: vllm-qwen36-27b-long-vision. Same prefill caveats as edit-the-default did.
• docker-compose.long-text.yml — Same config + --language-model-only + max-model-len 205K. Matches R3''' (50.11 narr / 65.84 code TPS). Container name: vllm-qwen36-27b-long-text.

Trade-off: 2 more compose files (now 11 vs 9). Net: every published bench row from the v714 formalization round (R2, R3, R3', R3''', R4, R6, R7) now boots cleanly with one -f flag — no error-prone editing for users who want to reproduce. R1 (eager) and R5 (longctx) stay deleted (obsolete, not niche).

Header references updated: model README compose table, USE_CASES.md frontier-context section, default's variant matrix, vllm/README.md "Pick a compose" code block.

2026-04-28 — Repo migration to club-3090

Configs migrated from the predecessor repos (qwen36-27b-single-3090, qwen36-dual-3090) into this repo's models/qwen3.6-27b/vllm/compose/ directory. File renames:

Old path	New path
qwen36-27b-single-3090/compose/docker-compose.yml	models/qwen3.6-27b/vllm/compose/docker-compose.yml
qwen36-27b-single-3090/compose/docker-compose.fast-chat.yml	models/qwen3.6-27b/vllm/compose/docker-compose.fast-chat.yml
qwen36-27b-single-3090/compose/docker-compose.tools-text.yml	models/qwen3.6-27b/vllm/compose/docker-compose.tools-text.yml
qwen36-27b-single-3090/compose/docker-compose.no-genesis-mtp.yml	models/qwen3.6-27b/vllm/compose/docker-compose.no-genesis-mtp.yml
qwen36-27b-single-3090/compose/docker-compose.minimal.yml	models/qwen3.6-27b/vllm/compose/docker-compose.minimal.yml
qwen36-dual-3090/compose/docker-compose.yml	models/qwen3.6-27b/vllm/compose/docker-compose.dual.yml
qwen36-dual-3090/compose/docker-compose.turbo.yml	models/qwen3.6-27b/vllm/compose/docker-compose.dual-turbo.yml
qwen36-dual-3090/compose/docker-compose.dflash.yml	models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash.yml
qwen36-dual-3090/compose/docker-compose.dflash-noviz.yml	models/qwen3.6-27b/vllm/compose/docker-compose.dual-dflash-noviz.yml
qwen36-27b-single-3090/patches/patch_tolist_cudagraph.py	models/qwen3.6-27b/vllm/patches/patch_tolist_cudagraph.py

Functional content identical — only paths changed. Anyone with scripts referencing the old paths needs to update; the old repos still serve the old paths read-only.

2026-04-28 — Compose rename: v7.14 is the zero-arg vLLM default (single-card)

Breaking change at the time (mitigated by being on a small-audience repo).

• docker-compose.v714.yml → docker-compose.yml. Running docker compose up -d (with no -f flag) now boots the production-safe TQ3 + Genesis v7.14 + MTP n=3 + 48K + 0.92 config.
• The previous zero-arg default (fp8 + MTP n=3 + 20K) → docker-compose.fast-chat.yml. Pick this one when you only need ≤20K context and want the maximum-TPS chat path (~5-7% faster than the new default).
• docker-compose.longctx-experimental.yml → deleted. Superseded by the default's opt-in 128K + 0.95 tier.

2026-04-28 — Prefill-OOM tests + safe v714 default

Triggered by ampersandru's production OOM report (noonghunna/qwen36-27b-single-3090#1) — a Hermes-class agent fetching ~25K tokens of news as a tool reply at 192K context crashed the engine.

Discovered two distinct activation-memory cliffs on this hardware:

• Cliff 1 — TurboQuant attention scratch + tool-response prefill, fires on ≥25K-token tool messages at high --gpu-memory-utilization. OOM site: TurboQuant forward (dequant scratch + mid_o/output buffers), ~138 MiB allocate.
• Cliff 2 — DeltaNet/GLA recurrent state buffer, fires on any single prompt above ~50-60K tokens regardless of mem-util. OOM site: fla.ops.chunk.chunk_gated_delta_rule_fwd_h.h.new_empty(...). NT grows linearly with prompt length; chunked-prefill doesn't help.

Shipped:

• verify-full.sh extended from 7 → 10 checks: #8 tool-response prefill OOM, #9 output quality / cascade detection, #10 MTP acceptance length threshold.
• verify-full.sh #7 long-context needle ladder treats engine HTTP 400 (oversize ctx rejection) as a clean "skipped at this depth" rather than a failure.
• vLLM single-card default lowered to 48K + 0.92 — below both cliffs. All 10 checks pass.
• README/docs document the full opt-in matrix (64K → 205K) with safe single-prompt + tool-prefill envelopes per tier.
• Three-layer defense documented: vLLM --max-model-len HTTP 400 rejection + agent-framework truncation + system-prompt limits.

TPS unchanged at the new default: 51 narr / 68 code TPS (CV ~2.3%). Hardware-bound.

2026-04-28 — Inherited prefill-OOM tests on dual-card

The dual-card stack adopted the new verify-full.sh checks for safety even though TP=2 + fp8 KV (the dual-card default) gives much wider safety margins than single-card TQ3 KV — the cliffs are not active failure modes on dual hardware.

All compose variants pinned to vllm/vllm-openai:nightly-07351e0883470724dd5a7e9730ed10e01fc99d08 (= vLLM dev205+g07351e088). Previously some tracked :nightly and drifted with upstream.

2026-04-27 — Full-matrix re-bench + substrate unification (single-card)

Discovered and fixed four real compose drift bugs during a complete re-bench cycle:

• Image split: composes had drifted across two different vLLM image pins. All six unified to vllm/vllm-openai:nightly-07351e0883470724dd5a7e9730ed10e01fc99d08.
• eager.yml config drift: shipped with gpu-memory-utilization=0.92 and max-model-len=131072 while @ampersandru's actual measurement was 0.97 + 125000. As-shipped failed to boot. Compose deleted entirely.
• v714.yml mount path: patch_tolist_cudagraph.py was mounted from a wrong path. Fixed.
• Bench harness regression: scripts/bench.sh had silently dropped the code-prompt arm. Restored.
• Genesis exoneration: A/B between default + Genesis vs no-Genesis confirmed Genesis is performance-neutral on the fp8+MTP path. Cross-rig confirmed by u/sudeposutemizligi on TP=2 + dev45 + no Genesis (55 narrative / 68 code, same hardware class).

2026-04-27 — Removed: docker-compose.eager.yml (single-card)

Originally proposed by @ampersandru as a 125K path that bypasses the cudagraph bug class via --enforce-eager. Re-bench cycle measured 25.5 narr / 32.3 code — strictly dominated by longctx-experimental.yml at the same 125K context (38/50 TPS). Compose removed.

2026-04-27 — Patch hardening (single-card)

• patches/patch_tolist_cudagraph.py was silently failing on (a) any non-docker setup (hardcoded dist-packages path) and (b) any vLLM nightly past the one we initially tested against. Fixed: patcher auto-discovers vLLM via import vllm and uses single-line regex anchors. Bug reported by @3dluvr in single-3090 #1.

2026-04-25 — Genesis v7.14 (Sandermage upstream)

Genesis v7.14 shipped with the P65 patch root-causing vllm#40880 — the silent tool-call cascade bug under MTP × TurboQuant × cudagraph. P65 forces cudagraph_mode=PIECEWISE for spec-decode → eager continuation runs the correct branch.

This shipped as a workaround. The proper fix is a custom multi-query Triton kernel (P67) that handles K+1 query against compressed cached KV under cudagraph capture — designed-but-not-implemented as of v7.14.

The dual-card Turbo variant (docker-compose.dual-turbo.yml) loads Genesis v7.14 with P64/P65/P66/P68/P69 enabled via env vars. ~25% per-stream TPS regression vs fp8 default but 4.59× concurrency at full 262K vs fp8's 2.36× — aggregate throughput exceeds fp8 above ~3 concurrent streams.

We adjusted two consumer-Ampere knobs vs Sandermage's A5000-class defaults: gpu-memory-utilization 0.92 → 0.85 and max-num-batched-tokens 8192 → 4128. Without these, deep-prefill (60K+) requests OOM on 24 GB cards.

2026-04-22 — DFlash N=5 + Qwen3.6-27B (Luce z-lab fork)

Luce z-lab's DFlash spec-decode draft model for Qwen3.6-27B clears verify-full.sh on dual-3090. Single-stream 78 / 128 TPS narr/code — substantially faster than MTP n=3's 71 / 89.

Two DFlash variants ship in the dual-card path:

• docker-compose.dual-dflash.yml — vision + DFlash N=5 + 185K context
• docker-compose.dual-dflash-noviz.yml — text-only + DFlash N=5 + 200K context

Required workaround: vllm#40334 (DFlash combine_hidden_states dtype mismatch) is open. Compose sets --dtype bfloat16 to match the draft's training dtype.

2026-04-15 — Marlin pad-sub-tile-n (PR #40361 — our patch)

Filed vllm#40361 — fixes a crash in vLLM's Marlin INT4 kernel where output features < 64 cause GPTQ_MARLIN_MIN_THREAD_N (64) > out_features on TP=2.

Status: OPEN, MERGEABLE, awaiting maintainer review. Until it lands, dual-card composes volume-mount our patched fork at /opt/ai/vllm-src/.

2026-04-08 — Initial dual-card release

vLLM-based dual-3090 recipe shipping at TP=2 with fp8 KV + MTP n=3, full feature parity with the single-card project plus the Marlin pad workaround. Was its own repo at the time; now lives here.

Earlier — Initial single-card release

Initial single-card release shipped a docker-compose.longctx-experimental.yml at 125K with cudagraph_mode=NONE as the long-context option. v7.14 superseded this; deprecated and removed during 2026-04-27 cleanup.