The C12 budget verdict used a flat absolute floor (`MIN_KV_GB = 0.05 if
qwen3-next-moe else 1.0`) and FAILed any config leaving <1 GB for the growing
KV pool. That false-FAILs KV-light models that demonstrably boot: gemma-4-26b
-a4b single-card AWQ + MTP projects 0.78 GB growing pool, yet the live boot
holds 17,490 tok >= 16,384 max_ctx and serves (sliding_window=1024 makes 25/30
layers' KV trivially cheap).
vLLM's real pre-check is token-capacity based: it boots iff the capped pool
holds >= ONE max_model_len sequence, NOT iff KV >= 1 GB. Growing KV scales
linearly with max_num_seqs, so one sequence's KV is
`kv_pool_requested_gb / max_num_seqs`. Use that as the floor — which also
generalizes (and removes) the old qwen3-next-moe 0.05 special-case.
CAP the floor at the legacy 1 GB: for dense/long-KV configs one sequence needs
many GB and a hard per-seq threshold would false-FAIL measured-working configs
sitting inside the estimator's +-1.5 GB band (e.g. gemma-dual-int8 @262K:
10.71 GB avail vs 10.84 GB/seq, 1.2% short but boots). Net: relax the floor
ONLY for KV-light models, leave dense on its exact prior >=1 GB behavior.
The fix lives in the estimator (where the gap is), NOT in disabling
kv_calc_supported for a config we actively ship the drafter on. Design
bounced with Codex (per-sequence floor); capped-at-1-GB refinement added here
after the pure per-seq form regressed the gemma-4-31b dense duals.
Verdicts after: gemma-26ba4b-single FAIL -> TIGHT (correct); dense duals
unchanged; --calibration still 7/7; full gate 42/42.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gemma-4-26b-a4b slugs were the last users of the purged
`vllm-nightly-clean` engine. Migrate them to AWQ-4bit on stock
`vllm-stable` v0.22.0, retiring the Ampere-dead AutoRound path.
Why AutoRound is dead on Ampere: AutoRound int4-MIXED packs to
`uint8b128`, which has no W4A16 kernel on sm_86 (Cutlass/Machete are
Hopper-only; Triton W4A16 takes only uint4b8/uint4). The cyankiwi
AWQ-4bit (pure uint4, compressed-tensors) resolves to Marlin WNA16 MoE
on sm_86 and serves cleanly.
Slug changes (registry 38 -> 36):
- REMOVE vllm/gemma-a4b, vllm/gemma-a4b-awq (AutoRound/AWQ-bf16 duals)
- RENAME vllm/gemma-a4b-single -> vllm/gemma-26ba4b-single (AWQ tp=1, :8040)
- RENAME vllm/gemma-a4b-awq-mtp -> vllm/gemma-26ba4b-dual (AWQ tp=2 + ext MTP, :8041)
Both on vllm-stable, kv bf16 (kv_cache_dtype=auto), kvcalc_key=SKIP
(experimental; rebench-full/soak pending before Production).
PR #40886 (AWQ compressed-tensors MoE key remap) is native in v0.22.0 —
the vendored overlay + install entrypoint are dropped; patch marked
deprecated_on 2026-06-06. vllm-nightly-clean deprecated (final
purged-nightly engine, now zero registry users); gemma4-swa-moe added
to vllm-stable supported_model_families; arch_patches engine_pin
flipped to [email protected].0 loads:true.
Boot-validated on 2x 3090 (2026-06-06): single tp=1 coherent + clean
gemma4 tool-call; dual tp=2 + external MTP (num_spec_tokens=4) coherent
+ clean tool-call, draft layers wired. 3 AutoRound/AWQ-bf16 composes
moved to _archive/ (kept for a future Hopper engine).
Gate: 42/42 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
#327 audit found all 3 DFlash slugs rode the purged nightly-e47c98ef image (the
same dead SHA as the deprecated vllm-nightly-full), and vllm/dual4-dflash was
mislabeled `production` while (a) on a dead image, (b) TP=4 (unrunnable on this
2-card rig), (c) DFlash is blocked on Qwen3-Next per the repo hardware truths
(DeltaNet rollback). Same treatment as the #254 Genesis tail:
- Archived dual-dflash, dual-dflash-noviz, dual4-dflash -> compose/_archive/
(+ manifest); removed their 3 registry entries (41 -> 38).
- DEFAULTS[(qwen,vllm,multi4)] removed -- no functional multi4 vLLM slug remains,
so the resolver degrades multi4 -> dual. The 4-card vLLM tier is now empty
(recoverable from git if demand returns).
- vllm-nightly-dflash engine -> stability: deprecated; its qwen3-next-hybrid arch
pin flipped loads:true -> false (zero registry users; not offered for derive).
- Cascade: calibration (-3 rows), kv-calc alias map + override, patches.yml
(-18 dflash refs), test fixtures (registry-disk 38/40, calibration 9/9;
launch-compat/list-topology-filter/model-default-resolver/pullgate-gates/pull
adapted to the now-empty multi4 + nightly-slug categories).
Completes the nightly-engine cleanup: vllm-nightly-mtp/full/dflash all deprecated
with zero registry users; only the gated vllm-nightly-clean (#326) remains.
42/42 gate green. Refs #327.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Self-descriptive <engine>/gemma-12b-<topology>-<quant>[-mtp] slugs:
vllm/gemma-12b-mtp -> vllm/gemma-12b-dual-bf16-mtp
vllm/gemma-12b-int8-mtp -> vllm/gemma-12b-single-int8-mtp
beellama/gemma-12b -> beellama/gemma-12b-single-q8kxl
llamacpp/gemma-12b -> llamacpp/gemma-12b-single-q8kxl
Pruned the two no-MTP base composes (MTP is output-lossless and fits the full
262144, so the bases bought nothing): removed vllm/gemma-12b (dual bf16 base) and
vllm/gemma-12b-int8 (single int8 base) + their compose files.
Cross-file: registry keys/entries, kv-calc COMPOSE_ALIAS_TEXT, calibration anchors
(dropped the removed dual-base anchor → kv-calc --calibration now 18/18, test
expectations updated), BENCHMARKS rows (rename 4 / collapse 2 pairs into the MTP
rows), UPSTREAM #39914 row, ADDING_MODELS slug example, disk-count 58/59 -> 56/57.
Hard rename, no back-compat aliases (slugs were day-old 🧪, no users). All stay 🧪
pending per-compose soaks + the ephemeral arch-preview image caveat.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vllm/gemma-12b-int8-mtp — base int8 + assistant external drafter, single 3090.
MTP fits the FULL 262144 on one card (drafter resident, KV pool ~310K tok, 1.18x
at 262K, ~20.7 GB — no ctx step-down vs base.yml).
n-sweep 2026-06-04 (code-gen probe @262144), accept-len in parens:
n=2 96.7 (2.62) · n=3 115.5 (3.33) · n=4 117.0 (3.67) · n=5 122.5 (4.06) TPS
vs ~50 no-MTP. Default SPEC_N=4 (near-peak, matches dual, robust; probe is
code-only — MTP's best case, so higher n risks draft-waste on narrative).
SPEC_N=5 for code-max.
Wiring: compose_registry vllm/gemma-12b-int8-mtp (tp=1, drafter gemma-12b-it-
assistant), kv-calc alias gemma-single-int8-mtp, disk-count 57/58 -> 58/59.
Guard suite 41/41 green. 8-pack quality (int8, lossless vs MTP) folds in next.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two LOW findings from the ship-with-notes review of 0f6f861:
- docstring: generalize the gemma 'full/sliding' layer counts (10/50 on 31B,
8/40 on 12B — read from spec, not hardcoded) + point at the measured override.
- calibration YAML: note the measured 22,816 B/tok/card is total-avail-KV/tokens,
NOT net of the ~0.08 GB/card sliding-fixed term (kv-calc adds that separately —
an intentional, tiny, conservative double-count, not an exact decomposition).
Comment-only; --calibration unchanged (19/19).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The 31B-derived gemma4-swa-dense global-only formula priced the 12B's growing KV
at 32,768 B/tok/card (8 full x 8 kv x 512 x bpe2 / TP2), but the live gemma4_unified
pool measured 22,816 B/tok/card (8.16 GiB / 384,019 tok @ 131K/TP2/0.90/MTP). Add a
gemma-12b-only measured override (measured_kv_growing_bpt_tp1=45,632) in the
gemma4-swa-dense KV branch — 31B keeps the formula (untouched, calibration byte-identical).
Token-capacity prediction 204K -> 315K vs measured 384K; residual is the conservative
shared cudagraph_overhead (handed to Codex to model principled-ly). Guard 41/41.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Registers the new gemma4_unified 12B model (vLLM PR #44429, merged 2026-06-03) as
catalog slugs vllm/gemma-12b (base) + vllm/gemma-12b-mtp (assistant drafter n=4),
so launch.sh/switch.sh resolve them by slug. Status: experimental.
- ModelProfile (family gemma4-unified; 48L/8 full+40 sliding, 16/8 GQA, head_dim
256/512, SWA 1024, max_ctx 131072 — past that vLLM CUDA-OOBs, vllm#39914).
- Engine profile vllm-gemma4-unified pins the gemma4-unified PREVIEW image
(NOT stable v0.22.0, which lacks the arch). Caveat: ephemeral tag — pin a digest
before any Production promotion.
- Drafter gemma-12b-it-assistant (n=4, 0.85GB), bf16 weights, 2 registry entries,
no DEFAULTS row (experimental). kv-calc made gemma-4-12b-aware via the shared
Gemma dense path + measured calibration anchor (384,019 tok / 8.16 GiB @ MTP/TP2/131072).
- ADDING_MODELS.md: name the central registry (compose_registry.py SoT → both
launchers) explicitly in the intro + path-3.
KNOWN: kv-calc under-predicts the live pool (~204K vs measured 384K tok, -47%) — the
shared Gemma dense formula over-prices gemma4_unified global-layer KV; --calibration
GB-verdict still PASSES (TIGHT). Recorded in the calibration YAML header; refine the
gemma4_unified global-KV model as a follow-up. Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
fp8-mtp.yml was only ever an A/B reference -- MTP is net-negative on this MoE
at vLLM TP=2 (n2 ~= -45% / n3 ~= -51% wall TPS; the built-in-MTP draft forward
pays inter-GPU sync the acceptance can't amortize). It served no production
purpose, so remove it and its wiring.
Removed:
- models/qwen3.6-35b-a3b/vllm/compose/dual/autoround-int4/fp8-mtp.yml
- compose_registry.py entry `vllm/qwen-a3b-preview-mtp` (port 8052)
- its profile_runtime.yml capture block + calibration anchor (-> 17 anchors)
- the kv-calc COMPOSE_ALIAS_TEXT token
Test/doc updates for the new counts:
- test-kv-generic-dense + test-submit-pull: calibration contract 18/18 -> 17/17
- test-compose-registry-disk: registry 50->49, disk 51->50 composes
- test-report-calib: container ref -> live vllm-qwen36-35b-a3b-dual
- BENCHMARKS.md + KV_MATH.md notes
The MTP-net-negative finding stays recorded in the BENCHMARKS production row
and learnings/qwen3.6-35b-a3b.md. Full tests/ suite green (sole red is the
pre-existing fixture-dependent test-submit-bench, identical on clean master).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Promote the 35B-A3B MoE dual compose out of preview after full live validation
on 2× RTX 3090 (2026-05-30): max-ctx probe (262K fits, 7.81× concurrency, 2.05M-
token KV pool), bench (178/174 wall · 182 decode TPS, CV <0.5%), verify-stress
(NIAH-clean to 240K), soak-continuous (0 growth / 0 err / 0 silent / 100%
retention), quality --full (det 79/90 = 88%, aider 13/30 — ≥ ik-llama ref), and a
live vision smoke (read a test image correctly @ 262K).
- compose: preview.yml → fp8.yml (serving-stack filename per layout convention),
preview-mtp.yml → fp8-mtp.yml; fp8.yml = 262K + vision-on + no-MTP, vLLM v0.22.0
stable (no overlays), ✅ Production header.
- registry: slug vllm/qwen-a3b-preview → vllm/qwen-35b-a3b-dual, max_ctx
16384 → 262144, compose_path, DEFAULTS, kvcalc alias (kv-calc.py).
- profile_runtime capture re-synced to fp8.yml; test-pull slug updated.
- BENCHMARKS row + ADDING_MODELS pointer.
- MTP intentionally OFF: built-in MTP is net-negative on this MoE under vLLM TP=2
at BOTH n=2 (≈−45%) and n=3 (≈−51%) — the draft forward pays inter-GPU sync the
acceptance can't amortize (ik-llama avoids this single-card). fp8-mtp.yml kept as
the A/B reference only.
GATED: the registry-wide test-diagnose-profile fails for this entry because
kv-calc's qwen-MoE branch does NOT divide weights by TP (a deliberate hack:
inflated weights ≈ the vLLM-filled peak, which coincidentally passes at 16K but
starves the 262K KV in the fit-check). The config is empirically validated; this
is a kv-calc MoE-VRAM-model limitation. Lands AFTER the kv-calc fix PR (separate)
that distinguishes diagnose fit-need from the calibration filled-peak for cheap-KV
MoE. Calibration anchor left untouched here (the re-baseline belongs to that PR).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
vLLM shards both attention and MoE expert weights across TP ranks, but
_weights_per_card_gb() returned the full quant weight for qwen3-next-moe
(no /tp). On dual-card that doubled the fixed footprint (20 GB instead of
10), so long-ctx configs (e.g. Qwen3.6-35B-A3B 262K dual) over-predicted
into a false FAIL -- even though the config fits and runs at 178/174 TPS
with a 2.05M-token KV pool (live-validated, see #259).
Root cause: the old comment inferred "weights resident per rank" from the
live ~=budget per-card peak. But that peak is the elastic KV pool filling
spare VRAM (vLLM allocates all available KV memory as blocks), not resident
weights. Fixed by dividing both autoround and gptq MoE weights by TP.
- qwen3.6-35b-a3b 262K/dual now PASS (was FAIL); diagnose-profile exit 0.
- Calibration holds 18/18 (verdict-direction; MoE rows now display like the
dense rows -- sub-budget predicted, verdict OK).
- Added an elastic-pool note so the now-sub-budget MoE 'predicted total'
reads as the single-seq floor, not spare VRAM.
- gemma4-swa-moe deliberately NOT changed: its TP=2 AWQ rows peak at
23.45-23.50/24 (TIGHT is protective there); needs its own boot-log
fixed-footprint anchor before /tp'ing -- annotated inline.
Verified: full tests/ suite green (test-kv-generic-dense 18/18,
test-diagnose-profile ok incl. simulated 262K registry, test-submit-pull,
test-kvcalc-version). Unblocks #259.
Closes#260.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Every gemma-4-31b vLLM compose pinned a purged Docker Hub nightly
(e47c98ef / bf610c2f), un-bootable for fresh users (#250, #167 class).
Prune the 9-variant set to 3 and repoint survivors onto the immutable
stable release tag vllm/vllm-openai:v0.21.0 (never :latest).
Survivors: vllm/gemma-mtp (bf16 dual, default), vllm/gemma-int8 (int8-PTH
dual), vllm/gemma-mtp-tp1 (fp8 single). Dropped: gemma-dflash,
gemma-dflash-int8, gemma-int8-tq3, gemma-bf16, gemma-awq; the 262K path
folds into gemma-int8 via a CTX env override. Gemma is decoupled from the
shared Qwen nightly profiles via a new vllm-gemma-stable engine profile;
Qwen nightly profiles are untouched.
Diagnose decoupling: the pruned gemma dflash patch dirs doubled as the
diagnose tool's cross-model disk-source proxies for two Qwen overlays
(vllm-pr41703-dflash, vllm-pr42102-dflash-kv-quant). Those capabilities
are image-baked in the pinned nightly, not mountable files, so mark them
image_baked and have diagnose skip the disk-source check for image-baked
overlays. Runtime-neutral (no Qwen compose mounts them).
switch.sh launcher: handle the new VLLM_IMAGE engine-pin export (the twin
loop in launch.sh had it; switch.sh did not -> "unexpected engine pin
export"), and guard the VLLM_NIGHTLY_SHA echo for the image-only stable
profile (was unbound under set -u).
Validated on 2x RTX 3090 (Ampere, v0.21.0): full scripts/tests/*.sh suite
35/0; live boots of gemma-mtp (bf16) and gemma-int8 serve coherent output,
qwen vllm/dual regression-clean. gemma-mtp-tp1 (fp8_e4m3) is correctly
SM-gated (required_sm=9.0) -- confirmed Ampere-incompatible (Triton
"fp8e4nv not supported on sm_86"), a documented 32 GB+/Hopper variant;
beellama remains the single-card gemma path on Ampere.
Refs #451, #250, #167.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
PR-B of the compose-quant-hierarchy refactor (follows PR-A #231). launch.sh no
longer hardcodes its variant/port/container/kvcalc maps — it derives them from
compose_registry.py via a shared emitter, the same single source of truth
switch.sh already uses. Adds a topology-aware <engine>/default resolver.
- scripts/lib/registry-emit.sh (new): shared emitter — emits switch/launch
tables from COMPOSE_REGISTRY, parses container_name from each compose, and
exposes registry_default_target() for <engine>[/<topology>]/default.
- launch.sh: drop the hardcoded `declare -A LAUNCH_*` maps; populate from the
emitter. PRIMARY_MODEL constant (default qwen3.6-27b) backs bare <engine>/default.
- switch.sh: drop inline derivation; resolve vllm/default, vllm/dual/default,
vllm/multi4/default via the shared emitter.
- compose_registry.py: add kvcalc_key to entries; tools/kv-calc.py aliases.
- bench-row-formatter.sh: DEFAULTS-aware compose_display (drops the now-dead
docker-compose.yml branch).
- tests: test-launch-registry-parity.sh + test-default-resolver.sh (new);
test-switch-registry-parity refreshed onto the shared emitter.
Default resolution is an alias layer over existing registry keys — no keys
renamed, no compose content changed. Side effect: the launcher<->registry drift
that left vllm/gemma-mtp pointing at a non-existent dual/.../fp8-mtp.yml is gone
(now derives the registry's bf16-mtp.yml).
Implemented via Codex per the PR-B brief; independently re-validated before
commit: 7/7 gates PASS (switch+launch parity, default-resolver, launch-compat,
registry-disk, mounts-resolve, kv-calc 22/22); live vllm/default->dual +
vllm/dual/default->dual both verify-full 8/8; leak-clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* Add llama.cpp bounded-thinking compose
* fix(structured-cot/llamacpp): ship no-underscore grammar variant + record on-rig validation
On-rig validation found the xgrammar deepseek-scratchpad.gbnf does NOT parse under
llama.cpp's GBNF — underscored rule names (line_char/line_chars) trip
"parse: error parsing grammar: expecting newline or end at _char", and llama-server
silently falls back to UNCONSTRAINED generation (FREE==FSM token counts — silent no-fire).
Fix: deepseek-scratchpad.llamacpp.gbnf — identical language, rule names with no
underscores (llama.cpp GBNF is [a-zA-Z0-9-] only). Validated 2026-05-24 on 1x 3090:
parses, fires (reasoning_content = PLAN/NOTE/VERDICT), coexists with --spec-type
draft-mtp (draft accept 0.66-0.88). FREE->FSM 2113->1009 tok (2.1x) and 934->331 (2.8x) @ temp 0.6.
Repointed the llama.cpp compose + READMEs + STRUCTURED_COT.md llama.cpp examples to the
variant; left the vLLM grammar + refs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* docs(structured-cot): redact pre-existing /home/wasif absolute paths
Pre-commit leak scan caught host-local /home/wasif/structured-cot/runs/... paths in
the 'What's saved on disk' section — pre-existing (2026-04-30 doc), not introduced by
this branch. Redacted to host-local relative form per no-internal-paths-in-public-docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
X-axis labels were still overlapping at 11 single-card configs even with
rotation + tighter labels. Restructured: labels show just the compose name
(rotated 30°), full descriptions render in a grouped legend block below.
Moved the narrative/code bar-color legend inside the chart (upper-left,
framed) to free the bottom margin for the description block + substrate.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The new llamacpp/* entries from v0.8.3 had compound labels that overlapped
neighbors at 13" figure width. Normalize all single-card llama entries to
the same `<name> <ctx>\n<modifier>` pattern the vLLM entries use, drop
redundant tokens (variant name already implies MTP/vision/default).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Compose files now live under `<model>/<engine>/compose/<topology>/<file>.yml`,
with topology as a folder rather than a filename prefix. Solves all 7
inconsistencies surfaced in the post-rename audit (single-card composes
without `single-` prefix, unsuffixed `docker-compose.yml` ambiguity,
fine-tunes encoding model name in filename, etc.) by making the directory
hierarchy enforce the convention.
Layout:
models/<model>/<engine>/compose/<topology>/<feature>.yml
Where:
- <model>: qwen3.6-27b, gemma-4-31b
- <engine>: vllm, llama-cpp, sglang
- <topology>: single, dual, multi3, multi4, multi8
- <feature>: docker-compose.yml (default) | turbo.yml | dflash.yml | etc.
Each topology subdir has a `docker-compose.yml` for the recommended
starter — bare `cd <topology> && docker compose up` works because
docker compose finds that filename automatically. Variants drop the
`docker-compose.` prefix since they're invoked via `-f` flag.
27 compose file moves total:
- 18 Qwen vLLM composes redistributed across single/dual/multi4
- 2 Qwen llama-cpp composes into single/
- 6 Gemma vLLM composes redistributed across single/dual
- 1 untracked qwopus-bf16mtp moved to dual/
Inside each compose: relative paths to `../patches/` and `../cache/`
bumped to `../../patches/` / `../../cache/`, and `../../../../models-cache`
to `../../../../../models-cache` (one extra `..` for the new depth).
Reference updates across 148 files (BENCHMARKS, all docs, CHANGELOGs,
sibling-table cross-references in compose headers, scripts, patch
READMEs, .github issue templates, tools/residency-instrument).
scripts/switch.sh VARIANTS map updated; tags themselves unchanged
(`vllm/dual` → `dual/docker-compose.yml`, `vllm/dual4` → `multi4/docker-compose.yml`,
`vllm/gemma-mtp` → `gemma-4-31b/.../dual/docker-compose.yml`, etc.).
AGENTS.md "Compose layout" section rewritten to describe the new
hierarchy, with concrete examples and the fine-tune exception
(`dual/carnice-bf16mtp.yml` carries the fine-tune name as a filename
prefix until the fine-tune graduates to its own model directory).
All switch.sh paths verified to resolve to actual files post-move.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Codex-authored harness for the Cliff 2b investigation that surfaced the
"fragmentation-dominated, not pool growth" finding (PN12 flat at 137 MiB
across the cliff fire). Three files:
- instrument.py — observational monkey-patches at request/engine/worker
boundaries; writes CSV snapshots of allocator state + KV pool deltas
- sitecustomize.py — auto-loaded by Python at container start via
PYTHONPATH mount; gates instrumentation on RESIDENCY_LOG_PATH env var
- run-instrumented-soak.sh — wrapper that boots a compose with the
sidecar mounted, runs soak, joins instrumentation rows to soak turns
README.md (new) frames it as research-grade tooling, not a shipped feature:
explicit "no SLA" note, cross-link to #41 + docs/CLIFFS.md, soft warning
against running on production composes (sitecustomize slows boot).
Notes that results/residency-* is gitignored by repo policy — we don't
auto-publish diagnostic local results because they're large, rig-specific,
and the analytical value lives in issue threads + commit log + memory
entries that distill what was learned. Cross-rig contributors should
share findings via issue comments (summary tables), not raw CSV dumps.
This closes the last open thread from #217 (results/ cleanup decision).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The results/ directory accumulated 21 MB of intermediate investigation
output during today's cliff/grammar work. Most of that (residency probes,
soak iteration matrices) lived its useful life in issue threads and the
gitignored docs/diagnostics/ memos, and isn't worth eternal evidence.
What's evidence (committed): 1.5 MB of raw bench output backing published
numbers in STRUCTURED_COT.md / BENCHMARKS.md / CHANGELOG:
- results/grammar-ab-20260503-224235/ — Phase 2 30-problem subset bench
(was the basis for the n=30 PROMPT_TERSE-wins finding that Phase 3
later disproved at scale)
- results/grammar-full-20260504-003118-gpu0/ — Phase 3 HE+ shard 1 (82 rows)
- results/grammar-full-20260504-003118-gpu1/ — Phase 3 HE+ shard 2 (82 rows)
- results/grammar-full-lcb-20260504-021003/ — Phase 3 LCB v6 (50 rows + shard
metadata; this is the post-bug-fix re-run after the harness LCB issue
surfaced and was patched)
- tools/grammar-eval/codex-check-scratchpad.gbnf — fourth grammar candidate
exploring DeepSeek + STATE/CHECK pre-VERDICT scaffolding (untested in
Phase 3; available for future bench)
What's investigation artifact (now gitignored): residency-* (Codex Cliff 2b
investigation probes, ~5 MB) + soak-* (~16 MB of cliff probe iterations,
final validation matrix already in #41 + cliff2_accumulated_ctx_finding.md
memory).
.gitignore pattern: `results/*` with explicit negations for `grammar-ab-*`,
`grammar-full-*`, and the existing `v0.20-migration` directory. Future
grammar bench runs auto-track when committed; future residency/soak runs
auto-ignore. The pattern works because git's directory-ignore precedence
applies to immediate children, not the parent itself, so negations
re-include named subdirs.
Total committed: 522 lines of bench output (jsonl + json + summary md).
The data is small (per-row generations cap at ~4096 tokens × 5 conditions
× 214 problems = ~4 MB raw, much of which compresses well in jsonl).
Held back for user decision: tools/residency-instrument/ (Codex's Cliff 2b
instrumentation harness — useful tool, no docs yet, may want a README
before publication).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
A directional estimator for "will this compose fit on my hardware?" that
runs before boot. Useful for:
- Comparative analysis (TQ3 vs fp8 KV trade-off, TP=1 vs TP=2 budget)
- Max-ctx solver (binary search the largest max_ctx that fits given
vram/tp/kv_format/max_num_seqs/mem_util)
- Educational breakdown (weights / KV pool / activation peak / cudagraph
workspace per-card components)
Anchored to published literature where applicable:
- PerfMamba (arxiv 2511.22849) — O(γ·D·N·L) scaling for the GDN forward
intermediate (this is our Cliff 2 mechanism)
- TurboQuant (arxiv 2504.19874, ICLR 2026) — TQ3 byte savings
- PagedAttention (arxiv 2309.06180) — KV pool layout
Calibrated empirically against BENCHMARKS.md cross-rig data. Current
verdict accuracy: 9/11 shipped composes (82%) with ±1.5 GB error band.
The two miscalibrations are over-predictions on max_num_seqs > 1
configs; vLLM's actual KV pool allocation rate-limits internally in
ways my naive demand formula doesn't capture. Documented as a known
limitation in docs/KV_MATH.md.
Usage examples:
bash tools/kv-calc.py --compose dual-turbo --vram 20 --mem-util 0.82
bash tools/kv-calc.py --solve-max-ctx --tp 2 --kv-format fp8_e5m2 --vram 16
bash tools/kv-calc.py --calibration
Documents the math at docs/KV_MATH.md including:
- Per-component formulas (weights / KV pool / activation peak / overhead)
- Per-KV-format byte tables
- The TQ3→fp8 swap rule on 20 GB cards (validated by @efschu, #47)
- Known limitations + when to trust kv-calc.py vs vLLM's boot log
Closes#226 (the v1 calculator). Future work that's NOT in this commit:
multi-model support (DeepSeek-V3, GLM-4.5 — requires per-architecture
spec tables), driver-class overhead modeling (4090 vs 3090 deltas
beyond what we currently fold into the ±1.5 GB error band), Cliff 2b
fragmentation modeling.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
After the full Phase 3 5-grammar A/B (HE+ 164 + LCB v6 50, n=214 problems
× 5 conditions = 1070 generations), bounded-thinking.yml is updated to
recommend the DeepSeek scratchpad grammar (PLAN/NOTE×0-15/VERDICT FSM at
tools/grammar-eval/deepseek-scratchpad.gbnf) as the default.
Phase 3 results (combined HE+ + LCB v6, all FSM-enforced):
- DeepSeek scratchpad: 87.4% Pass@1 (+1 net vs andthattoo, +4pp on LCB)
- andthattoo G/A/E: 86.9% (the originally-published technique)
- Holiday tagline: 86.4%
- PROMPT_TERSE (no FSM): 82.2% (Phase 2's n=30 win was subset-selection bias)
- FREE (no constraint): 78.0% (baseline)
Phase 1 reproducibility is exact: HE+ FSM Δ +4.3pp / LCB v6 Δ +24.0pp,
both match Phase 1's published numbers — validates the bench harness.
Compose ships unchanged engine-side (same vLLM image, same Genesis stack,
same TQ3 KV, same MTP n=3, same enable_in_reasoning flag). Only the
docstring's recommended-grammar pointer + the on-disk grammar files in
tools/grammar-eval/ change. Three grammars are now validated and available
client-side via extra_body={"structured_outputs": {"grammar": ...}}:
- DeepSeek scratchpad (default, best LCB)
- andthattoo G/A/E (originally-published, ~4× tighter think budget)
- Holiday tagline (extreme 24-token compression, wins LCB by 4pp too)
Combined-accuracy spread is within noise (0.5pp at n=214), so we ship one
compose rather than three siblings — choice is at the client, not at the
compose level.
Bug fix bundled: tools/grammar-eval/subset-bench.py --full --include-lcb
mode now correctly threads dataset kind through run_condition so LCB
problems use mod.run_tests_livecodebench instead of HE+ assertion-based
testing. Without this, the Phase 3 LCB shard crashed with KeyError:
'prompt' on the HE→LCB transition.
This wraps active research on bounded-thinking. Reopen if upstream FSM-
regress cluster behavior changes (Genesis pin bump, vLLM grammar engine
swap, model-family change), or if user demand surfaces for a sibling
compose pinning a non-default grammar.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- BENCHMARKS.md (new) — fills the gap that PR template + CONTRIBUTING.md
reference. Consolidates measured numbers from MULTI_CARD/SINGLE_CARD/
STRUCTURED_COT into a single by-model table. Includes the canonical
bench prompt definition, the verify-stress + soak-continuous matrix
per variant, and the dual4 + dual4-dflash rows promised on the
PR #44 merge (Whamp's 4× 3090 cross-rig data, including the bench-vs-
soak inversion noted on dual4-dflash).
- subset-bench.py — Phase 3 mode: --full (all 164 HE+) + --include-lcb
(50 LCB v6 release_v6/leetcode/2025-01-01 cutoff) + --he-start/--he-end
+ --lcb-start/--lcb-end + --label for parallel dual-GPU sharding.
The flags compose so a 2-GPU rig can split 0–82/82–164 + 0–25/25–50
across two endpoints, halving wall time from ~5h to ~2.5h on RTX 3090.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- Add .github/PULL_REQUEST_TEMPLATE.md with the rig-report / verify /
soak-continuous / bench / BENCHMARKS row checklist that PR #44 surfaced
as missing. New compose variants now have an explicit gate list.
- CONTRIBUTING.md: new "Submitting a new compose variant — full gate list"
section explaining the why behind each gate, cross-linked to docs/CLIFFS.md
and #41 for the soak-continuous rationale.
- numbers-from-your-rig.yml: optional v2 SOAK_MODE=continuous summary field
for cross-rig Cliff 2b validation.
- docs/STRUCTURED_COT.md: replace "Phase 2 pending" stub with measured
results — Holiday tagline (4/6 rescue, 23-token think), DeepSeek scratchpad
(5/6 rescue, 387-token think), PROMPT_TERSE (5/6 rescue at 75 tokens with
no FSM mask). Headline reframe: PROMPT_TERSE rescuing 5/6 with the same
G/A/E shape as current suggests FSM enforcement is the mechanism causing
those regressions, not absence of structure. Phase 3 plan included.
- tools/grammar-eval: deepseek-scratchpad.gbnf (PLAN/NOTE×0-15/VERDICT)
and subset-bench.py wired for the 5-condition harness.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
New output: docs/img/performance-single-vllm.{png,svg} — just the two
recommended single-3090 vLLM routes (long-vision 198K + long-text 218K).
Designed for the launch tweet so the image scope matches the tweet text
exactly (no dual, no llama.cpp, no Luce).
Also made the substrate footer line conditional on which engines actually
appear in the chart — vLLM-only charts no longer show llama.cpp/Luce
substrate cruft.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two issues caught in review of the previously-merged chart fix:
1. Combined performance.png had group labels "1× 3090 — llama.cpp" and
"1× 3090 — Luce DFlash *experimental*" overlapping horizontally at
figsize=(15, 7.0) with 11 bars. Bumped to (18, 7.5) and split labels
into 2 lines (cards / engine) so each group label fits over its band
without colliding.
2. VRAM dual chart title didn't make it explicit that all 4 dual configs
run vLLM (someone might assume "dual-dflash" uses Luce DFlash, the
single-card project). Now reads "Dual 3090 — vLLM (TP=2), per-card
breakdown — all 4 configs run vLLM; dual-dflash uses vLLM's DFlash
spec-decode."
Files: tools/charts/gen-perf.py + gen-vram.py + 6 regenerated SVG/PNG.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The 7-bar single-card chart was cramped at figsize=(11, 6.0) — the
"single 3090 — Luce DFlash *experimental*" group label collided with
the value labels above the long-text 218K bars.
Bumped:
- single chart figsize 11→13 wide, 6.0→6.5 tall
- combined chart figsize 14→15 wide, 6.5→7.0 tall
- title pad 22→36 (more room above for group-label band)
- y_band lifted from 1.10× to 1.18× max bar height
- ylim raised from 1.20× to 1.30× max bar height
- group label fontsize 10→9.5 to fit longer labels
Group labels now sit cleanly above value labels with adequate
separation. Twitter-ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Re-tested Luce-Org/lucebox-hub on Qwen3.6-27B Q4_K_M + matched z-lab
3.6 draft, 2026-04-30 PM. Significant progress since 2026-04-22 but
several gaps still keep it off the recommended list.
Measured (RTX 3090, greedy, single-stream, n_gen=1000):
Narrative: 37-47 TPS (mean ~40, AL ~3.7)
Code: 63-76 TPS (mean ~72, AL ~7.0)
vs vLLM long-text 218K: 50/66 narr/code
Files:
- docs/SINGLE_CARD.md: new "Watch list — Luce DFlash" section after the
fallback variants. Documents what works (tool calls, streaming, daemon
mode, stress-passing TQ3 + 65K config) and what still keeps it off the
recommended list (greedy only, under-trained 3.6 draft, no vision,
thinking-mode quirk, build fragility, daemon-mode empty-prompt bug).
- docs/UPSTREAM.md: new Luce DFlash watch table with 6 specific items —
z-lab draft training, build fragility (submodule ref drift on main
HEAD), daemon-mode regression, enable_thinking handling, greedy-only,
and prefill OOM in fattn-chunked.cu (closed by TQ3 KV at 65K).
- tools/charts/gen-perf.py: added Luce DFlash 3.6+3.6 row with new
"single-luce-watch" group (purple, *experimental* label). Footer
caveat under chart explains why it's experimental. Title bumped to
2026-04-30. Substrate line includes Luce dflash@f12a87c.
- docs/img/performance.{svg,png}, performance-single.{svg,png}: regen.
Re-test trigger: z-lab marks Qwen3.6-27B-DFlash training-complete OR
Luce-Org tags a release with the daemon-mode bug fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
SINGLE_CARD.md and DUAL_CARD.md embed scoped charts now — single-card
configs only on the single page, dual on the dual page. Combined views
stay on top-level README and the model README.
- docs/img/performance-{single,dual}.{svg,png} — new, scoped TPS charts
- docs/img/vram-budget-{single,combined}.{svg,png} — new
- docs/img/vram-budget-dual.{svg,png} — content swap: was combined,
now genuinely dual-only. Old combined content lives in -combined.
- tools/charts/gen-{perf,vram}.py — matplotlib sources, idempotent.
Re-run with: uv run --with matplotlib --with numpy python3 ...
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
noonghunna