12 Commits

Author SHA1 Message Date
noonghunna
75b29556c1 catalog-baselines slice 1: baselines.yml + registry-emit join + guards; catalog drops the BENCHMARKS scrape
The catalog's measured columns (TPS / 8pk) now come from ONE productized
source: scripts/lib/profiles/baselines.yml — the shipped, PR-reviewed
'bar' (accepted display projection of a validated gate run, with pin/rig/
power provenance) — joined per-slug into registry-emit --json with an
emit-computed staleness verdict. Consumers never read baselines.yml or
BENCHMARKS.md directly (BENCHMARKS stays the public human/cross-rig
ledger). Design: the catalog-baselines note (2026-07-02, §2/§5 slice 1).

- baselines.yml SEED WAVE 1: 10 rows with airtight traceability only
  (rebench tags on disk: agents-a1 golden specimen + gemma-31b-dual;
  unambiguous decode-class BENCHMARKS rows for the rest). 4 rows are
  HONESTLY born-stale with documented reasons (llamacpp rolling-tag-era
  x2, beellama pre-#296 image, 35B v0.22.0) — the guard's demo cases.
  9 slugs still owed rows are listed as wave-2 gaps, no guessed numbers.
- registry-emit join: per-variant 'baseline' field; current pin resolved
  the way launchers actually resolve it (engine-profile install.spec,
  compose-image-default fallback for ik/llama.cpp) → 'stale' =
  measured-pin != current-pin, null when undeterminable.
- test-baselines.sh: schema + slug-membership + ctx-parity (compose ctx
  default == registry max_ctx, functional slugs) RED; pin-staleness
  WARN-only (pin bumps must not block on immediate re-bench — the debt
  stays visible). test-registry-json gains the 'baseline' contract key.
- c3: enrich_measurements = pure in-memory map off the joined field —
  deletes BOTH the per-slug --explain fan-out (~4s/slug, the #439
  option-3 leg) and the BENCHMARKS.md scrape from the catalog path
  (Explain modal + cross-rig explorer keep their readers). Stale rows
  render a † on the TPS cell + a status-line legend; full badge/overlay
  treatment is slice 2. Measurement gains source='baseline' + stale.
- results/baselines/README: the three-store relationship (regression
  corpus vs measurement records vs display bar) + the no-drift rule the
  slice-2 induction tool enforces.

Verified: guard test green (10 rows, 4 stale-warned); live join emits
57 variants / 10 baselines; live c3 catalog shows all 10 with daggers on
exactly the stale 4 (A1 serving during the check: 154/154 - 105/150);
c3 suite 764/764; full scripts gate green serial. (First gate pass
tripped classifier/dedup by running CONCURRENTLY with the c3 suite —
.pull-captures pollution class; serial rerun clean.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
2026-07-04 15:25:36 +00:00
noonghunna
9426a03e85 Add curated quality-baseline corpus + auto-diff (#252 Phase 1) (#397)
Phase 1 (offline): the wrapper, the corpus home, and the test gate. Phase 2
(GPU) curates the actual baselines against the live RECOMMENDED_DEFAULT_MODELS
configs.

- scripts/quality-baseline.sh: thin wrapper over quality-test.sh --full that
  captures (--capture -> --save-json) or diffs (default -> --previous-result)
  an n>=3 aggregate per (registry-slug, thinking-mode). no-thinking is
  canonical (temp-0); enable-thinking is the reasoning-on companion. --dry-run
  prints the resolved command; extra args pass through to benchlocal-cli.
- scripts/quality-test.sh: forward --repeat / --previous-result and honor a
  --save-json path override, so the wrapper's blessed layout works.
- results/baselines/: committed corpus home (whitelisted in .gitignore) + a
  README documenting the convention, usage, and an empty index table.
- scripts/tests/test-quality-baseline.sh: offline gate (--dry-run) — asserts
  command/path resolution per mode + the required-slug / valid-mode /
  positive-repeat / missing-baseline guards.
- docs/QUALITY_TEST.md: regression-baseline subsection pointing at the corpus.

Co-authored-by: noonghunna <10742901+noonghunna@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 19:11:35 +05:00
noonghunna
ebca0c8921 docs(benchmarks): PFlash long-context bench — 131K source ceiling on 1× 3090 (#230)
Closes task #230. Measured PFlash NIAH compression at 16K-260K source
contexts on 1× 24 GB / 3090 single-card.

Result: PFlash works flawlessly up to 131K source. Compresses 131,068
tokens to 6,524 (5%) in 10.8s with NIAH key + answer both retained.
Vanilla llama.cpp pp131072 takes ~257s per Luce's published numbers,
so PFlash alone is ~24× faster at this context. End-to-end TTFT
(PFlash + target prefill on 6.5K) would be ~12-13s vs ~257s = ~20×.

Above 131K, drafter ephemeral forward-pass tensors (K_curr/V_curr/Q_last
at full sequence length) exceed 24 GB. K-cache quantization
(--pflash-k-type q8_0) doesn't help — the failing allocs are
forward-pass not cache, confirmed by separate bench at 200K/260K with
identical OOM at the same layer numbers.

@weicj's PR #78 claim of 24K → 262K dual-GPU phase split is neither
refuted nor reproduced. Their setup was 2× 22 GB Ti with target also
loaded co-resident on one card; the "24K" was target+drafter
combined. Our 131K is drafter-alone on 24 GB. Reproducing 262K
specifically would require investigation of their drafter config
(chunk_size, lookahead, BSA window) — drafter activation footprint
at 200K+ is the binding constraint regardless of GPU count.

Practical recommendation for 24 GB / 3090 single-card users: PFlash
is shippable for source contexts ≤ 131K. The ~24× TTFT speedup is
genuine and quality holds. Above 131K, fall back to vanilla llama.cpp
prefill or wait for upstream drafter optimizations.

Adds:
- BENCHMARKS.md "PFlash long-context compression on 1× 3090" subsection
  with full per-context table + drafter ceiling explanation
- results/lucebox-pflash-niah-20260504-150321/ (BF16 K cache run)
- results/lucebox-pflash-niah-q8k-20260504-150600/ (q8_0 K cache run)

This closes our active investigation of the Luce surface — three
benches done (DFlash same-card 73.97 mean, K8V4 same-card 74.68 mean,
PFlash compression ceiling 131K). Recommendation surface narrows to:
PFlash at ≤131K is the one piece of Luce that beats vLLM dual.yml on
TTFT for that workload class.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 15:08:12 +00:00
noonghunna
e78eaa1148 docs(benchmarks): K8V4 result + P2P-CNS finding on lucebox-hub dual-GPU (#229)
K8V4 same-card bench (-ctk q8_0 -ctv q4_0): 74.68 mean tok/s vs 73.97
default KV — basically identical (+1%). KV-format optimization doesn't
help at HumanEval-scale (<150-tok prompts × 128-tok gen) where the KV
pool isn't the bottleneck. Asymmetric quant available via PR #56/#54
(merged 2026-04-28 in lucebox-hub).

P2P-CNS finding (more important): the dual-GPU split bench from
cb089e1 ran on a chipset that reports "Chipset Not Supported" for
GPU↔GPU peer access (PHB topology, common consumer-board limitation).
The lucebox-hub split path requires CUDA P2P for direct draft-feature
transfers; without it, falls back to host-staging copies (CPU↔GPU
bouncing). The +1.7% we observed is therefore NOT a fair test of the
split's value — it's measuring same-card vs same-card-with-host-staging
overhead.

@weicj's published 51.86 tok/s on dual 2080 Ti 22GB (PR #80) presumably
ran with P2P available. Our negative result on PHB-only consumer boards
is rig-specific, not a refutation of the technique. Updated BENCHMARKS
row to reflect this honestly.

Setup gotcha now documented in BENCHMARKS: check `nvidia-smi topo -p2p r`
before configuring --target-gpu / --draft-gpu. NVLink-bonded setups
would typically expose P2P (cross-rig confirmation needed on lucebox
specifically; @JusefPol's NVLink win was on vLLM TP=2, not lucebox).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 14:49:26 +00:00
noonghunna
030f780f24 chore(gitignore): allow results/lucebox-*/ — evidence for BENCHMARKS lucebox row
Mirrors the existing negations for grammar-ab-* / grammar-full-* / v0.20-migration.
The lucebox-dual-gpu-20260504-142832/ run from cb089e1 backs the published
BENCHMARKS row; tracking it here keeps the evidence reproducible.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 14:33:49 +00:00
noonghunna
d82e89807a chore(results): commit grammar bench evidence + gitignore investigation artifacts (#217)
The results/ directory accumulated 21 MB of intermediate investigation
output during today's cliff/grammar work. Most of that (residency probes,
soak iteration matrices) lived its useful life in issue threads and the
gitignored docs/diagnostics/ memos, and isn't worth eternal evidence.

What's evidence (committed): 1.5 MB of raw bench output backing published
numbers in STRUCTURED_COT.md / BENCHMARKS.md / CHANGELOG:

- results/grammar-ab-20260503-224235/ — Phase 2 30-problem subset bench
  (was the basis for the n=30 PROMPT_TERSE-wins finding that Phase 3
  later disproved at scale)
- results/grammar-full-20260504-003118-gpu0/ — Phase 3 HE+ shard 1 (82 rows)
- results/grammar-full-20260504-003118-gpu1/ — Phase 3 HE+ shard 2 (82 rows)
- results/grammar-full-lcb-20260504-021003/ — Phase 3 LCB v6 (50 rows + shard
  metadata; this is the post-bug-fix re-run after the harness LCB issue
  surfaced and was patched)
- tools/grammar-eval/codex-check-scratchpad.gbnf — fourth grammar candidate
  exploring DeepSeek + STATE/CHECK pre-VERDICT scaffolding (untested in
  Phase 3; available for future bench)

What's investigation artifact (now gitignored): residency-* (Codex Cliff 2b
investigation probes, ~5 MB) + soak-* (~16 MB of cliff probe iterations,
final validation matrix already in #41 + cliff2_accumulated_ctx_finding.md
memory).

.gitignore pattern: `results/*` with explicit negations for `grammar-ab-*`,
`grammar-full-*`, and the existing `v0.20-migration` directory. Future
grammar bench runs auto-track when committed; future residency/soak runs
auto-ignore. The pattern works because git's directory-ignore precedence
applies to immediate children, not the parent itself, so negations
re-include named subdirs.

Total committed: 522 lines of bench output (jsonl + json + summary md).
The data is small (per-row generations cap at ~4096 tokens × 5 conditions
× 214 problems = ~4 MB raw, much of which compresses well in jsonl).

Held back for user decision: tools/residency-instrument/ (Codex's Cliff 2b
instrumentation harness — useful tool, no docs yet, may want a README
before publication).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 13:24:06 +00:00
noonghunna
f6613c869a v7.69 + #35975 + Codex P103 gate fix — Cliff 2 closure recipes
Six rounds of cross-rig bisect with Codex/ChatGPT diagnosis closes
Cliff 2 at 60K on TP=1 + 24GB. Two shippable variants:

Balanced MTP (long-text.yml updated):
- Genesis v7.69 dev tip (commit 2db18df)
- Codex r1 P103 gate fix applied to nested Genesis tree (cu_seqlens=[0,T]
  treated as dense single-seq, not multi-seq varlen). Sent to Sander as
  v7.70 proposal — diff in /tmp/cliff2_v770_cu_seqlens_response.md.
- vllm#35975 backport sidecar (patch_inputs_embeds_optional.py)
- mem-util 0.93 (down from 0.95)
- max_model_len 180000 (admission ceiling at this mem-util)
- MTP K=3 retained
- 60K probe: HTTP 200 in 623s, recall correct, MTP AL=4.00

Max-context safety (long-text-no-mtp.yml — NEW):
- Same patches, but MTP off + mem-util 0.95
- max_model_len 200000 (admission unlocked by removing MTP residency)
- 60K probe: HTTP 200 in 537s, recall correct
- 90K probe: indeterminate within 25-min curl budget
- For long single-shot RAG / codebase analysis

Diagnostic chain:
1. Codex r1 identified P103 gate as too broad (cu_seqlens != None
   bypasses chunking even for single-seq [0,T] case). Applied fix.
2. T=4128 distribution showed chunked path never engages on real
   serving (vLLM's outer chunked-prefill caps T well below MAX_T).
3. Codex r2: real Cliff 2 source is residency, not gate logic.
4. PR #35975 backport (skip inputs_embeds for text-only) frees
   444 MiB at boot — necessary but not sufficient at 0.95 mem-util.
5. mem-util sweep at 0.92/0.93 with MTP+#35975 closed Cliff 2 at 60K.
6. MTP-off + 0.95 + 200K admission validated max-context variant.

Codex's P103 gate fix is semantically correct and worth shipping in
Genesis v7.70 even though it's not what closes 60K Cliff 2 on this
config (the FLA call sees T=4128 already, well below MAX_T).

Full diagnostic trail: results/v0.20-migration/v769-codex-r1-test.summary

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 15:24:49 +00:00
noonghunna
7a7efbea0d bump Genesis pin 753344b → fc89395 (v7.66 dev tip)
v7.66 ships 3 new patches relevant to our config:
- PN33 (default ON): spec-decode warmup K-aware sizing, vllm#37521 backport
  EXTENDED beyond EAGLE to cover MTP/ngram. Sander claimed it closes both
  ampersandru's mid-stream OOM AND our workspace_lock AssertionError.
- PN25 v7.66: refactored from `@torch.library.custom_op` to
  `direct_register_custom_op` + `Library("genesis", "FRAGMENT")` at module
  level. Schema introspection at import time eliminates the
  `infer_schema skipped frame` Dynamo crash class.
- PN32 (default OFF): GDN chunked-prefill for Cliff 2 single-24GB-GPU OOM.

Cross-rig validation findings on 1×3090 TP=1
--------------------------------------------

**PN33 partial — narrows but does not close workspace_lock on TP=1.**

Sander's claim was that PN33 closes both ampersandru's mid-stream OOM
AND our workspace_lock AssertionError. Tested both:

| Test                                       | PN33 result      |
|--------------------------------------------|------------------|
| Engine boot (profile_run workspace lock)   |  closed       |
| Runtime decode (`turboquant_attn.py:1350`) |  still fires  |

Engine boots cleanly without `patch_workspace_lock_disable.py` sidecar
when PN33 is on, BUT the first decode request crashes with the same
`AssertionError: Workspace is locked but allocation from
turboquant_attn.py:1350:_decode_attention requires 0.76 MB`.

Net: keep `patch_workspace_lock_disable.py` sidecar mounted. PN33
narrows the bug surface but doesn't close it for our config.

**PN25 v7.66 still doesn't work on TP=1.**

Sander's `direct_register_custom_op` + `Library("genesis", "FRAGMENT")`
approach replaces v7.65's `@torch.library.custom_op`, eliminating the
`infer_schema` skipped-frame issue. But on TP=1 the new failure mode is
`Library("genesis", "FRAGMENT")` itself failing inside dynamo trace at
`instantiate_user_defined_class_object` (different mechanism, same root
cause: Library construction inside trace context disallowed on TP=1).

Net: keep `patch_pn25_genesis_register_fix.py` v3 (import-time approach).
Our patch text-patches activation.py to register the op at module-import
time as a cached global, BEFORE any trace context exists. Survives both
the v7.65 `@custom_op` and v7.66 `Library` failure modes because we
register outside the trace entirely.

**PN30 dst-shaped temp fix carries forward cleanly.**

Our `patch_pn30_dst_shaped_temp_fix.py` anchor still matches v7.66's
PN30 wiring file. All 4 TQ3 composes still pass probes 4 + 5 (multi-turn
agent, LCB-coding) which would otherwise crash with Sander's upstream
PN30 a9977d8 (compact `.contiguous()` row-stride corruption — see
genesis-vllm-patches#17 reply for the diagnosis).

**PN31 still doesn't fit on 24 GB.** Same memory pressure as v7.65 round.

Validation matrix on v7.66
--------------------------

| Compose            | Probes (verify-stress.sh)                |
|--------------------|-------------------------------------------|
| long-text          | 6/7  (Cliff 2 only fail)               |
| long-vision        | 6/7  (Cliff 2 only fail)               |
| bounded-thinking   | 6/7  (Cliff 2 only fail)               |
| dual-turbo (TP=2)  | 6/7  (Cliff 2 only fail)               |

Same coverage as v7.65 + our patches. No new regressions on v7.66.

Net effect of pin bump
----------------------

- Get Sander's v7.66 + PN33 (validated improvement, even if partial)
- Get PN32 available for opt-in (Cliff 2 mitigation, untested by us)
- Same 3 local sidecars retained (PN25 v3, PN30 fix, workspace_lock)
- No simplification possible yet

Per-config + cross-rig summary in
results/v0.20-migration/v766-pin-results.summary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 03:40:06 +00:00
noonghunna
b875624f2d extend PN25 v3 + PN30 dst-shaped temp fix to all 4 TQ3 composes
PR #23 + PR a62ad78 + PR 9af1a52 shipped PN25 v3 + PN30 dst-shaped temp
fix on long-text only. This commit extends the same patch stack to the
remaining 3 TQ3 composes (long-vision, bounded-thinking, dual-turbo) and
validates each independently.

What changed
------------

- **long-vision.yml**: 198K + 0.98 → **145K + 0.95** + DS layout +
  PN25 v3 + PN30. Vision tower residence forces deeper backoff than
  long-text (engine pre-check returned `estimated max 148608` at 175K +
  0.95, settled at 145K with safety margin).
- **bounded-thinking.yml**: 214K + 0.985 → **180K + 0.95** + DS layout +
  PN25 v3 + PN30. Parity with long-text — same patch stack, same backoff,
  structured-CoT grammar still works on top.
- **dual-turbo.yml**: 262K context preserved + DS layout + PN25 v3 + PN30.
  TP=2 splits state across both cards, fits the patch stack at 0.85
  mem-util cleanly.

Validation per compose (verify-stress.sh, 7 probes)
---------------------------------------------------

| Variant            | Pass count | Failure              |
|--------------------|------------|----------------------|
| long-text          | 6 / 7      | Cliff 2 architectural |
| long-vision        | 6 / 7      | Cliff 2 architectural |
| bounded-thinking   | 6 / 7      | Cliff 2 architectural |
| dual-turbo (TP=2)  | 6 / 7      | Cliff 2 architectural |

All non-architectural probes pass — IDE-agent one-shot, multi-turn agent,
LCB-coding, reasoning-heavy, 25K tool RETURN, small-rung longctx. Cliff 2
(60K+ single prompt DeltaNet GDN forward state OOM) fails on every
variant including TP=2 because GDN state is per-rank not split — that's
fundamental, not addressable on this config class.

Per-config summaries written to:
- results/v0.20-migration/long-vision-pn30.summary
- results/v0.20-migration/bounded-thinking-pn30.summary
- results/v0.20-migration/dual-turbo-pn30.summary

Docs partial update (SINGLE_CARD.md): updated TL;DR table with new ctx
ceilings and removed the Cliff 1 mech B "limitation to know" since both
mechanisms (PN12 eager + PN25 v3 compile) now close it. Kept Cliff 2 as
the one remaining limitation. More doc updates in follow-up commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-02 02:22:21 +00:00
noonghunna
0bdcb69fa3 results: re-bench dual.yml + dual-dflash + dual-dflash-noviz on v0.20
Closes the "dual variants not yet re-benched on v0.20" caveat from PR #23.
Confirms no v0.20 regression on fp8 / FP16 paths — code TPS within bench
variance of chart values across all 3.

n=5 measured + 3 warmup per prompt:

| Variant            | Chart (prior) narr / code | v0.20 measured narr / code | Δ                       |
|--------------------|----------------------------|------------------------------|-------------------------|
| dual.yml           | 69.05 / 88.58              | 68.61 / 90.71  (CV 1.8% both) | narr -0.6% / code +2.4% |
| dual-dflash.yml    | 81.94 / 124.93             | 77.12 / 125.97 (CV 2-4%)      | narr -5.9% / code +0.8% |
| dual-dflash-noviz  | 78.19 / 126.99             | 78.94 / 123.18 (CV 2-3%)      | narr +1.0% / code -3.0% |

dual-dflash narrative is the only delta outside CV (-5.9%); could be
substrate (different driver / power state at chart capture) or a minor
DFlash N=5 spec-decode regression on v0.20. Chart value stays — within
±5pp of measured, within bench noise band.

Both dual.yml and dual-dflash* are "Genesis-less by design" (zero Genesis
env vars). The migration only bumped the image SHA on these — no env-var
changes — which is consistent with the flat result. The +50% TPS jump
on TQ k8v4 we reported to Sander in discussion #19 came from enabling
his full PROD env-var stack on the TQ KV path; fp8 / FP16 dual paths
don't share the same patches and don't exhibit a similar bump.

Updated `docs/DUAL_CARD.md` performance summary table with the new
measured numbers + a per-variant Δ column. Per-config summaries written
to `results/v0.20-migration/dual-{yml,dflash,dflash-noviz}.summary`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 19:34:28 +00:00
noonghunna
077228e81b results: dual-turbo re-bench with corrected env vars (PN22 / PN26 naming fix)
Post-merge follow-up to PR #23. Captures the dual-turbo TP=2 bench after the
env-var naming fixes (PN22 / PN26 sub-config) actually fire, vs the prior
bench where they were silently no-op.

Honest finding: PN22's advertised "+9-30% TPS on TP>=2" (per PR author of
vllm#39419) doesn't materialize at our small bench sample sizes (n=2 runs/
stream, n=4 streams = 8 total runs at high CV).

| Metric | Prior (PN22 silently OFF) | Fixed (PN22 firing) | Δ |
|---|---|---|---|
| n=1 code wall_TPS    | 76.01 | 76.71 | +0.9% (within CV 4-5%) |
| n=4 aggregate        | 269.03 | 242.94 | -9.7% (within bench-size variance) |
| PN22 boot status     | "PN22=1" matches no env_flag — silent OFF | APPLY (vllm#39419 backport) |
| PN26 sparse-V status | BLOCK_KV / NUM_WARPS / THRESHOLD silently using defaults | 27B-tuned values applied |

The naming corrections are real bugs regardless of TPS impact — PN22 / PN26
sub-config are now actually firing on master. Effect may be more visible at
sustained high batch sizes (Sandermage's PROD bench at 100t × 50-req sustained)
where local-argmax dominates the draft-path latency budget more than at our
single-shot 800-token bench.

Headline numbers from PR #23 (76.01 code / 269 aggregate at n=4) remain the
right reference for docs and charts — re-bench is within noise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 18:56:35 +00:00
noonghunna
5aa97a25d9 v0.20 migration + Genesis v7.65 dev tip + cold-start cache + env-var alignment
This branch migrates the entire vLLM stack from `dev205+g07351e088` + Genesis
v7.64 to `0.20.1rc1.dev16+g7a1eb8ac2` + Genesis v7.65 dev tip (commit
`d89a089`). v7.65 is on Sandermage's `dev` branch — explicitly the cross-rig
testing surface he requested in discussion #19; he'll merge dev→main once we
both confirm stable. Pin gates restated when that lands.

What changes
------------

Pin migration:
- vLLM image: nightly-07351e08... → nightly-7a1eb8ac2... (dev205 → v0.20.1rc1.dev16)
- Genesis: 64dd18b (v7.64) → d89a089 (v7.65 dev tip)

Sidecar churn:
- DROPPED: patch_pn12_ffn_pool_anchor.py (PN12 native on v0.20)
- DROPPED: patch_pn12_compile_safe_custom_op.py (Genesis P38B in-source hook)
- DROPPED: patch_fa_max_seqlen_clamp.py (Genesis PN17 + P15B)
- ADDED: patch_workspace_lock_disable.py (relaxes vllm#39226 strict assertion;
  P98 covers same surface but auto-skips on v0.20 due to drift-marker false
  positive — pending Sandermage marker fix)

Env-var alignment to Sandermage's PROD set (start_27b_int4_TQ_k8v4.sh@dev):
- FIXED naming bugs that silently no-op'd patches:
  - PN9_INDEPENDENT_DRAFTER_ATT → _ATTN (was silently OFF)
  - PN22 → PN22_LOCAL_ARGMAX_TP (was silently OFF)
  - PN26_BLOCK_KV → PN26_SPARSE_V_BLOCK_KV (fell back to default 4, not 8)
  - PN26_NUM_WARPS → PN26_SPARSE_V_NUM_WARPS
  - PN26_THRESHOLD → PN26_SPARSE_V_THRESHOLD (fell back to default 0.001, not 0.01)
- ADDED explicit-OFFs to match Sander's PROD verbatim:
  - P78_TOLIST_CAPTURE_GUARD=0 (we use our own patch_tolist_cudagraph.py)
  - P81_FP8_BLOCK_SCALED_M_LE_8=0 (FP8-specific, no-op on TQ3)
  - P82=0, P82_THRESHOLD_SINGLE=0.3
- Cap divergence (justified): PROFILE_RUN_CAP_M=4128 + PREALLOC_TOKEN_BUDGET=4128
  (Sander uses 4096 — vLLM `interface.py:639` forces our config's Mamba
  block_size to 4128 due to TQ3 + TP=1 page-size math; lower values
  AssertionError at boot)
- Carry-forward (intentional): P4 (hybrid TQ required), P65 (TQ spec-CG
  downgrade — pending v0.20 verification that #40880 closure makes it
  redundant)

Cold-start cache mounts (closes #22):
- All 10 composes now mount torch_compile_cache + Triton cache from
  `models/qwen3.6-27b/vllm/cache/`. First boot warms (~6 min); warm boot
  drops to ~3.2 min (47% faster). Per-stage savings on long-text:
  - Dynamo bytecode transform: 18s → 5s (-73%)
  - torch.compile: 57s → 9s (-85%)
  - Initial profiling/warmup: 51s → 7s (-87%)

Mamba block_size cap fix:
- v0.20 enforces `long_prefill_token_threshold >= block_size`; on hybrid
  Mamba+TQ3, vLLM forces block_size=4128. Bumped GENESIS_PROFILE_RUN_CAP_M
  and PREALLOC_TOKEN_BUDGET 4096→4128 across all 5 main composes.

Default 48K compose:
- Required workspace_lock_disable sidecar after initial v0.20 boot hit
  vllm#39226 strict assertion. Caught during validation, fixed.

Context restored vs dev205 backoffs (validated 33K + 50K stress on v0.20):
- long-text:        185K → 214K (+16%)
- long-vision:      140K → 198K (+41%)
- bounded-thinking: 185K → 214K (+16%)

Bench results (n=5, results/v0.20-migration/):
- long-text 214K        narr 49.74 / code 67.39 (CV 2.6/2.7%)
- long-vision 198K      narr 50.32 / code 66.12 (CV 2.3/4.1%)
- bounded-thinking 214K narr 49.77 / code 65.80 (CV 1.4/2.3%)
- tools-text 75K (fp8)  narr 53.32 / code 69.66 (CV 2.3/1.4%)
- dual-turbo 262K (TP=2) narr 58.33 / code 76.01 per-stream
                         269 TPS aggregate at n=4 streams (3.63x speedup)
- default 48K           narr 48.82 / code 65.98 (n=3)

Validation: verify-full 8/8 on every variant. verify-stress 33K AND 50K
tool-prefill PASS on every variant — the cliff that fired on EVERY dev205
config no longer reproduces.

Docs + charts:
- README + SINGLE_CARD + DUAL_CARD + CLIFFS + EXAMPLES + STRUCTURED_COT
  + FAQ + UPSTREAM + 3 engine docs + model README + INTERNALS + CHANGELOG
  all updated with new pin, ctx, TPS numbers, and "v0.20 unblock" section
- performance.{png,svg} + variants regenerated with measured TPS
- vram-budget.{png,svg} + variants regenerated with measured VRAM
- UPSTREAM tracker: 5 issues moved  closed (PR #12, #13, #14, #15, P104
  superseded by PN17 + P15B)

Issues addressed:
- #16 (Cliff 1 mech B leaks past PN12 on inductor-compiled FFN) — partial:
  v0.20's revised TQ FA paths close the synthetic stress; PN25 (Sander's
  proper compile-path opaque-op fix) is on dev but explicitly opt-in pending
  worker-fork registration fix. Workarounds documented (tools-text fp8 path
  / --enforce-eager) until Sander ships PN25 default-on.
- #20 (launch.sh port + container-name mismatch) — already closed by
  77ca576 (post-issue-filing).
- #22 (cold-start caching) — closed by cache mounts above.

Remaining caveats:
- Cliff 2 (DeltaNet GDN forward, single prompt ≥50-60K) unchanged —
  architectural, applies to all single-card vLLM TQ3 paths. Mitigation:
  dual-turbo TP=2 (state splits across cards) or llama.cpp 262K.
- Default 48K narr_TPS (48.82) slightly under chart's 55 reference —
  bench variance + sample size n=3; not regression.
- Dual.yml / dual-dflash* not re-benched on v0.20; numbers carry forward
  from dev205 (fp8 paths were not TPS-changed by the migration).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 18:33:01 +00:00