Codifies the policy we already apply: rolling tag for unpatched engines
(llama.cpp :server-cuda), pinned nightly digest for engines where we vendor
overlays (vLLM Genesis sidecars / Marlin pad / INT8 PTH / DFlash). New
table per engine + rule for adding/dropping patches in the same commit
as the pin/unpin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to commit 1b2a76c which added the speed-vs-context section to
models/qwen3.6-27b/llama-cpp/README.md. The new learning (smaller -ub
buys more ctx, especially when mmproj F16 is loaded) needs to be
discoverable from the high-level user-facing docs too — not just buried
in the model-specific README.
- SINGLE_CARD.md: corrected the "49K is the safe-headroom max with mmproj"
claim (it's the speed-optimal max; ub=512 lever pushes to 192K) +
prominent override snippet right under the profile description.
- CLIFFS.md: added a 2026-05-20 refinement note pointing out that `-ub`
is doing two jobs (cliff-survival + KV-budget), and the optimal value
is configuration-conditional on vision-on/off.
- FAQ.md: inline note in the canonical switch.sh recipe for mtp-vision
pointing users at the override.
All three back-reference the README's full sweep data as the canonical
source; surfaced by @JensJN in #170.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to JensJN's question on #170 about pushing mtp-vision beyond 49K
context. Validated end-to-end 2026-05-20:
Config ctx VRAM narr TPS verify-stress
shipped (ub=1024) 49K 22.0 GB 56.5 ✓ (shipped)
override (ub=512 CTX=131072) 131K 21.0 GB 50.0 ✓ 7/7
override (ub=512 CTX=196608) 192K 22.5 GB 50.9 ✓ 7/7
~10% TPS hit buys 4× more context for agentic vision workloads. Keep
defaults stable (don't break shipped TPS numbers), but document the trade
+ give users explicit env-override examples so they can rebalance for
their workload.
Also corrects a stale `UBATCH_SIZE` default in the table (showed 2048,
actual is 1024 since v0.8.3 cliff-survival re-tune) + adds CTX_SIZE +
KV_TYPE to the table for completeness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to 4a53eda. With llama.cpp now on a rolling :server-cuda tag, the
plain image tag in report.sh (e.g. "ghcr.io/ggml-org/llama.cpp:server-cuda")
no longer pins down which bytes the user is running. Add docker-inspect
queries for image digest + OCI labels — every well-behaved upstream image
(both ggml-org and vLLM) ships these, so bug reports become reproducible.
Surfaced by zemaphore in discussion #170 after the rolling-tag switch:
their report would have left us guessing about build number.
New fields:
- Image digest: immutable sha256 of the actual bytes pulled
- Build tag (OCI version): e.g. "b9246" for ggml-org/llama.cpp
- Upstream commit (OCI revision): e.g. "871b0b70f..." git SHA from the image
- Upstream source: e.g. github.com/ggml-org/llama.cpp
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up on c3e7c7e: pinning to a specific build (b9246) was cargo-culted from
our vLLM pattern, but the vLLM pinning serves a real purpose (Genesis-patch
anchor, Docker Hub purge resistance) that doesn't apply here. llama.cpp on the
club-3090 stack is stock upstream — no patches, no Genesis equivalent — and
GHCR tag retention is more reliable than Docker Hub.
Switch to rolling `:server-cuda` tag so users automatically get MTP improvements,
EAGLE3 fixes, kernel updates from upstream without us being a bottleneck.
Override path preserved via `LLAMACPP_IMAGE` env if a future upstream build
regresses and a user needs to pin reactively.
Bench numbers in #170 footnoted as "measured on b9246"; expect ±5% drift on
newer builds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v0.8.3 shipped composes (llamacpp/default, llamacpp/mtp, llamacpp/mtp-vision) all
reference `image: llama-cpp:local`, a custom image that exists ONLY on the
maintainer's rig. There is no Dockerfile, no build script, and no setup.sh hook
to produce it for users. Anyone running `bash scripts/switch.sh llamacpp/mtp`
on a fresh clone hits "image llama-cpp:local not found" and dies at boot.
The custom image was a v0.8.3-dev artifact from when MTP PR #22673 was bleeding
edge. The official upstream `ghcr.io/ggml-org/llama.cpp:server-cuda` now has it
merged (build b9246 = commit 871b0b70f, 2026-05-20) — pinning to b9246 reproduces
the v0.8.3 numbers (50.25 narr / 58.04 code on single 3090, vs shipped 51.28/59.72).
Surfaced by @zemaphore in discussion #170. README.md was also lying: claimed
"both use the official ghcr.io image, no custom build needed" while composes
referenced llama-cpp:local.
Override the pin via `LLAMACPP_IMAGE=ghcr.io/.../server-cuda-bXXXX` env if you
want to follow upstream master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Post-v0.8.3 the llama.cpp profile family lands containers on :8020
(llamacpp/default/mtp) and :8030 (llamacpp/mtp-vision). The status block was
missing :8020 entirely and hardcoded :8030 as "gemma-mtp / Gemma 4 31B".
Add :8020 probe, and on :8030 detect the active container so the label tracks
whichever engine (vLLM Gemma vs llama.cpp Qwen) is actually serving.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
X-axis labels were still overlapping at 11 single-card configs even with
rotation + tighter labels. Restructured: labels show just the compose name
(rotated 30°), full descriptions render in a grouped legend block below.
Moved the narrative/code bar-color legend inside the chart (upper-left,
framed) to free the bottom margin for the description block + substrate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The new llamacpp/* entries from v0.8.3 had compound labels that overlapped
neighbors at 13" figure width. Normalize all single-card llama entries to
the same `<name> <ctx>\n<modifier>` pattern the vLLM entries use, drop
redundant tokens (variant name already implies MTP/vision/default).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Conceptual narrative (hardware → engine → model size → quant → templates)
that the doc set lacked: GETTING_STARTED is commands-only, GLOSSARY is
isolated definitions, FAQ is troubleshooting. Cross-linked from README,
GETTING_STARTED, and GLOSSARY. Carries a scope banner (stack is
NVIDIA/3090-tested; other vendors are context, not a support commitment)
and an RTX 3090 FP8 caveat (no FP8 weight compute on Ampere; AWQ/GPTQ
for weights, fp8_e5m2 KV only). Links to PULL.md / KV_MATH.md for the
stack's actual fit-math path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ARCHITECTURE.md described the v0.8.0 pull→gate→emit→boot→loop pipeline but
not v0.8.2's user surfaces. Adds: the --recommend verdict UX, the
--submit-last/--submit failure on-ramp, the broadened arch registry
(native built-ins reach a clean serve verdict, per-repo remote-code still
fail-closed), and the optional non-NVIDIA hwdetect subprocess; scripts-tree
pull.sh line annotated. README universal-pull bullet refreshed
(v0.8.0; extended in v0.8.2 — recommend + on-ramp + wider coverage).
Docs only; no behaviour change.
The compose generator copies --max-model-len / --gpu-memory-utilization /
--max-num-seqs / KV dtype verbatim from the captured reference profile and
does not solve fit for the user's actual GPU (smaller card may not boot;
larger card wastes headroom; derived path keeps native bf16 KV). This was
a deliberate scope decision ("reproduce + flag, NEVER repair") whose
user-facing consequence was previously implicit. COMPOSE_GENERATOR.md gets
a dedicated subsection; PULL.md Path-A gets a callout pointing users to
--recommend / kv-calc --solve-max-ctx for the real fit and the
env-overridable MAX_MODEL_LEN. Documentation only; no behaviour change.
The v0.8.2 release branch ships the four pull CONTRACTs plus two
orthogonal non-pull items that landed on the same branch: N-GPU NVLink
auto-detection and a documentation restructure. The prior headline +
readiness ledger described a CONTRACT-only scope, which under-claimed
the actual shipped reality. Correct both to state the true bundled scope
honestly without over-claiming; the GGUF §9 cross-engine design-unlock
deferral statement is preserved verbatim.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_render_recommendation keyed solely on res.ok, so a confirm→proceed /
override-accepted terminal (raw_verdict=fits-clean, res.ok=False because
the run needs an explicit --yes/--force-download) fell into the generic
"DOES NOT FIT / BLOCKED" branch — dishonest by imprecision (the model
fits; only acceptance is pending; CONTRACT-4 is "honest recommendation —
fits?"). Add a presentation-only needs-acceptance classification: a
fits-clean acceptance terminal now renders "FITS (estimated) — NOT YET
ACCEPTED" with the acceptance-gate guidance (not the failure on-ramp);
genuine hard-blocks still render DOES NOT FIT / BLOCKED. Pure
presentation — still derived only from res, no decision logic. Caught by
the V5 on-rig gate (microsoft/phi-2 --dry-run). test-pull.sh rec(2a)
updated to assert the honest rendering; suite 25/25, kv-calc N/N.
CONTRACT-4: add `--recommend` — an honest aggregated recommendation that
is PURE presentation/aggregation over the SHIPPED run_pull verdict. Every
line is read straight off the real PullResult (ok/confidence/raw_verdict/
terminal/stratum/abort_reason/notices/emitted); it introduces no decision
logic and does not change the exit code. Carries the §7 boot-fit≠runtime
caveat + soak-continuous pointer ONLY when the gate itself marked the run
boot-fit-satisfied (echoed from res.notices, never re-derived), states
which gate decided, is vLLM-only by construction, and never implies a
non-emitted artifact (the compose line appears only when res.emitted).
CONTRACT-1 user doc: docs/PULL.md gains a "Report a failed pull" section
documenting the SHIPPED V1/V2 on-ramp (capture-on-hard-block → surfaced
pointer → scripts/pull.sh --submit-last / --submit <dir>, consent prompt,
gh + gh-less). Every documented command/flag/output string was verified
verbatim against the live shipped CLI on this branch (docs-fidelity RED-
LINE). Leak-clean: only repo-relative .pull-captures/<slug>/<ts> forms,
no absolute paths.
§9-reconciliation: the release headline AND the readiness ledger in
docs/PULL.md now state explicitly that GGUF is deferred to a §9 cross-
engine design-unlock proposal, and that v0.8.2's scope is the failure
on-ramp + registry-expansion + whichllm-hw-detect + recommend — same
location/pattern v0.8.0 used for its §9-headline reconciliation.
Zero decision-logic change: gates.py / deriver.py / capture.py /
loop_input.py / classifier.py / dedup.py / submit_pull.py / hwdetect.py /
failure_fingerprints.yml / arch_patches.yml all byte-unchanged. pull.py
is a pure addition (zero removed lines): a new _render_recommendation()
function + a --recommend flag + one presentation-only call site.
test-pull.sh adds the CONTRACT-4 V5 section asserting the recommendation
TRACKS a real differing verdict — four genuinely-different real outcomes
(fit+emitted / confirm→proceed-blocked / estimated-lower-bound-fit /
hard-block) render four pairwise-different blocks, each matching its own
real res; rig-independent leak assertion (str(root) absent), not a
substring allowlist. Full shipped suite 25/25 green in the CI condition;
kv-calc --calibration 11/11 unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CONTRACT-3 §8: an OPTIONAL, bounded subprocess that augments hardware
ENUMERATION for the eval path where nvidia-smi does not apply (AMD ROCm /
Apple / other-vendor). Strictly detect-only; never feeds kv-calc (kv-calc
stays the sole fit authority); no new hard dependency.
New isolated leaf module scripts/lib/profiles/hwdetect.py:
- detect_non_nvidia_hw()/detect_non_nvidia_sm(): bounded `whichllm list
--json` subprocess, defensively parsed into a structured HwDetectResult;
maps a recognised non-NVIDIA device class to an SM-equivalent for the
[C0] SM gate ONLY.
- Every non-delivery path (tool absent / failed / timeout / unparseable /
NVIDIA-only / unrecognised) degrades to None and NEVER raises out.
Additive consume-point wiring in run_pull (the eval path): a new optional
`hwdetect_fn` kwarg, consulted ONLY inside the existing
`if hardware_sm is None:` stratum-3 block — i.e. only when nvidia-smi
already returned nothing. The NVIDIA majority never enters the seam, so
that path is byte-identical whether the augment is absent OR
present-but-degrading. On a recognised non-NVIDIA device the eval path
gets an SM-equivalent (the [C0] SM gate runs instead of the blind-refuse
`hardware-sm-undetermined` terminal) plus an additive notice/diagnostic;
no shipped decision field is mutated and kv-calc is not consulted.
BOTH RED-LINE halves covered + proven by scripts/tests/test-hwdetect.sh:
(a) safety — optional/no-hard-dep, graceful degrade, NVIDIA-path
byte-identity, never feeds kv-calc; (b) delivery — a simulated
(explicit, deterministic) non-nvidia env yields a structured enumeration
the eval path observably consumes (outcome moves OFF the degrade
terminal). Rig-independent leak assertion (str(abs_dir) not in shared).
All 9 shipped v0.8.0/V1/V2/V3 decision modules byte-unchanged
(dedup.py:262 FInput.dedup_hash(_EffProxy()) idiom fenced/untouched).
Full scripts/tests/test-*.sh suite green in the CI condition (25/25);
kv-calc --calibration unchanged (11/11).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first expansion marked ALL added arches requires_trust_remote_code:
unverified, so a registry-recognised model only moved no-arch-row ->
needs-trust-remote-code-ack — a lateral relabel, NOT the "materially more
models pass [C0] engine-supported" CONTRACT-2 requires (net newly-passing:
zero; caught on-rig via microsoft/phi-2).
Two-class TRC posture: long-standing native vLLM built-in classes (no
remote code — a documented upstream constraint) carry
requires_trust_remote_code:"false" with a documented-constraint evidence
anchor; arch families with genuine remote-code lineage
(Phi3SmallForCausalLM, InternLM2ForCausalLM) stay unverified/fail-closed.
Zero-false-pass preserved: gates.py's has_auto_map is an INDEPENDENT
OR-term, so a repo shipping auto_map still hard-blocks needs-trc-ack
regardless of the row flag — "false" removes only the arch-row-level
over-refusal, never the per-repo trust boundary.
On-rig (2026-05-18): microsoft/phi-2 (no auto_map) -> engine-supported
clean; PhiForCausalLM+auto_map -> needs-trc-ack; absent arch ->
no-arch-row; Phi3Small/InternLM2 -> needs-trc-ack. Suite 24/24,
kv-calc 22/22. test-pullgate-gates updated to the two-class invariant.
switch.sh now DERIVES its VARIANTS + VARIANT_DEFAULT_PORT tables from
compose_registry.py (the single source of truth) instead of a hardcoded
`declare -A` map that had drifted: 20 registered composes (incl.
vllm/dual-int8 — shipped as dual/int8.yml but unlaunchable, which cost a
real A/B a config pivot) were not launchable. All 42 registered composes
are now launchable; zero launcher-only ghosts.
New deterministic test-switch-registry-parity.sh (no docker/GPU/network)
fails CI on ANY registry↔launcher mismatch in EITHER direction: registry ⊆
launcher (zero registered-but-unlaunchable), launcher ⊆ registry (zero
ghosts) — driven through the FULL shipped `switch.sh --list` path so a
manual post-derivation ghost is caught too — plus spec parity, port parity,
and every resolved compose file exists on disk. Negative-case verified: a
synthetic registry/launcher mismatch makes the test exit 1.
Additive: no [C0]/decision-logic change; no shipped compose touched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CONTRACT-2 (§10-R4) arch-family registry expansion: +13 safetensors arch
rows in arch_patches.yml (PhiForCausalLM — the microsoft/phi-2 STEP V1
on-rig no-arch-row anchor — Phi3Small, Gemma/Gemma3/Gemma3-CG, Starcoder2,
Cohere, InternLM2, Mixtral/Qwen2Moe/Qwen3Moe MoE, Qwen2-VL). Additive data
only, zero [C0]/decision-logic change. Zero false-pass by construction:
each follows the established estimated-lower-bound/unverified-TRC precedent
so [C0] still resolves needs-trust-remote-code-ack (fail-closed, bypassable
ONLY by --trust-remote-code) — the expansion drops only the
--experimental-arch requirement, never auto-passes; an arch still absent
still hard-blocks no-arch-row. test-pullgate-gates.sh proves both, plus the
#146-shape worked acceptance case (a hand-added awq_bf16_int4 weights
variant the expanded flag schema/parity machinery absorbs cleanly).
CONTRACT-2b-i chat-template attribution + behavioral drift_guard: new
`chat_template` delivery class (VALID_DELIVERY_MECHANISM); froggeric (22
composes — 18 direct + 4 nvlink* via REAL Docker Compose extends: merge)
and carnice (mount-only) brought under load_bearing_when + a behavioral
drift_guard whose check encodes the self-contained symmetric restart+settle
protocol (identical docker restart both arms, /v1/models healthy, 60s
settle, >=3 bench runs/arm, grand-mean same-segment compare, flag only a
3/3 deterministic regression). Effective coverage uses REAL merge
semantics: docker compose config (preferred) or a deterministic offline
extends: merge applying the same rules (additive sequence merge; `!reset`
removal) — never the unsound single-base text concat. .jinja artifact
discovery catches an orphan vendored template. test-patch-attribution.sh
adds the class checks + an H4 fixture asserting a `!reset` child AND a
stopped-extending child both lose coverage (the false-negative is the
dangerous direction). Generator emit kept in lock-step with reaches().
Documented as PATCH_POLICY.md §3.1. Rig-independent leak assertions added
(str(abs_dir) not in shared; repo-relative-only — never a /opt|/home
substring allowlist).
RED-LINE: gates.py/pull.py/deriver.py/capture.py/loop_input.py/
classifier.py/dedup.py/submit_pull.py/kv-calc.py/failure_fingerprints.yml
byte-unchanged; no shipped compose changed; patch_attribution.py c0_state/
is_artifact/compose_text/service_body byte-identical (additive only). Full
test-*.sh suite green in the CI condition; kv-calc --calibration N/N.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The gh-less paste fallback embedded the absolute bundle dir into the
PUBLIC issue body ("full redacted bundle at `/abs/.../.pull-captures/...`"),
violating the acceptance that nothing the on-ramp tells a user to share
contains an unredacted absolute path. Render a repo-relative
`.pull-captures/<slug>/<ts>` pointer instead. Strengthen the gh-less
leak assertion to a rig-independent check (the absolute bundle dir must
not appear; only the relative pointer may) — the prior /opt|/home check
passed under a tmp sandbox dir and missed this.
CONTRACT-1.2: pull prints the honest one-line on-ramp pointer whenever a
gate bundle was emitted for the run, keyed on the V1-recorded capture dir
— explicitly NOT gated on the exit code (the bypassable no-arch-row C0
advisory path exits 0 yet emits the #1 §10-R9 bundle). Gate path stays
I/O-free: a single stdout line, no network/prompt/auto-send. It does not
classify (suppression is loop-side at submit).
CONTRACT-1.3: scripts/pull.sh --submit-last / --submit <dir> is a distinct
top-level verb parsed before the slug/--profile-like requirement.
--submit-last re-reads the V1 shared .last marker at submit (the race
defense — surfaces the CURRENT bundle, never a silent wrong-bundle).
Re-shows bundle identity + the exact already-redacted payload, requires an
explicit y before any network, then reuses the shipped F5 dedup.submit
(effective_dedup_hash, bounded loop:dedup-<hash> labels, +1-or-open,
collision-safe verify, suppression/review-queue) — not reimplemented.
gh-less fallback runs post-F2 classification, gated on should_file:
should_file=True -> a prefilled public issues/new URL with the
loop:dedup-<hash> label and the deterministic title template; review-
queued (unknown / correct-refusal) -> the local _review-queue spool path
and the no-public-issue line, with NO public issues/new URL. Never raises;
degrades to the local spool + printed paste-path. Console is never a
submission source — only the redacted artifact is emitted.
New scripts/tests/test-submit-pull.sh (mocked gh, zero network): the
.last-marker race re-read, the bundle-emitted-but-exit-0 surfacing,
F5-reuse, the gh-less should_file branch with no public URL for review-
queued, gate-path I/O-free, and leak-hygiene. Full shipped suite green in
the CI condition; kv-calc --calibration unchanged at 22/22; safetensors
decision path byte-unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CONTRACT-1.1 capture-on-hard-block (additive only; zero v0.8.0 decision-logic
change — the safetensors/GGUF paths are byte-unchanged):
- capture.py: new SEPARATE emit_gate_capture() (the emit_override_capture
byte-preserving precedent — NOT invoked by emit_capture()) writing a
pt1-gate.json + schema:2 manifest.json (outcome:hard-block, exact shipped
abort_reason, failure_class:null) per the per-abort-stratum key table
(model/arch/quant null pre-deriver; topology best-effort/nullable, capture-
only resolve; post-C0 always null). New shared write_last_marker() helper
(atomic tmp+os.replace) called from BOTH emit_capture() and the gate
emitter (centralization mandate — gate-only is the commonest failure).
- pull.py: pass-through capture on the 7 terminal hard-block return paths
(deriver / profile-like / hardware-sm-undetermined / C0 / C2a / no-fit-model
/ C1) — emits a bundle before the existing `return res`; the decision is
byte-unchanged; injectable gate_capture_fn; never raises.
- loop_input.py: BaseCaptureBundle typing.Protocol (Optional[dict] pt2-5);
FInput satisfies it by construction (verified: no isinstance(finput,FInput)
anywhere in F2/F5 — pure static retype, schema==1 byte-identical incl.
dedup_hash); new FInputGate + read_gate_bundle() (schema==2; validates ONLY
the always-present row + outcome==hard-block + failure_class is None — does
NOT reuse the 22-key validator); FInputGate.dedup_tuple() uses .get(k,None)
(behaviour-neutral schema-1, crash-safe schema-2, deterministic null-topo).
- classifier.py / dedup.py: F2+F5 parameter annotations retyped FInput ->
BaseCaptureBundle. The dedup.py FInput.dedup_hash(_EffProxy()) unbound-class
idiom is FENCED (unchanged — not tidied). Additive gate_abort_reason
_match_condition kind (reads pt1_gate.abort_reason; bool like sibling kinds;
no enum/routing change).
- failure_fingerprints.yml: seeded gate_abort_reason rules keyed on the
verified shipped abort_reason strings — only engine-support-unknown/
no-arch-row -> kernel-unsupported (public-filed); runtime-incompatible /
disk-short / hard-block / catch-alls -> unknown (review-queued, not filed).
- tests: extended test-{pull,pullemit-capture,loop-input,classifier,dedup}.sh
with the V1 RED-LINE proofs (emit_capture() still writes ONLY pt1-4; a
schema==1 bundle yields byte-identical FInput / ClassificationResult /
dedup_hash / effective_dedup_hash pre/post the protocol lift; the dedup
fence holds) + gate-emitter / read_gate_bundle / gate_abort_reason routing
/ shared .last marker coverage; all 22 shipped test-*.sh green in the CI
condition (gitignored .pull-captures absent), kv-calc --calibration 11/11.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
deriver.py:343 and :743 told users GGUF/.bin is "not supported until
v0.8.1". v0.8.1 has shipped (the fix/docs-fidelity stack) and GGUF was
deliberately de-scoped from the v0.8.2 feature work too (cross-engine
serving = a deferred §2/§9 design-unlock, not a near-term version). The
strings actively mislead users on master ("wait for v0.8.1" — which
exists and won't add it). Re-anchored both to accurate, version-free
wording: GGUF/.bin not supported — this path is vLLM + safetensors only.
String-only; zero decision-logic change. Surfaced by the v0.8.2 brief
r1 review (Major finding). Same docs-fidelity class as the v0.8.1 stack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Counter-frames duart's #161 ("Proxmox needs HugePages/pinning, 3x"):
our reference rig runs under Proxmox PCIe-only, no NVLink, untuned, at
full dual baselines — out-of-box Proxmox is not a tax. The fragile
element is NVLink across passed-through GPUs collapsing to a slow
fallback on wrong IOMMU/ACS/NUMA; PCIe-only has no such path. Cross-ref
#137 (NVLink-not-engaging under passthrough). Narrow, accurate framing
— NOT a mandatory-tuning guide; report.sh --full is the diagnostic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
launch.sh formats kv-calc.py JSON dot-decimals ("9.0") with printf
"%.2f". Under a comma-decimal LC_NUMERIC locale (de_DE etc.) bash
printf rejects the dot — `printf: 9.0: Ungültige Zahl` — and the
launcher aborts at the VRAM-budget print. Export LC_NUMERIC=C early
(LC_CTYPE/encoding untouched, UTF-8 UI glyphs still render). Reported
by @chrischd83 (German locale) in discussion #159.
bash -n clean; export precedes all executable printf; LC_NUMERIC=C
printf "%.2f" 9.0 -> 9.00 verified. Exact de_DE failure not
reproducible on our rig (no comma-decimal locale data installed) —
mechanism is textbook; reporter on de_DE is the definitive validator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First dual-3090 PCIe-x4 data point. 76.15/117.31 wall (decode
76.94/121.75), 200K, ~23.8 GB/card, 2026-05-18. Captioned accurately:
the aikitoria patched driver is installed host-side but the run used
the stack-default NCCL_P2P_DISABLE=1 + custom-all-reduce OFF (boot log
confirms) — P2P NOT engaged, so this is an x4-lanes/P2P-off number,
NOT a patched-P2P anchor. Only −2% narr / −8% code vs the x16-ish
baseline → dual-dflash-noviz is far less interconnect-bound than the
x4 warning implies. Also the first in-the-wild exercise of the #156
lspci report subsection (sudo-less degradation rendered correctly).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Maintainer rig A/B 2026-05-18 (today-anchored, vllm/dual, identical
engine, template-only delta): hermesagent-20 +10pp (50→60%, +2 scen),
7 other packs flat, streaming tool-calls OK both arms (#145 clear),
soak-continuous PASS 0/25 both arms. TPS-neutral: symmetric
restart+settle 3 bench/arm, CODE 89.06→87.62 (−1.6%, within noise),
NARRATIVE flat — the first-pass −7% was an asymmetric-protocol
artifact, not reproduced. Reproduces the prior froggeric finding on a
clean today-anchored baseline; #35936 control confound resolved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-vendors the froggeric Qwen3.6 chat template from the latest upstream
snapshot for maintainer re-evaluation only.
Provenance:
- Previous upstream revision: unrecoverable. The existing local snapshot was
introduced by local commit 84498d47aa with
SHA256 94e944287ffaf8c3ed8b5840a0c92fd4ca3caefa721f4f5e31e92605e63f1ad4,
but no exact upstream commit/release match was found in available froggeric
qwen3.5/qwen3.6 archive v8-v19 or main history.
- New upstream revision: c31fd393e531dbacd92b6deb99a2037cc949f950,
timestamp 2026-05-16T13:44:07Z, release label v19,
SHA256 4649b3fa3db3fda4d51173ed4ff0175fde7ece8bbceb9d595d04d862020c9746.
Structured diff summary:
- System/tools setup: system/developer content is now extracted once and
reused for tool and non-tool paths; tool definitions render function payloads
rather than raw tool wrappers when present.
- enable_thinking gate: ns_flags now tracks has_tools, last_tool_failed, and
consecutive_failures in addition to enable_thinking. The template still
honors enable_thinking plus think_on/think_off markers, but preserve_thinking
now defaults toward preserving non-empty reasoning unless explicitly false.
- Tool-call and streaming-sensitive formatting: tool instructions now require
reasoning inside a think block before a tool call, final answers immediately
after closing think when no tool is needed, and one fully closed tool_call per
function. Tool-call emission is normalized to newline-started XML blocks.
No explicit streaming branch was added, but these boundary/whitespace changes
should be covered by the #145 streaming/tool-call smoke.
- Reasoning delimiters: the template now handles </think>, </thinking>,
malformed </ think>, and </think > endings, removes embedded tool_call
content from assistant text when structured tool calls are present, and avoids
emitting empty historical think blocks.
- Qwen 4: no Qwen 4-specific branch was found in the upstream template.
- Whitespace/Jinja control: multiple spacing and newline changes affect
system/tool prompts, think/content boundaries, tool_call boundaries, and the
thinking-disabled generation prompt. Treat tokenization and prefix-cache
behavior as validation targets.
Validation:
- bash scripts/tests/test-patch-attribution.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EXAMPLES.md asserted thinking is on by default (lines 22/43/228), but
every shipped Qwen3.6 compose sets
--default-chat-template-kwargs '{"enable_thinking": false}'
(bounded-thinking.yml is the only exception). Same docs-vs-shipped
class as the v0.8.0 docs-fidelity gaps. Corrected the 3 inaccurate
spots, added a canonical "thinking is OFF by default + how to enable
per-request + bounded-thinking exception" note under the max_tokens
table. Consistent with the disc #151 public answer and the
enable_thinking-default rationale; does not pre-judge the #150
froggeric re-eval. Doc-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
nvidia-smi only reports negotiated PCIe gen/width; it cannot show
trained link state vs capability side-by-side, ACS state on the
upstream bridge, or the real PCIe topology tree — the three things
that actually decide whether GPU↔GPU P2P engages (recurring class:
#137 KVM passthrough, #351 in-container NVLink-not-engaging).
Adds a "PCIe / P2P detail (lspci)" subsection co-located with the
existing nvidia-smi Topology subsection (keeps PCIe data together).
It captures:
- lspci -t — the PCIe topology tree
- per NVIDIA VGA/3D-controller function: lspci -vvv filtered to
only LnkCap/LnkSta/ACSCap/ACSCtl (never the full verbose block,
keeping the report compact + redaction-safe), AND its resolved
upstream PCIe bridge — ACS lives on the bridge, not the GPU
endpoint, and ACS-redirect there is exactly what blocks P2P
- lspci -nnk | grep -A3 -i nvidia — driver binding + device IDs
(spots vfio-pci vs nvidia in-guest)
Graceful degradation, mirroring existing optional captures:
- gated on `have lspci`; one-line skip note if pciutils absent
- uses sudo lspci -vvv when passwordless sudo is available (ACS
is in root-only extended config space); otherwise degrades to
non-sudo lspci (LnkSta still accurate) and prints a one-line
note. Never blocks or prompts.
Output is piped through the existing redact helper and wrapped in
the existing collapsible details block, matching the Topology
subsection's structure exactly.
Refs #137, #351Closes#148
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #154 vendored models/gemma-4-31b/vllm/patches/vllm-pr41800-truncate-prompt-tokens/
to fix#153, but did not add its patch-attribution entry. test-patch-attribution
flags the install.sh as an orphan artifact (lacks patches.yml entry) → rc=1.
The repo has no PR-CI so the merge didn't catch it; master is red on this test.
Caught by the v0.8.1 pre-tag gate (full suite in CI condition) — exactly the
v0.8.0-lesson failure class that per-step verification misses.
Fix: add `gemma-vllm-pr41800-truncate-prompt-tokens` to patches.yml mirroring
the canonical `qwen-vllm-pr41800-truncate-prompt-tokens` entry (model=gemma-4-31b,
the 6 gemma dual compose registry ids that wire it, same delivery_spec/drift_guard/
upstream block), and list it in the Gemma4ForConditionalGeneration arch
required_patches for modeling consistency with the qwen arch. Engine-level overlay,
no behavior change. Full scripts/tests suite + kv-calc calibration 13/13 GREEN.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 Gemma 4 31b dual composes (int8, awq, int8-tq3, dflash, dflash-int8)
bind-mount `../../patches/vllm-pr41800-truncate-prompt-tokens/install.sh`
but commit 1d7aad1 ("vendor ... across all pre-fix engines", #139) wired
the mounts without copying the overlay dir into the gemma-4-31b tree —
only the qwen3.6-27b tree got it. Per the per-model-tree `../../patches/`
convention every other Gemma patch mount already follows, the relative
path resolves into the gemma tree, so the mount source was missing and
`docker compose up` failed on all 5.
All 5 composes route through pre-`d5b31c95` engines (vllm-nightly-full
`e47c98ef` / vllm-nightly-dflash `e47c98ef`/`01d4d1ad`) that genuinely
need the kwarg fix, so dropping the mount is NOT correct — the fix is to
vendor the dir. install.sh is engine-level / model-agnostic, copied
byte-identical from the canonical qwen3.6-27b copy. README's compose
list re-scoped to the Gemma tree + a canonical-source pointer added so
the co-located doc isn't misleading.
Verified: all 5 composes now resolve the bind-mount path and parse via
`docker compose config`; install.sh diff-identical to canonical.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sethbrasile's controlled 9-run matrix on a headless 3090 + driver
595.71.05 / CUDA 13.2 shows the same env-override pattern as the 4090
display-overhead case: the newer driver's vLLM activation-profile
reserve measured ~2.87 GiB vs ~1.5 GiB on the bare-metal reference rig,
shrinking the KV pool and capping long-text.yml at MAX_MODEL_LEN=105000
(vs 180K default). Added as the 3090 sibling anchor next to the
@laurimyllari 4090 -> 90000 data point so newer-driver 3090 users start
from the right number. Tuning-data contribution, not a bug
(corroborates the known Cliff 2a-under-v7.72.2 / genesis#22 picture).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
