* LMCache: in-repo L2 default + LMCACHE_L2=1 toggle, preflight disk-check, corrected sizing
- LMCACHE_L2=1 enables the L2 disk tier at a gitignored repo-root lmcache-kv/ (zero
config); LMCACHE_L2_ADAPTER still fully overrides. L2 stays OFF by default.
- preflight_lmcache_ram now also soft-warns on low L2 disk space at the L2 host dir.
- CORRECTED capacity (measured > estimated): the LMCache offload cache is
~131 KB/token MEASURED (lmcache_mp_l1_memory_usage_bytes 4.93 GB + 4.6 GB on L2
disk, 36,808-token session) — ~7x the GPU's 18.9 int8-PTH rate, which only sets
max servable context. Prior docs used 18.9 for capacity and overstated it ~7x:
--l1-size-gb 30 holds ~4 x 50K sessions (not ~33), <1 full 262K.
- Added a RAM/disk-vs-context sizing table to INTERNALS.md.
- Measured L2 rehydrate: 4.8s vs 43s cold re-prefill (~9x), cross-restart persistence
confirmed; use the fs adapter (not nixl_store — this image's NIXL is broken).
Refs #133.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* LMCache: document fp8 serde is broken on this model (no reduced-size L2 yet)
Measured 2026-06-17: LMCache's fp8 serde shape-errors on this model's KV
('[2,16,1584,260]' invalid — wrong layout for the hybrid GDN/head_dim-256/int8-PTH
source), 60 serialize fails, L2 stays empty. CacheGen-MP is 'coming soon'. So L2 is
uncompressed (~131 KB/token); the fp8-serde toggle was NOT wired (it doesn't work).
Re-test trigger noted. Refs #133.
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Layers an LMCache tiered persistent prefix-KV cache (MP/HMA connector) onto the
dual-max fidelity profile (FP8 + int8-PTH KV + MTP n=3 @262K). Controlled A/B
shows ZERO decode penalty (74 narr / 94 code TPS == without LMCache, MTP intact
~83% accept); reuses cached prefix KV across long / multi-session workloads
(cold->warm TTFT ~7-8x) instead of re-prefilling.
- New vllm-lmcache engine profile: lmcache/vllm-openai DIGEST-pinned (the tag is
mutable — bundled vLLM 0.22.1-dev then 0.23.1-dev under the same tag).
- preflight_lmcache_ram guard: rejects --l1-size-gb over-allocation even under
--force (the l1=100-on-94GB host-OOM that forced a reboot, #133).
- env-gated L2 disk tier hook (LMCACHE_L2_ADAPTER, off by default).
- INTERNALS.md RAM/disk sizing section; registry entry + count bumps; full
guard suite green.
Incubating (hidden, --force to launch): runs LMCache's third-party image with a
newer vLLM than our v0.22.0 pin; L2 latency unmeasured on-rig; 38 GB on-demand.
Refs #133.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
adds an optional `revision:` key per weights variant. weights.py emits
WEIGHT_REVISION; setup.sh threads it into `hf download --revision` and
pins the post-download sha-verify etag lookup to the same revision, so a
stale pin can't false-fail against a newer HEAD. preflight's manual hint
mirrors the flag. unset = track HEAD, so behavior is unchanged for every
current entry (nothing sets revision: today).
this is the weights half of #316: upstream quant repos re-quant
silently, and we had no lever to pin the bytes a BENCHMARKS row was
measured against. engine images already pin; weights didn't. mechanism
only, no real entry is pinned in this PR (that's a per-entry,
rig-validated call that's yours to make).
refs #319, #316
report.sh autodetects the running container + URL + engine but not the
served model name, so verify.sh / verify-full.sh / verify-stress.sh /
bench.sh fell back to a hardcoded MODEL=qwen3.6-27b-autoround. Against a
non-qwen vLLM endpoint (e.g. vllm/gemma-26ba4b-single serving
gemma-4-26b-a4b-awq) every request 404'd with "The model
`qwen3.6-27b-autoround` does not exist", failing every check (#372).
llama.cpp ignores the request's model field, so the same wrong default
silently "worked" there (#371) — which masked the bug.
Add a shared preflight_autodetect_model helper that resolves the served
name from the endpoint's /v1/models (first id) when MODEL is unset, and
call it in the four affected scripts before their qwen fallback. This
mirrors what soak-test.sh / bench-agentic.sh / quality-test.sh already
do. An explicit MODEL= still wins (important for llama-swap/multi-model
endpoints); the qwen literal stays as a last resort if detection no-ops.
Validated: unit (resolve / respect-explicit / unreachable-fallback),
end-to-end against a mock /v1/models, and the full scripts/tests suite
stays green (the one pre-existing test-compose-registry-disk failure is
unrelated — untracked nex-n2-mini WIP composes).
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
preflight_compose_deps falsely reported "missing model files" for any compose
whose vLLM --model / --speculative-config path uses an env-default, e.g.
--model /root/.cache/huggingface/${MODEL_SUBDIR:-gemma-4-12b-autoround-int8}
The path grep's char-class excluded `:` and `}`, truncating the token to
`${MODEL_SUBDIR` before _preflight_compose_path_default (which already resolves
${VAR:-default}) ever ran → switch.sh aborted launching ALL 4 gemma-12b composes.
Two hardenings:
- path grep: stop only at real delimiters (quote/space/comma), not `:`/`}`, so
`${VAR:-default}` survives to the resolver. Also skip comment lines (`^\s*#`)
so a path written in prose (e.g. "...expects it at /root/.cache/.../assistant.")
isn't mistaken for a dependency.
- _preflight_compose_path_default: strip a trailing `}` left when a path is the
DEFAULT inside an outer ${VAR:-/root/.cache/huggingface/<path>} (the grep
anchors mid-expansion → captures `<path>}`).
Verified: all 4 gemma-12b composes now pass preflight_compose_deps; qwen3.6-27b
+ gemma-4-31b composes still pass (no regression). Guard suite 41/41 (the lone
transient fail was bench-mock autodetect colliding with a concurrently-running
soak service — unrelated, passes with PREFLIGHT_NO_AUTODETECT=1).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
bench.sh / verify.sh / verify-stress.sh share preflight_autodetect_endpoint,
which only recognised a hardcoded list of container-name prefixes (qwen36-27b,
gemma-4-31b, ...). Any model on a differently-named container (gemma-4-12b,
qwen-35b-a3b, beellama, any BYO compose) fell through and the tools silently
defaulted to port 8020 → ran against the wrong endpoint (#310, syangsao).
Fix: detect by the ENGINE-INTERNAL port mapping instead — vLLM 8000 /
llama.cpp 8080 / sglang 30000. Prefer a recognised club-3090 engine-family
prefix when several match; otherwise take the first and print how to override
(CONTAINER=/URL=). Validated live: picks vllm-gemma-4-12b-int8-mtp on :8038
(missed by the old allowlist) and ignores non-engine containers (benchlocal
sandboxes :9000, litellm :4000). URL/CONTAINER env still win.
Guard suite 41/41 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
When the Genesis/nightly single-card vLLM family was deprecated 2026-05-31
(vllm/default, long-text, long-vision, long-text-no-mtp, bounded-thinking,
tools-text), the hand-written user-facing hints weren't repointed (the
deprecation-checklist step 4 was missed) — surfaced by a #35 commenter who
read them as live. The DEFAULTS resolver was fine (vllm/default → vllm/minimal),
but the docs still advertised the dead composes.
- switch.sh header: the "Single-card vLLM" subset listed 6 deprecated composes
as recommended options AND omitted beellama/dflash (the actual single-card
default). Rewrote to lead with beellama/dflash + the functional vllm/minimal,
with a dated deprecation note (mirrors the dual-section note).
- preflight.sh: the "start with: switch.sh vllm/default" hint now points at
beellama/dflash (the single-card default).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
beellama.cpp composes (ghcr.io/{anbeeld/beellama.cpp,noonghunna/beellama-cpp})
were not matched by the is_llamacpp image regex, so they fell into the vLLM
HF-cache branch and skipped the model-presence check entirely. And even on the
llama.cpp branch, only `-m`/`--model` (the target) was checked — never the
`--spec-draft-model` drafter. Result: a missing target OR drafter GGUF passed
preflight and surfaced as a cryptic in-container "failed to open GGUF file"
crash instead of preflight's "download this:" hint (#288 eddie + George).
- Add `beellama` to the is_llamacpp image regex.
- Collect drafter paths from --spec-draft-model/--model-draft/-md into their own
presence check, with a DRAFT_FILE env override mirroring GGUF_FILE/MMPROJ_FILE.
- New test case: beellama target-present + drafter-missing must refuse with the
drafter path + its `hf download Anbeeld/...` hint; both present must pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Rename vllm/gemma-int8→gemma-int8-mtp + vllm/gemma-mtp→gemma-bf16-mtp (both carry MTP n=4; only KV format differs); kv-calc alias fix. Ladder gemma-bf16-mtp default 32K→131K (measured BF16 KV pool 196,527 tok @ 0.95; verify-stress 8/8, NIAH→120K, bench 118.8/154.3, zero TPS regression). Codify the dual-card max-context-priority rule in DUAL_CARD.md. Suite 41/41.
Bump vllm/gemma-mtp (dual, bf16+MTP) to stable v0.22.0 (validated 5/5 coherent). Deprecate vllm/gemma-mtp-tp1: fp8 KV is hardware-impossible for Gemma 4 on Ampere sm_86 (fp8e4nv kernel unsupported; fp8_e5m2 rejected by gemma4 attention allowlist; nvfp4 Blackwell-only), live-confirmed on v0.22.0. Remove the dead DEFAULTS[(gemma-4-31b,vllm,single)] row; repoint launch/setup/preflight single-card Gemma -> beellama/gemma-dflash. Also fixes a pre-existing unterminated NEXT_STEPS_NOTE string in setup.sh's gemma-4-31b arm. Suite 38/38.
When a user launches with a 4090 or 5090, preflight now prints a single
hint line pointing at the matching FAQ entry. Caught one class of UX
gap where users land on launch.sh without reading the README + run into
unexpected ctx ceilings (4090) or miss the 32 GB-headroom story (5090).
Detection: greps `nvidia-smi -L` output for "RTX 4090" / "RTX 5090".
Quiet on 3090-only rigs. Pure-additive — no behavior change beyond the
extra echo lines.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
soak-test.sh::auto_container only matched vllm-* → rc=2 "no running
container" on the soak step of llama.cpp / ik_llama rebench-full, aborting
the chain before the aider step. Mirror preflight's canonical engine-prefix
set into auto_container, and probe internal 8080/30000 (not just 8000) in
endpoint_from_container. Also add ik-llama-qwen36-27b to
preflight_autodetect_endpoint + preflight_running globs — the ik leg's
autodetect was a latent gap (only llama-cpp was listed). Closes#403.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Adds a clean front-door to rebench-full.sh so users running OpenAI-compatible
endpoints outside our pre-baked Docker composes (llama-swap, ramalama, host-
build llama-server, ik_llama.cpp, raw vLLM) can run the canonical 5-phase
suite without script edits or env-soup.
Today the entry chain assumes a `vllm-qwen36-27b`-style container exists and
calls `docker logs` / `docker inspect` for ENGINE_KIND detection + spec-decode
metric scraping. The infra to bypass this was already 80% there (CONTAINER=none
host mode already in soak-test.sh + verify-stress.sh, docker-inspect guards
already in bench.sh line 82), but there was no top-level flag, and the silent
preflight pipefail bug below meant users hit a confusing no-op exit instead of
a useful error.
Changes:
* rebench-full.sh: add --url/--model/--engine. When --url is set we export
PREFLIGHT_NO_AUTODETECT=1 + CONTAINER=none so the chained scripts skip the
Docker scrapes (they no-op cleanly).
* rebench-full.sh: container snapshot block now wrapped in
`[[ "$CONTAINER" != "none" ]] && command -v docker` so it doesn't try to
inspect the wrong (or no) container in endpoint-first mode.
* rebench-full.sh: endpoint-not-responding error now distinguishes the --url
case (check host/port) from the default case (start a compose or pass --url).
* bench.sh: same CONTAINER=none guard for the trailing `docker logs ... grep
SpecDecoding metrics` block.
* preflight.sh: fix a pre-existing silent-exit when no container matches the
autodetect regex. `grep -E` returns 1, which under `set -euo pipefail` in
the caller killed rebench-full.sh before it reached its own endpoint-error
path. Adding `|| true` to the pipe restores the intended fall-through.
* preflight.sh: regex now also recognises sglang-qwen36-27b containers + port
30000 (added with v0.7.x sglang experimental tree, not yet auto-detected).
* docs/FAQ.md: document the new --url flag under the "Numbers from your rig"
section.
Tested:
- bash -n on all three scripts: OK
- --help renders the new endpoint-first block
- Default no-flag mode now exits cleanly with a helpful error (was silent
exit 1 pre-fix)
- --url FAKE skips autodetect and emits the external-endpoint-specific hint
- --url + non-existent host: clean error, no Docker side-effects
Motivated by ampersandru's external eval in discussion #152 — running
ik_llama via llama-swap forced him to set PREFLIGHT_NO_AUTODETECT=1 manually
and edit scripts to bypass docker assumptions. With this PR the invocation
becomes:
bash scripts/rebench-full.sh \
--url http://192.168.29.235:8887 \
--model 'Qwen3.6-27B MTP ik_llama:instruct' \
--engine llama-cpp
— and the whole 5-phase chain (bench + verify-stress + quality-full + soak
+ aider-polyglot) works against any OpenAI-API endpoint.
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
User docs were hardcoding the dev rig path (/mnt/models/huggingface/...) as
if it was canonical. It's not — cross-rig users have models at /data/models,
~/models, /mnt/nvme/llms, etc. Setting MODEL_DIR per their setup is the
intended UX (the compose already supports it via env-var default).
Updates:
- models/qwen3.6-27b/llama-cpp/README.md: download examples now use
\$MODEL_DIR/qwen3.6-27b-gguf/ instead of /mnt/models/huggingface/...
- models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml: header
comment uses \$MODEL_DIR/qwen3.6-27b-gguf/ for download examples + says
"MODEL_DIR=/your/models/dir docker compose up -d" instead of our path.
- docs/engines/LLAMA_CPP.md: same treatment + cleaned up Qwen3.5 + DFlash
draft path examples to also use \$MODEL_DIR.
- scripts/preflight.sh: hf download hint shows literal \${MODEL_DIR} so user
knows what to set, plus explicit "set MODEL_DIR first" line. Previously
echoed the resolved relative path (../../../../models-cache) which lands
outside the repo if pwd isn't the compose dir.
Caught by RobH589 in club-3090#116 — they hit the path-resolved-to-root-of-drive
case from the relative-path default. Closes the doc UX side; the compose's
env-var override mechanism was already correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
After the GGUF dir move (/mnt/models/gguf/qwen3.6-27b/ → /mnt/models/
huggingface/qwen3.6-27b-gguf/), four refs were not updated and led to
a path-mismatch loop reported in club-3090#116:
- scripts/preflight.sh `hf download` hint pointed at qwen3.6-27b/, but
the compose default expects qwen3.6-27b-gguf/. Same for the mv hint
for mmproj relocation, and the in-container mmproj default at line 329.
- models/qwen3.6-27b/llama-cpp/README.md example command still said
`MODEL_DIR=/mnt/models/gguf` (now /mnt/models/huggingface).
- models/qwen3.6-27b/llama-cpp/compose/single/{docker-compose,concurrent}.yml
header comment said "Q5_K_XL" but the actual default has been Q3_K_XL
for a while (this one predates the reorg — just stale doc).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Three related fixes for running benchmarks without IDE agent interference:
1. All 18 vLLM compose files: port binding is now
${BIND_HOST:-0.0.0.0}:${PORT:-<n>}:8000
Setting BIND_HOST=127.0.0.1 in .env restricts the API to localhost,
preventing IDE agents (Cline, Cursor) from competing for the
max-num-seqs=1 slot and causing verify-stress HTTP 000 failures.
2. scripts/preflight.sh: port auto-detection regex now matches
127.0.0.1:<port>->8000/tcp in addition to 0.0.0.0: and [::]:
Previously all verify-*/bench scripts silently produced no output
when BIND_HOST=127.0.0.1 was set.
3. scripts/report.sh: SOAK_TIMEOUT_S is now forwarded to soak-test.sh
from the shell environment. Previously the variable was read from
compose .env (docker-compose only) and silently ignored by the
script, always using the 1800s default regardless of what was set.
Docs: .env.example gains BIND_HOST and SOAK_TIMEOUT_S entries.
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
* add Gemma 4 31B + Google MTP drafter — first Ampere consumer cross-rig data
Adds models/gemma-4-31b/ tree with two compose variants (TP=2 + TP=1),
vendored vLLM PR #41745 overlay, switch.sh / launch.sh wiring, plus the
script-level extensions needed to make `bash scripts/{bench,verify-*,
soak-test}.sh` auto-detect Gemma containers.
Validated 2026-05-05 on 2× RTX 3090 (Ampere sm_86, PCIe-only):
- canonical bench (3 warm + 5 measured per prompt):
narrative wall TPS = 108.87 (CV 3.2%)
code wall TPS = 142.25 (CV 2.3%)
- soak-continuous (5 sessions × 5 turns × 4 prompts = 100 turns):
verdict PASS, 0 errors, 0 silent-empty, 0 MiB VRAM growth,
98.3% TPS retention, p50 decode 111.79
- VRAM 22.5 GB/card. Speedup vs no-spec-decode baseline: 1.79× narr / 2.31× code.
First published Ampere consumer numbers on Google's Gemma 4 MTP "assistant"
drafters (released 2026-05-05). Discussion #67 has the announcement +
upstream context. BENCHMARKS.md gets a new "Gemma 4 31B" section with
both TP=2 (working) and TP=1 (upstream-blocked on Ampere) rows.
Components landed:
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp.yml (TP=2)
- models/gemma-4-31b/vllm/compose/docker-compose.gemma-mtp-tp1.yml
(TP=1 — preserved for re-test when upstream Ampere fp8 path lands)
- models/gemma-4-31b/vllm/patches/vllm-gemma4-mtp/ — vendored overlay
of vllm-project/vllm#41745 (lucianommartins/gemma4-mtp). 7 modified
Python files RO-mounted over the stock nightly image. Same shape as
vllm-marlin-pad. Drop the entire tree when PR merges + propagates.
- scripts/switch.sh + launch.sh: vllm/gemma-mtp + vllm/gemma-mtp-tp1
variants registered. RUNNING_PATTERN extended for vllm-gemma-4-31b*.
- scripts/preflight.sh: autodetect_endpoint extended for gemma containers.
- scripts/soak-test.sh: container grep extended.
- scripts/soak-helper.py: SOAK_NO_CHAT_TEMPLATE_KWARGS=1 env knob to
skip the Qwen3-specific `chat_template_kwargs.enable_thinking` body
field for non-Qwen models that reject it (Gemma 4, etc.).
- BENCHMARKS.md: new Gemma 4 31B section with both TP rows.
- docs/UPSTREAM.md: PR #41745 row + the Ampere fp8-blocked finding +
transformers 5.8.0 dependency entry.
Pre-merge dependencies (drop when both land):
1. vllm-project/vllm#41745 → drop the patches/ tree + the volume block
2. transformers ≥ 5.8.0 → drop the entrypoint pip install line
Out of scope:
- TP=1 single-card. Tested + upstream-blocked on Ampere consumer:
fp8 KV needed for fit; fp8_e4m3 hits Triton "fp8e4nv not supported"
on sm_86; fp8_e5m2 rejected by gemma4_mm.py:1336 allowlist. Compose
is preserved with the failing config baked in for future re-test.
- Gemma 4 26B-A4B MoE single-card. Active params ~4B → should fit
cleanly without fp8. Queued as the obvious follow-up.
- vllm/vllm-openai:gemma4-0505-cu129 image swap. The recipe-image is
Hopper-tagged and may not include sm_86 kernels; testing it would
collapse this whole 4-layer wrangle to a one-line image bump but
needs explicit Ampere validation first.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* models/gemma-4-31b/vllm/cache: gitignore + README (mirror qwen3.6 pattern)
Replaces the .gitkeep stubs with the standard .gitignore (ignore everything
except .gitignore + README.md) + README documenting the cache lifecycle.
Mirrors models/qwen3.6-27b/vllm/cache/ exactly.
Boot times this enables (validated tonight):
- cold first boot: ~3-7 min (TP=1 / TP=2)
- warm subsequent boot: ~2-3 min (cache hit)
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Verify-full / verify-stress / bench previously hardcoded
URL=http://localhost:8020 + CONTAINER=vllm-qwen36-27b. That assumption
silently broke for anyone running a non-default variant — dual-turbo on
8011, dual-dflash on 8012, etc. — making the embedded report.sh chain
emit false negatives. Reported by sudepo on club-3090#52.
Add preflight_autodetect_endpoint() that:
- scans `docker ps` for one of our container patterns
(vllm-qwen36-27b* / llama-cpp-qwen36-27b*)
- extracts the host port from its 0.0.0.0:<port>->{8000,8080}/tcp
mapping
- sets URL + CONTAINER, but ONLY for fields the user didn't already
set explicitly (env-var override always wins)
- prints one [autodetect] line so the user sees what was picked
- falls back silently to the existing hardcoded defaults if nothing
is detected (no behaviour regression for fresh setups)
Wired into the three test scripts. Skip via PREFLIGHT_NO_AUTODETECT=1
for the rare case where the user wants to point at a non-running
container or remote endpoint.
Verified locally:
- autodetect with no env vars → picks up vllm-qwen36-27b-dual-turbo on
port 8011 (matches `docker ps`)
- autodetect with URL=... CONTAINER=... env set → preserves both
- all four scripts pass `bash -n`
Direct commit per the docs/cosmetic-direct-to-master convention; this
is small, additive, override-preserving and falls back to existing
behaviour on detection miss.
* fix(preflight): catch missing llama.cpp GGUF before container boot
preflight_compose_deps was vLLM-only — it always required the AutoRound
INT4 weights and never validated llama.cpp GGUF / mmproj mounts. Result:
fresh users following the README to switch to llamacpp/default would hit
a 600s container timeout buried in docker logs (syangsao on #58).
Make preflight engine-aware: detect llama.cpp composes by image, parse
the `-m /models/<path>` and `--mmproj /models/<path>` args (resolving
${GGUF_FILE:-...} / ${MMPROJ_FILE:-...} defaults, env-var overrides win),
and surface a copy-pasteable `hf download unsloth/Qwen3.6-27B-GGUF ...`
one-liner when files are missing.
Verified:
- llamacpp/default + missing GGUF → preflight errors with clear fix
- llamacpp/concurrent → same correct behavior
- vllm/dual on /mnt/models/huggingface → no regression
Closes the trip-hazard on #58. setup.sh stays vLLM-default so fresh
clones don't pay 16 GB for a path most users won't take first.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
* fix(preflight): correct hf download CLI form (positional, not --include)
syangsao tested PR #63 and the suggested one-liner only fetched
mmproj-F16.gguf (928 MB), not the main 14 GB GGUF. Root cause:
hf download unsloth/Qwen3.6-27B-GGUF \
--include 'Qwen3.6-27B-UD-Q3_K_XL.gguf' 'mmproj-F16.gguf' \
--local-dir <dir>
`hf download` parses positional [FILENAMES]... args; when both
--include and positional filenames are passed, --include is silently
ignored:
UserWarning: Ignoring `--include` since filenames have being
explicitly set.
So the second arg ('mmproj-F16.gguf') becomes the only filename to
fetch, and the main weights are skipped. Fix: drop --include and
list both files as positional args, matching `hf download`'s own
example in --help:
hf download meta-llama/Llama-3.2-1B-Instruct config.json tokenizer.json
Also clarified the mmproj-move comment: this isn't a "sometimes" —
unsloth ships mmproj-F16.gguf at the top of the GGUF repo, so it
always lands inside the --local-dir we pass. The compose default
points at qwen3.6-27b/mmproj-F16.gguf (one level up from
unsloth-q3kxl/), so the move is mandatory, not conditional.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Closes the UX gap raised by @snoby on #37 — previously, launching a
compose that needs the DFlash draft model without first running
WITH_DFLASH_DRAFT=1 setup.sh would fail with vLLM's generic
"Invalid repository ID or local directory" pydantic error.
Adds three new preflight functions in scripts/preflight.sh:
1. preflight_hf_token (soft warn) — wired into setup.sh before the model
download. Catches missing HF_TOKEN early and tells the user exactly
what to do (visit hf.co/settings/tokens, accept T&C, export the token).
Skip via PREFLIGHT_NO_HF_TOKEN=1.
2. preflight_compose_deps (hard error) — wired into switch.sh just before
docker compose up. Parses the target compose YAML for known model-path
patterns (qwen3.6-27b-dflash, qwen3.6-27b-mtp-head, qwen3.6-27b-autoround-int4)
and verifies each exists on host. If missing, emits a clear "Fix:" line
with the exact setup.sh invocation needed (e.g. WITH_DFLASH_DRAFT=1).
Refuses to proceed with exit 1. Skip via PREFLIGHT_NO_COMPOSE_DEPS=1.
3. preflight_kv_format_hint (soft warn) — wired into switch.sh as the last
pre-up check. Detects smallest VRAM via nvidia-smi; if <24 GB and the
target compose uses turboquant_3bit_nc KV, emits the @efschu finding
from #47 (TQ3 → fp8_e5m2 swap rule) with cross-link to docs/HARDWARE.md
+ a tools/kv-calc.py one-liner to predict the user's specific config.
Skip via PREFLIGHT_NO_KV_HINT=1.
Tested on this rig:
- HF_TOKEN unset → warns
- Missing DFlash dir → hard errors with WITH_DFLASH_DRAFT=1 hint
- Missing main model → hard errors with generic setup.sh hint
- 24 GB rig + TQ3 compose → silent (correct, TQ3 is right pick)
- Simulated 20 GB rig + TQ3 compose → fires the fp8_e5m2 hint
All preflights are individually skippable via PREFLIGHT_NO_<NAME>=1
env vars (matches existing PREFLIGHT_NO_FETCH=1 / PREFLIGHT_NO_GENESIS_PIN=1
patterns).
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Two-part addition for the most common stale-setup pattern: user cloned
weeks ago, master has moved (Genesis pin bumps, compose changes, vendored
patch updates), they re-run their compose, hit a stale config, and file
an issue we already solved on master. Wispborne's _register_op_once and
GuiPerPT's pre-pull boot OOM both surfaced through this loop.
scripts/preflight.sh — preflight_repo_drift:
- Skips silently if not a git repo, on a non-master branch, or if
PREFLIGHT_NO_FETCH=1 (offline rigs / CI / forks tracking elsewhere).
- Verifies origin remote is noonghunna/club-3090 (avoids false positives
on forks pointing elsewhere).
- timeout 5 git fetch --quiet origin master — bounded so flaky networks
don't block boot.
- On behind > 0: WARN with commit count, last-fetch age (h/d), and the
one-line fix command. Soft-warning, never blocks. Tells user about
PREFLIGHT_NO_FETCH=1 for opting out.
Wired into both launch.sh and switch.sh, alongside the existing
preflight_genesis_pin so users get one consolidated stale-setup signal.
scripts/update.sh — the easy upgrade path:
- Refuses on dirty tree (git status --porcelain) — surfaces the local
edits and tells the user to commit or stash first. We don't clobber
the rare user who's been editing a compose locally.
- Refuses on non-master branch — feature branches and fork-trackers
should pull manually; this script is the master-from-origin path.
- git pull --ff-only — no merge commits, no rebase ambiguity. Diverged
branches get an explicit error pointing at git pull --rebase.
- Re-runs setup.sh (idempotent — re-pins Genesis, re-vendors Marlin).
- Tells the user to restart their variant via switch.sh — doesn't auto-
restart, so they can A/B old-vs-new if they want.
- --dry-run shows the plan without changing anything.
- --force re-runs setup.sh even when up-to-date (for "I edited Genesis
by hand and want it re-pinned" cases).
Why detection-then-explicit-command instead of "press y to auto-update":
the user will rarely have local commits (they're consumers of the recipes,
not vLLM contributors), but we still want consent — they should see what
they're committing to. The dirty-tree guard handles the rare custom-edit
case without nagging the common path.
Catches the failure mode @wispborne hit in #32: user pulled latest
club-3090 (which bumped GENESIS_PIN in scripts/setup.sh from old →
new), but didn't re-run setup.sh. The on-disk Genesis tree at
models/qwen3.6-27b/vllm/patches/genesis/ stays at the OLD pin while
setup.sh's declared pin advances. vLLM boots against the outdated
tree and hits mysterious patch failures (e.g. v7.66 PN25's
infer_schema crash that v7.69 PN25 worker-spawn registration fixed).
Adds new `preflight_genesis_pin` function to scripts/preflight.sh:
- Parses `GENESIS_PIN="${GENESIS_PIN:-<default>}"` from setup.sh to
extract the declared default value
- Reads on-disk HEAD via `git -C <genesis-dir> rev-parse --short HEAD`
- Compares (declared short-form vs on-disk short-form match)
- On mismatch: emits a [preflight] WARN block with both pins + a
"Fix: bash scripts/setup.sh qwen3.6-27b" hint
- Soft-warning only — does not block boot
Wired into both entry points:
- scripts/launch.sh (the wizard-driven path) — adds the check after
preflight_running, before the variant pick
- scripts/switch.sh (the direct stateless switcher) — adds the check
inside up_variant() right before the docker compose up call
Skips silently if the Genesis tree hasn't been cloned yet (caller
should run setup.sh first; preflight isn't the place to handle that
case). Also skips if setup.sh isn't present at the expected path
(weird state — silent skip rather than false warnings).
Tested locally:
- Current state (declared 2db18df = on-disk 2db18df): silent ✓
- bash -n passes on all three modified scripts
Coverage gap: users who bypass switch.sh + launch.sh and run
`docker compose -f docker-compose.X.yml up -d` directly won't get
the warning. That's the canonical-path-vs-bypass tradeoff; we lead
with switch.sh in the docs.
Closes the procedural follow-up flagged in
[#32 comment-4364895764](https://github.com/noonghunna/club-3090/issues/32#issuecomment-4364895764)
("setup.sh should be re-run after every git pull that bumps
GENESIS_PIN. We could add a check for this — compare setup.sh's
declared pin vs the on-disk tree's HEAD, warn if they differ.").
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
- scripts/preflight.sh (new) — sourceable library: docker, GPU >= N,
disk free, GPU-idle warning, running-container note. Each error has
an actionable Fix: hint instead of a cryptic mid-run crash.
- scripts/setup.sh + scripts/launch.sh wire pre-flight in early.
launch.sh adds --no-preflight escape hatch.
- launch.sh wizard inverted: cards → workload → auto-pick engine.
Newcomers can answer "how many GPUs" and "what do I want to do" but
rarely "vLLM or llama.cpp" — engine falls out of the pick with a
one-paragraph why. --engine override still works (filters the
workload list to that engine).
- Embedded charts swapped SVG → PNG in README + SINGLE_CARD +
DUAL_CARD + qwen3.6-27b/README. Clicking a PNG on GitHub opens a
viewable image; SVGs open as raw XML. SVG remains the editable
source — re-export PNG when SVG changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
