club-3090/docs/PULL.md

Pull — serve any HF safetensors model (v0.8.0)

User guide. You have a Hugging Face model and you want to know: will it run on my GPUs, and if so, how? scripts/pull.sh answers that — it evaluates the repo against this stack's precise KV math before you download anything, and is honest about how much it trusts the answer.

v0.8.0 headline: "Evaluate any safetensors HF repo; pull only vLLM-loadable supported ones, and only when the gates pass (or an explicit override is accepted)."

v0.8.2 adds (additive only — no v0.8.0 decision logic changed): a failure on-ramp (a redacted-diagnostics submit path when a pull fails), arch-registry expansion (materially more safetensors models reach engine-supported), an optional hardware-detect slice for non-NVIDIA enumeration, and the --recommend UX (an honest aggregated recommendation over the same verdict). This release also bundles two non-pull items that shipped on the same branch: N-GPU NVLink auto-detection (wired into the multi-4 and Gemma-4-26B dual composes) and a documentation restructure (a quick-start-first README, a new GETTING_STARTED.md, model READMEs, and an updated docs index) — both are independent of and orthogonal to the pull decision path. GGUF is deferred — it is not a v0.8.2 item: cross-engine generation (GGUF / llama.cpp serving via pull) is a separate §9 cross-engine design-unlock proposal, not this release. Pointing pull at a GGUF-only repo is a known scope boundary, not a stack failure (see the readiness ledger below).

This is the user front door. For the contributor/maintainer internals of the same pipeline (gate strata, classifier, trust pipeline) start at docs/README.md → the Contributor track.

---

Quickstart

One command. Replace the slug with your model; --profile-like borrows a curated runtime shape (a COMPOSE_REGISTRY key like vllm/minimal — see Usage below for what the keys mean).

# Just check — never downloads, never boots:
scripts/pull.sh <org/Model> --profile-like vllm/minimal --dry-run

# Evaluate, then (if it passes) download + emit a compose + boot it:
scripts/pull.sh <org/Model> --profile-like vllm/minimal --yes

What you'll see — exactly one of:

Outcome	Exit	Means
proceed / confirm→proceed	0 / 3	Fits. 0 = clean; 3 = re-run with the named flag (e.g. --yes) to continue.
hard-block	2	Honest stop with a precise reason (unsupported engine/arch, won't-fit, disk, needs --trust-remote-code). Nothing downloaded.
override-accepted	0	You explicitly accepted a non-pass path (e.g. --force-download); proceeds with the caveat recorded.

First-run heads-up: many common models (anything Qwen2ForCausalLM — Qwen2.5 & a large family, plus other custom-code archs) hard-block at [C0] needs-trust-remote-code-ack on the very first try — even with --dry-run. That's the gate working, not a failure. After you've checked what code the repo would run, add --trust-remote-code to that same command to clear it. See --trust-remote-code — a security decision below.

It is honest about confidence and never silently passes. A "fits" verdict is a boot-time check — read Boot-fit ≠ runtime-stability before relying on it for sustained agent workloads. Full detail below.

---

What changed in v0.8.0

Older releases worked one way: the repo formally supported a fixed list of models, and you picked from that list. That still works and still ships — see docs/SINGLE_CARD.md / docs/DUAL_CARD.md / docs/MULTI_CARD.md, the curated catalog is unchanged.

v0.8.0 adds a model-agnostic front door. The stack no longer needs to "formally support model X" per release to be useful for X. Instead:

• You hand pull any safetensors HF repo slug.
• It derives the model's shape from the repo's own config.json and runs it through this stack's KV math.
• It returns a verdict with an explicit confidence tier, and tells you which gate decided.

The curated catalog doesn't go away — it becomes the calibration backbone: the measured corpus the math is anchored against. Curated models get an exact confidence tier; arbitrary repos get an honest lower-bound estimate. Both are first-class; the difference is stated, never hidden.

---

Usage

scripts/pull.sh <hf-slug> --profile-like <COMPOSE_REGISTRY-key> [opts]

--profile-like is required: it names a curated registry key that supplies the runtime shape (engine, KV format, TP) to evaluate against.

Path A — curated pull-and-emit

The slug is a curated, generator-emittable model. On a gate-passing run pull hands the validated key to the #141 compose generator and emits a ready compose.

scripts/pull.sh Lorbus/Qwen3.6-27B-int4-AutoRound \
    --profile-like vllm/minimal --out qwen.yml

The emitted compose carries the reference profile's capacity values, not a fit tuned to your GPU. --max-model-len, --gpu-memory-utilization, --max-num-seqs and the KV dtype are copied from the captured profile — not re-solved for your card. It is a known-safe starting point: add --recommend (or run tools/kv-calc.py --solve-max-ctx) for the honest fit on your hardware, and tune the emitted env-overridable ${MAX_MODEL_LEN} accordingly. See docs/COMPOSE_GENERATOR.md § "Capacity values are the reference profile's".

Path B — universal evaluate (never downloads, never emits)

Any non-curated slug, or --dry-run on anything, takes Path B. It prints a confidence-tiered verdict and never calls the generator and never downloads weights.

scripts/pull.sh some-org/Some-Llama-7B --profile-like vllm/minimal --dry-run

Options

Opt	Meaning
--yes	Accept a confirm→proceed terminal (§4.1 — see "Reading the verdict").
--force-download	Advisory low-confidence wont-fit → override-accepted. No-op + notice this phase (download deferred to a later phase).
--experimental-arch	Bypass only a [C0] engine-support-unknown (no arch row) hard-block; attempt with default vLLM settings. Path B only this phase.
--trust-remote-code	Bypass a [C0] needs-trust-remote-code-ack hard-block (security decision — see below).
--hf-home DIR	Override the HF_HOME resolution chain (where disk is checked / weights would land).
--out FILE	Path A: write the emitted compose here.
--hardware SM	Override detected GPU compute capability (e.g. 8.6 for RTX 3090); default = nvidia-smi.

Exit codes

Code	Meaning
0	Download-eligible / clean verdict.
3	Needs a flag — a confirm→proceed or advisory terminal that is not yet satisfied (re-run with the named flag).
2	Honest hard-stop — a gate aborted, or a hard-block terminal.
64	Usage error — missing/unknown argument (distinct from 2, so a typo is distinguishable from an honest gate-block).

Note: the 64 usage-vs-2 hard-stop split is a post-v0.8.0 fix — present on master/the next release; the v0.8.0 release tag still exits 2 for argument errors.

---

Reading the verdict

Every run prints the confidence tier and which gate decided — this is non-negotiable: the gate never silently passes anything except an exact × fits-clean case.

Two axes combine:

• confidence ∈ {exact, estimated-lower-bound} (a derived tier is reserved for a future phase). exact = a curated calibration anchor; estimated-lower-bound = a derived estimate where the modelled VRAM is a floor and is likely under-modelled.
• raw_verdict ∈ {fits-clean, fits-constrained, wont-fit} — the KV math's pure measurement, no policy.

The two map to a terminal ∈ {proceed, confirm→proceed, hard-block, override-accepted}:

confidence	fits-clean	fits-constrained	wont-fit
exact	proceed (silent — the only silent pass)	confirm→proceed (--yes; a constraint changed your requested config)	hard-block (math trusted; closest-fit suggested)
estimated-lower-bound	confirm→proceed (--yes; VRAM is a floor, likely under-modelled)	confirm→proceed (--yes + floor + constraint notice)	advisory → --force-download → override-accepted

The output names the stratum/gate that decided. An illustrative Path B line (shape derived from the tool's actual print sites — your values will differ):

[pull] OK path=B stratum=DECIDED slug=some-org/Some-Llama-7B profile-like=vllm/minimal
[pull] confidence=estimated-lower-bound raw_verdict=fits-clean terminal=confirm→proceed
[pull] Path B verdict: [C1] estimated-lower-bound×fits-clean → confirm→proceed (VRAM is a floor; likely under-modeled)
[pull] note: boot-fit satisfied; this does NOT guarantee stability under sustained / accumulated-context workloads — validate with soak-continuous before relying on it (recommend: scripts/soak.sh SOAK_MODE=continuous).

override-accepted is not a gate-pass. It is the deliberate, explicit path for forcing a low-confidence wont-fit: it records the outcome as a calibration signal, it does not record "fit validated".

---

Boot-fit ≠ runtime-stability — read this

The KV math is a static, boot-time allocation check. Passing it is necessary but not sufficient for real workloads. On this hardware class, measured failure modes exist that a static check cannot see:

• Cliff 2 — degradation/OOM at roughly 21–26K accumulated context under accumulated-context agent workloads (hermes/openhands style).
• Prefill cliffs at single-prompt sizes well below the static ceiling.
• Cliff 2b — only detectable under a continuous soak, not a single request.

So a fits-clean / proceed config can still degrade or OOM once a real agent accumulates context. Honesty is non-negotiable on this stack, so the verdict output always carries this caveat verbatim:

boot-fit satisfied; this does NOT guarantee stability under sustained / accumulated-context workloads — validate with soak-continuous before relying on it (recommend: scripts/soak.sh SOAK_MODE=continuous).

Before you rely on any fits-clean / proceed config in production, run scripts/soak.sh SOAK_MODE=continuous — it is the only test that catches Cliff 2b. A fits-clean that silently dies under soak is exactly the confidently-wrong outcome this design forbids; the soak makes the predicted side honest about its scope. See docs/CLIFFS.md for the full diagnosis of these failure modes.

---

--trust-remote-code — a security decision

Some HF repos ship custom modelling code that the loader executes. If the architecture's matrix entry requires it, the gate hard-blocks with needs-trust-remote-code-ack and prints what code origin would run. It does not proceed until you explicitly pass --trust-remote-code. This is a deliberate fail-closed security gate — do not reflexively pass the flag to clear an error; understand what code you are authorizing first.

A genuinely unknown architecture (no entry in the patch matrix at all) hard-blocks differently — with engine-support-unknown. Pass --experimental-arch to attempt it anyway with default vLLM settings; the outcome is captured to inform support coverage.

---

What happens after a pass

pull itself stays user-level — it evaluates and (Path A) emits. For the depth behind a passing run:

• The download → boot → smoke path for a download-eligible derived model: docs/PULL_EMIT_DERIVED.md (the [E] stage).
• The contribution loop — how a boot/OOM outcome becomes a classified, deduped, consensus-keyable calibration signal: docs/LOOP.md (the [F] stage).
• The compose the generator emits and how it is shaped: docs/COMPOSE_GENERATOR.md.

---

--recommend — the honest one-line answer

Add --recommend to any pull invocation and, after the gate runs, you get an aggregated plain-language recommendation: does it fit, on which profile/variant, at what confidence, and which gate decided — plus the boot-fit≠runtime caveat verbatim when the verdict reached the fit math.

scripts/pull.sh <org/Model> --profile-like vllm/minimal --dry-run --recommend

It is presentation only: every line is read straight off the same verdict the gate already produced — --recommend never changes the decision, the exit code, or what gets downloaded/emitted. It is honest by construction:

• It echoes the real confidence tier; an estimated-lower-bound fit is stated as a floor, never dressed up as a guarantee.
• It is vLLM-only (the gate is vLLM-only; --recommend only echoes that).
• It never implies an artifact that was not produced — the "compose emitted" line appears only when a compose was actually emitted (Path A); a Path B / --dry-run recommendation says so explicitly.
• A FITS verdict carries the §7 caveat and the scripts/soak.sh SOAK_MODE=continuous pointer; a pre-fit-math hard-block does not (it never reached the fit math, so it makes no soak claim).

When the verdict is blocked, --recommend points you at the failure on-ramp below.

---

Report a failed pull

When a pull fails (a gate hard-block, or a fits-clean that then fails to boot), the run leaves a redacted diagnostics bundle on disk and prints exactly where it is and the one command to send it back:

[pull] Diagnostics captured (redacted, no paths/tokens): .pull-captures/<slug>/<ts>
[pull] Help improve the fit math — submit with: scripts/pull.sh --submit-last

This is the success-path's mirror image: --recommend tells you what to run; this tells you "that failed — help us fix the fit math." It is entirely opt-in and the pull run itself does no network — capture is a local file write only.

The on-ramp, step by step

1. Capture is automatic and redacted. On any failure terminal pull writes a bundle under .pull-captures/<slug>/<ts> and records it as the most-recent capture. The bundle is already scrubbed of paths and tokens — you never paste terminal scrollback (your console output is not a safe source; the artifact is).

2. You submit it deliberately, in a separate command. Submission is a distinct, explicit, consented step — never automatic, never a phone home:

   # Submit the most-recent capture:
   scripts/pull.sh --submit-last
   
   # Or submit a specific bundle by directory:
   scripts/pull.sh --submit .pull-captures/<slug>/<ts>
   

   --submit-last and --submit <dir> need neither a slug nor --profile-like — they are a different verb from a gate run.

3. You see the exact payload, then consent. Before anything leaves your machine the command prints the resolved bundle identity and the exact already-redacted payload that would be sent, then asks:

   [submit] resolved bundle: <abs dir>
   [submit] identity: slug='<org/Model>' utc_ts='<ts>' outcome='hard-block' schema=2 ...
   [submit] this is the EXACT already-redacted payload that will be sent (no terminal scrollback, no paths/tokens):
   ------------------------------------------------------------------------
   ... the redacted report ...
   ------------------------------------------------------------------------
   [submit] §6.1 class=<class> should_file=<bool>
   [submit] submit this redacted report? [y/N]
   

   Anything other than a leading y/Y (including just Enter, or EOF in a non-interactive shell) is a decline:

   [submit] declined — nothing sent (no network performed).
   

   Network happens only after an explicit y.

4. With gh (the GitHub CLI) installed and authenticated, a consented submit reuses the deduped contribution path: equivalent reports are coalesced (a duplicate adds a +1 rather than opening a new issue), and a correct-refusal class is spooled to the maintainer triage queue instead of opening a public issue. You'll see a line like [submit] F5 action=... dedup_hash=... issue=... (and a [submit] spool: ... line when it was queued, not filed).

5. Without gh, the same consented submit degrades cleanly: for a solicited (actionable) class it prints a prefilled GitHub issue URL you can paste into a browser; for a correct-refusal / unactionable class it prints the local triage-spool path and a "captured for maintainer triage; not a public issue" line — and no public-issue URL. Either way, the only thing you are ever asked to share is the already-redacted artifact, never a filesystem path or terminal output.

If --submit-last finds no recent capture it tells you plainly and points at the explicit-directory form:

[submit] no recent capture; use `scripts/pull.sh --submit <dir>`

---

Release readiness ledger — honest deferrals

pull evaluates and serves safetensors via vLLM only. This is a deliberate scope boundary, stated up front so a §9 reader is not surprised. v0.8.2's shipped scope is the four pull items below plus two orthogonal, non-pull items that landed on the same release branch (N-GPU NVLink auto-detection and a documentation restructure) — listed here so the bundled scope is stated honestly, not under-claimed:

Item	Status in v0.8.2
Safetensors evaluate + (Path A) emit + boot	shipped (v0.8.0)
Failure on-ramp (capture + consented --submit*)	shipped (v0.8.2)
Arch-registry expansion (more models reach engine-supported)	shipped (v0.8.2)
Optional non-NVIDIA hardware-detect slice	shipped (v0.8.2, optional)
--recommend UX	shipped (v0.8.2)
N-GPU NVLink auto-detection (multi-4 + Gemma-4-26B dual composes)	shipped (v0.8.2, bundled — not a pull item)
Documentation restructure (quick-start README, GETTING_STARTED.md, model READMEs, docs index)	shipped (v0.8.2, bundled — not a pull item)
GGUF repos (evaluate and serve)	deferred — see below
.bin / non-safetensors weight layouts	deferred
Cross-engine generation (llama.cpp serving via pull)	deferred — see below

GGUF is deferred to a §9 cross-engine design-unlock proposal, not a later patch release of this line. The reason is structural, not a backlog slip: pull emits and serves vLLM by design, so a GGUF path would be either a thin evaluate-only calculator (no launcher) or full llama.cpp serving — and cross-engine generation is §9 "deferred indefinitely", i.e. it requires its own design-unlock and review round before any implementation. If GGUF matters to you, the next artifact is that design-unlock proposal, not a GGUF feature in this line.

If you point pull at a GGUF-only or .bin-only repo today, that is this documented scope boundary, not a stack failure.