# Pull — serve any HF safetensors model (v0.8.0) **User guide.** You have a Hugging Face model and you want to know: *will it run on my GPUs, and if so, how?* `scripts/pull.sh` answers that — it evaluates the repo against this stack's precise KV math before you download anything, and is honest about how much it trusts the answer. > **v0.8.0 headline:** *"Evaluate any safetensors HF repo; pull only > vLLM-loadable supported ones, and only when the gates pass (or an explicit > override is accepted)."* > > **v0.8.2 adds** (additive only — no v0.8.0 decision logic changed): a > failure on-ramp (a redacted-diagnostics submit path when a pull fails), > arch-registry expansion (materially more safetensors models reach > `engine-supported`), an optional hardware-detect slice for non-NVIDIA > enumeration, and the `--recommend` UX (an honest aggregated > recommendation over the same verdict). This release also **bundles two > non-`pull` items that shipped on the same branch**: N-GPU NVLink > auto-detection (wired into the multi-4 and Gemma-4-26B dual composes) > and a documentation restructure (a quick-start-first README, a new > `GETTING_STARTED.md`, model READMEs, and an updated docs index) — both > are independent of and orthogonal to the `pull` decision path. **GGUF > is deferred** — it is not a v0.8.2 item: cross-engine generation (GGUF > / llama.cpp serving via `pull`) is a separate **§9 cross-engine > design-unlock proposal**, not this release. Pointing `pull` at a > GGUF-only repo is a known scope boundary, not a stack failure (see [the > readiness ledger](#release-readiness-ledger--honest-deferrals) below). This is the **user front door**. For the contributor/maintainer internals of the same pipeline (gate strata, classifier, trust pipeline) start at [`docs/README.md`](README.md) → the Contributor track. --- ## Quickstart One command. Replace the slug with your model; `--profile-like` borrows a curated runtime shape (a `COMPOSE_REGISTRY` key like `vllm/minimal` — see [Usage](#usage) below for what the keys mean). ```bash # Just check — never downloads, never boots: scripts/pull.sh --profile-like vllm/minimal --dry-run # Evaluate, then (if it passes) download + emit a compose + boot it: scripts/pull.sh --profile-like vllm/minimal --yes ``` What you'll see — exactly one of: | Outcome | Exit | Means | |---|---|---| | `proceed` / `confirm→proceed` | `0` / `3` | Fits. `0` = clean; `3` = re-run with the named flag (e.g. `--yes`) to continue. | | `hard-block` | `2` | Honest stop with a precise reason (unsupported engine/arch, won't-fit, disk, needs `--trust-remote-code`). Nothing downloaded. | | `override-accepted` | `0` | You explicitly accepted a non-pass path (e.g. `--force-download`); proceeds with the caveat recorded. | > **First-run heads-up:** many common models (anything `Qwen2ForCausalLM` — Qwen2.5 & a large family, plus other custom-code archs) hard-block at `[C0] needs-trust-remote-code-ack` on the *very first* try — **even with `--dry-run`**. That's the gate working, not a failure. After you've checked what code the repo would run, add **`--trust-remote-code`** to that same command to clear it. See [`--trust-remote-code` — a security decision](#--trust-remote-code--a-security-decision) below. It is **honest about confidence and never silently passes.** A "fits" verdict is a *boot-time* check — read [Boot-fit ≠ runtime-stability](#boot-fit--runtime-stability--read-this) before relying on it for sustained agent workloads. Full detail below. --- ## What changed in v0.8.0 Older releases worked one way: the repo *formally supported* a fixed list of models, and you picked from that list. That still works and still ships — see [`docs/SINGLE_CARD.md`](SINGLE_CARD.md) / [`docs/DUAL_CARD.md`](DUAL_CARD.md) / [`docs/MULTI_CARD.md`](MULTI_CARD.md), the curated catalog is unchanged. v0.8.0 **adds** a model-agnostic front door. The stack no longer needs to "formally support model X" per release to be useful for X. Instead: - You hand `pull` *any* safetensors HF repo slug. - It derives the model's shape from the repo's own `config.json` and runs it through this stack's KV math. - It returns a verdict **with an explicit confidence tier**, and tells you which gate decided. The curated catalog doesn't go away — it becomes the **calibration backbone**: the measured corpus the math is anchored against. Curated models get an `exact` confidence tier; arbitrary repos get an honest lower-bound estimate. Both are first-class; the difference is stated, never hidden. --- ## Usage ``` scripts/pull.sh --profile-like [opts] ``` `--profile-like` is **required**: it names a curated registry key that supplies the runtime shape (engine, KV format, TP) to evaluate against. ### Path A — curated pull-and-emit The slug is a curated, generator-emittable model. On a gate-passing run `pull` hands the validated key to the #141 compose generator and emits a ready compose. ``` scripts/pull.sh Lorbus/Qwen3.6-27B-int4-AutoRound \ --profile-like vllm/minimal --out qwen.yml ``` > **The emitted compose carries the reference profile's capacity values, not a fit tuned to your GPU.** `--max-model-len`, `--gpu-memory-utilization`, `--max-num-seqs` and the KV dtype are copied from the captured profile — *not* re-solved for your card. It is a known-safe starting point: add `--recommend` (or run `tools/kv-calc.py --solve-max-ctx`) for the honest fit on your hardware, and tune the emitted env-overridable `${MAX_MODEL_LEN}` accordingly. See `docs/COMPOSE_GENERATOR.md` § "Capacity values are the reference profile's". ### Path B — universal evaluate (never downloads, never emits) Any non-curated slug, or `--dry-run` on anything, takes Path B. It prints a confidence-tiered verdict and **never calls the generator and never downloads weights**. ``` scripts/pull.sh some-org/Some-Llama-7B --profile-like vllm/minimal --dry-run ``` ### Options | Opt | Meaning | |---|---| | `--yes` | Accept a `confirm→proceed` terminal (§4.1 — see "Reading the verdict"). | | `--force-download` | Advisory low-confidence `wont-fit` → `override-accepted`. **No-op + notice this phase** (download deferred to a later phase). | | `--experimental-arch` | Bypass *only* a `[C0] engine-support-unknown` (no arch row) hard-block; attempt with default vLLM settings. Path B only this phase. | | `--trust-remote-code` | Bypass a `[C0] needs-trust-remote-code-ack` hard-block (security decision — see below). | | `--hf-home DIR` | Override the `HF_HOME` resolution chain (where disk is checked / weights would land). | | `--out FILE` | Path A: write the emitted compose here. | | `--hardware SM` | Override detected GPU compute capability (e.g. `8.6` for RTX 3090); default = `nvidia-smi`. | ### Exit codes | Code | Meaning | |---|---| | `0` | Download-eligible / clean verdict. | | `3` | Needs a flag — a `confirm→proceed` or advisory terminal that is not yet satisfied (re-run with the named flag). | | `2` | Honest hard-stop — a gate aborted, or a `hard-block` terminal. | | `64` | Usage error — missing/unknown argument (distinct from `2`, so a typo is distinguishable from an honest gate-block). | > *Note: the `64` usage-vs-`2` hard-stop split is a post-`v0.8.0` fix — present on `master`/the next release; the `v0.8.0` release tag still exits `2` for argument errors.* --- ## Reading the verdict Every run prints the **confidence tier** and **which gate decided** — this is non-negotiable: the gate never silently passes anything except an `exact × fits-clean` case. Two axes combine: - **confidence** ∈ `{exact, estimated-lower-bound}` (a `derived` tier is reserved for a future phase). `exact` = a curated calibration anchor; `estimated-lower-bound` = a derived estimate where the modelled VRAM is a *floor* and is likely under-modelled. - **raw_verdict** ∈ `{fits-clean, fits-constrained, wont-fit}` — the KV math's pure measurement, no policy. The two map to a **terminal** ∈ `{proceed, confirm→proceed, hard-block, override-accepted}`: | confidence | `fits-clean` | `fits-constrained` | `wont-fit` | |---|---|---|---| | `exact` | **proceed** (silent — the only silent pass) | **confirm→proceed** (`--yes`; a constraint changed your requested config) | **hard-block** (math trusted; closest-fit suggested) | | `estimated-lower-bound` | **confirm→proceed** (`--yes`; VRAM is a floor, likely under-modelled) | **confirm→proceed** (`--yes` + floor + constraint notice) | advisory → `--force-download` → **override-accepted** | The output names the stratum/gate that decided. An illustrative Path B line (shape derived from the tool's actual print sites — your values will differ): ``` [pull] OK path=B stratum=DECIDED slug=some-org/Some-Llama-7B profile-like=vllm/minimal [pull] confidence=estimated-lower-bound raw_verdict=fits-clean terminal=confirm→proceed [pull] Path B verdict: [C1] estimated-lower-bound×fits-clean → confirm→proceed (VRAM is a floor; likely under-modeled) [pull] note: boot-fit satisfied; this does NOT guarantee stability under sustained / accumulated-context workloads — validate with soak-continuous before relying on it (recommend: scripts/soak.sh SOAK_MODE=continuous). ``` `override-accepted` is **not** a gate-pass. It is the deliberate, explicit path for forcing a low-confidence `wont-fit`: it records the outcome as a calibration signal, it does not record "fit validated". --- ## Boot-fit ≠ runtime-stability — read this The KV math is a **static, boot-time allocation** check. Passing it is *necessary but not sufficient* for real workloads. On this hardware class, measured failure modes exist that a static check cannot see: - **Cliff 2** — degradation/OOM at roughly **21–26K accumulated context** under accumulated-context agent workloads (hermes/openhands style). - **Prefill cliffs** at single-prompt sizes well below the static ceiling. - **Cliff 2b** — only detectable under a *continuous soak*, not a single request. So a `fits-clean` / `proceed` config can still degrade or OOM once a real agent accumulates context. Honesty is non-negotiable on this stack, so the verdict output always carries this caveat verbatim: > *boot-fit satisfied; this does NOT guarantee stability under sustained / > accumulated-context workloads — validate with soak-continuous before > relying on it (recommend: scripts/soak.sh SOAK_MODE=continuous).* **Before you rely on any `fits-clean` / `proceed` config in production, run** `scripts/soak.sh SOAK_MODE=continuous` — it is the only test that catches Cliff 2b. A `fits-clean` that silently dies under soak is exactly the confidently-wrong outcome this design forbids; the soak makes the *predicted* side honest about its scope. See [`docs/CLIFFS.md`](CLIFFS.md) for the full diagnosis of these failure modes. --- ## `--trust-remote-code` — a security decision Some HF repos ship custom modelling code that the loader executes. If the architecture's matrix entry requires it, the gate **hard-blocks** with `needs-trust-remote-code-ack` and prints what code origin would run. It does not proceed until you explicitly pass `--trust-remote-code`. This is a deliberate fail-closed security gate — *do not reflexively pass the flag to clear an error*; understand what code you are authorizing first. A genuinely unknown architecture (no entry in the patch matrix at all) hard-blocks differently — with `engine-support-unknown`. Pass `--experimental-arch` to attempt it anyway with default vLLM settings; the outcome is captured to inform support coverage. --- ## What happens after a pass `pull` itself stays user-level — it evaluates and (Path A) emits. For the depth behind a passing run: - **The download → boot → smoke path** for a download-eligible derived model: [`docs/PULL_EMIT_DERIVED.md`](PULL_EMIT_DERIVED.md) (the `[E]` stage). - **The contribution loop** — how a boot/OOM outcome becomes a classified, deduped, consensus-keyable calibration signal: [`docs/LOOP.md`](LOOP.md) (the `[F]` stage). - **The compose the generator emits** and how it is shaped: [`docs/COMPOSE_GENERATOR.md`](COMPOSE_GENERATOR.md). --- ## `--recommend` — the honest one-line answer Add `--recommend` to any `pull` invocation and, after the gate runs, you get an aggregated plain-language recommendation: does it fit, on which profile/variant, at what confidence, and **which gate decided** — plus the boot-fit≠runtime caveat verbatim when the verdict reached the fit math. ```bash scripts/pull.sh --profile-like vllm/minimal --dry-run --recommend ``` It is **presentation only**: every line is read straight off the same verdict the gate already produced — `--recommend` never changes the decision, the exit code, or what gets downloaded/emitted. It is honest by construction: - It echoes the real confidence tier; an `estimated-lower-bound` fit is stated as a floor, never dressed up as a guarantee. - It is **vLLM-only** (the gate is vLLM-only; `--recommend` only echoes that). - It **never implies an artifact that was not produced** — the "compose emitted" line appears only when a compose was actually emitted (Path A); a Path B / `--dry-run` recommendation says so explicitly. - A `FITS` verdict carries the §7 caveat and the `scripts/soak.sh SOAK_MODE=continuous` pointer; a pre-fit-math hard-block does **not** (it never reached the fit math, so it makes no soak claim). When the verdict is *blocked*, `--recommend` points you at the failure on-ramp below. --- ## Report a failed pull When a `pull` fails (a gate hard-block, or a `fits-clean` that then fails to boot), the run leaves a **redacted diagnostics bundle** on disk and prints exactly where it is and the one command to send it back: ``` [pull] Diagnostics captured (redacted, no paths/tokens): .pull-captures// [pull] Help improve the fit math — submit with: scripts/pull.sh --submit-last ``` This is the success-path's mirror image: `--recommend` tells you what to run; this tells you "that failed — help us fix the fit math." It is entirely opt-in and the `pull` run itself does **no** network — capture is a local file write only. ### The on-ramp, step by step 1. **Capture is automatic and redacted.** On any failure terminal `pull` writes a bundle under `.pull-captures//` and records it as the most-recent capture. The bundle is already scrubbed of paths and tokens — you never paste terminal scrollback (your console output is *not* a safe source; the artifact is). 2. **You submit it deliberately, in a separate command.** Submission is a distinct, explicit, consented step — never automatic, never a phone home: ```bash # Submit the most-recent capture: scripts/pull.sh --submit-last # Or submit a specific bundle by directory: scripts/pull.sh --submit .pull-captures// ``` `--submit-last` and `--submit ` need **neither a slug nor `--profile-like`** — they are a different verb from a gate run. 3. **You see the exact payload, then consent.** Before anything leaves your machine the command prints the resolved bundle identity and the **exact already-redacted payload that would be sent**, then asks: ``` [submit] resolved bundle: [submit] identity: slug='' utc_ts='' outcome='hard-block' schema=2 ... [submit] this is the EXACT already-redacted payload that will be sent (no terminal scrollback, no paths/tokens): ------------------------------------------------------------------------ ... the redacted report ... ------------------------------------------------------------------------ [submit] §6.1 class= should_file= [submit] submit this redacted report? [y/N] ``` Anything other than a leading `y`/`Y` (including just Enter, or EOF in a non-interactive shell) is a decline: ``` [submit] declined — nothing sent (no network performed). ``` Network happens **only** after an explicit `y`. 4. **With `gh` (the GitHub CLI) installed and authenticated**, a consented submit reuses the deduped contribution path: equivalent reports are coalesced (a duplicate adds a `+1` rather than opening a new issue), and a correct-refusal class is spooled to the maintainer triage queue instead of opening a public issue. You'll see a line like `[submit] F5 action=... dedup_hash=... issue=...` (and a `[submit] spool: ...` line when it was queued, not filed). 5. **Without `gh`**, the same consented submit degrades cleanly: for a solicited (actionable) class it prints a prefilled GitHub issue URL you can paste into a browser; for a correct-refusal / unactionable class it prints the **local** triage-spool path and a "captured for maintainer triage; not a public issue" line — and **no** public-issue URL. Either way, the only thing you are ever asked to share is the already-redacted artifact, never a filesystem path or terminal output. If `--submit-last` finds no recent capture it tells you plainly and points at the explicit-directory form: ``` [submit] no recent capture; use `scripts/pull.sh --submit ` ``` --- ## Release readiness ledger — honest deferrals `pull` evaluates and serves **safetensors via vLLM only**. This is a deliberate scope boundary, stated up front so a §9 reader is not surprised. v0.8.2's shipped scope is the four `pull` items below **plus** two orthogonal, non-`pull` items that landed on the same release branch (N-GPU NVLink auto-detection and a documentation restructure) — listed here so the bundled scope is stated honestly, not under-claimed: | Item | Status in v0.8.2 | |---|---| | Safetensors evaluate + (Path A) emit + boot | shipped (v0.8.0) | | Failure on-ramp (capture + consented `--submit*`) | shipped (v0.8.2) | | Arch-registry expansion (more models reach `engine-supported`) | shipped (v0.8.2) | | Optional non-NVIDIA hardware-detect slice | shipped (v0.8.2, optional) | | `--recommend` UX | shipped (v0.8.2) | | N-GPU NVLink auto-detection (multi-4 + Gemma-4-26B dual composes) | shipped (v0.8.2, bundled — not a `pull` item) | | Documentation restructure (quick-start README, `GETTING_STARTED.md`, model READMEs, docs index) | shipped (v0.8.2, bundled — not a `pull` item) | | **GGUF** repos (evaluate **and** serve) | **deferred — see below** | | **`.bin`** / non-safetensors weight layouts | deferred | | Cross-engine generation (llama.cpp serving via `pull`) | deferred — see below | **GGUF is deferred to a §9 cross-engine design-unlock proposal**, not a later patch release of this line. The reason is structural, not a backlog slip: `pull` emits and serves vLLM by design, so a GGUF path would be either a thin evaluate-only calculator (no launcher) or full llama.cpp serving — and **cross-engine generation is §9 "deferred indefinitely"**, i.e. it requires its own design-unlock and review round before any implementation. If GGUF matters to you, the next artifact is that design-unlock proposal, not a GGUF feature in this line. If you point `pull` at a GGUF-only or `.bin`-only repo today, that is this documented scope boundary, **not** a stack failure.