docs: cross-link the v0.8.0 universal pull flow from the existing user guides

Post-release additive pass — the pre-existing high-traffic guides didn't
know `pull` exists. All edits additive, curated path untouched (same
discipline as the README migration nudge):

- FAQ.md: new Q "model not in the supported list — can I still run it?";
  GGUF Q gets a v0.8.0 note (safetensors-only eval, GGUF→llama.cpp stays
  curated/manual, cross-engine deferred); launch.sh answer points
  non-catalog models at `pull`.
- SINGLE_CARD / DUAL_CARD / MULTI_CARD: one blockquote cross-link each to
  docs/PULL.md ("not in the configs / any HF safetensors repo — both
  paths work").
- ADDING_MODELS.md: reframed catalog-onboarding vs just-run-a-model
  (`pull`); the doc is the heavier calibration-catalog promotion task,
  not a prerequisite for serving.
- GLOSSARY.md: new "Universal pull (v0.8.0)" table (pull, dry-run,
  confidence tier, boot-fit≠runtime, calibration backbone).

Leak-clean; all links resolve on master; CommonMark structure verified
(blockquotes/headings blank-line separated). Docs-only — triggers no CI
(only tags do); lands as post-v0.8.0 polish on master.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
noonghunna
2026-05-17 17:02:19 +00:00
co-authored by Claude Opus 4.7
parent b720840d2e
commit afe56f763f
6 changed files with 25 additions and 3 deletions
+3 -1
View File
@@ -1,6 +1,8 @@
# Adding a model to the club-3090 stack
End-to-end workflow for onboarding a new model into the v0.7.0 profile catalog + serving infrastructure. Pairs with [KV_MATH.md](KV_MATH.md) (math reference) and [ARCHITECTURE.md](ARCHITECTURE.md) (current stack state).
End-to-end workflow for onboarding a new model into the **curated profile catalog** + serving infrastructure. Pairs with [KV_MATH.md](KV_MATH.md) (math reference) and [ARCHITECTURE.md](ARCHITECTURE.md) (current stack state).
> **Just want to run a model, not add it to the catalog?** As of v0.8.0 you don't need this workflow — `scripts/pull.sh <org/Model> --profile-like vllm/minimal` evaluates *any* safetensors HF repo against the KV math and boots it if it passes (see [docs/PULL.md](PULL.md)). This page is for the heavier task of promoting a model into the **measured/calibration catalog** (real benchmarks, validated composes, calibration anchors, per-model gotchas) — the high-confidence backbone, not a prerequisite for serving.
## When to add a new model vs a new quant of an existing one
+2
View File
@@ -2,6 +2,8 @@
You have **2× RTX 3090s**. This page is the front door for picking a config and knowing what dual-card unlocks vs single. Model-specific deep dives (quants, Genesis, engine internals) live in the model directory — links at the bottom.
> **Model not in the configs below / want any HF safetensors repo?** → [`docs/PULL.md`](PULL.md): `scripts/pull.sh` evaluates any model against the KV math (honest, no download) and boots it if it passes. The curated configs on this page are the measured path; both work.
**NVLink auto-detection** (since 2026-05-14): the dual-card composes now auto-detect whether an NVLink bridge is present. If you have one, you get the NVLink-optimized path automatically. If not, PCIe mode is used. Override with `NVLINK_MODE=force_on|force_off` in your `.env`. See the "NVLink auto-detection" section below.
> **Have 3+ GPUs?** See [`MULTI_CARD.md`](MULTI_CARD.md) — derivation of TP=4 / TP=8 configs from `dual.yml`, valid TP values for Qwen3.6-27B (1, 2, 4, 5, 8, 10), and what scales vs what doesn't.
+6 -2
View File
@@ -64,9 +64,13 @@ Use LM Studio if you prefer a GUI and don't need the engineering. Use this repo
We tried EAGLE — it's blocked on Qwen3-Next (the family Qwen3.5/3.6 belong to) by DeltaNet hybrid attention's lack of KV rollback support in vLLM/SGLang. MTP works because it's a different protocol (multi-token prediction at draft-head level, not a separate draft model). See [INTERNALS.md "Speculative decoding"](../models/qwen3.6-27b/INTERNALS.md) for the full forensic chain. **Re-test triggers:** if vllm#39931 lands or DeltaNet rollback support arrives upstream, EAGLE becomes viable again.
### The model I want isn't in the supported list — can I still run it?
Yes, if it's a **safetensors** repo. As of v0.8.0, `scripts/pull.sh <org/Model> --profile-like vllm/minimal --dry-run` evaluates *any* safetensors HF repo against this stack's KV math — no download — and tells you honestly whether it fits and at what confidence. Drop `--dry-run` (add `--yes`) and, if it passes the gates, it downloads, generates a minimal compose, and boots it. Non-fits stop with a precise reason, not a crash. Full guide: [docs/PULL.md](PULL.md). Limits: safetensors + vLLM only; GGUF / `.bin` repos hard-block with a clear message (see next Q).
### Why not GGUF on vLLM for this model?
Multiple gates blocked. Qwen3.6-27B GGUF on vLLM hits a chain of "fixed but-not-quite" issues — multimodal config routing, ParallelLMHead skip, the `Qwen35TensorProcessor._reverse_reorder_v_heads` weight loader producing garbage output on the 27B layout (transformers PR #45283 only validated on 0.8B). Tracked in [INTERNALS.md](../models/qwen3.6-27b/INTERNALS.md#qwen36-27b-gguf-on-vllm). Use llama.cpp for GGUF on this model.
Multiple gates blocked. Qwen3.6-27B GGUF on vLLM hits a chain of "fixed but-not-quite" issues — multimodal config routing, ParallelLMHead skip, the `Qwen35TensorProcessor._reverse_reorder_v_heads` weight loader producing garbage output on the 27B layout (transformers PR #45283 only validated on 0.8B). Tracked in [INTERNALS.md](../models/qwen3.6-27b/INTERNALS.md#qwen36-27b-gguf-on-vllm). Use llama.cpp for GGUF on this model. **Note (v0.8.0):** `pull` evaluates *safetensors* repos only — GGUF→llama.cpp is **not** served via `pull` (it stays the curated/manual path; cross-engine generation is deliberately deferred). A GGUF repo hard-blocks at derive with a clear message, not a crash.
### Why AutoRound INT4 not GPTQ / AWQ?
@@ -134,7 +138,7 @@ If your numbers on the same compose look different from ours by >15%, the most l
For a first install, run `bash scripts/setup.sh` with no model argument in a normal terminal. It opens a hardware-aware model picker, marks Qwen / Gemma / Both as eligible or not for your detected GPUs, then continues into the existing download flow.
After setup, run `bash scripts/launch.sh`. The wizard asks which model (filtered to what you've downloaded), then which GPU(s) to use, auto-picks TP for homogeneous sets (PP for heterogeneous), filters variants by hardware fit, shows a per-card VRAM projection from `tools/kv-calc.py` for the suggested default, then boots and runs `verify-full.sh`. Power-user forms still work: `bash scripts/setup.sh qwen3.6-27b`, `bash scripts/launch.sh --variant vllm/dual`, partial flags like `bash scripts/launch.sh --model qwen3.6-27b --gpus 0,1` (skips prompts), `--tp 4 --pp 2` to override parallelism, plus `setup.sh --help` / `launch.sh --help` for the full flag list.
After setup, run `bash scripts/launch.sh`. The wizard asks which model (filtered to what you've downloaded), then which GPU(s) to use, auto-picks TP for homogeneous sets (PP for heterogeneous), filters variants by hardware fit, shows a per-card VRAM projection from `tools/kv-calc.py` for the suggested default, then boots and runs `verify-full.sh`. Power-user forms still work: `bash scripts/setup.sh qwen3.6-27b`, `bash scripts/launch.sh --variant vllm/dual`, partial flags like `bash scripts/launch.sh --model qwen3.6-27b --gpus 0,1` (skips prompts), `--tp 4 --pp 2` to override parallelism, plus `setup.sh --help` / `launch.sh --help` for the full flag list. This wizard covers the **curated catalog**; for a model *not* in the catalog (any safetensors HF repo), use `scripts/pull.sh` instead — see [docs/PULL.md](PULL.md).
### `bash scripts/setup.sh qwen3.6-27b` is downloading 20+ GB. Where does it go? / Can I put models on a different drive?
+10
View File
@@ -6,6 +6,16 @@ Plain-language definitions for terms used throughout the docs. Roughly grouped b
---
## Universal `pull` (v0.8.0)
| Term | What it means |
|---|---|
| **`pull`** | `scripts/pull.sh <hf-repo> --profile-like <key>` — evaluates *any* safetensors HF repo against this stack's KV math and, if it passes the gates, downloads + generates a minimal compose + boots it. The model-agnostic front door; the curated catalog still works unchanged. See [PULL.md](PULL.md). |
| **dry-run** | `pull … --dry-run` — evaluate only: never downloads, never boots, just prints the verdict. |
| **confidence tier** | How much the fit verdict is trusted: `exact` (a measured/curated profile) vs `estimated-lower-bound` (derived from the repo's own config — a floor, likely under-modeled). Always shown with the verdict. |
| **boot-fit ≠ runtime-stability** | A "fits" verdict is a *boot-time* allocation check. It's necessary-not-sufficient: a config that boots clean can still degrade/OOM under sustained accumulated-context agent workloads (see [CLIFFS.md](CLIFFS.md)). Validate with soak-continuous before relying on it. |
| **calibration backbone** | The curated catalog's role under v0.8.0 — the measured anchor set the KV math is calibrated against (vs. being the only supported models). |
## Throughput / latency
| Term | What it means |
+2
View File
@@ -7,6 +7,8 @@ This page explains what scales (and what doesn't) when going beyond TP=2,
the constraints to know, and how to derive your own compose when `multi4.yml`
isn't your topology.
> **Model not in the configs here / want any HF safetensors repo?** → [`docs/PULL.md`](PULL.md): `scripts/pull.sh` evaluates any model against the KV math (honest, no download) and boots it if it passes. The recipes on this page are the measured/derivation path; both work.
> **Validation note:** the maintainer rig is **2× RTX 3090 PCIe**, but
> Whamp's 4× RTX 3090 PCIe rig validated the TP=4 fp8/MTP baseline in
> [discussion #26](https://github.com/noonghunna/club-3090/discussions/26)
+2
View File
@@ -2,6 +2,8 @@
You have **one RTX 3090 (24 GB VRAM)**. This page is the front door for picking a config and knowing what to expect. The model-specific deep dives (quants, Genesis patches, engine internals) live elsewhere — links at the bottom.
> **Model not in the configs below / want any HF safetensors repo?** → [`docs/PULL.md`](PULL.md): `scripts/pull.sh` evaluates any model against the KV math (honest, no download) and boots it if it passes. The curated configs on this page are the measured path; both work.
---
## ⚠️ Critical — read first if you're running an agentic coding client