Conceptual narrative (hardware → engine → model size → quant → templates)
that the doc set lacked: GETTING_STARTED is commands-only, GLOSSARY is
isolated definitions, FAQ is troubleshooting. Cross-linked from README,
GETTING_STARTED, and GLOSSARY. Carries a scope banner (stack is
NVIDIA/3090-tested; other vendors are context, not a support commitment)
and an RTX 3090 FP8 caveat (no FP8 weight compute on Ampere; AWQ/GPTQ
for weights, fp8_e5m2 KV only). Links to PULL.md / KV_MATH.md for the
stack's actual fit-math path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Post-release additive pass — the pre-existing high-traffic guides didn't
know `pull` exists. All edits additive, curated path untouched (same
discipline as the README migration nudge):
- FAQ.md: new Q "model not in the supported list — can I still run it?";
GGUF Q gets a v0.8.0 note (safetensors-only eval, GGUF→llama.cpp stays
curated/manual, cross-engine deferred); launch.sh answer points
non-catalog models at `pull`.
- SINGLE_CARD / DUAL_CARD / MULTI_CARD: one blockquote cross-link each to
docs/PULL.md ("not in the configs / any HF safetensors repo — both
paths work").
- ADDING_MODELS.md: reframed catalog-onboarding vs just-run-a-model
(`pull`); the doc is the heavier calibration-catalog promotion task,
not a prerequisite for serving.
- GLOSSARY.md: new "Universal pull (v0.8.0)" table (pull, dry-run,
confidence tier, boot-fit≠runtime, calibration backbone).
Leak-clean; all links resolve on master; CommonMark structure verified
(blockquotes/headings blank-line separated). Docs-only — triggers no CI
(only tags do); lands as post-v0.8.0 polish on master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds docs/DTYPE_MATRIX.md — a reference table mapping NVIDIA GPU
architectures (Pascal → Blackwell DC) to native vs emulated Tensor Core
support for each dtype + quant scheme.
Sections:
- At-a-glance compute-dtype matrix (9 archs × 10 dtypes, ✓/SW/✗ marks)
- Weight-quantization schemes (GPTQ / AWQ / AutoRound / NF4 / SmoothQuant
/ FP8 weights / MXFP8 / MXFP4 / NVFP4 / GGUF / HQQ-AQLM-SqueezeLLM)
with storage-vs-compute paths
- Weight-only vs weight+activation axis (W4A16 vs W8A8 vs W4A8)
- KV-cache dtype support (FP16 / FP8 / INT8 PTH / TQ3 / TQ4 / k8v4)
- Per-arch compose recommendations (which compose to ship for which
GPU class)
- Runtime detection (points at Genesis guards.py)
- Corner cases (Ada FP8 vs Hopper FP8, Blackwell consumer vs DC, NVFP4
vs MXFP4 block-size differences, MX* family overview, FP6)
- References (NVIDIA whitepapers, Marlin, Genesis, BENCHMARKS)
Cross-linked from:
- HARDWARE.md GPU-compat table
- GLOSSARY.md Quantization section
- FAQ.md as a new "What dtype/quant should I pick for my GPU?" Q under
Hardware
This sets the foundation for future per-arch compose optimization —
detecting compute capability at boot and picking the right KV dtype /
weight quant scheme automatically based on what the hardware actually
accelerates rather than what's nominally supported.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
