Commit Graph

8 Commits

Author SHA1 Message Date
noonghunna
fb114bfde9 docs(ARCHITECTURE): drop pinned version on 'surfaces (current)' marker
Was '**v0.8.2 surfaces (current):**' — stale at v0.8.7. The four pull-gate
bullets are still the current surface, so reword to '**Pull-gate surfaces
(current):**' (no pinned version → can't go stale on the next release).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 23:37:07 +00:00
noonghunna
c5c8f469b0 docs(architecture): bring current-state docs up to v0.8.2 (recommend / submit on-ramp / arch-registry / hwdetect)
ARCHITECTURE.md described the v0.8.0 pull→gate→emit→boot→loop pipeline but
not v0.8.2's user surfaces. Adds: the --recommend verdict UX, the
--submit-last/--submit failure on-ramp, the broadened arch registry
(native built-ins reach a clean serve verdict, per-repo remote-code still
fail-closed), and the optional non-NVIDIA hwdetect subprocess; scripts-tree
pull.sh line annotated. README universal-pull bullet refreshed
(v0.8.0; extended in v0.8.2 — recommend + on-ramp + wider coverage).
Docs only; no behaviour change.
2026-05-18 23:58:42 +00:00
noonghunna
bef766d4ae v0.8.0 [docs] PULL.md Quickstart (command-first, top-of-doc) + ARCHITECTURE one-liner: stage names are internal, users run one command
User feedback: the runnable command sat ~40 lines below prose. Adds a
copy-paste Quickstart (the one command + the 3 outcomes/exit codes) at the
TOP of docs/PULL.md, and a one-liner in ARCHITECTURE.md clarifying the
[C0]/[B]/[C1] stage taxonomy is internal — users just run scripts/pull.sh.
Verified: the referenced `--list` flag does NOT exist (dropped — cited a
real COMPOSE_REGISTRY key + the Usage section instead); leak-clean; docs-only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 12:20:35 +00:00
noonghunna
1cdda195a7 v0.8.0 [docs] ARCHITECTURE.md: add the universal pull→gate→emit→loop pipeline to the mental model + scripts tree (current-state, was stale for v0.8.0)
ARCHITECTURE.md is the canonical current-state orientation doc but had no
mention of pull.sh / generate-compose.sh / the gate→emit→loop pipeline /
.pull-captures. Adds a "Two ways in: curated catalog + universal pull"
section (curated = measured backbone; pull = honest, never-silently-passes
front door) with the pipeline diagram + the boot-fit≠runtime caveat, and
updates the scripts/ tree. Present-tense current-state voice; leak-clean;
all links resolve. Ships with v0.8.0 under #147.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 11:48:39 +00:00
noonghunna
3742244e4d Split charts per GPU-count page; chart sources land in tools/charts/
SINGLE_CARD.md and DUAL_CARD.md embed scoped charts now — single-card
configs only on the single page, dual on the dual page. Combined views
stay on top-level README and the model README.

- docs/img/performance-{single,dual}.{svg,png} — new, scoped TPS charts
- docs/img/vram-budget-{single,combined}.{svg,png} — new
- docs/img/vram-budget-dual.{svg,png} — content swap: was combined,
  now genuinely dual-only. Old combined content lives in -combined.
- tools/charts/gen-{perf,vram}.py — matplotlib sources, idempotent.
  Re-run with: uv run --with matplotlib --with numpy python3 ...

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 14:42:09 +00:00
noonghunna
26ac8118de Restructure docs around hardware axis: SINGLE_CARD.md + DUAL_CARD.md
User feedback: navigating to relevant docs was cumbersome. The natural
first decision is "1 GPU or 2 GPUs?" and the existing docs mixed model-
specific reference with deployment guidance.

New navigation:
- README.md adds a "Pick your path" pivot pointing at hardware-axis pages
- docs/SINGLE_CARD.md — 1× 3090 deployment menu (workload → compose →
  TPS) with all single-card configs (vLLM + llama.cpp), VRAM budget,
  prefill cliffs explained operationally, what single-card can't do
- docs/DUAL_CARD.md — 2× 3090 mirror (4 dual variants + TP=2 explainer +
  what dual unlocks vs single + Marlin pad fork dependency)

Slimming:
- models/qwen3.6-27b/README.md: dropped duplicated variant tables
  (now in GPU-count pages); kept model-specific content (quants, Genesis
  patch surface table, what's working / not, VRAM diagram)
- models/qwen3.6-27b/USE_CASES.md: deleted. Per-workload content
  absorbed into the GPU-count pages (deduplicated). Troubleshooting
  list moved to docs/FAQ.md as a new "Troubleshooting" subsection.
  Image-token cost / vision specifics absorbed into SINGLE_CARD.md.

Reference updates: 8 files updated (engines/VLLM.md, engines/README.md,
COMPARISONS.md, ARCHITECTURE.md, EXAMPLES.md, FAQ.md, INTERNALS.md,
top-level README) — all USE_CASES.md links re-pointed to SINGLE_CARD/
DUAL_CARD where appropriate.

Net delta: -167 lines (was 197 in USE_CASES + duplicated tables in
model README; now 342 lines split between SINGLE_CARD + DUAL_CARD with
content deduplicated against each other).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 13:44:16 +00:00
noonghunna
5060e22a6c Split verify-full.sh → verify-full.sh (fast functional) + verify-stress.sh (boundary)
Recent additions to verify-full.sh (#8 tool-prefill OOM, #9 cascade
detection, #10 MTP AL) made the script slow — the longctx needle ladder
(#7) alone could run 5+ min, and the full 10-check suite was approaching
10 min. Awkward for "is the stack functional" iteration during dev work.

verify-full.sh (8 fast checks, ~1-2 min)
  1. Server reachable
  2. Genesis patches applied
  3. Basic completion (Paris)
  4. Tool calling
  5. Streaming (SSE)
  6. Thinking / reasoning mode
  7. Output quality / cascade detection (was #9)
  8. MTP acceptance length threshold (was #10)
Run: after every config change to confirm the stack still serves cleanly.

verify-stress.sh (2 boundary checks, ~5-10 min)
  1. Long-context needle ladder (4 depths, 10K / 30K / 60K / 90K) — was #7
  2. Tool response prefill OOM (~25K-token mock tool message) — was #8
Run: before publishing or when investigating prefill-OOM regressions
specifically.

Smoke-tested against dual.yml on dual-card:
  verify-full.sh:    8/8 green in 65 seconds
  verify-stress.sh:  2/2 green (skipped longctx for this smoke), 15s

Same env-var conventions (URL, MODEL, CONTAINER, SKIP_LONGCTX,
SKIP_TOOL_PREFILL, PREFILL_TARGET_CHARS).

Doc updates:
  - top-level README repo layout: lists both scripts with timing/scope
  - docs/ARCHITECTURE.md: scripts/ section + design rules updated
  - models/qwen3.6-27b/USE_CASES.md: tool-prefill reference points at
    verify-stress.sh now
  - models/qwen3.6-27b/CHANGELOG.md: dated entry documenting the split
2026-04-28 12:44:07 +00:00
noonghunna
3fa33332ce Initial commit — club-3090: model-agnostic LLM serving recipes for RTX 3090
Consolidates and supersedes:
  - noonghunna/qwen36-27b-single-3090
  - noonghunna/qwen36-dual-3090

The two predecessor repos partitioned by card count (1× vs 2×). This
repo partitions by engine instead, which matches how users actually
decide ("vLLM or llama.cpp?" before "1 card or 2"). Card count becomes
a config variant within each engine.

Structure (model-agnostic from day 1):

  docs/                       cross-model engine + hardware docs
    engines/                    vLLM / llama.cpp / SGLang comparison + per-engine deep dives
    HARDWARE.md                 Ampere SM 8.6+, NVLink, power, VRAM ceilings
    GLOSSARY.md                 plain-language definitions
    img/                        illustrations (vram-budget.svg)
    ARCHITECTURE.md             how this stack thinks about LLM serving on 24 GB

  models/<model-name>/        everything specific to a model
    qwen3.6-27b/                today's only model
      README.md / INTERNALS.md / USE_CASES.md / CHANGELOG.md
      vllm/                     vLLM-specific configs for this model
        compose/                  docker-compose files (single + dual variants)
        patches/                  tolist_cudagraph + Marlin pad notes
      llama-cpp/                llama.cpp recipes for this model
        recipes/                  shell scripts (single-card default + 262K max-ctx)
      sglang/                   SGLang status (currently blocked)

  scripts/                    shared, model-aware
    setup.sh                    bash setup.sh <model> → downloads + verifies
    verify.sh / verify-full.sh  smoke + functional tests
    bench.sh                    canonical TPS bench

vLLM compose variants (all under models/qwen3.6-27b/vllm/compose/):

  Single-card:
    docker-compose.yml              DEFAULT — TQ3 + Genesis P65, 48K, 51/68 TPS
    docker-compose.fast-chat.yml   fp8 + 20K, 55/70 TPS — fastest at small ctx
    docker-compose.tools-text.yml  fp8 + 75K, 53/70 TPS — best for long single prompts
    docker-compose.no-genesis-mtp.yml control variant
    docker-compose.minimal.yml     no spec-decode

  Dual-card:
    docker-compose.dual.yml              fp8 + 262K + MTP + vision, 71/89 TPS
    docker-compose.dual-turbo.yml       TQ3 + Genesis v7.14 — 4-stream concurrency
    docker-compose.dual-dflash.yml      DFlash N=5 + 185K + vision — 78/128 TPS
    docker-compose.dual-dflash-noviz.yml DFlash + 200K text-only

llama.cpp recipes (under models/qwen3.6-27b/llama-cpp/recipes/):

  single-card-default.sh    Q4_K_M + 65K
  single-card-max-ctx.sh    Q4_K_M + q4_0 KV at full 262K — the standout recipe

Old repos remain readable for issue history + external links (Medium,
Reddit, Twitter, Sandermage's PR threads). New issues should be filed
here.

Credits in README. Apache 2.0.
2026-04-28 10:24:14 +00:00