feat(catalog): add 🐣 Incubating status tier + VibeThinker-3B (incubating)
Introduce a new pre-experimental status tier, `incubating`, for composes that work but aren't ready for the actionable list — niche specialists or models that fail the standard functional gate by design. Incubating composes are HIDDEN from `switch.sh --list` by default (revealed by `--list --all`) and launch-gated behind `--force` (non-functional), so a half-validated model is catalogued and discoverable without cluttering the recommended set. Tier wiring (reuses the existing status plumbing — no new emit column): - compose_registry.py: `incubating` in STATUS_VALUES + `🐣` in COMPOSE_STATUS_EMOJI (stays out of FUNCTIONAL_STATUSES → --force-gated, never auto-defaulted) - switch.sh: both --list filters skip incubating unless --all, with a `(+N incubating hidden — --all)` header note - AGENTS.md: Status enum row + Caveats-required references - docs/ADDING_MODELS.md: new rule — NEW MODELS START at 🐣 Incubating, promote up the enum (🐣 → 🧪 → ⚠️/✅) as they earn the actionable list First occupant — VibeThinker-3B (WeiboAI, Qwen2 dense reasoning fine-tune): - `vllm/vibethinker-3b-single` — bf16 weights + fp8_e5m2 KV, single 3090, full 131072 ctx, mem_util 0.40 (~9.8 GB, single-concurrency sized), --reasoning-parser qwen3, no tool-calling - Live-validated 2026-06-16: serves clean correct reasoning/code (~110 TPS), qwen3 parser splits <think>. fp8 WEIGHTS rejected (break the quant-sensitive 3B: non-terminating empty output). Always-reasoning + no-tools → fails verify-full's fixed-small-budget checks (5/9) by design → incubating, not gate-passing. Guard suite 47/47 green (nex-n2-mini parked aside; it's an unrelated local experiment). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
@@ -143,7 +143,7 @@ Every compose starts with a `Profile (at-a-glance)` block declaring the (Model,
|
||||
# Max ctx: <e.g. 262K>
|
||||
# Genesis: <none | v7.72.2 | N/A — Genesis is Qwen3-Next-specific>
|
||||
# Status: <REQUIRED — exactly one of the enum values below>
|
||||
# Caveats: <REQUIRED if Status is ⚠️ / 👁️ / ⏸️ / 🗑️; otherwise omit>
|
||||
# Caveats: <REQUIRED if Status is ⚠️ / 🐣 / 👁️ / ⏸️ / 🗑️; otherwise omit>
|
||||
# Quality: <OPTIONAL — populated by `bash scripts/quality-test.sh --medium`>
|
||||
# e.g. "ToolCall-15 14/15 (93%) · InstructFollow-15 13/15 (87%)
|
||||
# · StructOutput-15 15/15 (100%) · DataExtract-15 12/15 (80%)
|
||||
@@ -160,13 +160,14 @@ Every compose starts with a `Profile (at-a-glance)` block declaring the (Model,
|
||||
| `✅ Production` | Recommended for users. | verify-full 8/8 + verify-stress 7/7 + bench (BENCHMARKS row) + soak-continuous PASS + `quality-test.sh --quick` PASS (no ≥10pp regression on ToolCall / InstructFollow vs the pre-change baseline). Quality numbers on the compose's `Quality:` schema field when `--medium` has been run. |
|
||||
| `⚠️ Production w/ caveats` | Works under documented constraints; not the same as broken. | Same gates as Production, but a known-and-disclosed limitation exists (e.g., Cliff 2b at >50K, or a >10pp drop on a specific quality pack). Caveats line MUST list the constraint. |
|
||||
| `🧪 Experimental` | Under active validation; may not boot or pass all tests. | Typically untracked in git. No production guarantee. |
|
||||
| `🐣 Incubating` | Pre-experimental: works, but not ready for the actionable list — a niche specialist or one that fails the standard gate by design (e.g. an always-reasoning model with no tool-calling). | **HIDDEN from `switch.sh --list` by default** (revealed by `--list --all`), and launch requires `--force` (non-functional). Caveats line MUST state why it's not gate-passing. Promote to 🧪/✅ when it earns the actionable list. |
|
||||
| `👁️ Preview` | Known quality issues; tracked but not for production. | E.g., quality regressions in soak / NIAH. Caveats line MUST list specific issues. |
|
||||
| `⏸️ Upstream-gated` | Exists but blocked by external action (PR merge, driver fix, hardware ceiling). | Boots only with vendored override OR doesn't boot until external dep lands. Caveats line MUST point at the external dep. |
|
||||
| `🗑️ Deprecated` | Kept for historical reference; will be removed. | N/A — flagged for cleanup. |
|
||||
|
||||
**Why this enum exists**: the previous "Status optional, only when not production" convention left readers guessing whether absence-of-status meant "validated production" or "author forgot to fill it in." Making Status required + enumerated removes that ambiguity. Users picking a config can scan to one field and know the lifecycle stage instantly; new contributors must consciously declare it when authoring.
|
||||
|
||||
The `Caveats:` line is REQUIRED whenever Status is ⚠️ / 👁️ / ⏸️ / 🗑️, OMITTED for ✅ / 🧪. Format: a single-line summary or a short bullet list, with links to issues / discussions / upstream PRs where relevant.
|
||||
The `Caveats:` line is REQUIRED whenever Status is ⚠️ / 🐣 / 👁️ / ⏸️ / 🗑️, OMITTED for ✅ / 🧪. Format: a single-line summary or a short bullet list, with links to issues / discussions / upstream PRs where relevant.
|
||||
|
||||
This rule applies to **shipped composes AND local-only test composes** — apply the convention even before deciding whether to ship; it avoids a rename later if the experiment graduates.
|
||||
|
||||
|
||||
@@ -261,7 +261,9 @@ Place at `models/<model-id>/<engine>/compose/<topology>/<quant-slug>/<serving>.y
|
||||
|
||||
### Profile header (mandatory)
|
||||
|
||||
Every compose opens with the `# Profile (at-a-glance):` block — `Model` / `Topology` / `Drafter` / `KV` / `Vision` / `Max ctx` / `Genesis` / **`Status`** / `Best for` — plus a sibling-comparison table. **`Status:` is required**: exactly one of `✅ Production` · `⚠️ Production w/ caveats` · `🧪 Experimental` · `👁️ Preview` · `⏸️ Upstream-gated` · `🗑️ Deprecated`, with a `Caveats:` line whenever it's `⚠️`/`👁️`/`⏸️`/`🗑️`. `test-compose-status-drift` asserts the header status matches the registry entry, so a missing/mismatched `Status` **fails CI**. Full schema: [`CLAUDE.md`](../CLAUDE.md) → "Profile schema header." For a non-Qwen3-Next model write `Genesis: N/A — Genesis is Qwen3-Next-specific`.
|
||||
Every compose opens with the `# Profile (at-a-glance):` block — `Model` / `Topology` / `Drafter` / `KV` / `Vision` / `Max ctx` / `Genesis` / **`Status`** / `Best for` — plus a sibling-comparison table. **`Status:` is required**: exactly one of `✅ Production` · `⚠️ Production w/ caveats` · `🧪 Experimental` · `🐣 Incubating` · `👁️ Preview` · `⏸️ Upstream-gated` · `🗑️ Deprecated`, with a `Caveats:` line whenever it's `⚠️`/`🐣`/`👁️`/`⏸️`/`🗑️`. `test-compose-status-drift` asserts the header status matches the registry entry, so a missing/mismatched `Status` **fails CI**. Full schema: [`CLAUDE.md`](../CLAUDE.md) → "Profile schema header." For a non-Qwen3-Next model write `Genesis: N/A — Genesis is Qwen3-Next-specific`.
|
||||
|
||||
> **New models START at `🐣 Incubating`.** When you first add a model, ship its compose **and** registry entry at `status="incubating"` (header `🐣 Incubating` + a `Caveats:` line stating what's unvalidated). Incubating is **hidden from `switch.sh --list`** (revealed by `--list --all`) and launch-gated behind `--force`, so a half-validated model is catalogued and discoverable without cluttering the actionable list or being mistaken for a recommended config. **Promote up the enum as it earns it**: `🐣 Incubating` → `🧪 Experimental` (boots + serves cleanly, under active validation) → `⚠️`/`✅` once it clears the full gate (verify-full 8/8, verify-stress, bench row, soak, quality). Don't open a new model directly at `🧪`/`✅` — start incubating, then graduate.
|
||||
|
||||
### Required env-var hooks (post-v0.7.0)
|
||||
|
||||
@@ -312,8 +314,9 @@ COMPOSE_REGISTRY = {
|
||||
default_port=8040, # MUST equal the compose's ${PORT:-NNNN} fallback (parity test)
|
||||
kvcalc_key="<model-id>:dual", # vLLM: "<model>:<kvcalc-profile>"; llama.cpp/ik-llama/beellama: "SKIP"
|
||||
required_engine_features=["turboquant_3bit_nc"],
|
||||
status="production", # production|caveats|experimental|preview|upstream-gated|deprecated
|
||||
# maps to the compose header ✅/⚠️/🧪/👁️/⏸️/🗑️ (test-compose-status-drift checks both match)
|
||||
status="incubating", # NEW MODELS START HERE. production|caveats|experimental|incubating|preview|upstream-gated|deprecated
|
||||
# maps to the compose header ✅/⚠️/🧪/🐣/👁️/⏸️/🗑️ (test-compose-status-drift checks both match)
|
||||
# incubating = hidden from `switch.sh --list` (see --all), --force to launch; promote as it validates
|
||||
status_note=None, # REQUIRED non-None string when status is caveats/preview/upstream-gated/deprecated
|
||||
),
|
||||
}
|
||||
|
||||
5
models/vibethinker-3b/vllm/cache/.gitignore
vendored
Normal file
5
models/vibethinker-3b/vllm/cache/.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
||||
# Per-variant compile/Triton artifacts. These get regenerated on first boot
|
||||
# of any variant whose config changed; not source. Don't commit.
|
||||
*
|
||||
!.gitignore
|
||||
!README.md
|
||||
157
models/vibethinker-3b/vllm/compose/single/bf16/fp8.yml
Normal file
157
models/vibethinker-3b/vllm/compose/single/bf16/fp8.yml
Normal file
@@ -0,0 +1,157 @@
|
||||
# ===========================================================================
|
||||
# Profile (at-a-glance):
|
||||
# Model: VibeThinker-3B (WeiboAI — bf16 safetensors; a verifiable-reasoning
|
||||
# SFT+RL fine-tune of Qwen2.5-Coder-3B. Qwen2 dense, GQA 16h/2kv.)
|
||||
# Topology: Single 3090 (TP=1)
|
||||
# Drafter: none (no spec-decode — no MTP head, no matched draft model)
|
||||
# KV: fp8_e5m2 (storage-only — bf16 weights untouched, so NO quality hit; halves the
|
||||
# cache. At the default single-concurrency mem_util 0.40 → 174K-token / 1.33x pool,
|
||||
# ~9.8 GB total, frees ~14 GB to co-reside w/ a 27B — full 131K retained)
|
||||
# Vision: no (text-only)
|
||||
# Max ctx: 131072 (full native; RL-trained at a 64K window) — fits a 24 GB card with room
|
||||
# Genesis: N/A — Genesis is Qwen3-Next-specific; VibeThinker is Qwen2 dense
|
||||
# Status: 🐣 Incubating (pre-experimental — hidden from `switch.sh --list`; see `--list --all`)
|
||||
# Caveats: Does NOT pass the standard functional gate (verify-full 5/9): an always-reasoning
|
||||
# model (emits <think> before every answer, no way to disable) that FAILS fixed-small-
|
||||
# budget checks + has NO tool-calling. Niche math/code/STEM reasoning specialist, not a
|
||||
# general/agentic serve. temp 1.0 (card default) is unstable on short prompts — TEMP=0.6
|
||||
# is steadier. Kept in the catalog for the co-resident-reasoner experiment (#403-adjacent).
|
||||
# Best for: single-card verifiable reasoning — competition math / coding / STEM
|
||||
# (AIME, LiveCodeBench, LeetCode). NOT an open-domain-knowledge or
|
||||
# prose model (the authors say so) — pick a larger general model for that.
|
||||
# ---------------------------------------------------------------------------
|
||||
# VibeThinker-3B on vLLM — single 3090, bf16 weights + fp8_e5m2 KV.
|
||||
# A 3B reasoning model: ~5.8 GB of bf16 weights leave the bulk of a 24 GB card free.
|
||||
# KV is quantized to fp8_e5m2 (storage-only — weights stay bf16, so quality is intact;
|
||||
# A/B'd 2026-06-16: math/code answers identical to bf16 KV). Combined with single concurrency
|
||||
# (max_num_seqs=1) the default GPU_MEMORY_UTILIZATION is 0.40 — sized for ONE full-131K stream,
|
||||
# not a fat multi-seq pool: ~9.8 GB total (5.8 GB weights + a 174K-token / 1.33x fp8-KV pool),
|
||||
# which FREES ~14 GB on the card to co-reside VibeThinker with a 27B — full 131K context kept.
|
||||
# ~9.8 GB is near the floor: the bf16 weights (5.8 GB) can't shrink, and one 131K fp8-KV stream
|
||||
# needs ~2.4 GB + ~1 GB overhead (mem-util ~0.37 = the 1.0x floor; 0.40 keeps a prefill margin).
|
||||
# fp8 *weights* were REJECTED (see --dtype block): they break this quant-sensitive 3B.
|
||||
#
|
||||
# Reasoning behaviour (important for clients):
|
||||
# VibeThinker emits Qwen-style <think>...</think> reasoning blocks (the chat template
|
||||
# doesn't INJECT the tags, but the model GENERATES them — verified live 2026-06-16),
|
||||
# so --reasoning-parser qwen3 splits the trace into reasoning_content and leaves the
|
||||
# final answer in content. For math/code/STEM give it a GENEROUS max_tokens (8K–32K) —
|
||||
# the model's own generation_config defaults to max_new_tokens=2048, which truncates
|
||||
# real reasoning traces. That's a client-side sampling concern, not a serve flag.
|
||||
#
|
||||
# Sampling: the technical report's vLLM eval uses temperature=1.0, top_p=0.95,
|
||||
# top_k=-1 (disabled) — set as the override defaults below (all env-overridable).
|
||||
#
|
||||
# Tool calling: NOT supported. The chat template is Qwen2.5 hermes-style, but the RL
|
||||
# fine-tune does not emit the <tool_call> wrapper (it dumps bare JSON, and only after a
|
||||
# long reasoning trace), so no vLLM tool parser extracts it. The authors document no
|
||||
# function-calling. Tool support is intentionally NOT wired (verified 2026-06-16).
|
||||
#
|
||||
# ⚠️ Always-reasoning model — DO NOT pin a small max_tokens. It emits a <think> block
|
||||
# before every answer; with a tiny budget (e.g. 30-256 tokens) it truncates mid-reason
|
||||
# with NO answer. Omit max_tokens (vLLM defaults to a large budget → it reasons briefly
|
||||
# and answers) or set it generously (8K-40K; the authors use max_new_tokens up to 40960
|
||||
# for hard math/code). This is intrinsic — no system prompt or /no_think suppresses it.
|
||||
#
|
||||
# Override defaults via .env or shell:
|
||||
# MODEL_DIR host HF cache mounted at /root/.cache/huggingface
|
||||
# MAX_MODEL_LEN context window (default: 131072 — full native)
|
||||
# GPU_MEMORY_UTILIZATION (default: 0.40 — single-concurrency sized: weights +
|
||||
# a 1.33x 131K fp8-KV pool in ~9.8 GB, freeing ~14 GB.
|
||||
# ~0.37 is the 1.0x floor for full 131K; raise for
|
||||
# more concurrency; lower needs a smaller MAX_MODEL_LEN.)
|
||||
# MAX_NUM_SEQS concurrent sequences (default: 1 — single concurrency)
|
||||
# KV_CACHE_DTYPE K/V cache dtype (default: fp8_e5m2; set 'auto' for bf16 KV)
|
||||
# TEMP / TOP_P / TOP_K / MIN_P (defaults: 1.0 / 0.95 / -1 / 0.0)
|
||||
# PORT host port (default: 8074)
|
||||
# VLLM_IMAGE engine image (default: vllm/vllm-openai:v0.22.0)
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hardware metadata (parsed by scripts/preflight.sh):
|
||||
# Requires-min-vram-gb: 10
|
||||
# Engine-profile: vllm-stable
|
||||
# Requires-min-gpu-count: 1
|
||||
# Tensor-parallel: 1
|
||||
services:
|
||||
vllm-vibethinker-3b:
|
||||
image: ${VLLM_IMAGE:-vllm/vllm-openai:v0.22.0}
|
||||
container_name: "${ESTATE_CONTAINER:-vllm-vibethinker-3b}"
|
||||
restart: ${CLUB3090_RESTART:-unless-stopped}
|
||||
ports:
|
||||
- "${BIND_HOST:-0.0.0.0}:${ESTATE_PORT:-${PORT:-8074}}:8000"
|
||||
volumes:
|
||||
- ${MODEL_DIR:-../../../../../../models-cache}:/root/.cache/huggingface
|
||||
# torch.compile + Triton kernel caches — first boot warms (~60-90 sec);
|
||||
# subsequent boots reuse cached graphs.
|
||||
- ../../../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
|
||||
- ../../../cache/triton:/root/.triton/cache
|
||||
environment:
|
||||
- NVIDIA_VISIBLE_DEVICES=${ESTATE_GPUS:-${NVIDIA_VISIBLE_DEVICES:-all}}
|
||||
# Uncomment the next line to pin to a specific GPU (e.g. GPU 0):
|
||||
# - CUDA_VISIBLE_DEVICES=0
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
|
||||
- HF_HUB_OFFLINE=${HF_HUB_OFFLINE:-${OFFLINE:-0}}
|
||||
- TRANSFORMERS_OFFLINE=${TRANSFORMERS_OFFLINE:-${OFFLINE:-0}}
|
||||
- VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- NCCL_CUMEM_ENABLE=0
|
||||
- NCCL_P2P_DISABLE=1
|
||||
- VLLM_NO_USAGE_STATS=1
|
||||
- PYTORCH_CUDA_ALLOC_CONF=${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True}
|
||||
- OMP_NUM_THREADS=1
|
||||
shm_size: "16gb"
|
||||
ipc: host
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
entrypoint:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# VLLM_ENFORCE_EAGER=1 in .env disables CUDA graphs — use on
|
||||
# hardware where graph capture causes OOM or instability (e.g. WSL2).
|
||||
exec vllm serve ${VLLM_ENFORCE_EAGER:+--enforce-eager} "$@"
|
||||
- --
|
||||
command:
|
||||
- --model
|
||||
- /root/.cache/huggingface/vibethinker-3b
|
||||
- --served-model-name
|
||||
- vibethinker-3b
|
||||
- --dtype
|
||||
- "${DTYPE:-bfloat16}"
|
||||
# bf16 weights ONLY. fp8 weight-only quant (Marlin W8A16) was A/B'd 2026-06-16 and
|
||||
# BREAKS this 3B: it boots + halves VRAM (6.2->3.2 GB) + is faster (~164 vs ~110 TPS),
|
||||
# but the model loses its stop/format behavior — non-terminating, empty responses even
|
||||
# on "capital of France" (2048 tok empty vs bf16's clean 149). A 3B reasoning model is
|
||||
# too quant-sensitive for fp8 here. Do NOT re-enable without re-validating output.
|
||||
- --tensor-parallel-size
|
||||
- "${TP:-1}"
|
||||
- --pipeline-parallel-size
|
||||
- "${PP:-1}"
|
||||
- --max-model-len
|
||||
- "${MAX_MODEL_LEN:-131072}"
|
||||
- --gpu-memory-utilization
|
||||
- "${GPU_MEMORY_UTILIZATION:-0.40}"
|
||||
- --max-num-seqs
|
||||
- "${MAX_NUM_SEQS:-1}"
|
||||
- --kv-cache-dtype
|
||||
- "${KV_CACHE_DTYPE:-fp8_e5m2}"
|
||||
- --trust-remote-code
|
||||
# NO tool-calling: VibeThinker is a math/code/STEM reasoning model — the authors
|
||||
# don't support function-calling, and it emits bare JSON (not <tool_call>) so no
|
||||
# vLLM parser extracts it. Advertising tools would be misleading. (verified 2026-06-16)
|
||||
- --enable-prefix-caching
|
||||
- --enable-chunked-prefill
|
||||
# No --speculative-config (no MTP head / no matched drafter for this model)
|
||||
# VibeThinker emits Qwen-style <think>...</think> reasoning blocks (verified
|
||||
# live 2026-06-16) → qwen3 parser splits reasoning_content from the answer.
|
||||
- --reasoning-parser
|
||||
- qwen3
|
||||
- --override-generation-config
|
||||
- '{"temperature":${TEMP:-${TEMPERATURE:-1.0}},"top_p":${TOP_P:-0.95},"top_k":${TOP_K:--1},"min_p":${MIN_P:-0.0},"repetition_penalty":${REPEAT_PENALTY:-1.0}}'
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- "8000"
|
||||
@@ -18,6 +18,9 @@ STATUS_VALUES = (
|
||||
"production", # ✅ Production — recommended, fully validated.
|
||||
"caveats", # ⚠️ Production w/ caveats — works under documented limits.
|
||||
"experimental", # 🧪 Experimental — under active validation; may not boot.
|
||||
"incubating", # 🐣 Incubating — pre-experimental: works but not ready for the
|
||||
# actionable list (niche / fails the standard gate by design).
|
||||
# HIDDEN from `switch.sh --list` by default; revealed by `--all`.
|
||||
"preview", # 👁️ Preview — known quality issues; tracked, not for prod.
|
||||
"upstream-gated", # ⏸️ Upstream-gated — blocked by external action (pin/PR/HW).
|
||||
"deprecated", # 🗑️ Deprecated — kept for reference; flagged for removal.
|
||||
@@ -33,6 +36,7 @@ COMPOSE_STATUS_EMOJI = {
|
||||
"✅": "production",
|
||||
"⚠️": "caveats",
|
||||
"🧪": "experimental",
|
||||
"🐣": "incubating",
|
||||
"👁️": "preview",
|
||||
"⏸️": "upstream-gated",
|
||||
"🗑️": "deprecated",
|
||||
@@ -714,6 +718,19 @@ COMPOSE_REGISTRY = {
|
||||
status_note="Dense 40B uncensored Qwen3.6 merge (Q6_K MTP GGUF, 31 GB) on dual 3090 llama.cpp. Arch CONFIRMED qwen35-dense (standard GQA, 97 layers) from the GGUF header. MTP n=2 sweet spot (~41.6 tok/s, 0.81 accept). 128K ctx ceiling @q8_0 KV (192K OOMs). Dual-only. verify-full 8/8, verify-stress 8/8, 8-pack 105/150 (MTP off==on, spec-dec lossless), soak-continuous PASS (0 MiB growth, 0/25 silent-empty). First uncensored + first dual-llama.cpp compose in the catalog.",
|
||||
category="uncensored",
|
||||
),
|
||||
|
||||
# VibeThinker-3B — WeiboAI verifiable-reasoning fine-tune of Qwen2.5-Coder-3B
|
||||
# (Qwen2 dense). First dense-family + first sub-4B model in the catalog.
|
||||
"vllm/vibethinker-3b-single": _entry(
|
||||
model="vibethinker-3b", weights_variant="bf16", workload="long-ctx-single",
|
||||
engine="vllm-stable", drafter=None, kv_format="fp8_e5m2",
|
||||
tp=1, max_ctx=131072, max_num_seqs=1, mem_util=0.40,
|
||||
compose_path="models/vibethinker-3b/vllm/compose/single/bf16/fp8.yml",
|
||||
default_port=8074,
|
||||
kvcalc_key="SKIP",
|
||||
status="incubating",
|
||||
status_note="VibeThinker-3B (WeiboAI) — bf16 Qwen2 dense verifiable-reasoning model (SFT+RL fine-tune of Qwen2.5-Coder-3B) on a single 3090, vLLM v0.22.0 (vllm-stable). bf16 weights (~5.8 GB) + fp8_e5m2 KV (storage-only A/B'd 2026-06-16 — math/code answers identical to bf16 KV; halves cache, quality-neutral). full 131072 ctx, ~110 TPS. Single-concurrency sized: max_num_seqs=1 + mem_util 0.40 → ~9.8 GB total (174K-token / 1.33x KV pool, full 131K kept), freeing ~14 GB to co-reside with a 27B; ~9.8 GB is near the floor (5.8 GB bf16 weights immovable). Output quality is temp-governed not mem-governed: temp 0.6 coherent, the card's temp 1.0 is unstable on short prompts (degenerate loops) — overridable via TEMP. fp8 WEIGHTS rejected (break this quant-sensitive 3B: non-terminating empty output). Sampling per the tech report: temp 1.0 / top_p 0.95 / top_k -1. Live-validated 2026-06-16: serves clean, correct reasoning + code (verify-full output-quality + thinking-mode PASS); --reasoning-parser qwen3 splits <think> blocks correctly. ⚠️ ALWAYS-REASONING: it emits a <think> trace before every answer with NO way to disable it (system prompt / /no_think ignored; authors document no controls). Omit max_tokens (vLLM's large default → reasons briefly then answers) or set generously (8K-40K; authors use up to 40960); a small explicit max_tokens truncates mid-reason with no answer. NO tool-calling (emits bare JSON not <tool_call>; authors don't support it — intentionally unwired). Consequence: FAILS verify-full's fixed-small-budget checks (basic 30 / streaming 120 / tool 200-256 tok) → 5/9 — a harness-vs-always-reasoning mismatch, NOT a serving defect. Stays 🧪: does not pass the standard functional gate; math/code/STEM reasoning only, not general/agentic.",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
|
||||
40
scripts/lib/profiles/models/vibethinker-3b.yml
Normal file
40
scripts/lib/profiles/models/vibethinker-3b.yml
Normal file
@@ -0,0 +1,40 @@
|
||||
schema_version: 1
|
||||
id: vibethinker-3b
|
||||
display_name: VibeThinker-3B
|
||||
# WeiboAI verifiable-reasoning fine-tune (SFT+RL) of Qwen2.5-Coder-3B.
|
||||
# Standard Qwen2 dense decoder, GQA (16 attn heads / 2 KV heads), SWA disabled.
|
||||
# Emits Qwen-style <think>...</think> reasoning blocks → served with --reasoning-parser qwen3.
|
||||
family: dense
|
||||
hidden_size: 2048
|
||||
intermediate_size: 11008
|
||||
num_hidden_layers: 36
|
||||
num_attn_heads: 16
|
||||
num_kv_heads: 2
|
||||
head_dim_attn: 128
|
||||
max_ctx_supported: 131072
|
||||
attention_k_eq_v: true
|
||||
# kv-calc has no branch for the generic `dense` family (curated models get
|
||||
# family-specific branches; `generic-dense` is reserved for auto-derived pulls).
|
||||
# A 3B trivially fits a 24 GB card at full ctx, so KV-fit gating is moot here →
|
||||
# kv_calc_supported: false and the registry entry uses kvcalc_key="SKIP".
|
||||
kv_calc_supported: false
|
||||
vision_capable: false
|
||||
weights:
|
||||
bf16:
|
||||
path: vibethinker-3b
|
||||
local_subdir: vibethinker-3b
|
||||
size_gb: 6.2
|
||||
format: bf16
|
||||
status: experimental
|
||||
hf_repo: WeiboAI/VibeThinker-3B
|
||||
engine: vllm
|
||||
manual_note: >-
|
||||
Original WeiboAI bf16 release (~6.2 GB, 2 safetensors shards). A 3B reasoning
|
||||
model — runs the full 131072 native context single-stream on one 24 GB card
|
||||
with headroom; no KV quant needed. Sampling per the technical report:
|
||||
temperature=1.0, top_p=0.95, top_k=-1. Give clients a generous max_tokens
|
||||
(8K-32K) — the model reasons inline and its own generation_config caps at 2048.
|
||||
default_weight_variant: bf16
|
||||
valid_tp:
|
||||
- 1
|
||||
requires_genesis: false
|
||||
@@ -353,7 +353,7 @@ list_variants() {
|
||||
# Counts: split into VISIBLE vs HIDDEN by the hardware filter, so the header
|
||||
# reflects what's actually shown (+ how many were hidden). Health split is
|
||||
# over the VISIBLE set; the by-topology hidden tally drives the note.
|
||||
local _prod=0 _cav=0 _na=0 _hidden=0 _dep_hidden=0 _gated_hidden=0
|
||||
local _prod=0 _cav=0 _na=0 _hidden=0 _dep_hidden=0 _gated_hidden=0 _inc_hidden=0
|
||||
declare -A _seen_models=() _hidden_by_topo=()
|
||||
for v in "${!VARIANTS[@]}"; do
|
||||
IFS='|' read -r _e _d _f <<< "${VARIANTS[$v]}"
|
||||
@@ -361,12 +361,15 @@ list_variants() {
|
||||
IFS=/ read -ra _fs <<< "$_f"
|
||||
local _vtopo="${_fs[0]:-unknown}" _vrank
|
||||
_vrank="$(topology_rank "$_vtopo")"
|
||||
# Hide non-active statuses by default: deprecated (tombstoned / going away) and
|
||||
# upstream-gated (PARKED — blocked on an external fix, not abandoned). --all reveals both.
|
||||
# Hide non-active statuses by default: deprecated (tombstoned / going away),
|
||||
# upstream-gated (PARKED — blocked on an external fix, not abandoned), and
|
||||
# incubating (pre-experimental — works but not ready for the actionable list).
|
||||
# --all reveals all three.
|
||||
if [[ "$show_all" != "1" ]]; then
|
||||
case "${VARIANT_STATUS[$v]:-production}" in
|
||||
deprecated) _dep_hidden=$((_dep_hidden + 1)); continue ;;
|
||||
upstream-gated) _gated_hidden=$((_gated_hidden + 1)); continue ;;
|
||||
incubating) _inc_hidden=$((_inc_hidden + 1)); continue ;;
|
||||
esac
|
||||
fi
|
||||
if [[ "$show_all" != "1" && "$_vrank" -gt "$max_rank" ]]; then
|
||||
@@ -408,7 +411,11 @@ list_variants() {
|
||||
if [[ "$_gated_hidden" -gt 0 ]]; then
|
||||
_gated_note=" (+${_gated_hidden} parked/upstream-gated hidden — --all)"
|
||||
fi
|
||||
echo " Models: ${#_seen_models[@]} · variants: ${_visible} (${_prod} production · ${_cav} caveats · ${_na} NA)${_hidden_note}${_dep_note}${_gated_note}"
|
||||
local _inc_note=""
|
||||
if [[ "$_inc_hidden" -gt 0 ]]; then
|
||||
_inc_note=" (+${_inc_hidden} incubating hidden — --all)"
|
||||
fi
|
||||
echo " Models: ${#_seen_models[@]} · variants: ${_visible} (${_prod} production · ${_cav} caveats · ${_na} NA)${_hidden_note}${_dep_note}${_gated_note}${_inc_note}"
|
||||
|
||||
{
|
||||
for v in "${!VARIANTS[@]}"; do
|
||||
@@ -418,7 +425,7 @@ list_variants() {
|
||||
topo="${fseg[0]:-unknown}"
|
||||
rank="$(topology_rank "$topo")"
|
||||
if [[ "$show_all" != "1" ]]; then
|
||||
case "${VARIANT_STATUS[$v]:-production}" in deprecated|upstream-gated) continue ;; esac
|
||||
case "${VARIANT_STATUS[$v]:-production}" in deprecated|upstream-gated|incubating) continue ;; esac
|
||||
fi
|
||||
if [[ "$show_all" != "1" && "$rank" -gt "$max_rank" ]]; then
|
||||
continue
|
||||
|
||||
@@ -25,8 +25,8 @@ def check(cond, msg):
|
||||
print(f"FAIL: {msg}")
|
||||
failures.append(msg)
|
||||
|
||||
check(len(COMPOSE_REGISTRY) == 49, f"registry has 49 entries (got {len(COMPOSE_REGISTRY)})")
|
||||
check(len(disk_paths) == 50, f"disk has 50 compose files (got {len(disk_paths)})")
|
||||
check(len(COMPOSE_REGISTRY) == 50, f"registry has 50 entries (got {len(COMPOSE_REGISTRY)})")
|
||||
check(len(disk_paths) == 51, f"disk has 51 compose files (got {len(disk_paths)})")
|
||||
check(registry_paths <= disk_paths, "all registry compose_path values exist on disk")
|
||||
parked_disk_only = disk_paths - registry_paths
|
||||
# Disk-only (non-registry) composes allowed: parked SGLang archives, plus the experimental
|
||||
|
||||
@@ -24,7 +24,7 @@ run_test "load_profiles parses all profile groups" <<'PY'
|
||||
from scripts.lib.profiles.compat import load_profiles
|
||||
p = load_profiles()
|
||||
assert len(p.hardware) == 9
|
||||
assert len(p.models) == 7
|
||||
assert len(p.models) == 8
|
||||
assert len(p.workloads) == 5
|
||||
assert len(p.engines) == 12
|
||||
assert len(p.drafters) == 11
|
||||
|
||||
Reference in New Issue
Block a user