noonghunnaandClaude Opus 4.8
d3cacfad0c
The performance-max VibeThinker path — prithivMLmods Q8_0 GGUF on mainline llama.cpp, single 3090, q8_0 KV, full 131K, -b 4096 -ub 2048. A sibling to the vLLM bf16 compose (#418), and the better one on every axis. Why it's better than the vLLM sibling: - ~166 TPS decode (vs ~110 vLLM bf16), ~6.2 GB at full 131K (vs ~9.8 GB), prefill ~6,630 tok/s (-b 4096/-ub 2048 = +20%), boot ~9s. - Q8_0 is near-lossless → quality INTACT, where vLLM's fp8 weight-quant broke this quant-sensitive 3B (non-terminating empty output). KV quant is fine (storage-only); fp8 *weights* are the problem. Validated 2026-06-16 (tool-free packs, temp 0.6, thinking-on): gsm-symbolic-30 100% · instructfollow-15 100% · reasonmath-15 80% · structoutput-15 80% · dataextract-15 40%. dataextract is a genuine extraction ceiling — full temp sweep {0:27%, 0.6:40%, 1.0:27%} can't lift it (value mismatches persist). A reasoning specialist, not an extractor. No tool-calling. verify-full 5/9 by design. Wiring: registry entry + dense-family added to llama-cpp-local engine + prithivmlmods-q8 weights variant + disk/registry counts (51/52). No MTP head in the GGUF (plain qwen2 conversion) → no spec-dec. Guard suite 47/47. Status: 🐣 incubating (hidden from switch.sh --list; --force to launch). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
31 lines
1.2 KiB
YAML
31 lines
1.2 KiB
YAML
schema_version: 1
|
|
id: llama-cpp-local
|
|
display_name: llama.cpp local CUDA server (MTP-enabled)
|
|
type: llama.cpp
|
|
stability: stable
|
|
install:
|
|
method: docker_image
|
|
spec: ghcr.io/ggml-org/llama.cpp:server-cuda
|
|
min_sm: 6.0
|
|
supported_model_families:
|
|
- qwen3-next-hybrid
|
|
- qwen3-next-moe
|
|
- qwen35-dense # Deckard-40B (standard GQA, arch=qwen35; MTP via embedded nextn head)
|
|
- gemma4-unified
|
|
- dense # plain transformer GGUFs (e.g. VibeThinker-3B, Qwen2 dense — no MTP)
|
|
features: {}
|
|
supported_kv_formats:
|
|
- q4_0
|
|
- q5_0
|
|
- q8_0
|
|
- k8v4
|
|
supported_drafters:
|
|
- mtp
|
|
supported_weight_formats:
|
|
- gguf
|
|
required_overlays: []
|
|
vendored_overlays: []
|
|
required_genesis: false
|
|
notes: "Cliff-immune single-card fallback for Qwen GGUF serving. Uses upstream `ghcr.io/ggml-org/llama.cpp`, pinned to build `server-cuda-b9246` (NOT the rolling `:server-cuda` tag — it regressed at b9282 with a broken-lib crash loop, #187). MTP PR #22673 merged upstream + draft-eagle3 + draft-mtp spec types available natively. Bump via env once a newer build is validated: `LLAMACPP_IMAGE=ghcr.io/.../server-cuda-bXXXX`. No club-3090 patches on llama.cpp; we follow upstream as-is."
|
|
|