llama-cpp: switch to rolling :server-cuda tag (no patches → no pin needed)
Follow-up on c3e7c7e: pinning to a specific build (b9246) was cargo-culted from
our vLLM pattern, but the vLLM pinning serves a real purpose (Genesis-patch
anchor, Docker Hub purge resistance) that doesn't apply here. llama.cpp on the
club-3090 stack is stock upstream — no patches, no Genesis equivalent — and
GHCR tag retention is more reliable than Docker Hub.
Switch to rolling `:server-cuda` tag so users automatically get MTP improvements,
EAGLE3 fixes, kernel updates from upstream without us being a bottleneck.
Override path preserved via `LLAMACPP_IMAGE` env if a future upstream build
regresses and a user needs to pin reactively.
Bench numbers in #170 footnoted as "measured on b9246"; expect ±5% drift on
newer builds.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
c3e7c7ed80
commit
4a53edab43
@@ -22,7 +22,7 @@ For full pros/cons + general llama.cpp tuning, see [`/docs/engines/LLAMA_CPP.md`
|
||||
|
||||
## Docker compose (recommended)
|
||||
|
||||
Three compose variants in [`compose/single/`](compose/single/) — all use the official `ghcr.io/ggml-org/llama.cpp:server-cuda` image (pinned at build `b9246`, contains MTP PR #22673), **no custom build needed**. Override the pinned tag via `LLAMACPP_IMAGE` env if you want a newer build (do verify MTP behavior hasn't drifted before relying on changed numbers).
|
||||
Three compose variants in [`compose/single/`](compose/single/) — all use the official rolling `ghcr.io/ggml-org/llama.cpp:server-cuda` image, **no custom build needed**, **no club-3090 patches** (unlike our vLLM track). MTP PR #22673 has merged upstream so the rolling tag has it natively. Pin to a specific build via `LLAMACPP_IMAGE=ghcr.io/ggml-org/llama.cpp:server-cuda-bXXXX` env if you want to lock against upstream drift; otherwise `docker compose pull` picks up upstream's latest. Bench numbers were measured on build `b9246` (2026-05-20); expect ±5% drift on newer builds.
|
||||
|
||||
### `docker-compose.yml` — max context, single slot, vision
|
||||
|
||||
|
||||
@@ -111,7 +111,7 @@
|
||||
|
||||
services:
|
||||
llama-cpp-qwen36-27b:
|
||||
image: ${LLAMACPP_IMAGE:-ghcr.io/ggml-org/llama.cpp:server-cuda-b9246}
|
||||
image: ${LLAMACPP_IMAGE:-ghcr.io/ggml-org/llama.cpp:server-cuda}
|
||||
container_name: "${ESTATE_CONTAINER:-llama-cpp-qwen36-27b}"
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
|
||||
@@ -68,7 +68,7 @@
|
||||
|
||||
services:
|
||||
llama-cpp-qwen36-27b-mtp-vision:
|
||||
image: ${LLAMACPP_IMAGE:-ghcr.io/ggml-org/llama.cpp:server-cuda-b9246}
|
||||
image: ${LLAMACPP_IMAGE:-ghcr.io/ggml-org/llama.cpp:server-cuda}
|
||||
container_name: "${ESTATE_CONTAINER:-llama-cpp-qwen36-27b}"
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
|
||||
@@ -72,7 +72,7 @@
|
||||
|
||||
services:
|
||||
llama-cpp-qwen36-27b-mtp:
|
||||
image: ${LLAMACPP_IMAGE:-ghcr.io/ggml-org/llama.cpp:server-cuda-b9246}
|
||||
image: ${LLAMACPP_IMAGE:-ghcr.io/ggml-org/llama.cpp:server-cuda}
|
||||
container_name: "${ESTATE_CONTAINER:-llama-cpp-qwen36-27b}"
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
|
||||
@@ -5,7 +5,7 @@ type: llama.cpp
|
||||
stability: stable
|
||||
install:
|
||||
method: docker_image
|
||||
spec: ghcr.io/ggml-org/llama.cpp:server-cuda-b9246
|
||||
spec: ghcr.io/ggml-org/llama.cpp:server-cuda
|
||||
min_sm: 6.0
|
||||
supported_model_families:
|
||||
- qwen3-next-hybrid
|
||||
@@ -22,5 +22,5 @@ supported_weight_formats:
|
||||
required_overlays: []
|
||||
vendored_overlays: []
|
||||
required_genesis: false
|
||||
notes: "Cliff-immune single-card fallback for Qwen GGUF serving. Upstream `ghcr.io/ggml-org/llama.cpp:server-cuda` pinned at build b9246 (commit 871b0b70f, 2026-05-20) — MTP PR #22673 merged + draft-eagle3 + draft-mtp spec types available natively. Override via env: `LLAMACPP_IMAGE=ghcr.io/.../server-cuda-bXXXX`."
|
||||
notes: "Cliff-immune single-card fallback for Qwen GGUF serving. Uses upstream `ghcr.io/ggml-org/llama.cpp:server-cuda` (rolling tag — auto-bumps to latest upstream build). MTP PR #22673 merged upstream + draft-eagle3 + draft-mtp spec types available natively. Pin to a specific build via env: `LLAMACPP_IMAGE=ghcr.io/.../server-cuda-bXXXX` (e.g. if upstream regresses). No club-3090 patches on llama.cpp; we follow upstream as-is."
|
||||
|
||||
|
||||
Reference in New Issue
Block a user