docs: use \$MODEL_DIR placeholder, not the dev rig's /mnt/models/huggingface/
User docs were hardcoding the dev rig path (/mnt/models/huggingface/...) as
if it was canonical. It's not — cross-rig users have models at /data/models,
~/models, /mnt/nvme/llms, etc. Setting MODEL_DIR per their setup is the
intended UX (the compose already supports it via env-var default).
Updates:
- models/qwen3.6-27b/llama-cpp/README.md: download examples now use
\$MODEL_DIR/qwen3.6-27b-gguf/ instead of /mnt/models/huggingface/...
- models/qwen3.6-27b/llama-cpp/compose/single/docker-compose.yml: header
comment uses \$MODEL_DIR/qwen3.6-27b-gguf/ for download examples + says
"MODEL_DIR=/your/models/dir docker compose up -d" instead of our path.
- docs/engines/LLAMA_CPP.md: same treatment + cleaned up Qwen3.5 + DFlash
draft path examples to also use \$MODEL_DIR.
- scripts/preflight.sh: hf download hint shows literal \${MODEL_DIR} so user
knows what to set, plus explicit "set MODEL_DIR first" line. Previously
echoed the resolved relative path (../../../../models-cache) which lands
outside the repo if pwd isn't the compose dir.
Caught by RobH589 in club-3090#116 — they hit the path-resolved-to-root-of-drive
case from the relative-path default. Closes the doc UX side; the compose's
env-var override mechanism was already correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This commit is contained in:
noonghunna
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
cf7f1959fd
commit
fbf343129c
+10
-10
@@ -86,7 +86,7 @@ This is exactly why our launch frame is **two routes, not one** ([README](../../
|
||||
|
||||
```bash
|
||||
# Use hf CLI (pip install 'huggingface-hub[hf_transfer]')
|
||||
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir /mnt/models/huggingface/qwen3.6-27b-gguf/
|
||||
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir $MODEL_DIR/qwen3.6-27b-gguf/
|
||||
```
|
||||
|
||||
Confirm size matches the HuggingFace listing. If a `sha256` is published, verify it.
|
||||
@@ -108,7 +108,7 @@ For a sane mid-context default (65K, plenty for chat + light agent work):
|
||||
|
||||
```bash
|
||||
/opt/llama.cpp/build/bin/llama-server \
|
||||
-m /mnt/models/huggingface/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
|
||||
-m $MODEL_DIR/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
|
||||
-c 65536 \
|
||||
--host 0.0.0.0 --port 8020 \
|
||||
-ngl 999 \
|
||||
@@ -131,7 +131,7 @@ Recipe (community-reported, validated by multiple users on r/LocalLLaMA):
|
||||
|
||||
```bash
|
||||
/opt/llama.cpp/build/bin/llama-server \
|
||||
-m /mnt/models/huggingface/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
|
||||
-m $MODEL_DIR/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
|
||||
-ngl 99 \
|
||||
-c 262144 \
|
||||
-np 1 \
|
||||
@@ -154,12 +154,12 @@ Sustained throughput at 262K with this config is typically **35-45 tok/s** on a
|
||||
|
||||
Download the `mmproj` model:
|
||||
```bash
|
||||
hf download unsloth/Qwen3.6-27B-GGUF mmproj-F16.gguf --local-dir /mnt/models/huggingface/qwen3.6-27b-gguf/
|
||||
hf download unsloth/Qwen3.6-27B-GGUF mmproj-F16.gguf --local-dir $MODEL_DIR/qwen3.6-27b-gguf/
|
||||
```
|
||||
|
||||
Add to launch:
|
||||
```bash
|
||||
--mmproj /mnt/models/huggingface/qwen3.6-27b-gguf/mmproj-F16.gguf
|
||||
--mmproj $MODEL_DIR/qwen3.6-27b-gguf/mmproj-F16.gguf
|
||||
```
|
||||
|
||||
### 5. Tool calls (limited)
|
||||
@@ -185,12 +185,12 @@ cmake -B build -DGGML_CUDA=ON
|
||||
cmake --build build --config Release -j
|
||||
|
||||
# Download draft model (~500 MB)
|
||||
hf download z-lab/Qwen3.6-27B-DFlash --local-dir /mnt/models/huggingface/z-lab/Qwen3.6-27B-DFlash/
|
||||
hf download z-lab/Qwen3.6-27B-DFlash --local-dir $MODEL_DIR/z-lab/Qwen3.6-27B-DFlash/
|
||||
|
||||
# Launch
|
||||
/opt/lucebox-hub/build/bin/llama-server \
|
||||
-m /mnt/models/huggingface/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
|
||||
--draft /mnt/models/gguf/qwen3.6-27b-dflash/dflash-N5.gguf \
|
||||
-m $MODEL_DIR/qwen3.6-27b-gguf/Qwen3.6-27B-Q4_K_M.gguf \
|
||||
--draft $MODEL_DIR/qwen3.6-27b-dflash-gguf/dflash-N5.gguf \
|
||||
--draft-max 5 \
|
||||
--draft-min 1 \
|
||||
-c 65536 \
|
||||
@@ -210,8 +210,8 @@ If you have two GPUs (e.g. 2× 3090), lucebox-hub now supports a heterogeneous-s
|
||||
```bash
|
||||
# Target on GPU 0, DFlash draft on GPU 1
|
||||
/opt/lucebox-hub/build/bin/llama-server \
|
||||
-m /mnt/models/gguf/qwen3.5-27b/Qwen3.5-27B-Q4_K_M.gguf \
|
||||
--draft /mnt/models/gguf/qwen3.5-27b-dflash/dflash-N5.gguf \
|
||||
-m $MODEL_DIR/qwen3.5-27b-gguf/Qwen3.5-27B-Q4_K_M.gguf \
|
||||
--draft $MODEL_DIR/qwen3.5-27b-dflash-gguf/dflash-N5.gguf \
|
||||
--target-gpu 0 --draft-gpu 1 \
|
||||
--draft-max 16 --draft-min 1 \
|
||||
-c 262144 \
|
||||
|
||||
@@ -30,7 +30,7 @@ Showcase: full **262K context** on one 3090 with vision + q4_0 KV.
|
||||
|
||||
```bash
|
||||
cd models/qwen3.6-27b/llama-cpp/compose
|
||||
MODEL_DIR=/mnt/models/huggingface docker compose up -d
|
||||
MODEL_DIR=/your/models/dir docker compose up -d
|
||||
```
|
||||
|
||||
Memory budget: 14.5 GB (Q3_K_XL) + 4.5 GB KV @ 262K + 0.8 GB mmproj ≈ 20 GB / 24 GB.
|
||||
@@ -66,7 +66,7 @@ The Q3_K_XL number at 262K is **lower than community-reported 35-45 tok/s** ([Re
|
||||
|
||||
```bash
|
||||
# 1. Get a GGUF quant (recommended: Unsloth's Q4_K_M)
|
||||
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir /mnt/models/huggingface/qwen3.6-27b-gguf/
|
||||
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q4_K_M.gguf --local-dir $MODEL_DIR/qwen3.6-27b-gguf/
|
||||
|
||||
# 2. Build llama.cpp with CUDA support
|
||||
git clone https://github.com/ggerganov/llama.cpp /opt/llama.cpp
|
||||
@@ -99,9 +99,9 @@ GGUFs of this model are at [unsloth/Qwen3.6-27B-GGUF](https://huggingface.co/uns
|
||||
## Vision (mmproj)
|
||||
|
||||
```bash
|
||||
hf download unsloth/Qwen3.6-27B-GGUF mmproj-F16.gguf --local-dir /mnt/models/huggingface/qwen3.6-27b-gguf/
|
||||
hf download unsloth/Qwen3.6-27B-GGUF mmproj-F16.gguf --local-dir $MODEL_DIR/qwen3.6-27b-gguf/
|
||||
|
||||
# Add to launch: --mmproj /mnt/models/huggingface/qwen3.6-27b-gguf/mmproj-F16.gguf
|
||||
# Add to launch: --mmproj $MODEL_DIR/qwen3.6-27b-gguf/mmproj-F16.gguf
|
||||
```
|
||||
|
||||
Vision works via the mmproj model. Sample text+image queries are OpenAI-compat.
|
||||
|
||||
@@ -43,15 +43,15 @@
|
||||
# 1. Get the GGUF + mmproj:
|
||||
# hf download unsloth/Qwen3.6-27B-GGUF \
|
||||
# --include "Qwen3.6-27B-UD-Q3_K_XL.gguf" "mmproj-F16.gguf" \
|
||||
# --local-dir /mnt/models/huggingface/qwen3.6-27b-gguf/unsloth-q3kxl
|
||||
# --local-dir $MODEL_DIR/qwen3.6-27b-gguf/unsloth-q3kxl
|
||||
# (mmproj sometimes ships at the repo root rather than under unsloth-q3kxl;
|
||||
# the MMPROJ env var below points at the canonical sidecar location.)
|
||||
# 2. From this directory:
|
||||
# MODEL_DIR=/mnt/models/huggingface docker compose up -d
|
||||
# MODEL_DIR=/your/models/dir docker compose up -d
|
||||
# 3. curl http://localhost:8020/v1/models → should list the model.
|
||||
#
|
||||
# Override defaults via .env or shell:
|
||||
# MODEL_DIR host dir to mount as /models (default: ../../../../models-cache for repo, /mnt/models/huggingface on this stack)
|
||||
# MODEL_DIR host dir to mount as /models (default: ../../../../models-cache for repo, /path/to/your/models on your stack)
|
||||
# GGUF_FILE path under /models (default: qwen3.6-27b-gguf/unsloth-q3kxl/Qwen3.6-27B-UD-Q3_K_XL.gguf)
|
||||
# MMPROJ_FILE path under /models (default: qwen3.6-27b-gguf/mmproj-F16.gguf)
|
||||
# CTX_SIZE total KV pool (default: 262144)
|
||||
|
||||
@@ -375,9 +375,10 @@ preflight_compose_deps() {
|
||||
if [[ $hint_gguf -eq 1 ]]; then
|
||||
echo "[preflight] hf download unsloth/Qwen3.6-27B-GGUF \\" >&2
|
||||
echo "[preflight] Qwen3.6-27B-UD-Q3_K_XL.gguf mmproj-F16.gguf \\" >&2
|
||||
echo "[preflight] --local-dir ${model_dir}/qwen3.6-27b-gguf/unsloth-q3kxl" >&2
|
||||
echo "[preflight] --local-dir \${MODEL_DIR}/qwen3.6-27b-gguf/unsloth-q3kxl" >&2
|
||||
echo "[preflight] # (set MODEL_DIR first: export MODEL_DIR=\${MODEL_DIR:-/path/to/your/models})" >&2
|
||||
echo "[preflight] # mmproj lands at unsloth-q3kxl/ — move it up so the default --mmproj path resolves:" >&2
|
||||
echo "[preflight] # mv ${model_dir}/qwen3.6-27b-gguf/unsloth-q3kxl/mmproj-F16.gguf ${model_dir}/qwen3.6-27b-gguf/" >&2
|
||||
echo "[preflight] # mv \${MODEL_DIR}/qwen3.6-27b-gguf/unsloth-q3kxl/mmproj-F16.gguf \${MODEL_DIR}/qwen3.6-27b-gguf/" >&2
|
||||
echo "[preflight] (~16 GB total. setup.sh today only fetches the vLLM AutoRound weights;" >&2
|
||||
echo "[preflight] GGUF must be fetched separately for any llamacpp/* variant.)" >&2
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user