7e9f5cb930
The shared served-model-name `qwen3.6-27b-autoround` (from #490) mislabels the non-autoround 27b scenes (fp8 dual-max, lmcache): /v1/models advertises "autoround" while `root` points at qwen3.6-27b-fp8. Same class on gemma-4-31b — after the v0.24.0 consolidation the default is cyankiwi qat-AWQ-INT4 (bf16 KV), yet the LiteLLM route still targeted `gemma-4-31b-autoround` (a latent #482 drift). Fix WITHOUT breaking anything, via vLLM multi-served-name: - Every 27b scene now serves `qwen3.6-27b <its-quant-name>`; every gemma-31b scene serves `gemma-4-31b <its-quant-name>`. The neutral name is PRIMARY (honest /v1/models id); the quant-specific name is retained as a live ALIAS. - LiteLLM: add `qwen3.6-27b` / `gemma-4-31b` canonical public routes; keep the `-autoround` routes as back-compat aliases (same upstream). Repairs the gemma drift. - Migrate our own MODEL= defaults + docs (bench/verify/quality/launch/setup, c3, tui-core, EXAMPLES, ...) to the neutral name. Weights slugs (`-autoround-int4`) untouched; CHANGELOG + results/ history left as-is. Retiring the `-autoround` alias entirely is a deliberate later step once nothing still asks for it. Live-verified on-rig (single/minimal, stock v0.24.0): /v1/models lists BOTH names (root=...-autoround-int4); chat to `qwen3.6-27b` AND `qwen3.6-27b-autoround` both return 200; `qwen3.6-27b-fp8` correctly 404s. Full shell gate 59/59 (1 = known worktree-fixture); c3 pytest 41 passed; served-name arg-order + YAML validated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
42 lines
2.0 KiB
Markdown
42 lines
2.0 KiB
Markdown
# Getting started — zero to `curl` in 5 minutes
|
|
|
|
The fastest path from `git clone` to serving your first response. No decisions, no menus — just commands.
|
|
|
|
> New to local AI and the terms below feel like jargon? Read [LOCAL_AI_PRIMER.md](LOCAL_AI_PRIMER.md) first — how hardware, engines, model sizes, and quants fit together in plain English.
|
|
|
|
```bash
|
|
# 1. Clone
|
|
git clone https://github.com/noonghunna/club-3090.git
|
|
cd club-3090
|
|
|
|
# 2. Download the model (Qwen3.6-27B, ~18 GB)
|
|
bash scripts/setup.sh qwen3.6-27b
|
|
|
|
# 3. Boot the default config for this model on your hardware
|
|
# (auto-picks: single-card → ik-llama/iq4ks-mtp; dual → vllm/dual)
|
|
bash scripts/launch.sh --variant qwen3.6-27b/default
|
|
|
|
# 4. Test it
|
|
curl -sf http://localhost:8020/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Capital of France?"}],"max_tokens":200}'
|
|
```
|
|
|
|
If you see `Paris` in the response, you're up and running.
|
|
|
|
> After the boot finishes, `launch.sh` asks **"Make `<slug>` your default for `qwen3.6-27b`? [y/N]"**. Say `y` and a bare `bash scripts/launch.sh` next time goes straight to that exact config — no flags, one keypress to launch. (Change or clear it anytime: `bash scripts/switch.sh --set-default <slug>` / `--clear-default qwen3.6-27b`.)
|
|
|
|
---
|
|
|
|
## Next steps
|
|
|
|
| You want | Go here |
|
|
|----------|---------|
|
|
| **Pick a config by workload** (long context, vision, dual-card, etc.) | [`docs/SINGLE_CARD.md`](SINGLE_CARD.md) or [`docs/DUAL_CARD.md`](DUAL_CARD.md) |
|
|
| **Understand the jargon** (TPS, KV, MTP, TP) | [`docs/GLOSSARY.md`](GLOSSARY.md) |
|
|
| **Client code snippets** (Python, curl, IDE setup) | [`docs/EXAMPLES.md`](EXAMPLES.md) |
|
|
| **Run the canonical benchmark** | `bash scripts/bench.sh` |
|
|
| **Update to the latest** | `bash scripts/update.sh` |
|
|
| **Hardware questions** (power caps, NVLink, 4090/5090) | [`docs/HARDWARE.md`](HARDWARE.md) |
|
|
| **File an issue or share bench numbers** | `bash scripts/report.sh --full > my-rig.md` and open an issue |
|