alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(vllm): opt-in offline/air-gap support across vLLM composes (#318)

Browse Source 
Add HF_HUB_OFFLINE + TRANSFORMERS_OFFLINE passthrough (default off) to all
31 vLLM composes, driven by one OFFLINE flag. Even with local weights and a
local --model path, vLLM/transformers resolve config/tokenizer metadata
against HF unless these are set, so air-gapped boots hang/fail (#318).
OFFLINE=1 (or either var) makes vLLM use local files only.

Compose only injects env vars it explicitly lists, so a host-shell
HF_HUB_OFFLINE never reached the container before this. Default 0 = unchanged
behavior online. Documents the air-gap flow in docs/FAQ.md (incl. the gated-
drafter gotcha for *-mtp composes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-06-05 09:01:39 +00:00
parent 47304065f4
commit edd77cfd46
32 changed files with 76 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/FAQ.md
View File

@@ -368,6 +368,20 @@ You'll usually find out you're behind before you ask: `launch.sh` and `switch.sh
If your *Genesis tree* (not the repo) is out of sync — the pin in `setup.sh` moved but you didn't re-run setup — `preflight_genesis_pin` warns separately and tells you to run `setup.sh`. That was the failure mode behind [#32](https://github.com/noonghunna/club-3090/issues/32) and wispborne's `_register_op_once` crash.
### How do I run fully offline / air-gapped (no Hugging Face access)?
Even with the weights already on disk and `--model` pointed at a local path, vLLM/transformers still reach out to Hugging Face to resolve config/tokenizer metadata — so a sealed network makes startup hang or fail. Two things:
1. **Set `OFFLINE=1`** (or the individual `HF_HUB_OFFLINE=1` / `TRANSFORMERS_OFFLINE=1`). Every vLLM compose passes these through to the container, default-off, so it's one flag:
```bash
OFFLINE=1 docker compose -f <compose>.yml up # or export it / put it in your .env
```
With it set, vLLM uses local files only and never phones home.
2. **Pre-download everything the compose loads — including gated drafters.** Speculative-decoding composes (`*-mtp.yml`, `dflash.yml`) also load an assistant/MTP drafter, often a **gated** repo (e.g. `google/gemma-4-12B-it-assistant`). If only the main model is local, boot will still try to fetch the drafter. Either grab it too (into `MODEL_DIR`, while you still have network), or drop the `--speculative-config` lines to run the main model alone.
`MODEL_DIR` is the directory *containing* the model folder; the container serves `/root/.cache/huggingface/<MODEL_SUBDIR>`. On WSL2, keep weights on ext4 (not `/mnt/<drive>`) — see [WSL_SETUP.md](WSL_SETUP.md).
### My GPU isn't card 0 — how do I change it?
Use the `--gpus` flag: `bash scripts/launch.sh --gpus 2` (single-card) or `bash scripts/launch.sh --gpus 2,3` (two cards). The wizard exports `CUDA_VISIBLE_DEVICES` for you. The older form `CUDA_VISIBLE_DEVICES=2 bash scripts/launch.sh --variant vllm/default` still works if you prefer to set the env yourself.