master
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
c25b9947ff |
c3+pull: detect in-progress downloads + per-repo lock (#617) (#644)
* c3+pull: detect in-progress downloads + per-repo lock (#617) Repeated ① Bring [D] presses spawned N concurrent `hf download` runs racing into the same `.incomplete` staging dir (5-deep live), each rmtree-ing and re-fetching the others' partial. Two disk-truth gaps behind that: 1. No dedup. `download_model` now acquires an atomic per-repo lock (`mkdir <pull_dir>/.download.lock` + a `pid` file = holder PID + UTC start) BEFORE the staging rmtree. A 2nd concurrent call for the same slug is REFUSED with `failure="in-progress"` (not a race); swap_apply surfaces it as `in_progress`, and `pull.sh --apply-swap` exits **rc=3** ("already downloading", distinct from rc=1 failure). A STALE lock (dead holder — a crashed/SIGKILL'd download that skipped `finally`) is reclaimed on the next call, so a leaked lock self-heals — more robust than a signal trap (a SIGKILL skips traps too). Released in `finally` on every return path. 2. No in-progress detection. Presence probes only saw the FINAL dir (empty until hf moves files out of `.incomplete`), so a mid-flight download read as absent → the false "download did not complete". `services.bring_download_in_ progress` now reads the lock (live PID) + `.incomplete` bytes for a best-effort pct; c3's fit-check render + [D] no-op guard + [k] cancel are disk-aware, so a running download is REFLECTED (and not duplicated) even across a c3 restart or a bare `pull.sh` started outside the session — the in-memory tracker (#643) couldn't see those. download_model is the shared chokepoint (bring apply-swap AND catalog fetch), so both paths get the lock; the body moved verbatim to `_download_model_impl` behind the locking wrapper — no behaviour change beyond the added guard. Tests: scripts/tests/test-download-lock.sh (live-refuse + stale-reclaim + release); c3 test_services (in-progress live/stale/absent + pct) + test_app_headless (disk-detected → downloading render, [D] no-op, [k] kill). 245 services/registry + 131 app blast-radius tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm * downloader: refuse a fresh pidless lock (close acquire-window race) The lock acquires with mkdir then writes its pid file a beat later. A 2nd caller landing in that window saw the lock dir but no pid → read_active returned None → the wrapper treated it as stale and rmtree'd + reclaimed it, letting BOTH callers proceed — the exact duplicate-download race the lock exists to prevent. Now: a pid-present-but-dead lock still reclaims (unchanged), but a PIDLESS lock only reclaims once it's older than a 10s grace — a fresh pidless lock is a holder mid-acquire and is refused, not stolen. Guard adds PASS 4 (fresh pidless → refuse) + PASS 5 (old pidless → reclaim). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm --------- Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
|
97d3523a9d |
fix(c3): hide [D] when brought weights on disk; ② Serve emits swap (#630)
A Route-C fit-check whose weights were already downloaded showed a self-contradiction: the verdict card hardcoded "→ Press [D] to download + serve" while the weights-line below it correctly said "✓ weights on disk → ② Serve". [D] is not only download — for a Route-C brought model it emits the serve-locally swap compose that ② Serve then serves — so it can't simply be hidden. Fix (presence-aware, "② Serve owns the emission"): - _byo_result_text takes weights_present (probed once in the fit-check handler, feeding BOTH the card and the weights-line so they can't disagree). On disk → the card points at ② Serve, no [D]; absent → the [D] download prompt as before. - pull.sh gains --emit-only (with --apply-swap): passes do_download=False so apply_swap emits the serve compose WITHOUT downloading — the present-weights path. - action_serve_untested: Route-C + weights present + no swap compose yet → run_bring_emit_and_serve emits via pull.sh --apply-swap --emit-only, then serves it. So a present-weights brought model serves straight from ② Serve with no [D] step. Tests: fixed test_bring_result_points_forward_to_serve (stale since #628 — it asserted "→ ② Serve" on an absent-weights card that now honestly shows [D]); added test_bring_result_present_points_to_serve (present → ② Serve, no [D]); extended the run_bring_download test for --emit-only. 139 touched-surface tests pass; pull.sh --emit-only validated end-to-end. Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm Co-authored-by: noonghunna <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
|
bc815b3272 |
feat(byo): apply-swap — download + serve a Route-C fine-tune (#628)
* feat(byo): apply-swap — download + serve a Route-C fine-tune
A curated-arch fine-tune (e.g. josefprusa/ThinkingCap-Qwen3.6-27B AutoRound
→ qwen3.6-27b) hard-stops at the pull gate's stratum-5 no-fit-model (correct
— nothing to price), so the c3 Bring [D] download failed with "download did
not complete". Route-C was pure guidance ("clone the compose, point --model
at your weights") with no code behind it — the deferred "swap_path apply".
Build it as a DISTINCT action that never touches the locked 6-stratum gate:
- scripts/lib/profiles/swap_apply.py (new): resolve the Route-C swap
(arch→curated sibling + has_mtp_head from deriver), download the brought
weights SHA-verified via downloader.download_model (a minimal EInput), and
emit a serve-locally compose that CLONES the --profile-like sibling's REAL
compose (keeping its curated chat-template / reasoning+tool parsers / MTP
wiring — NOT the derived-vllm template, which drops all that) with --model
re-pointed at a `<weights>:/brought-model:ro` mount, --served-model-name set
to the brought basename, and --speculative-config kept iff the checkpoint
carries an MTP head (else dropped). Written alongside the sibling compose as
`_brought-<san>.yml` (gitignored) so its relative ../ mounts resolve.
- scripts/pull.sh: intercept `--apply-swap` in the wrapper (strictly additive,
like --json) → run the apply-swap action; prints `[apply-swap] compose: …`.
The gate path is byte-unchanged without the flag.
- c3: services.run_bring_download gains `apply_swap=` (appends --apply-swap +
captures the emitted compose via last_swap_compose()); the [D] worker passes
it on route==C and stashes the compose; ② Serve serves that swap compose
directly (serve_generated) instead of a reproduction of the sibling's own
catalog compose.
- pull.py: fix the stale NOTE — "the bf16 base won't fit and lacks the MTP
head" → the base HAS the head; now just the size reason.
Tests: test-pull-swap.sh section 4 (emit: --model repointed, MTP kept/dropped
by head presence, curated flags + JSON args survive the YAML round-trip,
distinct container_name); test_services apply_swap flag + compose capture. The
locked-gate hard-stop tests stay green; --json fit-check byte-identical.
Validated live: pull.sh --json still no-fit-model+route-C, and apply_swap emits
a correct ThinkingCap swap compose (MTP kept).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
* c3: reframe Route-C fit-check card as ✓ Servable + a clear [D] action
The fit-check card for a curated-arch fine-tune (Route-C swap) was
self-contradicting: a RED "not eligible" + a bare "no-fit-model" token at
the top, then a GREEN "② Serve is armed with <sibling>" at the bottom —
which read as a dead-end AND named the wrong model (the sibling, not the
brought fine-tune). The engine's "no-fit-model" is the *means* (generic
fit-math can't price a curated-hybrid arch), not the user's answer; the
outcome is servable via the sibling's recipe.
Reframe the Route-C case in _byo_result_text:
- green "✓ Servable — a fine-tune of <sibling>" instead of red "not eligible",
- one plain "How it serves" line (reuses the sibling's chat-template / tools /
spec-dec with your weights) instead of the --quantization/impl bullets,
- MTP kept/dropped line from has_mtp_head,
- the next-step is an explicit "→ Press [D] to download + serve <BROUGHT model>"
(the fine-tune, not the sibling) — which #628's --apply-swap now fulfils,
- raw verdict + backend note dimmed for debugging, not the headline.
Non-swap cases (eligible / Route A / B / plain no-fit) are byte-unchanged.
Regression test added; c3 fast suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm
---------
Co-authored-by: noonghunna <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
noonghunnaandClaude Opus 4.8
|
fdbd68bc3a |
Detect MTP head in BYO Route-C swap instead of blanket spec-drop
The BYO fit-check's Route-C weight-swap (pull.sh _swap_path) hardcoded `drop_spec_config: True` on the premise "a generic repo carries no MTP head." That's false for a fine-tune that PRESERVES the head — so any such model (e.g. ThinkingCap-Qwen3.6-27B, whose AutoRound INT4 ships the full 15-tensor BF16 mtp head in model_mtp_bf16.safetensors) was silently served MTP-off, losing spec-dec despite the head being present. Fix: detect it. `deriver.detect_mtp_head(config, api)` returns True when the config DECLARES the MTP layers (`mtp_num_hidden_layers` / nested `num_nextn_predict_layers`) AND a dedicated mtp weights file is present — ground-truth for the separate-file layout fine-tune re-quants use, using signals the deriver already has in hand (no extra fetch). Exposed as an additive `has_mtp_head` on der.profile; _swap_path sets `drop_spec_config = not has_mtp_head`. An embedded-head repo (head baked into the shards with no named file) still falls back to drop — conservative, and the named-file layout is the norm. Validated on real HF metadata: josefprusa/ThinkingCap AutoRound → has_mtp_head=True (keep --speculative-config); a plain AWQ re-quant → False (drop). test-pull-swap replaces its tautology check with a real detect_mtp_head unit test (declares+file, plain, nested text_config, declares-but-no-file). Pull-gate suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01EfF565T9eSLaqGzidyJ1Pm |
||
|
noonghunnaandClaude Opus 4.8
|
d09a063317 |
Phase 2b: add --json/CLI contracts to 7 stack scripts (additive)
Data-layer contracts the cockpit (and any jq user) consumes — all strictly
additive (existing human output byte-identical), full guard suite green (54/54):
- registry-emit.sh --json : {variants,defaults,profiles{engines,models,hardware,drafters}}
- tools/kv-calc.py --fit <slug|model> --card <gpu> --json : structured fit verdict
- gpu-mode.sh --list-modes [--json] : scene catalog (serving/studio/ops)
- estate_cli.py report-state/diagnose --json : structured estate read
- pull.sh --profile-like --dry-run --json : structured swap_path (not a message blob)
- health.sh CONTAINER= : Doctor probes any engine container (was qwen36-27b-hardcoded)
- switch.sh --explain <slug> [--json] : joined registry/engine/model/hw/drafter + fit + bench
Built + adversarially reviewed via workflow. The review caught a real
switch<->kv-calc seam defect (switch fed hyphenated 'rtx-3090', kv-calc matched
only 'rtx3090' -> fit silently 'unavailable' on the 3090 rig); fixed: kv-calc
accepts hyphenated hardware-profile ids + hyphen-strip fallback; switch surfaces
kv-calc's structured verdict regardless of RC and renders its real keys; both
tests now exercise the seam. No shared-module edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
noonghunnaandClaude Opus 4.7
|
e1cdcb53c7 |
feat(pull): v0.8.2 STEP V2 — surface pointer + --submit-last/--submit (gh + gh-less, consented, F5 reuse)
CONTRACT-1.2: pull prints the honest one-line on-ramp pointer whenever a gate bundle was emitted for the run, keyed on the V1-recorded capture dir — explicitly NOT gated on the exit code (the bypassable no-arch-row C0 advisory path exits 0 yet emits the #1 §10-R9 bundle). Gate path stays I/O-free: a single stdout line, no network/prompt/auto-send. It does not classify (suppression is loop-side at submit). CONTRACT-1.3: scripts/pull.sh --submit-last / --submit <dir> is a distinct top-level verb parsed before the slug/--profile-like requirement. --submit-last re-reads the V1 shared .last marker at submit (the race defense — surfaces the CURRENT bundle, never a silent wrong-bundle). Re-shows bundle identity + the exact already-redacted payload, requires an explicit y before any network, then reuses the shipped F5 dedup.submit (effective_dedup_hash, bounded loop:dedup-<hash> labels, +1-or-open, collision-safe verify, suppression/review-queue) — not reimplemented. gh-less fallback runs post-F2 classification, gated on should_file: should_file=True -> a prefilled public issues/new URL with the loop:dedup-<hash> label and the deterministic title template; review- queued (unknown / correct-refusal) -> the local _review-queue spool path and the no-public-issue line, with NO public issues/new URL. Never raises; degrades to the local spool + printed paste-path. Console is never a submission source — only the redacted artifact is emitted. New scripts/tests/test-submit-pull.sh (mocked gh, zero network): the .last-marker race re-read, the bundle-emitted-but-exit-0 surfacing, F5-reuse, the gh-less should_file branch with no public URL for review- queued, gate-path I/O-free, and leak-hygiene. Full shipped suite green in the CI condition; kv-calc --calibration unchanged at 22/22; safetensors decision path byte-unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
noonghunnaandClaude Opus 4.7
|
adf7a3bf13 |
v0.8.0 Pull-Gate P4: stratum-5 + [C1] §4.1 total fn + stratum-6 [D] dry-run + pull orchestrator + exhaustive test-pull.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> |