alex/club-3090
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: laptop EC-managed power + TQ3 vs fp8 KV naming-trap; verify-stress: auto-bump curl timeout under VLLM_ENFORCE_EAGER

Browse Source 
Three documentation/script follow-ups from @easel's #102 re-bench
on RTX 5090 Laptop:

- HARDWARE.md: new "Laptop GPUs — EC-managed power" subsection.
  nvidia-smi -pl returns N/A on laptop-class GPUs (EC owns the
  envelope, not the OS). Documents clock-lock as the only
  software characterization path on laptops.

- CLIFFS.md: new "naming trap" callout in the KV-format section.
  fp8_e5m2 is 8 bits/token; turboquant_3bit_nc packs 3 bits.
  At 180K on 24GB, TQ3 fits where fp8 OOMs (4.36 GiB available
  vs 6.64 GiB needed for fp8). Pin: TQ3 = long-context KV;
  fp8 = short-context throughput.

- verify-stress.sh: auto-detect VLLM_ENFORCE_EAGER=1 in the
  running container's env via docker inspect; when set, bump
  STRESS_LONGCTX_TIMEOUT_S 300→600s and STRESS_TOOL_PREFILL_-
  TIMEOUT_S 240→480s. Eager-mode prefill at 60K-140K runs
  200-290s and was false-positiving as HTTP 000 (curl timeout)
  in @easel's run. Both env vars also exposed for manual override.

Refs: noonghunna/club-3090#102

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
noonghunna
2026-05-09 10:19:23 +00:00
parent 1528b591c3
commit fe23eff8f0
3 changed files with 67 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/CLIFFS.md
View File

@@ -417,6 +417,21 @@ On 24 GB / 3090 the per-card budget absorbs TQ3's activation peak and the smalle
The general principle: **the variant matrix is per-card-budget × KV-format-tradeoff aware**. Compose defaults are tuned for 24 GB / 3090; users on different VRAM classes may need to override `--kv-cache-dtype` to relocate the activation/pool balance for their hardware.
#### The naming trap — fp8 is *larger* than TQ3 per token
The KV-format names are misleading. **fp8_e5m2 stores 8 bits per cached element; turboquant_3bit_nc packs 3 bits.** At long single-prompt contexts on memory-tight rigs, this flips the conventional intuition:
| KV format | Bytes / cached token | Verdict at 180K on 24 GB single-card |
|---|---:|---|
| `turboquant_3bit_nc` (TQ3) | 0.375 | ✅ fits at 180K with mem-util 0.93 |
| `fp8_e5m2` | 1.0 | ❌ OOMs — needs 6.64 GiB KV pool, only 4.36 GiB available |
**Validated by [@easel #102](https://github.com/noonghunna/club-3090/issues/102#issuecomment-4412264989) on RTX 5090 Laptop**: switching from TQ3 → fp8_e5m2 at 180K context produced `ValueError: 6.64 GiB KV cache is needed, larger than available (4.36 GiB)`. Reverted to TQ3 and the boot succeeded.
**Pin**: on 24 GB single-card, **TQ3 is the long-context KV; fp8 is for short-context throughput**. The TQ3 activation-peak trade discussed above is real but ~1 GB; the fp8 KV-pool inflation at long-ctx is several GB. At 180K the activation-peak trade is dominated by the pool-size trade.
**On dual-card rigs** the analysis is the same per-card; TQ3 is still the long-context-on-tight-budget choice. fp8 starts to make sense again when (a) context is short enough that pool size doesn't dominate, or (b) you have generous per-card headroom (32 GB+ cards, or shorter max_model_len giving you VRAM to spare).
### Why llama.cpp doesn't have Cliff 2
llama.cpp's Qwen3-Next implementation processes DeltaNet/GDN layers with **online state updates** (incremental) rather than materializing the full intermediate. State is updated per-token or per-tile, never as a single multi-GB tensor. Different algorithm, different memory profile.