kv-calc: calibrate gemma4_unified global-KV per-token to measured anchor
The 31B-derived gemma4-swa-dense global-only formula priced the 12B's growing KV at 32,768 B/tok/card (8 full x 8 kv x 512 x bpe2 / TP2), but the live gemma4_unified pool measured 22,816 B/tok/card (8.16 GiB / 384,019 tok @ 131K/TP2/0.90/MTP). Add a gemma-12b-only measured override (measured_kv_growing_bpt_tp1=45,632) in the gemma4-swa-dense KV branch — 31B keeps the formula (untouched, calibration byte-identical). Token-capacity prediction 204K -> 315K vs measured 384K; residual is the conservative shared cudagraph_overhead (handed to Codex to model principled-ly). Guard 41/41. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
noonghunna
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
aff9890720
commit
0f6f86195e
@@ -9,16 +9,21 @@ model: gemma-4-12b
|
||||
# implied non-KV resident = 21.6 − 8.76 (= 8.16 GiB) ≈ 12.84 GB/card.
|
||||
#
|
||||
# INTEGRITY NOTE: these rows are the measured ground truth, NOT back-fit to make
|
||||
# the predictor agree. The shared Gemma dense formula UNDER-predicts the live KV
|
||||
# pool: it prices the available-KV pool at ~6.69 GB (≈204K tokens) vs the
|
||||
# measured 8.16 GiB (≈384K tokens) — a ~−47% token-capacity gap, from (a) the
|
||||
# fixed-budget being over-priced by ~2 GB (cudagraph/activation conservatism) and
|
||||
# (b) the per-token growing bytes (8 global layers × 8 kv-heads × 512 × 2 / TP2 =
|
||||
# 32,768 B) being higher than the live pool's effective ~22,816 B/token. The
|
||||
# constants are intentionally left UNTUNED (shared with gemma-4-31b); the verdict
|
||||
# stays directionally correct (TIGHT, boots, peak < 24 GB). Re-tune the Gemma
|
||||
# dense activation/per-token terms only with ≥4 gemma-4-12b anchors across KV
|
||||
# format / ctx / TP — do NOT fit to this single boot.
|
||||
# the predictor agree. REFINED 2026-06-04 — the gemma4_unified GLOBAL-KV per-token
|
||||
# term is now CALIBRATED to the measurement: kv-calc's gemma4-swa-dense branch uses
|
||||
# `measured_kv_growing_bpt_tp1=45,632` (= 22,816 B/tok/card) for the 12B, because
|
||||
# the 31B-derived global-only formula priced it at 32,768 B/tok/card
|
||||
# (8 global × 8 kv × 512 × bpe2 / TP2) — 1.44× too high. That fix moves predicted
|
||||
# token capacity ~204K → ~315K (vs the measured 384K). The shared activation +
|
||||
# cudagraph/overhead constants are intentionally LEFT UNTUNED (shared with
|
||||
# gemma-4-31b): the residual ~315K-vs-384K gap is the deliberately-conservative
|
||||
# `cudagraph_overhead_gb` (estimates ~1.70 GB/card at mem_util 0.90, but the
|
||||
# gemma4-unified image captured only 0.11 GiB — a tiny [1,2,4,8] graph set). Left
|
||||
# conservative ON PURPOSE: under-predicting capacity is the SAFE direction, and
|
||||
# trimming a SHARED term to fit one model would corrupt qwen / gemma-31b. The
|
||||
# verdict stays correct (TIGHT, boots, peak < 24 GB). To close the residual, add a
|
||||
# gemma4-unified-specific overhead calibration ONLY with ≥3 more anchors across
|
||||
# KV-format / ctx / TP — do NOT fit the shared term to this single boot.
|
||||
rows:
|
||||
- compose: vllm/gemma-12b-mtp
|
||||
vram_gb: 24
|
||||
|
||||
+28
-10
@@ -110,10 +110,13 @@ def _load_model_specs_from_yaml(profiles):
|
||||
# there is no int4/awq variant, so all three weight-size keys point at the
|
||||
# one real bf16 blob (23.9 GB) — this keeps _weights_per_card_gb()'s
|
||||
# int4/awq/bf16 branch from KeyError'ing regardless of which the resolver
|
||||
# asks for. Activation/overhead constants are the SHARED Gemma dense
|
||||
# constants — deliberately NOT re-tuned for this model.
|
||||
# asks for. Activation/overhead constants stay the SHARED Gemma dense
|
||||
# constants (NOT re-tuned). The ONE measured calibration is the growing
|
||||
# KV per-token — `measured_kv_growing_bpt_tp1` (see kv_pool_per_card_bytes):
|
||||
# gemma4_unified's global-layer KV measured 1.44x LOWER than the 31B-derived
|
||||
# global-only formula predicts, so we ride the measurement for the 12B only.
|
||||
g12_bf16 = _weight_size(gemma12, "bf16")
|
||||
g12spec = {"model_id": gemma12.id, "model_family": "gemma4-swa-dense", **{k: getattr(gemma12, k) for k in g_fields}, "valid_tp": list(gemma12.valid_tp), "weights_int4_gb": g12_bf16, "weights_awq_gb": g12_bf16, "weights_bf16_gb": g12_bf16, "drafter_mtp_gb": float(profiles.drafters["gemma-12b-it-assistant"].vram_footprint_gb), "mtp_n_default": profiles.drafters["gemma-12b-it-assistant"].n_default}
|
||||
g12spec = {"model_id": gemma12.id, "model_family": "gemma4-swa-dense", **{k: getattr(gemma12, k) for k in g_fields}, "valid_tp": list(gemma12.valid_tp), "weights_int4_gb": g12_bf16, "weights_awq_gb": g12_bf16, "weights_bf16_gb": g12_bf16, "drafter_mtp_gb": float(profiles.drafters["gemma-12b-it-assistant"].vram_footprint_gb), "measured_kv_growing_bpt_tp1": 45632, "mtp_n_default": profiles.drafters["gemma-12b-it-assistant"].n_default}
|
||||
return {
|
||||
"qwen3.6-27b": qspec,
|
||||
"qwen3.6-35b-a3b": qmspec,
|
||||
@@ -453,13 +456,28 @@ def kv_pool_per_card_bytes(spec, kv_format, max_ctx, max_num_seqs, tp, mtp_n=0):
|
||||
|
||||
elif spec["model_family"] == "gemma4-swa-dense":
|
||||
# K==V tied → ×1 storage
|
||||
per_token_growing = (
|
||||
spec["num_full_attn_layers"]
|
||||
* spec["num_kv_heads"]
|
||||
* spec["global_head_dim"]
|
||||
* 1 # K==V tied; vLLM stores once
|
||||
* bpe
|
||||
)
|
||||
measured_bpt_tp1 = spec.get("measured_kv_growing_bpt_tp1")
|
||||
if measured_bpt_tp1 is not None:
|
||||
# gemma4_unified (gemma-4-12b) MEASURED calibration. The 31B-derived
|
||||
# global-only formula (below) predicts 65,536 B/tok TP1 for the 12B
|
||||
# (8 full x 8 kv x 512 global_head_dim x bpe2), but the LIVE
|
||||
# gemma4_unified pool measured 22,816 B/tok/card = 45,632 TP1
|
||||
# (8.16 GiB available KV / 384,019 tokens @ 131K / TP2 / mem-util
|
||||
# 0.90, MTP, 2026-06-04) — 1.44x LOWER. Cause: gemma4_unified's
|
||||
# unified-KV global layers + vLLM's hybrid-SWA pool accounting differ
|
||||
# from the 31B's gemma4-swa-dense. bf16 baseline; scale by bpe for
|
||||
# quantized KV. Single-anchor measured calibration; the precise arch
|
||||
# decomposition is a TODO (needs vLLM kv_cache_utils source or >1
|
||||
# anchor). Set ONLY on the 12B spec, so the 31B keeps the formula.
|
||||
per_token_growing = measured_bpt_tp1 * (bpe / 2.0)
|
||||
else:
|
||||
per_token_growing = (
|
||||
spec["num_full_attn_layers"]
|
||||
* spec["num_kv_heads"]
|
||||
* spec["global_head_dim"]
|
||||
* 1 # K==V tied; vLLM stores once
|
||||
* bpe
|
||||
)
|
||||
# No MTP draft-token bump on Gemma — drafter is a separate model
|
||||
growing = (per_token_growing / tp) * max_ctx * max_num_seqs
|
||||
|
||||
|
||||
Reference in New Issue
Block a user