power-cap-sweep: time-bounded streaming bench (Codex Option A redesign)
Replace token-bounded bench.sh per-cap with inline time-bounded streaming curl. Per-cap wall is now constant ~23s regardless of cap or card class, fixing the cross-card portability issue where token-counted benches took 2-4× longer at low caps than high caps. Implementation: - New flag --target-cap-seconds (default 10): seconds of streaming per direction (narrative + code). Total per-cap = 2 × N + ~3s settle. - Inline curl with stream=true, max_tokens=99999, --max-time=N. Curl exits 28 when wall budget hits — that's the bench boundary, not an error. - TPS = streamed token-chunks / wall_seconds, with fallback to usage.completion_tokens if engine emits final usage before timeout. - Per-cap UTC start/end timestamps + wall-time line in output. - Updated runtime estimate to use TARGET_CAP_SECONDS math for decode-single. Validated 2026-05-07 on @noonghunna's 3090 water-cooled rig: - 21-cap sweep (190-390W): 8m12s total wall (target was ≤12 min) - Per-cap wall: 23.4s consistent (target was 15-45s) - Power sampler: 35-37 util>50% samples per cap = 17-18.5s of data (target was ≥10s) - TPS values reproduce prior data within ~1% (290W: 32.16 narr / 0.111 TPS/W matches our prior 32.08 / 0.111 measurement) Cross-card extrapolation (linear in cap count, not throttle): - 5090 (300-600W, 31 caps): ~12 min - 4090 (230-600W, 38 caps): ~15 min - 3090 (190-390W, 21 caps): ~8 min ✓ Removed env-var token overrides for decode-single (BENCH_MAX_TOKENS_* no longer apply since we don't pass token counts). BENCH_WARMUPS and BENCH_RUNS_PER_CAP also moot for decode-single now. Other load modes (decode-concurrent, prefill-heavy) unchanged. Refreshed 3090 chart with clean 21-cap data (no more cold-cache anomaly on cap 220W since time-bounded approach has no warmup/measurement gap). HARDWARE.md canonical-command section updated to recommend decode-single as the default and document the time-bounded methodology. Implementation by Codex via mcp__codex__codex per brief at docs/diagnostics/power-cap-sweep-cross-card-codex-brief.md.
This commit is contained in:
noonghunna
+14
-6
@@ -84,6 +84,14 @@ Run `sudo bash scripts/power-cap-sweep.sh --cooling air|water|aio` on a new rig
|
||||
|
||||
**Canonical cross-rig anchor command** (production-grade data — what to paste into [disc #86](https://github.com/noonghunna/club-3090/discussions/86) for a real cross-rig efficiency anchor):
|
||||
|
||||
```bash
|
||||
sudo bash scripts/power-cap-sweep.sh \
|
||||
--cooling air|water|aio \
|
||||
--load-mode decode-single
|
||||
```
|
||||
|
||||
For larger cards where single-stream doesn't saturate compute (5090, RTX PRO 6000), use `decode-concurrent`:
|
||||
|
||||
```bash
|
||||
sudo bash scripts/power-cap-sweep.sh \
|
||||
--cooling air|water|aio \
|
||||
@@ -92,13 +100,13 @@ sudo bash scripts/power-cap-sweep.sh \
|
||||
--bench-runs 3
|
||||
```
|
||||
|
||||
Three flags matter for anchor data:
|
||||
- **`--bench-runs 3`** — medians three batches per cap. Without this, single-batch variance can be 10-30%, making adjacent-cap deltas noise rather than signal.
|
||||
- **`--concurrency auto`** — picks N via plateau-detection at the highest cap (selects the highest N where both TPS and draw improve >3% over previous N). Avoids both under-loading (smaller GPUs at default N=4 plateau early) and over-loading (concurrency contention drops TPS).
|
||||
- **`--load-mode decode-concurrent`** — surfaces the curve on cards that don't saturate at single-stream decode (5090, larger Ada/Blackwell). 3090s often work fine at default `decode-single`, but `decode-concurrent` is safer cross-class.
|
||||
How `decode-single` is timed (the new default since 2026-05-07):
|
||||
- **Time-bounded streaming bench**: 10s narrative + 10s code per cap (configurable via `--target-cap-seconds`). Per-cap wall is constant ~23s regardless of cap or card class.
|
||||
- **Cross-card portable**: a 3090 sweep (190-390W, 21 caps) takes ~8 min; a 5090 sweep (300-600W, 31 caps) ~12 min; a 4090 sweep (230-600W, 38 caps) ~15 min — runtime scales linearly with cap count, not throttle severity.
|
||||
- **Power sampler stability**: the 23s/cap window provides 35-37 sampler readings (0.5s interval) where util>50%, well above the 10s minimum needed for stable median.
|
||||
|
||||
**Default step-size is 10W.** Don't override unless you know why:
|
||||
- `--step-size 10` (default) → ~30 caps × ~30 sec/cap = ~15-20 min total. The right resolution for finding the actual knee.
|
||||
- `--step-size 10` (default) → 21-38 caps depending on card class. The right resolution for finding the actual knee.
|
||||
- `--step-size 50` → ~5-6 caps total. Quick smoke / single-rig sanity only — too coarse to pin down the efficiency knee for a cross-rig anchor.
|
||||
|
||||
| GPU | Cooling | Engine | Model | Cap | Narr TPS | Code TPS | TPS/W | Source |
|
||||
@@ -135,7 +143,7 @@ For rigs where we have full 10W-resolution sweeps, the curves below show TPS + T
|
||||
|
||||
|
||||
|
||||
*3090 water-cooled + Qwen3.6-27B Q3_K_XL + mainline llama.cpp, 18-cap sweep 200-390W (mixed bench shapes). Yellow callout: 290W sweet spot (0.111 TPS/W) at **78% of stock 370W TDP**. Orange-shaded zone 340-370W: firmware boost-state plateau where caps 340/350/360/370W all draw identical ~334W actual. At 380W cap, draw escapes to 361W; at 390W cap, draw reaches 388W — so the apparent "ceiling" at 334W isn't a hardware limit, it's a discrete boost-state behavior. Source script: [`img/power-cap-3090-qwen36.py`](img/power-cap-3090-qwen36.py).*
|
||||
*3090 water-cooled + Qwen3.6-27B Q3_K_XL + mainline llama.cpp, 21-cap sweep 190-390W via time-bounded streaming bench (10s/direction). **Total wall: 8m12s.** Yellow callout: 290W sweet spot (0.111 TPS/W) at **78% of stock 370W TDP**. Orange-shaded zone 340-370W: firmware boost-state plateau where caps 340/350/360/370W all draw identical ~334W actual. At 380W cap, draw escapes to 361W; at 390W cap, draw reaches 388W — so the apparent "ceiling" at 334W isn't a hardware limit, it's a discrete boost-state behavior. Source script: [`img/power-cap-3090-qwen36.py`](img/power-cap-3090-qwen36.py).*
|
||||
|
||||
**Cross-rig pattern**: efficiency knee falls at **~60-85% of stock TDP** across consumer Ampere/Ada — start there for a new card class and zoom in. Ada (4090) is proportionally more aggressive than Ampere (3090) — 4090 cuts 33% of stock TDP for ~7% TPS loss; 3090 cuts 15% of stock for ~5% loss.
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 193 KiB After Width: | Height: | Size: 197 KiB |
@@ -1,48 +1,46 @@
|
||||
"""Generate 3090 power-cap efficiency chart from @noonghunna's water-cooled rig.
|
||||
|
||||
Source data: this run, 2026-05-07, dual-3090 rig (GPU 0 used), water-cooled.
|
||||
Source data: 2026-05-07 sweep, dual-3090 rig (GPU 0 used), water-cooled.
|
||||
Engine: mainline llama.cpp (ghcr.io/ggml-org/llama.cpp:server-cuda) +
|
||||
Qwen3.6-27B-UD-Q3_K_XL.gguf, single-stream decode-single.
|
||||
|
||||
Sweep stages:
|
||||
200-210W: full bench (1 warm + 2 measured, 500/400 max_tokens) — sweep v2
|
||||
220W: quick bench (0 warm + 1 measured, 250/200 tokens) — sweep v3
|
||||
narr 10.28 was cold-cache biased (first cap of quick-bench run);
|
||||
code 18.72 retained (cache warmed by narr run that just preceded)
|
||||
230-240W: same quick bench shape; cache warm from previous cap
|
||||
250-390W: quick bench with warmups=1 (1 warm + 1 measured, 250/200) — sweep v4
|
||||
Sweep methodology: time-bounded streaming bench (10s/direction at each cap).
|
||||
Total wall: 8m12s for 21 caps from 190-390W in 10W increments. The time-bounded
|
||||
approach (vs token-bounded) makes per-cap wall constant ~23s regardless of cap,
|
||||
so total runtime scales linearly with cap count, not throttle severity.
|
||||
"""
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# (cap_W, narr_TPS, code_TPS, actual_W, eff_TPS_per_W)
|
||||
# (cap_W, narr_TPS, code_TPS, actual_W, eff_TPS_per_W) — full 21-cap clean sweep
|
||||
data = [
|
||||
(200, 15.56, 15.56, 199.67, 0.078),
|
||||
(210, 17.55, 17.42, 209.67, 0.084),
|
||||
(220, None, 18.72, 219.64, None), # narr cold-cache biased; skip
|
||||
(230, 21.07, 21.06, 229.66, 0.092),
|
||||
(240, 23.29, 23.15, 239.7, 0.097),
|
||||
(250, 26.21, 26.11, 249.69, 0.105),
|
||||
(260, 27.75, 27.32, 259.80, 0.107),
|
||||
(270, 29.18, 26.06, 269.91, 0.108),
|
||||
(280, 30.74, 30.41, 279.95, 0.110),
|
||||
(290, 32.08, 31.74, 289.64, 0.111), # ⭐ sweet spot
|
||||
(300, 32.74, 32.44, 299.61, 0.109),
|
||||
(310, 33.23, 32.98, 309.18, 0.107),
|
||||
(320, 33.81, 33.60, 319.51, 0.106),
|
||||
(330, 34.25, 34.05, 328.89, 0.104),
|
||||
(340, 34.39, 34.14, 334.24, 0.103), # boost-state plateau begins
|
||||
(350, 34.38, 34.18, 334.08, 0.103),
|
||||
(360, 34.41, 34.23, 334.08, 0.103),
|
||||
(370, 34.46, 34.20, 334.20, 0.103), # stock TDP, plateau holds
|
||||
(380, 35.24, 35.04, 361.19, 0.098), # plateau ends, draw jumps to 361
|
||||
(390, 35.84, 35.66, 388.58, 0.092), # max — 388W draw at 390W cap
|
||||
(190, 13.88, 13.69, 189.73, 0.073),
|
||||
(200, 15.58, 15.68, 199.71, 0.078),
|
||||
(210, 17.68, 17.48, 209.77, 0.084),
|
||||
(220, 19.38, 19.28, 219.73, 0.088),
|
||||
(230, 21.27, 21.07, 229.71, 0.093),
|
||||
(240, 23.17, 22.97, 239.84, 0.097),
|
||||
(250, 24.97, 24.77, 249.80, 0.100),
|
||||
(260, 26.77, 26.57, 259.86, 0.103),
|
||||
(270, 28.57, 28.47, 269.56, 0.106),
|
||||
(280, 30.36, 30.77, 279.75, 0.109),
|
||||
(290, 32.16, 32.06, 289.37, 0.111), # ⭐ sweet spot
|
||||
(300, 32.76, 32.76, 299.30, 0.109),
|
||||
(310, 33.36, 33.26, 309.59, 0.108),
|
||||
(320, 33.86, 33.76, 319.47, 0.106),
|
||||
(330, 34.37, 34.26, 329.47, 0.104),
|
||||
(340, 34.46, 34.25, 333.70, 0.103), # boost-state plateau begins
|
||||
(350, 34.46, 34.36, 334.00, 0.103),
|
||||
(360, 34.36, 34.37, 333.97, 0.103),
|
||||
(370, 34.36, 34.26, 334.02, 0.103), # stock TDP, plateau holds
|
||||
(380, 35.36, 35.26, 361.30, 0.098), # plateau ends, draw jumps to 361
|
||||
(390, 36.06, 35.96, 388.72, 0.093), # max — 388W draw at 390W cap
|
||||
]
|
||||
|
||||
caps = [d[0] for d in data]
|
||||
narr = [d[1] if d[1] is not None else float('nan') for d in data]
|
||||
narr = [d[1] for d in data]
|
||||
code = [d[2] for d in data]
|
||||
draw = [d[3] for d in data]
|
||||
eff = [d[4] if d[4] is not None else float('nan') for d in data]
|
||||
eff = [d[4] for d in data]
|
||||
|
||||
plt.rcParams.update({
|
||||
"font.family": "sans-serif",
|
||||
@@ -65,8 +63,8 @@ ax1.plot(caps, code, "s-", color=color_code, linewidth=2.2, markersize=6,
|
||||
label="Code TPS", zorder=3)
|
||||
ax1.set_xlabel("Power cap (W)", fontsize=13)
|
||||
ax1.set_ylabel("Wall TPS (single-stream, llama.cpp mainline)", fontsize=13)
|
||||
ax1.set_xlim(195, 395)
|
||||
ax1.set_ylim(13, 39)
|
||||
ax1.set_xlim(185, 395)
|
||||
ax1.set_ylim(11, 39)
|
||||
ax1.grid(True, alpha=0.3, zorder=0)
|
||||
ax1.tick_params(axis="both", labelsize=11)
|
||||
|
||||
@@ -82,8 +80,8 @@ ax2.set_ylim(0.07, 0.118)
|
||||
# Sweet spot annotation: 290W
|
||||
ax1.axvline(290, color="goldenrod", linestyle=":", alpha=0.5, linewidth=1.5)
|
||||
ax1.annotate(
|
||||
"★ 290W cap\n0.111 TPS/W (best efficiency)\n32.1 narr / 31.7 code\n78% of stock TDP",
|
||||
xy=(290, 32.08),
|
||||
"★ 290W cap\n0.111 TPS/W (best efficiency)\n32.2 narr / 32.1 code\n78% of stock TDP",
|
||||
xy=(290, 32.16),
|
||||
xytext=(220, 27),
|
||||
fontsize=10.5,
|
||||
fontweight="bold",
|
||||
@@ -94,7 +92,7 @@ ax1.annotate(
|
||||
|
||||
# Boost-state plateau region (340-370W → all 334W draw)
|
||||
ax1.axvspan(335, 375, alpha=0.10, color="orange", zorder=0)
|
||||
ax1.text(355, 14.5, "boost-state plateau\n(caps 340-370W → ~334W draw)",
|
||||
ax1.text(355, 12.5, "boost-state plateau\n(caps 340-370W → ~334W draw)",
|
||||
fontsize=9.5, ha="center", color="#aa5500", fontstyle="italic")
|
||||
|
||||
# Stock TDP marker at 370W
|
||||
@@ -119,7 +117,7 @@ ax1.set_title(
|
||||
fig.text(
|
||||
0.5, 0.92,
|
||||
"1× 3090 water-cooled (GPU 0 of dual-3090 rig), mainline llama.cpp + Q3_K_XL GGUF, "
|
||||
"single-stream | data: @noonghunna",
|
||||
"time-bounded single-stream | data: @noonghunna",
|
||||
ha="center", fontsize=10, color="#666",
|
||||
style="italic",
|
||||
)
|
||||
|
||||
+167
-35
@@ -116,6 +116,10 @@ BENCH_RUNS=1 # repeated measured batches for decode-concurrent/prefill-
|
||||
MAX_CONCURRENCY_PROBE=16
|
||||
LOAD_TARGET=0.92 # target actual-power/cap ratio for --concurrency auto
|
||||
CONCURRENCY_STRETCH=0 # add N to auto-detected concurrency (probe headroom past plateau pick)
|
||||
TARGET_CAP_SECONDS=10 # decode-single time-bounded streaming bench seconds per direction
|
||||
# (narrative + code). This keeps per-cap wall stable
|
||||
# across card classes while giving the sampler >=10s
|
||||
# of util>50% data per cap.
|
||||
CALIBRATION_NOTE=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
@@ -130,6 +134,7 @@ while [ $# -gt 0 ]; do
|
||||
--max-concurrency-probe) MAX_CONCURRENCY_PROBE="$2"; shift 2 ;;
|
||||
--load-target) LOAD_TARGET="$2"; shift 2 ;;
|
||||
--concurrency-stretch) CONCURRENCY_STRETCH="$2"; shift 2 ;;
|
||||
--target-cap-seconds) TARGET_CAP_SECONDS="$2"; shift 2 ;;
|
||||
--no-reset) RESET=0; shift ;;
|
||||
-h|--help)
|
||||
sed -n '1,/^set -euo/p' "$0" | grep '^#' | sed 's/^# \?//'
|
||||
@@ -170,6 +175,10 @@ if [ "$CONCURRENCY_STRETCH" -gt 0 ] && [ "$CONCURRENCY_AUTO" -ne 1 ]; then
|
||||
echo "[error] --concurrency-stretch only applies with --concurrency auto" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! [[ "$TARGET_CAP_SECONDS" =~ ^[1-9][0-9]*$ ]]; then
|
||||
echo "[error] --target-cap-seconds must be a positive integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! python3 - "$LOAD_TARGET" <<'PY' >/dev/null 2>&1
|
||||
import sys
|
||||
x = float(sys.argv[1])
|
||||
@@ -279,6 +288,118 @@ cleanup() {
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
bench_decode_single_for_seconds() {
|
||||
local kind="$1"
|
||||
local seconds="$2"
|
||||
local cap="$3"
|
||||
local log_file="$4"
|
||||
local req_file out_file start_ns end_ns wall_s tokens tps prompt max_time
|
||||
|
||||
req_file="/tmp/power-cap-N${cap}-${kind}.req.json"
|
||||
out_file="/tmp/power-cap-N${cap}-${kind}.sse"
|
||||
max_time="$seconds"
|
||||
|
||||
case "$kind" in
|
||||
narrative) prompt="Write a detailed 800-word essay explaining transformer attention." ;;
|
||||
code) prompt="Implement quicksort in Python with detailed comments." ;;
|
||||
*) echo "[error] unknown decode-single prompt kind: $kind" >&2; return 1 ;;
|
||||
esac
|
||||
|
||||
python3 - "$req_file" "$MODEL" "$prompt" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
path, model, prompt = sys.argv[1:4]
|
||||
body = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": 99999,
|
||||
"temperature": 0.6,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20,
|
||||
"stream": True,
|
||||
}
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(body, f)
|
||||
PY
|
||||
|
||||
start_ns=$(date +%s%N)
|
||||
# curl exits 28 when --max-time cuts the stream. That is expected here: the
|
||||
# wall clock is the benchmark boundary, not a completed max_tokens response.
|
||||
curl -sS --no-buffer --max-time "$max_time" "${URL}/v1/chat/completions" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "@${req_file}" \
|
||||
-o "$out_file" 2>>"$log_file" || true
|
||||
end_ns=$(date +%s%N)
|
||||
|
||||
wall_s=$(python3 - "$start_ns" "$end_ns" <<'PY'
|
||||
import sys
|
||||
start, end = map(int, sys.argv[1:3])
|
||||
print(f"{(end - start) / 1e9:.3f}")
|
||||
PY
|
||||
)
|
||||
tokens=$(python3 - "$out_file" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
path = sys.argv[1]
|
||||
chunks = 0
|
||||
usage_tokens = None
|
||||
chars = 0
|
||||
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8", errors="ignore") as f:
|
||||
for raw in f:
|
||||
line = raw.strip()
|
||||
if not line.startswith("data:"):
|
||||
continue
|
||||
data = line[5:].strip()
|
||||
if not data or data == "[DONE]":
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(data)
|
||||
except Exception:
|
||||
continue
|
||||
usage = obj.get("usage")
|
||||
if isinstance(usage, dict):
|
||||
completion = usage.get("completion_tokens")
|
||||
if isinstance(completion, int) and completion > 0:
|
||||
usage_tokens = completion
|
||||
for choice in obj.get("choices", []):
|
||||
text = ""
|
||||
delta = choice.get("delta")
|
||||
if isinstance(delta, dict):
|
||||
text = delta.get("content") or ""
|
||||
if not text:
|
||||
text = choice.get("text") or ""
|
||||
if text:
|
||||
chunks += 1
|
||||
chars += len(text)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
if usage_tokens:
|
||||
print(usage_tokens)
|
||||
elif chunks:
|
||||
print(chunks)
|
||||
elif chars:
|
||||
print(max(1, round(chars / 4)))
|
||||
else:
|
||||
print(0)
|
||||
PY
|
||||
)
|
||||
tps=$(python3 - "$tokens" "$wall_s" <<'PY'
|
||||
import sys
|
||||
tokens = int(sys.argv[1])
|
||||
wall = float(sys.argv[2])
|
||||
print(f"{tokens / max(wall, 0.001):.2f}")
|
||||
PY
|
||||
)
|
||||
|
||||
echo "[$kind] ${tokens} streamed token-chunks in ${wall_s}s -> ${tps} TPS" | tee -a "$log_file"
|
||||
printf "%s\n" "$tps"
|
||||
}
|
||||
|
||||
run_concurrency_probe() {
|
||||
local n="$1"
|
||||
local cap="$2"
|
||||
@@ -427,13 +548,15 @@ else
|
||||
AUTO_DERIVED=0
|
||||
fi
|
||||
NUM_CAPS=$(echo "$CAPS" | tr ',' '\n' | wc -l | tr -d ' ')
|
||||
# Runtime estimate: ~30s/cap base for default bench shape (1 warm + 2 measured
|
||||
# of 500/400 tokens) at normal operating points. Real time scales with bench
|
||||
# shape (BENCH_RUNS_PER_CAP) and cap range (sub-50%-stock caps run 3-5× slower
|
||||
# due to GPU throttle). Estimate is conservative for the default shape; if you
|
||||
# customize via env vars or --caps below 50% stock, expect 1.5-3× longer.
|
||||
EST_MIN=$(( (NUM_CAPS * 30 + 59) / 60 ))
|
||||
EST_MAX=$(( EST_MIN * 3 ))
|
||||
if [ "$LOAD_MODE" = "decode-single" ]; then
|
||||
EST_MIN=$(( (NUM_CAPS * (TARGET_CAP_SECONDS * 2 + 5) + 59) / 60 ))
|
||||
EST_MAX=$(( (NUM_CAPS * (TARGET_CAP_SECONDS * 2 + 10) + 59) / 60 ))
|
||||
else
|
||||
# Runtime estimate for non-time-bounded modes: ~30s/cap base at normal
|
||||
# operating points; low explicit caps can stretch longer due to throttle.
|
||||
EST_MIN=$(( (NUM_CAPS * 30 + 59) / 60 ))
|
||||
EST_MAX=$(( EST_MIN * 3 ))
|
||||
fi
|
||||
HIGHEST_CAP=$(python3 - "$CAPS" <<'PY'
|
||||
import sys
|
||||
print(max(int(float(x.strip())) for x in sys.argv[1].split(",") if x.strip()))
|
||||
@@ -573,7 +696,7 @@ else
|
||||
echo "[setup] sweep caps: $NUM_CAPS caps (user-specified)"
|
||||
echo "[setup] $CAPS W"
|
||||
fi
|
||||
echo "[setup] load mode: $LOAD_MODE$([ "$LOAD_MODE" = "decode-concurrent" ] && echo " (concurrency=$CONCURRENCY)")$([ "$LOAD_MODE" != "decode-single" ] && echo " (bench-runs=$BENCH_RUNS)")"
|
||||
echo "[setup] load mode: $LOAD_MODE$([ "$LOAD_MODE" = "decode-single" ] && echo " (${TARGET_CAP_SECONDS}s × 2 timed streams)")$([ "$LOAD_MODE" = "decode-concurrent" ] && echo " (concurrency=$CONCURRENCY)")$([ "$LOAD_MODE" != "decode-single" ] && echo " (bench-runs=$BENCH_RUNS)")"
|
||||
[ -n "$CALIBRATION_NOTE" ] && echo "[setup] calibration: $CALIBRATION_NOTE"
|
||||
echo "[setup] estimated runtime: ${EST_MIN}-${EST_MAX} min (${NUM_CAPS} caps; range varies with cap throttle + bench shape)"
|
||||
echo "[setup] reset at end: $([ $RESET -eq 1 ] && echo yes || echo no)"
|
||||
@@ -656,7 +779,7 @@ RESULTS_FILE=/tmp/power-cap-summary.md
|
||||
echo ""
|
||||
echo "**GPU:** $GPU_NAME **VRAM:** ${GPU_VRAM} MiB **Stock TDP:** ${STOCK_TDP}W **Cooling:** ${COOLING}"
|
||||
echo "**Model:** \`${MODEL}\` **Engine:** \`${CONTAINER}\` **Endpoint:** ${URL}"
|
||||
echo "**Load mode:** \`${LOAD_MODE}\`$([ "$LOAD_MODE" = "decode-concurrent" ] && echo " (concurrency=${CONCURRENCY})")$([ "$LOAD_MODE" != "decode-single" ] && echo " (bench-runs=${BENCH_RUNS})")"
|
||||
echo "**Load mode:** \`${LOAD_MODE}\`$([ "$LOAD_MODE" = "decode-single" ] && echo " (${TARGET_CAP_SECONDS}s × 2 timed streams)")$([ "$LOAD_MODE" = "decode-concurrent" ] && echo " (concurrency=${CONCURRENCY})")$([ "$LOAD_MODE" != "decode-single" ] && echo " (bench-runs=${BENCH_RUNS})")"
|
||||
[ -n "$CALIBRATION_NOTE" ] && echo "**Calibration:** ${CALIBRATION_NOTE}"
|
||||
echo "**Date:** $(date -u +%Y-%m-%dT%H:%M:%S)Z"
|
||||
echo ""
|
||||
@@ -676,10 +799,13 @@ RESULTS_FILE=/tmp/power-cap-summary.md
|
||||
} > "$RESULTS_FILE"
|
||||
|
||||
IFS=',' read -ra CAP_ARRAY <<< "$CAPS"
|
||||
|
||||
for CAP in "${CAP_ARRAY[@]}"; do
|
||||
CAP=$(echo "$CAP" | tr -d ' ')
|
||||
CAP_START_NS=$(date +%s%N)
|
||||
CAP_START_UTC=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
echo "================================================"
|
||||
echo "=== Cap: ${CAP}W (GPU $GPU_INDEX) ==="
|
||||
echo "=== Cap: ${CAP}W (GPU $GPU_INDEX) @ ${CAP_START_UTC} ==="
|
||||
echo "================================================"
|
||||
|
||||
# Apply cap
|
||||
@@ -724,37 +850,32 @@ for CAP in "${CAP_ARRAY[@]}"; do
|
||||
LOG_FILE="/tmp/power-cap-N${CAP}.log"
|
||||
case "$LOAD_MODE" in
|
||||
decode-single)
|
||||
# Single-stream: original bench.sh path. Captures decode-bottleneck on
|
||||
# cards where compute is the limit (3090, 4090); shows flat curve on
|
||||
# cards over-provisioned for the workload (5090 + small models).
|
||||
#
|
||||
# Bench shape is env-overridable for fast sweeps on slow caps (e.g. 200W
|
||||
# on a 3090 throttles TPS to ~15, making 500/400 tokens × 3 runs take
|
||||
# ~3 min/cap). Run via `sudo -E ...` to preserve env across sudo:
|
||||
# BENCH_WARMUPS=0 BENCH_RUNS_PER_CAP=1 \
|
||||
# BENCH_MAX_TOKENS_NARR=250 BENCH_MAX_TOKENS_CODE=200 \
|
||||
# sudo -E bash scripts/power-cap-sweep.sh ...
|
||||
# Defaults preserve the canonical 1+2 / 500+400 shape (~30s/cap on cards
|
||||
# at decent operating points, ~3min/cap at heavy throttle).
|
||||
echo "[bench] decode-single @ ${CAP}W cap (output: $LOG_FILE)"
|
||||
if ! WARMUPS=${BENCH_WARMUPS:-1} RUNS=${BENCH_RUNS_PER_CAP:-2} \
|
||||
MAX_TOKENS_NARR=${BENCH_MAX_TOKENS_NARR:-500} \
|
||||
MAX_TOKENS_CODE=${BENCH_MAX_TOKENS_CODE:-400} \
|
||||
bash "$BENCH" 2>&1 | tee "$LOG_FILE" | tail -8; then
|
||||
# Single-stream decode is time-bounded instead of token-bounded. Fixed
|
||||
# token counts make low caps take 2-4× longer than high caps; fixed wall
|
||||
# seconds keep sweep runtime portable across 3090/4090/5090/A-series while
|
||||
# still providing sustained under-load samples for the power median.
|
||||
echo "[bench] decode-single @ ${CAP}W cap (${TARGET_CAP_SECONDS}s narrative + ${TARGET_CAP_SECONDS}s code, output: $LOG_FILE)"
|
||||
: > "$LOG_FILE"
|
||||
if ! NARR_TPS=$(bench_decode_single_for_seconds narrative "$TARGET_CAP_SECONDS" "$CAP" "$LOG_FILE"); then
|
||||
kill $SAMPLER_PID 2>/dev/null || true
|
||||
wait $SAMPLER_PID 2>/dev/null || true
|
||||
SAMPLER_PID=""
|
||||
echo "[warn] bench.sh failed at ${CAP}W"
|
||||
echo "[warn] narrative timed bench failed at ${CAP}W"
|
||||
continue
|
||||
fi
|
||||
NARR_TPS=$(echo "$NARR_TPS" | tail -1)
|
||||
if ! CODE_TPS=$(bench_decode_single_for_seconds code "$TARGET_CAP_SECONDS" "$CAP" "$LOG_FILE"); then
|
||||
kill $SAMPLER_PID 2>/dev/null || true
|
||||
wait $SAMPLER_PID 2>/dev/null || true
|
||||
SAMPLER_PID=""
|
||||
echo "[warn] code timed bench failed at ${CAP}W"
|
||||
continue
|
||||
fi
|
||||
CODE_TPS=$(echo "$CODE_TPS" | tail -1)
|
||||
kill $SAMPLER_PID 2>/dev/null || true
|
||||
wait $SAMPLER_PID 2>/dev/null || true
|
||||
SAMPLER_PID=""
|
||||
echo
|
||||
|
||||
# Extract from bench summary lines
|
||||
NARR_TPS=$(grep -A1 "summary \[narrative\]" "$LOG_FILE" | grep "wall_TPS" | head -1 | grep -oE 'mean= *[0-9]+\.[0-9]+' | head -1 | grep -oE '[0-9]+\.[0-9]+' || echo "?")
|
||||
CODE_TPS=$(grep -A1 "summary \[code\]" "$LOG_FILE" | grep "wall_TPS" | head -1 | grep -oE 'mean= *[0-9]+\.[0-9]+' | head -1 | grep -oE '[0-9]+\.[0-9]+' || echo "?")
|
||||
;;
|
||||
|
||||
decode-concurrent)
|
||||
@@ -1039,8 +1160,19 @@ else:
|
||||
EFFICIENCY="?"
|
||||
fi
|
||||
|
||||
printf "[result] %sW cap → %s narr / %s code TPS @ %sW actual draw, %s°C, eff %s TPS/W\n\n" \
|
||||
CAP_END_NS=$(date +%s%N)
|
||||
CAP_END_UTC=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
CAP_WALL_S=$(python3 - "$CAP_START_NS" "$CAP_END_NS" <<'PY'
|
||||
import sys
|
||||
start, end = map(int, sys.argv[1:3])
|
||||
print(f"{(end - start) / 1e9:.1f}")
|
||||
PY
|
||||
)
|
||||
|
||||
printf "[result] %sW cap → %s narr / %s code TPS @ %sW actual draw, %s°C, eff %s TPS/W\n" \
|
||||
"$CAP" "$NARR_TPS" "$CODE_TPS" "$ACTUAL_POWER" "$GPU_TEMP" "$EFFICIENCY"
|
||||
printf "[time] %sW cap wall=%ss start=%s end=%s\n\n" \
|
||||
"$CAP" "$CAP_WALL_S" "$CAP_START_UTC" "$CAP_END_UTC"
|
||||
|
||||
printf "| %s | %s | %s | %s | %s | %s |\n" \
|
||||
"$CAP" "$NARR_TPS" "$CODE_TPS" "$ACTUAL_POWER" "$GPU_TEMP" "$EFFICIENCY" \
|
||||
@@ -1063,8 +1195,8 @@ fi
|
||||
echo "**Notes:**"
|
||||
case "$LOAD_MODE" in
|
||||
decode-single)
|
||||
echo "- Load mode: \`decode-single\` — original bench.sh path, 1 warm + 2 measured runs of canonical narr (500-token essay) + code (400-token quicksort)."
|
||||
echo "- TPS columns are per-request wall TPS from bench.sh summaries."
|
||||
echo "- Load mode: \`decode-single\` — time-bounded streaming requests: ${TARGET_CAP_SECONDS}s narrative + ${TARGET_CAP_SECONDS}s code per cap."
|
||||
echo "- TPS columns are streamed token-chunks / wall seconds. If an engine emits final streaming usage before timeout, completion_tokens is used instead."
|
||||
;;
|
||||
decode-concurrent)
|
||||
echo "- Load mode: \`decode-concurrent\` — ${CONCURRENCY} parallel chat completions for narr, then ${CONCURRENCY} parallel chat completions for code."
|
||||
|
||||
Reference in New Issue
Block a user