🪙 Rune/Charm Demand (Traderie listings as currency)
+
+
Small Charm
+
235 listings
+
≈0.5 El
+
+
Jah Rune
+
204 listings
+
≈50 El
+
+
Ber Rune
+
71 listings
+
≈60 El
+
+
Zod Rune
+
59 listings
+
≈80 El
+
+
Lo Rune
+
31 listings
+
≈30 El
+
+
Cham Rune
+
19 listings
+
≈35 El
+
+
Ohm Rune
+
17 listings
+
≈25 El
+
+
Vex Rune
+
12 listings
+
≈22 El
+
+
Ist Rune
+
11 listings
+
≈15 El
+
+
Um Rune
+
3 listings
+
≈10 El
+
+
Lem Rune
+
2 listings
+
≈8 El
+
+
Gul Rune
+
2 listings
+
≈18 El
+
+
Hel Rune
+
2 listings
+
≈3 El
+
+
Pul Rune
+
1 listings
+
≈9 El
+
+
Mal Rune
+
1 listings
+
≈12 El
+
+
+
+
📈 Price History (3 days)
+
+
+
Date
Items
FG Items
Trade Items
+
2026-06-12
190
61
144
2026-06-17
140
0
140
2026-08-23
164
0
164
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/docs/melee_hunt_test_log.md b/docs/melee_hunt_test_log.md
new file mode 100644
index 0000000..8463d32
--- /dev/null
+++ b/docs/melee_hunt_test_log.md
@@ -0,0 +1,154 @@
+# Paladalla Melee Hunt Test Log
+
+Date: 2026-08-23
+Workspace: `C:\Users\alex\my-botty`
+Character/profile: `paladalla`
+Script: `tools/melee_hunt.py`
+
+## Current State
+
+- Testing was stopped at the user's request.
+- No `melee_hunt` Python process was running when checked.
+- Paladalla was alive in the Rogue Encampment at the end of the session.
+- The four visible belt potions were manually consumed. The last screenshot shows
+ partial life and an empty belt, so refill and fully heal before the next run.
+- The game must not be controlled automatically until the user asks to resume.
+
+Final-state screenshot: [healed_from_belt.png](../log/screenshots/manual/healed_from_belt.png)
+
+## Commands Used
+
+The successful short run used:
+
+```powershell
+C:\Users\alex\miniforge3\envs\botty\python.exe tools\melee_hunt.py --debug --button left
+```
+
+The later long tests used:
+
+```powershell
+C:\Users\alex\miniforge3\envs\botty\python.exe tools\melee_hunt.py --debug --button left --minutes 1440
+```
+
+`F12` is intended to stop the script. It worked during one run but was not
+reliable while an in-game overlay was open. `Ctrl+C` in the script terminal was
+the reliable emergency stop.
+
+## Verified Results
+
+### Initial ten-minute run
+
+- Started outside town and completed its configured timer normally.
+- Result: `engaged 107, roamed 15`.
+- XP changed from `261/500` to `357/500`.
+- Final reported life was `69%`.
+- The process then exited because the default duration is ten minutes. Paladalla
+ later died because nothing was controlling or protecting the character.
+
+### Corpse recovery and level gain
+
+- The death prompt was cleared and the game was cycled through Save and Exit.
+- The corpse appeared beside Paladalla in town and was recovered successfully.
+- Weapon and attack slots were restored.
+- Paladalla was manually routed across the bridge into the Blood Moor.
+- Manual movement/attack clicks later produced a level-up. The next script start
+ read XP as `564/1500`, confirming level 2 and working melee damage.
+
+Evidence:
+
+- [corpse_recovered.png](../log/screenshots/manual/corpse_recovered.png)
+- [blood_moor_ready2.png](../log/screenshots/manual/blood_moor_ready2.png)
+- [deep_moor_restart.png](../log/screenshots/manual/deep_moor_restart.png)
+
+## Failures Found
+
+### 1. Ten-minute default looked like a crash
+
+The first hunt was not dead or hung. It reached its default ten-minute deadline
+and exited cleanly. Long unattended tests must pass an explicit `--minutes`
+value or implement a true continuous mode.
+
+### 2. Friendly bridge NPC was treated as motion target
+
+The first 24-hour attempt remained near the town bridge. At the two-minute
+checkpoint it showed repeated attacks, full life, and unchanged XP
+(`357/500`). The screenshot showed it attacking the friendly rogue near the
+bridge. Motion alone cannot distinguish that NPC from a monster.
+
+Evidence: [hunt_2min_checkpoint.png](../log/screenshots/manual/hunt_2min_checkpoint.png)
+
+Workaround used: stop the script and move several screens deeper into the Blood
+Moor before restarting. A permanent NPC rejection/stall detector is still
+needed.
+
+### 3. Animated HUD controls were treated as targets
+
+After reaching level 2, the pulsing new-stat/new-skill controls appeared above
+the original HUD mask. The motion detector could target that region. During the
+second long attempt an in-game Loot Filter overlay opened and the script kept
+issuing clicks behind it.
+
+Evidence: [restart_live_40s.png](../log/screenshots/manual/restart_live_40s.png)
+
+### 4. No-potion condition was unsafe
+
+Life fell through `18%`, `14%`, `10%`, `8%`, and eventually `0%`. The script
+logged `no healing potion in belt!` but continued its loop. Belt detection also
+reported no potion even though red potions were visibly present, so belt
+detection or its profile assumptions need separate investigation.
+
+### 5. XP OCR became invalid after level-up/UI changes
+
+The log reported XP OCR values such as `'-'` and `''`, then incorrectly printed
+`*** LEVEL UP *** now 0/0`. Invalid OCR should not be interpreted as a level-up,
+and `xp_last` should retain the last valid sample.
+
+## Code Changes Made
+
+`tools/melee_hunt.py` was changed in two ways:
+
+1. The target scan bottom boundary was raised from `640` to `510`, excluding the
+ animated level-up/stat controls and the interactive HUD from motion targets.
+2. Failure to detect or use a healing potion at the configured health threshold
+ now raises `SafetyStop`, exits the hunt loop, and prints a safety-stop reason.
+
+The edited script passed:
+
+```powershell
+C:\Users\alex\miniforge3\envs\botty\python.exe -m py_compile tools\melee_hunt.py
+```
+
+These changes were not runtime-verified after editing because testing was
+stopped. The narrower scan may ignore monsters near the bottom of the viewport;
+that tradeoff must be checked visually.
+
+## Next Test Checklist
+
+1. Confirm no old hunt process is running.
+2. Refill the belt with healing potions and fully heal at Akara.
+3. Save and Exit, re-enter, and recover any corpse if one appears.
+4. Spend or dismiss the new stat/skill notifications if practical.
+5. Walk across the bridge and several screens into the Blood Moor before launch.
+6. Start with a short five-minute test, not a 24-hour run:
+
+```powershell
+C:\Users\alex\miniforge3\envs\botty\python.exe tools\melee_hunt.py --debug --button left --minutes 5
+```
+
+7. Capture and inspect screenshots at launch, two minutes, and five minutes.
+8. At each checkpoint verify: outside town, no modal open, attacking a hostile,
+ XP increasing, life stable, movement occurring, and no repeated target lock.
+9. Force a controlled low-health/no-potion condition only after normal combat is
+ proven, and verify `SAFETY STOP` exits immediately.
+10. Use `Ctrl+C` if any modal opens, XP remains unchanged across two checkpoints,
+ life drops rapidly, or the character approaches the town bridge NPC.
+
+## Follow-up Engineering Work
+
+- Add a stall detector based on unchanged XP plus repeated target coordinates.
+- Temporarily blacklist a target position after excessive swings without XP.
+- Reject invalid XP samples instead of converting them into `0/0` level-ups.
+- Investigate why `belt.drink_potion("health", ...)` missed visible red potions.
+- Detect death and modal overlays explicitly and stop before sending more input.
+- Consider an explicit `--continuous` option instead of using a large minute
+ count.
diff --git a/src/input_layer/win_input.py b/src/input_layer/win_input.py
index d1ff3b4..abe79cf 100644
--- a/src/input_layer/win_input.py
+++ b/src/input_layer/win_input.py
@@ -300,6 +300,11 @@ def mouse_move(x: int, y: int):
Logger.warning(f"SetCursorPos failed for ({target_x}, {target_y}); continuing with current cursor position")
except Exception:
pass
+ # Ensure the cursor is exactly on target before any click (SetCursorPos can
+ # land 1-2px off on some DPI/virtual-desktop setups; D2R UI hitboxes are tight).
+ cur_x, cur_y = get_cursor_pos()
+ if abs(cur_x - target_x) > 0 or abs(cur_y - target_y) > 0:
+ user32.SetCursorPos(target_x, target_y)
def mouse_down(button: str = "left"):
"""Press mouse button down."""
diff --git a/tools/assets/templates/a5_wp.png b/tools/assets/templates/a5_wp.png
new file mode 100644
index 0000000..0f2dea0
Binary files /dev/null and b/tools/assets/templates/a5_wp.png differ
diff --git a/tools/build_price_dashboard.py b/tools/build_price_dashboard.py
new file mode 100644
index 0000000..bf1f306
--- /dev/null
+++ b/tools/build_price_dashboard.py
@@ -0,0 +1,313 @@
+#!/usr/bin/env python
+"""Generate D2R Price Dashboard - FG (d2jsp) + Rune Trade Prices (Traderie).
+
+Reads:
+ config/daily_prices.json - merged daily prices (traderie + fg)
+ config/daily_prices_history.json - history of daily prices
+ config/fg_daily_estimates.json - FG estimates from d2jsp
+ config/traderie_prices.json - latest traderie scrape
+
+Writes:
+ config/price_dashboard.html - self-contained HTML dashboard
+"""
+import json
+import html
+from pathlib import Path
+from datetime import datetime
+
+REPO = Path(r"C:\Users\alex\my-botty")
+CONFIG = REPO / "config"
+OUT = CONFIG / "price_dashboard.html"
+
+# Rune value ladder (relative to El=1) for normalizing trade prices
+RUNE_VALUES = {
+ "el rune": 1, "eld rune": 1, "tir rune": 1, "nef rune": 1,
+ "eth rune": 1, "ith rune": 1,
+ "tal rune": 2, "ral rune": 3, "ort rune": 4, "amn rune": 5,
+ "sol rune": 6, "shael rune": 3, "dol rune": 3, "hel rune": 3,
+ "io rune": 4, "ko rune": 4, "lum rune": 4,
+ "fal rune": 7, "lem rune": 8, "pul rune": 9, "um rune": 10,
+ "mal rune": 12, "ist rune": 15, "gul rune": 18, "vex rune": 22,
+ "ohm rune": 25, "lo rune": 30, "cham rune": 35,
+ "jah rune": 50, "ber rune": 60, "koh rune": 70,
+ "zod rune": 80,
+}
+CHARM_VALUES = {"small charm": 0.5, "medium charm": 1, "large charm": 2, "grand charm": 5}
+CURRENCY_VALUES = {**RUNE_VALUES, **CHARM_VALUES}
+
+
+def load_json(name, default=None):
+ p = CONFIG / name
+ if p.exists():
+ try:
+ return json.loads(p.read_text(encoding="utf-8"))
+ except Exception as e:
+ print(f"WARNING: could not load {name}: {e}")
+ return default if default is not None else {}
+
+
+def normalize_trade_price(currencies: dict) -> float:
+ """Convert a currency dict {name: {count, median...}} to a normalized value (El=1)."""
+ total = 0.0
+ for cur, stats in currencies.items():
+ val = CURRENCY_VALUES.get(cur.lower(), 0)
+ count = stats.get("count", 1) if isinstance(stats, dict) else 1
+ total += val * count
+ return total
+
+
+def build_dashboard():
+ daily = load_json("daily_prices.json", {})
+ history = load_json("daily_prices_history.json", [])
+ fg_est = load_json("fg_daily_estimates.json", {})
+ traderie = load_json("traderie_prices.json", {})
+
+ prices = daily.get("prices", {})
+ date = daily.get("date", "unknown")
+ generated = daily.get("generated_at", "")
+ n_listings = daily.get("traderie_listings_scanned", 0)
+
+ # Build item rows: name, fg_price, trade_price (normalized), trade_detail, listings, fg_name
+ rows = []
+ for name, data in prices.items():
+ tr = data.get("traderie")
+ fg = data.get("fg")
+ fg_name = data.get("fg_name")
+
+ trade_norm = None
+ trade_detail = ""
+ listings = 0
+ if tr:
+ listings = tr.get("total_listings", 0)
+ curs = tr.get("currencies", {})
+ overall = tr.get("overall", {})
+ trade_norm = overall.get("median")
+ # Build human-readable detail
+ parts = []
+ for cur, stats in curs.items():
+ cnt = stats.get("count", 1) if isinstance(stats, dict) else 1
+ med = stats.get("median", "?") if isinstance(stats, dict) else "?"
+ parts.append(f"{cnt}x {cur} ({med})" if cnt > 1 else f"{cur} ({med})")
+ trade_detail = " + ".join(parts)
+
+ rows.append({
+ "name": name,
+ "fg": fg,
+ "fg_name": fg_name,
+ "trade_norm": trade_norm,
+ "trade_detail": trade_detail,
+ "listings": listings,
+ "has_fg": fg is not None,
+ "has_trade": tr is not None,
+ })
+
+ # Sort: items with both first, then by max price
+ def sort_key(r):
+ fg_v = r["fg"] if isinstance(r["fg"], (int, float)) else 0
+ tr_v = r["trade_norm"] if isinstance(r["trade_norm"], (int, float)) else 0
+ return (-(1 if r["has_fg"] and r["has_trade"] else 0), -(max(fg_v, tr_v)))
+ rows.sort(key=sort_key)
+
+ # History for sparklines: date -> {item: value}
+ hist_by_date = {}
+ for entry in history:
+ d = entry.get("date")
+ if d:
+ hist_by_date[d] = entry.get("prices", {})
+ hist_dates = sorted(hist_by_date.keys())
+
+ # FG estimates (d2jsp forum)
+ fg_estimates = fg_est.get("estimates", {})
+ topics_scanned = fg_est.get("topics_scanned", 0)
+
+ # Traderie rune price summary (what runes are worth in listings)
+ rune_summary = {}
+ for name, data in prices.items():
+ tr = data.get("traderie")
+ if not tr:
+ continue
+ for cur, stats in tr.get("currencies", {}).items():
+ if cur.lower() in RUNE_VALUES or cur.lower() in CHARM_VALUES:
+ cnt = stats.get("count", 0) if isinstance(stats, dict) else 0
+ rune_summary[cur] = rune_summary.get(cur, 0) + cnt
+
+ # Build HTML
+ now = datetime.now().strftime("%Y-%m-%d %H:%M")
+ html_parts = []
+ html_parts.append(f"""
+
+
+
+
+D2R Price Dashboard
+
+
+
+
🪙 Rune/Charm Demand (Traderie listings as currency)
+
""")
+ for cur in sorted(rune_summary.keys(), key=lambda c: -rune_summary[c]):
+ val = CURRENCY_VALUES.get(cur.lower(), 0)
+ html_parts.append(f"""
+
{html.escape(cur)}
+
{rune_summary[cur]} listings
+
≈{val} El
+
""")
+ html_parts.append("""
+
""")
+
+ # History section
+ if len(hist_dates) > 1:
+ html_parts.append(f"""
+
+
📈 Price History ({len(hist_dates)} days)
+
+
+
Date
Items
FG Items
Trade Items
+ """)
+ for d in hist_dates:
+ p = hist_by_date[d]
+ n_items = len(p)
+ n_fg = sum(1 for v in p.values() if v.get("fg") is not None)
+ n_tr = sum(1 for v in p.values() if v.get("traderie") is not None)
+ html_parts.append(f"""
{html.escape(d)}
{n_items}
{n_fg}
{n_tr}
""")
+ html_parts.append("""
+
+
+
""")
+
+ html_parts.append(f"""
+
+
+
+
+""")
+
+ OUT.write_text("".join(html_parts), encoding="utf-8")
+ print(f"Dashboard written to {OUT}")
+ print(f" Items: {len(rows)}")
+ print(f" With FG: {sum(1 for r in rows if r['has_fg'])}")
+ print(f" With Trade: {sum(1 for r in rows if r['has_trade'])}")
+ print(f" Both: {sum(1 for r in rows if r['has_fg'] and r['has_trade'])}")
+ print(f" History days: {len(hist_dates)}")
+ print(f" Rune currencies: {len(rune_summary)}")
+
+
+if __name__ == "__main__":
+ build_dashboard()
diff --git a/tools/melee_hunt.py b/tools/melee_hunt.py
index d65f69a..af00462 100644
--- a/tools/melee_hunt.py
+++ b/tools/melee_hunt.py
@@ -33,7 +33,9 @@ def main():
ap = argparse.ArgumentParser()
ap.add_argument("--minutes", type=float, default=10.0)
ap.add_argument("--stop-key", default="f12")
- ap.add_argument("--swings", type=int, default=5, help="clicks per target before rescanning")
+ ap.add_argument("--swings", type=int, default=10, help="clicks per target before rescanning")
+ ap.add_argument("--button", default="right", choices=["left", "right"],
+ help="mouse button to attack with (default: right)")
ap.add_argument("--min-area", type=int, default=80, help="smallest motion blob to treat as a monster")
ap.add_argument("--max-area", type=int, default=9000, help="largest motion blob to accept")
ap.add_argument("--debug", action="store_true", help="save an annotated scan image each cycle")
@@ -75,7 +77,10 @@ def main():
W, H = 1280, 720
CX, CY = W // 2, 340 # character sits slightly above centre
SELF_R = 85 # ignore her own animation
- HUD_TOP, HUD_BOTTOM = 70, 640 # ignore the name banner and the control bar
+ # The level-up/stat controls pulse above the main control bar and look like
+ # monsters to a motion detector. Keep the entire interactive HUD out of the
+ # target image so a click can never open a panel during combat.
+ HUD_TOP, HUD_BOTTOM = 70, 510
fm = Config().char["force_move"]
stand_still = Config().char["stand_still"]
@@ -111,6 +116,9 @@ def main():
cv2.imwrite("log/screenshots/manual/hunt_debug.png", dbg)
return out
+ class SafetyStop(RuntimeError):
+ pass
+
def check_health():
"""Drink a healing potion if life has dropped to the threshold.
@@ -126,9 +134,14 @@ def main():
print(f" LIFE {hp*100:.0f}% <= {opts.heal_at*100:.0f}% — drinking potion")
try:
if not belt.drink_potion("health", stats=[hp, 1.0]):
- print(" no healing potion in belt!")
+ raise SafetyStop(
+ f"life is {hp*100:.0f}% and no healing potion was detected")
+ except SafetyStop:
+ raise
except Exception as e:
print(f" potion failed: {e}")
+ raise SafetyStop(
+ f"life is {hp*100:.0f}% and potion use failed") from e
wait(0.4, 0.6)
return hp
@@ -188,9 +201,9 @@ def main():
if stopping():
break
mouse.move(mx, my, randomize=6, delay_factor=[0.15, 0.25])
- mouse.click(button="left")
+ mouse.click(button=opts.button)
wait(0.22, 0.32)
- if i and i % 2 == 0:
+ if i and i % 4 == 0:
hp = check_health()
finally:
keyboard.send(stand_still, do_press=False)
@@ -240,6 +253,8 @@ def main():
f"life={hp_txt} xp={xp_txt} "
f"{max(0, deadline - time.time()):5.0f}s left ")
print(CR + status, end="", flush=True)
+ except SafetyStop as e:
+ print(f"\nSAFETY STOP: {e}")
except KeyboardInterrupt:
print("\ninterrupted")
diff --git a/tools/push_prices_to_db.py b/tools/push_prices_to_db.py
new file mode 100644
index 0000000..ff66607
--- /dev/null
+++ b/tools/push_prices_to_db.py
@@ -0,0 +1,257 @@
+#!/usr/bin/env python
+"""Push D2R price data to MariaDB on .96.
+
+Reads config/daily_prices.json + config/fg_prices.json from the repo,
+pushes all prices to the d2r_prices database on 192.168.1.96:3306.
+
+FG prices come from fg_prices.json rules (name → fg value).
+Trade prices come from daily_prices.json (traderie data with raw currency detail).
+
+Usage:
+ python tools/push_prices_to_db.py
+"""
+import json
+import re
+import sys
+from datetime import datetime
+from pathlib import Path
+
+try:
+ import pymysql
+except ImportError:
+ print("ERROR: pymysql not installed. Run: pip install pymysql")
+ sys.exit(1)
+
+REPO = Path(__file__).resolve().parent.parent
+CONFIG = REPO / "config"
+
+DB_CONFIG = {
+ "host": "192.168.1.96",
+ "port": 3306,
+ "user": "d2r",
+ "password": "D2rPrices@2026!",
+ "database": "d2r_prices",
+ "charset": "utf8mb4",
+}
+
+
+def load_json(name, default=None):
+ p = CONFIG / name
+ if p.exists():
+ try:
+ return json.loads(p.read_text(encoding="utf-8"))
+ except Exception as e:
+ print(f"WARNING: could not load {name}: {e}")
+ return default if default is not None else {}
+
+
+def build_fg_lookup(fg_config):
+ """Build a lookup from fg_prices.json rules: item_name → fg_price."""
+ lookup = {}
+ rules = fg_config.get("rules", [])
+ for rule in rules:
+ name = rule.get("name", "")
+ fg = rule.get("fg")
+ if name and isinstance(fg, (int, float)):
+ lookup[name.lower()] = float(fg)
+ return lookup
+
+
+def match_fg(item_name, fg_lookup):
+ """Try to match an item name to an FG price rule."""
+ # Exact match first
+ key = item_name.lower()
+ if key in fg_lookup:
+ return fg_lookup[key]
+ # Try without common suffixes
+ for suffix in [" rune", " charm", " jewel", " amulet"]:
+ if key.endswith(suffix):
+ base = key[:-len(suffix)]
+ if base in fg_lookup:
+ return fg_lookup[base]
+ return None
+
+
+def format_trade_detail(tr):
+ """Format trade detail as '2 Jah Rune + 1 Ber Rune' (not El Rune normalized)."""
+ if not tr:
+ return ""
+ parts = []
+ for cur, stats in tr.get("currencies", {}).items():
+ if isinstance(stats, dict):
+ cnt = stats.get("count", 1)
+ med = stats.get("median", "?")
+ if cnt > 1:
+ parts.append(f"{cnt} {cur} ({med})")
+ else:
+ parts.append(f"{cur} ({med})")
+ else:
+ parts.append(f"{cur} ({stats})")
+ return " + ".join(parts)
+
+
+def extract_currency_prices(prices):
+ """Extract per-currency median prices from all trade listings.
+
+ Returns: {currency_name: {"median": float, "listings": int}}
+ """
+ currency_data = {}
+ for name, data in prices.items():
+ tr = data.get("traderie")
+ if not tr:
+ continue
+ for cur, stats in tr.get("currencies", {}).items():
+ if not isinstance(stats, dict):
+ continue
+ med = stats.get("median")
+ cnt = stats.get("count", 0)
+ if med is None or not isinstance(med, (int, float)):
+ continue
+ if cur not in currency_data:
+ currency_data[cur] = {"prices": [], "listings": 0}
+ currency_data[cur]["prices"].append(float(med))
+ currency_data[cur]["listings"] += cnt
+
+ # Compute median for each currency
+ result = {}
+ for cur, d in currency_data.items():
+ if d["prices"]:
+ sorted_p = sorted(d["prices"])
+ n = len(sorted_p)
+ median = sorted_p[n // 2] if n % 2 == 1 else (sorted_p[n // 2 - 1] + sorted_p[n // 2]) / 2
+ result[cur] = {"median": median, "listings": d["listings"]}
+ return result
+
+
+def push():
+ daily = load_json("daily_prices.json", {})
+ fg_config = load_json("fg_prices.json", {})
+
+ if not daily:
+ print("ERROR: daily_prices.json not found or empty")
+ sys.exit(1)
+
+ prices = daily.get("prices", {})
+ date = daily.get("date", datetime.now().strftime("%Y-%m-%d"))
+ generated = daily.get("generated_at", "")
+ if generated and "T" in generated:
+ generated = generated.replace("T", " ").split("+")[0].split(".")[0]
+ n_listings = daily.get("traderie_listings_scanned", 0)
+
+ # Build FG lookup from rules
+ fg_lookup = build_fg_lookup(fg_config)
+ print(f"FG rules loaded: {len(fg_lookup)}")
+
+ # Build rows
+ rows = []
+ n_fg = n_trade = n_both = 0
+ for name, data in prices.items():
+ tr = data.get("traderie")
+ fg_data = data.get("fg")
+
+ # FG price: try merged data first, then rule match
+ fg_price = None
+ if isinstance(fg_data, (int, float)):
+ fg_price = float(fg_data)
+ else:
+ fg_price = match_fg(name, fg_lookup)
+
+ # Trade: keep raw currency detail, NOT El Rune normalized
+ trade_value = None
+ trade_detail = ""
+ listings = 0
+ if tr:
+ listings = tr.get("total_listings", 0)
+ trade_detail = format_trade_detail(tr)
+ # Use the primary currency's median as trade_value (not El Rune)
+ currencies = tr.get("currencies", {})
+ if currencies:
+ # Pick the most common currency
+ primary = max(currencies.items(), key=lambda x: x[1].get("count", 0) if isinstance(x[1], dict) else 0)
+ cur_name, cur_stats = primary
+ if isinstance(cur_stats, dict) and isinstance(cur_stats.get("median"), (int, float)):
+ trade_value = float(cur_stats["median"])
+
+ has_fg = fg_price is not None
+ has_trade = bool(trade_detail)
+ if has_fg:
+ n_fg += 1
+ if has_trade:
+ n_trade += 1
+ if has_fg and has_trade:
+ n_both += 1
+
+ source = "both" if (has_fg and has_trade) else ("fg" if has_fg else "trade")
+
+ rows.append((
+ date, name, fg_price, None, trade_value,
+ trade_detail, listings, int(has_fg), int(has_trade), source,
+ ))
+
+ # Add FG-only items that aren't in the Traderie data
+ # fg_lookup keys are lowercase, so we need the original names
+ for rule in fg_config.get("rules", []):
+ rule_name = rule.get("name", "")
+ rule_fg = rule.get("fg")
+ if not rule_name or not isinstance(rule_fg, (int, float)):
+ continue
+ # Check if already added (case-insensitive)
+ if any(r[1].lower() == rule_name.lower() for r in rows):
+ continue
+ rows.append((
+ date, rule_name, float(rule_fg), None, None,
+ "", 0, 1, 0, "fg",
+ ))
+ n_fg += 1
+
+ # Connect and push
+ conn = pymysql.connect(**DB_CONFIG)
+ try:
+ with conn.cursor() as cur:
+ cur.executemany("""
+ INSERT INTO prices (scrape_date, item_name, fg_price, fg_name,
+ trade_value, trade_detail, listings, has_fg, has_trade, source)
+ VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+ ON DUPLICATE KEY UPDATE
+ fg_price=VALUES(fg_price), fg_name=VALUES(fg_name),
+ trade_value=VALUES(trade_value), trade_detail=VALUES(trade_detail),
+ listings=VALUES(listings), has_fg=VALUES(has_fg),
+ has_trade=VALUES(has_trade), source=VALUES(source)
+ """, rows)
+
+ cur.execute("""
+ INSERT INTO scrape_meta (scrape_date, generated_at, traderie_listings,
+ d2jsp_topics, items_total, items_fg, items_trade, items_both)
+ VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
+ ON DUPLICATE KEY UPDATE
+ generated_at=VALUES(generated_at), traderie_listings=VALUES(traderie_listings),
+ d2jsp_topics=VALUES(d2jsp_topics), items_total=VALUES(items_total),
+ items_fg=VALUES(items_fg), items_trade=VALUES(items_trade),
+ items_both=VALUES(items_both)
+ """, (date, generated, n_listings, 0, len(rows), n_fg, n_trade, n_both))
+
+ conn.commit()
+
+ # Push currency prices
+ currency_prices = extract_currency_prices(prices)
+ if currency_prices:
+ cur_rows = [(date, cur, d["median"], d["listings"]) for cur, d in currency_prices.items()]
+ cur2 = conn.cursor()
+ cur2.executemany("""
+ INSERT INTO currency_prices (scrape_date, currency_name, median_price, listings)
+ VALUES (%s, %s, %s, %s)
+ ON DUPLICATE KEY UPDATE
+ median_price=VALUES(median_price), listings=VALUES(listings)
+ """, cur_rows)
+ conn.commit()
+ print(f" Currency prices: {len(cur_rows)} currencies")
+
+ print(f"Pushed {len(rows)} prices for {date} to 192.168.1.96")
+ print(f" FG: {n_fg}, Trade: {n_trade}, Both: {n_both}")
+ print(f" Traderie listings: {n_listings}")
+ finally:
+ conn.close()
+
+
+if __name__ == "__main__":
+ push()
diff --git a/tools/run_continuous_scrape.bat b/tools/run_continuous_scrape.bat
new file mode 100644
index 0000000..9d5fde4
--- /dev/null
+++ b/tools/run_continuous_scrape.bat
@@ -0,0 +1,40 @@
+@echo off
+REM Continuous D2R Price Scrape - runs all night
+REM Scrape every 30 minutes, push to .96
+setlocal
+set PYTHON=C:\Python313\python.exe
+set REPO=C:\Users\alex\my-botty
+set LOG=%REPO%\log\fg_scrape_continuous.log
+set INTERVAL=1800 REM 30 minutes in seconds
+
+echo ==========================================>> "%LOG%"
+echo [%date% %time%] Starting continuous price scrape>> "%LOG%"
+echo ==========================================>> "%LOG%"
+
+:loop
+cd /d "%REPO%"
+
+echo [%date% %time%] Running scrape cycle...>> "%LOG%"
+
+REM Scrape d2jsp
+"%PYTHON%" tools\playwright_fg_scraper.py --max-forum-pages 40 --max-topics 800 --delay-s 0.5 >> "%LOG%" 2>&1
+
+REM Parse d2jsp
+"%PYTHON%" tools\fg_market_scraper.py --offline-dir data/d2jsp_pages --ladder-start-date 2026-05-12 --days 60 >> "%LOG%" 2>&1
+
+REM Scrape Traderie
+"%PYTHON%" tools\traderie_price_scraper.py >> "%LOG%" 2>&1
+
+REM Push to .96
+scp -o StrictHostKeyChecking=no -r data/d2jsp_pages/* alex@192.168.1.96:/tmp/d2jsp_pages/ >> "%LOG%" 2>&1
+scp -o StrictHostKeyChecking=no config/daily_prices.json alex@192.168.1.96:/tmp/ >> "%LOG%" 2>&1
+scp -o StrictHostKeyChecking=no config/fg_daily_estimates.json alex@192.168.1.96:/tmp/ >> "%LOG%" 2>&1
+
+ssh -o StrictHostKeyChecking=no alex@192.168.1.96 "sudo mkdir -p /opt/d2r-prices/data/d2jsp_pages /opt/d2r-prices/config && sudo cp /tmp/d2jsp_pages/* /opt/d2r-prices/data/d2jsp_pages/ 2>/dev/null; sudo cp /tmp/daily_prices.json /opt/d2r-prices/config/ 2>/dev/null; sudo cp /tmp/fg_daily_estimates.json /opt/d2r-prices/config/ 2>/dev/null; sudo rm -rf /tmp/d2jsp_pages /tmp/daily_prices.json /tmp/fg_daily_estimates.json; sudo /opt/d2r-prices/run_scrape.sh" >> "%LOG%" 2>&1
+
+echo [%date% %time%] Cycle complete. Waiting %INTERVAL% seconds...>> "%LOG%"
+
+REM Wait 30 minutes
+timeout /t %INTERVAL% /nobreak >nul
+
+goto loop
diff --git a/tools/run_fg_scrape.bat b/tools/run_fg_scrape.bat
new file mode 100644
index 0000000..9cf52c4
--- /dev/null
+++ b/tools/run_fg_scrape.bat
@@ -0,0 +1,37 @@
+@echo off
+REM Daily D2R Price Pipeline - .211
+REM Scrape d2jsp + Traderie -> push data to .96 -> .96 runs DB + dashboard
+setlocal
+set PYTHON=C:\Python313\python.exe
+set REPO=C:\Users\alex\my-botty
+set LOG=%REPO%\log\fg_scrape.log
+
+echo ==========================================>> "%LOG%"
+echo [%date% %time%] Starting daily price scrape>> "%LOG%"
+echo ==========================================>> "%LOG%"
+
+cd /d "%REPO%"
+
+echo [1/3] Scraping d2jsp (40 pages, 800 topics)... >> "%LOG%"
+"%PYTHON%" tools\playwright_fg_scraper.py --max-forum-pages 40 --max-topics 800 --delay-s 0.5 >> "%LOG%" 2>&1
+if errorlevel 1 (
+ echo [ERROR] d2jsp scrape failed >> "%LOG%"
+ goto :end
+)
+
+echo [2/3] Scraping Traderie... >> "%LOG%"
+"%PYTHON%" tools\traderie_price_scraper.py >> "%LOG%" 2>&1
+
+echo [3/3] Pushing data to .96 and running pipeline... >> "%LOG%"
+REM Push d2jsp HTML files
+scp -o StrictHostKeyChecking=no -r data/d2jsp_pages/* alex@192.168.1.96:/tmp/d2jsp_pages/ >> "%LOG%" 2>&1
+REM Push Traderie + FG data
+scp -o StrictHostKeyChecking=no config/daily_prices.json alex@192.168.1.96:/tmp/ >> "%LOG%" 2>&1
+scp -o StrictHostKeyChecking=no config/fg_daily_estimates.json alex@192.168.1.96:/tmp/ >> "%LOG%" 2>&1
+
+REM Move to correct location and run .96 pipeline
+ssh -o StrictHostKeyChecking=no alex@192.168.1.96 "sudo mkdir -p /opt/d2r-prices/data/d2jsp_pages /opt/d2r-prices/config && sudo cp /tmp/d2jsp_pages/* /opt/d2r-prices/data/d2jsp_pages/ 2>/dev/null; sudo cp /tmp/daily_prices.json /opt/d2r-prices/config/ 2>/dev/null; sudo cp /tmp/fg_daily_estimates.json /opt/d2r-prices/config/ 2>/dev/null; sudo rm -rf /tmp/d2jsp_pages /tmp/daily_prices.json /tmp/fg_daily_estimates.json; sudo /opt/d2r-prices/run_scrape_96.sh" >> "%LOG%" 2>&1
+
+:end
+echo [%date% %time%] Pipeline complete >> "%LOG%"
+echo Done.