Bestiary Sanitization — Progress Report

Last updated: 2026-07-02 (session in progress)

Why this exists

The extracted bestiary was badly corrupted: distinct creatures shared mis-aligned statblocks (e.g. Civilian / Servo-Skull / Imperial Agent all got the Hive Tyrant's line — T78 / W120), and several NPCs stored Strength/Toughness bonuses (single digits) where the characteristic should be. This report tracks the repair.

Effective data source (important)

Runtime reads public/deathwatch-bestiary-extracted.json (34 entries). loadBestiaryData() tries a MariaDB bestiary table first, but that table does not exist, so it falls back to this JSON. Fixing the JSON fixes the app.
database/deathwatch-bestiary-extracted.json is a stale 24-entry subset, not used at runtime. (Left mostly alone; safe to regenerate from public later.)
Sources of truth for stats: the rulebook PDFs in database/rules/ (CR.pdf = Core Rulebook, MoX.pdf = Mark of the Xenos), read with pdftotext (raw mode linearizes statblocks; -layout -f P -l P per page pairs name→profile). Plus database/remote-extract/all_deathwatch_statblocks.json (5 NPCs).

DONE ✅

Roller code fixes (src/components/DeathwatchRoller.jsx, SkillRoller.jsx)

GM "Roll as" any player/enemy; players added to Target/defender list.
Fate now loads for the acting-as player (fateSubject).
Player self-load reads lower-case chars + {total,current} wounds (was broken).
Enemy 0/again wounds coerced robustly (strings, 0, missing → usable tracker).
/api/bestiary/enemies now returns the full bestiary (filter relaxed) and transformBestiaryEntry coerces wounds + honours stats.toughnessBonus.
Fire-mode +10/+20, jam 96/94/100, Advanced-untrained skill gate, Dodge/Parry burst negation = DoS (earlier in session). Full CRA build compiles.

Bestiary data

scripts/sanitize-bestiary-canonical.js written and run against public/…json. Backups written (*.pre-sanitize.<ts>.json).
Result on the 34 public entries:
- PDF-verified (9): Chaos Space Marine, Chaos Heretic, Gun Drone, Hive Tyrant, Hormagaunt, Tau Commander, Tau Fire Warrior, Tau Stealth Suit, Tyranid Warrior.
- Canonical-fill, review (13): Termagant, Renegade Militia, Imperial Guardsman, Daemon Prince, Ork Nob, Battle/Monotask/Industrial Servitor, Servo-Skull, Civilian, Chapter Serf, Imperial Agent, Imperial Guard Field/Senior Officer.
- OCR-restored (5): caele Rylus Darkscourge, Father Marius, Infected Auran warrior, Feynir Surbryte, Arcus Dyre.
- Structural-only, review (8): the TEP named NPCs (Agamorr, Skarjaw, Vex, Genestealer Patriarch, Zephon, Chaos Cultist Champion, Prince-Prefect Allewis, Alexei Drahj — S/T bonus→char fixed).
Each touched entry records stats.statSource ('pdf'|'canon'|'ocr'|'structural') and stats.needsReview, and an effective stats.toughnessBonus (includes Unnatural Toughness, e.g. Tyranid Warrior 10, Hive Tyrant 15, Genestealer 8).

PDF-verified statblocks (authoritative, WS BS S T Ag Int Per WP Fel | W | armour)

Chaos Space Marine  50 50 65 45 40 35 40 43 17 | 29  | Power Armour H8/A8/B10/L8   CR363
Tyranid Warrior     55 30 60 50 44 20 35 50 –  | 48  | Reinforced Chitin All8      CR370
Hive Tyrant         78 33 60 53 45 45 49 70 –  | 120 | Bonded Exoskeleton All10    CR371
Hormagaunt          45 20 35 30 55 10 40 30 –  | 9   | Chitinous Carapace All3     CR372
Tau Fire Warrior    25 35 30 30 25 30 20 30 30 | 12  | Fire Warrior Armour All6    CR369
Tau Stealth Suit    30 42 42 47 38 30 40 40 40 | 30  | XV15 H8/B8/A7/L7            CR368
Gun Drone           20 25 20 40 40 15 25 20 10 | 15  | (machine)                  CR368
Chaos Heretic       35 35 35 35 30 20 30 35 20 | 10  | Flak All4                  CR361
Tau Commander       41 57 50 57 48 45 45 50 50 | 90  | XV-8 Crisis Suit All9      MoX
Genestealer         55 0  45 40 50 20 40 40 10 | 16  | Reinforced Chitin All4      MoX44
Apostle Bellephrades75 60 66 45 48 48 48 75 30 | 80  | Armour of Chaos All12       CR363

DONE ✅ (continued)

Sanitizer now also ADDS missing canonical creatures and mirrors public→DB copy. Public + database JSON are both 38 entries and identical. New entries added: Genestealer, Ork Nob, Apostle Bellephrades, Industrial Servitor.
Bestiary UI redesign (src/components/BestiaryTab.jsx): colour-coded stat chips (WS/BS/S/T/Ag/Int/Per/WP/Fel each with its own colour), tier bar + badge (Troops/Elite/Master by wounds), Wounds + TB pills, verified/review badges from statSource/needsReview, and smart armour rendering ({all:8}→"All 8", location objects → "Head 8 · Body 10 · …"). Build compiles clean.
Long-entry cleanup: scripts/sanitize-bestiary-canonical.js now strips raw pageText/snippet extraction text, removes copied root-level spillover from canonical entries, and trims OCR/structural skills/talents/traits/gear at section boundaries. Public + database JSON remain identical, 38 entries, with no rendered text field over 300 chars.
40k RPG Tools index staging: scripts/import-40krpgtools-bestiary-index.js downloads the public master bestiary index from https://www.40krpgtools.com/bestiary/ and writes Deathwatch-only missing references to database/remote-extract/40krpgtools-bestiary-index.json. This is an index only (name/type/affiliation/book/page/source URL), not statblocks, so it does not write directly into the live bestiary.
DB-backed live bestiary: database/mariadb.js now creates the bestiary table and exposes replace/upsert helpers. scripts/import-bestiary-json-to-db.js imports the sanitized active bestiary into MariaDB. Current DB import: 84 rows, 57 PDF-verified, all 84 are roller-ready because /api/bestiary/enemies reads the same bestiary table before falling back to JSON.
40k RPG Tools hydration: scripts/hydrate-bestiary-from-40krpgtools.js uses the staged index as a page map and local PDFs as the stat source. First Mark of the Xenos pass added 46 compact statblocks to the active bestiary. The remaining index-only rows stay out of the live combat table until a real profile and wounds can be extracted.
First Founding pass: hydrated 4 additional compact statblocks from the local FF.pdf: The Headsman of Cellebos, Blight-Master Ussax, Atum Sum, and Warsmith Koros. Re-imported MariaDB afterward; current live DB count is 88 bestiary rows. Verified these four through /api/bestiary/enemies with valid Wounds/TB/armour output for the combat roller.

TODO / NEXT (in priority order)

More NEW creatures: the PDF scan found ~40 more clean profiles whose names need pairing via per-page -layout extraction (Ork Boy, Broadside XV-88, Kroot Carnivore, Bloodletters/greater daemons, more Tyranids). Extend the CANON table in scripts/sanitize-bestiary-canonical.js (auto-adds if missing). The 40k RPG Tools staging file currently has 344 Deathwatch index rows not already in the active bestiary; use it to prioritize local PDF page extraction. Do not import index-only rows into the live bestiary table because live bestiary rows automatically appear in the combat roller.
Re-verify canonical-fill entries against PDFs (Termagant, Ork Nob, Daemon Prince, servitors, officers) to promote them from needsReview to pdf (verified).
ComfyUI (192.168.1.96) image generation for players/beasts — user asked; feasible. Plan: backend POST /api/imagegen proxy → ComfyUI /prompt API with a stored workflow JSON, poll /history/{id}, save PNG under /avatars or a new /beast-images; UI "Generate portrait" buttons on PlayerTab + BestiaryTab. NOT STARTED (needs the ComfyUI workflow JSON + confirmation the box is reachable).

How to continue

Re-run sanitizer: node scripts/sanitize-bestiary-canonical.js (idempotent; writes fresh backups each run).
Extract a creature page: pdftotext -layout -f <page> -l <page> database/rules/CR.pdf - (or MoX.pdf). Profile = the 9-number row; Wounds/Armour/Weapons lines follow.
The DoS/roller conventions and the jsdom-segfault test workarounds are in project memory (~/.claude/projects/-nas-git-dwroller/memory/).

8.2 KiB Raw Blame History Unescape Escape