# Bestiary Sanitization — Progress Report

_Last updated: 2026-07-02 (session in progress)_

## Why this exists
The extracted bestiary was badly corrupted: distinct creatures shared mis-aligned
statblocks (e.g. Civilian / Servo-Skull / Imperial Agent all got the Hive Tyrant's
line — T78 / W120), and several NPCs stored **Strength/Toughness bonuses** (single
digits) where the characteristic should be. This report tracks the repair.

## Effective data source (important)
- Runtime reads **`public/deathwatch-bestiary-extracted.json`** (34 entries).
  `loadBestiaryData()` tries a MariaDB `bestiary` table first, but **that table does
  not exist**, so it falls back to this JSON. Fixing the JSON fixes the app.
- `database/deathwatch-bestiary-extracted.json` is a **stale 24-entry subset**, not
  used at runtime. (Left mostly alone; safe to regenerate from public later.)
- Sources of truth for stats: the rulebook PDFs in `database/rules/`
  (`CR.pdf` = Core Rulebook, `MoX.pdf` = Mark of the Xenos), read with `pdftotext`
  (raw mode linearizes statblocks; `-layout -f P -l P` per page pairs name→profile).
  Plus `database/remote-extract/all_deathwatch_statblocks.json` (5 NPCs).

## DONE ✅
### Roller code fixes (src/components/DeathwatchRoller.jsx, SkillRoller.jsx)
- GM "Roll as" any player/enemy; players added to Target/defender list.
- Fate now loads for the acting-as player (`fateSubject`).
- Player self-load reads lower-case chars + `{total,current}` wounds (was broken).
- Enemy 0/again wounds coerced robustly (strings, 0, missing → usable tracker).
- `/api/bestiary/enemies` now returns the **full** bestiary (filter relaxed) and
  `transformBestiaryEntry` coerces wounds + honours `stats.toughnessBonus`.
- Fire-mode +10/+20, jam 96/94/100, Advanced-untrained skill gate, Dodge/Parry
  burst negation = DoS (earlier in session). Full CRA build compiles.

### Bestiary data
- `scripts/sanitize-bestiary-canonical.js` written and **run** against
  `public/…json`. Backups written (`*.pre-sanitize.<ts>.json`).
- Result on the 34 public entries:
  - **PDF-verified (9):** Chaos Space Marine, Chaos Heretic, Gun Drone, Hive Tyrant,
    Hormagaunt, Tau Commander, Tau Fire Warrior, Tau Stealth Suit, Tyranid Warrior.
  - **Canonical-fill, review (13):** Termagant, Renegade Militia, Imperial Guardsman,
    Daemon Prince, Ork Nob, Battle/Monotask/Industrial Servitor, Servo-Skull,
    Civilian, Chapter Serf, Imperial Agent, Imperial Guard Field/Senior Officer.
  - **OCR-restored (5):** caele Rylus Darkscourge, Father Marius, Infected Auran
    warrior, Feynir Surbryte, Arcus Dyre.
  - **Structural-only, review (8):** the TEP named NPCs (Agamorr, Skarjaw, Vex,
    Genestealer Patriarch, Zephon, Chaos Cultist Champion, Prince-Prefect Allewis,
    Alexei Drahj — S/T bonus→char fixed).
- Each touched entry records `stats.statSource` ('pdf'|'canon'|'ocr'|'structural')
  and `stats.needsReview`, and an effective `stats.toughnessBonus` (includes
  Unnatural Toughness, e.g. Tyranid Warrior 10, Hive Tyrant 15, Genestealer 8).

## PDF-verified statblocks (authoritative, WS BS S T Ag Int Per WP Fel | W | armour)
```
Chaos Space Marine  50 50 65 45 40 35 40 43 17 | 29  | Power Armour H8/A8/B10/L8   CR363
Tyranid Warrior     55 30 60 50 44 20 35 50 –  | 48  | Reinforced Chitin All8      CR370
Hive Tyrant         78 33 60 53 45 45 49 70 –  | 120 | Bonded Exoskeleton All10    CR371
Hormagaunt          45 20 35 30 55 10 40 30 –  | 9   | Chitinous Carapace All3     CR372
Tau Fire Warrior    25 35 30 30 25 30 20 30 30 | 12  | Fire Warrior Armour All6    CR369
Tau Stealth Suit    30 42 42 47 38 30 40 40 40 | 30  | XV15 H8/B8/A7/L7            CR368
Gun Drone           20 25 20 40 40 15 25 20 10 | 15  | (machine)                  CR368
Chaos Heretic       35 35 35 35 30 20 30 35 20 | 10  | Flak All4                  CR361
Tau Commander       41 57 50 57 48 45 45 50 50 | 90  | XV-8 Crisis Suit All9      MoX
Genestealer         55 0  45 40 50 20 40 40 10 | 16  | Reinforced Chitin All4      MoX44
Apostle Bellephrades75 60 66 45 48 48 48 75 30 | 80  | Armour of Chaos All12       CR363
```

## DONE ✅ (continued)
- **Sanitizer now also ADDS missing canonical creatures** and mirrors public→DB
  copy. Public + database JSON are both **38 entries** and identical. New entries
  added: **Genestealer, Ork Nob, Apostle Bellephrades, Industrial Servitor**.
- **Bestiary UI redesign** (`src/components/BestiaryTab.jsx`): colour-coded stat
  chips (WS/BS/S/T/Ag/Int/Per/WP/Fel each with its own colour), tier bar + badge
  (Troops/Elite/Master by wounds), Wounds + TB pills, `verified`/`review` badges
  from `statSource`/`needsReview`, and smart armour rendering (`{all:8}`→"All 8",
  location objects → "Head 8 · Body 10 · …"). Build compiles clean.
- **Long-entry cleanup**: `scripts/sanitize-bestiary-canonical.js` now strips raw
  `pageText`/`snippet` extraction text, removes copied root-level spillover from
  canonical entries, and trims OCR/structural `skills`/`talents`/`traits`/`gear`
  at section boundaries. Public + database JSON remain identical, 38 entries,
  with no rendered text field over 300 chars.
- **40k RPG Tools index staging**: `scripts/import-40krpgtools-bestiary-index.js`
  downloads the public master bestiary index from `https://www.40krpgtools.com/bestiary/`
  and writes Deathwatch-only missing references to
  `database/remote-extract/40krpgtools-bestiary-index.json`. This is an index
  only (name/type/affiliation/book/page/source URL), not statblocks, so it does
  not write directly into the live bestiary.
- **DB-backed live bestiary**: `database/mariadb.js` now creates the `bestiary`
  table and exposes replace/upsert helpers. `scripts/import-bestiary-json-to-db.js`
  imports the sanitized active bestiary into MariaDB. Current DB import: **84
  rows**, **57 PDF-verified**, all 84 are roller-ready because `/api/bestiary/enemies`
  reads the same `bestiary` table before falling back to JSON.
- **40k RPG Tools hydration**: `scripts/hydrate-bestiary-from-40krpgtools.js`
  uses the staged index as a page map and local PDFs as the stat source. First
  Mark of the Xenos pass added **46** compact statblocks to the active bestiary.
  The remaining index-only rows stay out of the live combat table until a real
  profile and wounds can be extracted.
- **First Founding pass**: hydrated **4** additional compact statblocks from the
  local `FF.pdf`: The Headsman of Cellebos, Blight-Master Ussax, Atum Sum, and
  Warsmith Koros. Re-imported MariaDB afterward; current live DB count is **88**
  bestiary rows. Verified these four through `/api/bestiary/enemies` with valid
  Wounds/TB/armour output for the combat roller.

## TODO / NEXT (in priority order)
1. **More NEW creatures**: the PDF scan found ~40 more clean profiles whose names
   need pairing via per-page `-layout` extraction (Ork Boy, Broadside XV-88, Kroot
   Carnivore, Bloodletters/greater daemons, more Tyranids). Extend the `CANON` table
   in `scripts/sanitize-bestiary-canonical.js` (auto-adds if missing).
   The 40k RPG Tools staging file currently has **344 Deathwatch index rows not
   already in the active bestiary**; use it to prioritize local PDF page extraction.
   Do not import index-only rows into the live `bestiary` table because live
   bestiary rows automatically appear in the combat roller.
2. Re-verify canonical-fill entries against PDFs (Termagant, Ork Nob, Daemon Prince,
   servitors, officers) to promote them from `needsReview` to `pdf` (verified).
3. **ComfyUI (192.168.1.96)** image generation for players/beasts — user asked;
   feasible. Plan: backend `POST /api/imagegen` proxy → ComfyUI `/prompt` API with a
   stored workflow JSON, poll `/history/{id}`, save PNG under `/avatars` or a new
   `/beast-images`; UI "Generate portrait" buttons on PlayerTab + BestiaryTab.
   NOT STARTED (needs the ComfyUI workflow JSON + confirmation the box is reachable).

## How to continue
- Re-run sanitizer: `node scripts/sanitize-bestiary-canonical.js` (idempotent;
  writes fresh backups each run).
- Extract a creature page: `pdftotext -layout -f <page> -l <page> database/rules/CR.pdf -`
  (or MoX.pdf). Profile = the 9-number row; Wounds/Armour/Weapons lines follow.
- The DoS/roller conventions and the jsdom-segfault test workarounds are in
  project memory (`~/.claude/projects/-nas-git-dwroller/memory/`).
```
```