128 lines
8.2 KiB
Markdown
128 lines
8.2 KiB
Markdown
# Bestiary Sanitization — Progress Report
|
||
|
||
_Last updated: 2026-07-02 (session in progress)_
|
||
|
||
## Why this exists
|
||
The extracted bestiary was badly corrupted: distinct creatures shared mis-aligned
|
||
statblocks (e.g. Civilian / Servo-Skull / Imperial Agent all got the Hive Tyrant's
|
||
line — T78 / W120), and several NPCs stored **Strength/Toughness bonuses** (single
|
||
digits) where the characteristic should be. This report tracks the repair.
|
||
|
||
## Effective data source (important)
|
||
- Runtime reads **`public/deathwatch-bestiary-extracted.json`** (34 entries).
|
||
`loadBestiaryData()` tries a MariaDB `bestiary` table first, but **that table does
|
||
not exist**, so it falls back to this JSON. Fixing the JSON fixes the app.
|
||
- `database/deathwatch-bestiary-extracted.json` is a **stale 24-entry subset**, not
|
||
used at runtime. (Left mostly alone; safe to regenerate from public later.)
|
||
- Sources of truth for stats: the rulebook PDFs in `database/rules/`
|
||
(`CR.pdf` = Core Rulebook, `MoX.pdf` = Mark of the Xenos), read with `pdftotext`
|
||
(raw mode linearizes statblocks; `-layout -f P -l P` per page pairs name→profile).
|
||
Plus `database/remote-extract/all_deathwatch_statblocks.json` (5 NPCs).
|
||
|
||
## DONE ✅
|
||
### Roller code fixes (src/components/DeathwatchRoller.jsx, SkillRoller.jsx)
|
||
- GM "Roll as" any player/enemy; players added to Target/defender list.
|
||
- Fate now loads for the acting-as player (`fateSubject`).
|
||
- Player self-load reads lower-case chars + `{total,current}` wounds (was broken).
|
||
- Enemy 0/again wounds coerced robustly (strings, 0, missing → usable tracker).
|
||
- `/api/bestiary/enemies` now returns the **full** bestiary (filter relaxed) and
|
||
`transformBestiaryEntry` coerces wounds + honours `stats.toughnessBonus`.
|
||
- Fire-mode +10/+20, jam 96/94/100, Advanced-untrained skill gate, Dodge/Parry
|
||
burst negation = DoS (earlier in session). Full CRA build compiles.
|
||
|
||
### Bestiary data
|
||
- `scripts/sanitize-bestiary-canonical.js` written and **run** against
|
||
`public/…json`. Backups written (`*.pre-sanitize.<ts>.json`).
|
||
- Result on the 34 public entries:
|
||
- **PDF-verified (9):** Chaos Space Marine, Chaos Heretic, Gun Drone, Hive Tyrant,
|
||
Hormagaunt, Tau Commander, Tau Fire Warrior, Tau Stealth Suit, Tyranid Warrior.
|
||
- **Canonical-fill, review (13):** Termagant, Renegade Militia, Imperial Guardsman,
|
||
Daemon Prince, Ork Nob, Battle/Monotask/Industrial Servitor, Servo-Skull,
|
||
Civilian, Chapter Serf, Imperial Agent, Imperial Guard Field/Senior Officer.
|
||
- **OCR-restored (5):** caele Rylus Darkscourge, Father Marius, Infected Auran
|
||
warrior, Feynir Surbryte, Arcus Dyre.
|
||
- **Structural-only, review (8):** the TEP named NPCs (Agamorr, Skarjaw, Vex,
|
||
Genestealer Patriarch, Zephon, Chaos Cultist Champion, Prince-Prefect Allewis,
|
||
Alexei Drahj — S/T bonus→char fixed).
|
||
- Each touched entry records `stats.statSource` ('pdf'|'canon'|'ocr'|'structural')
|
||
and `stats.needsReview`, and an effective `stats.toughnessBonus` (includes
|
||
Unnatural Toughness, e.g. Tyranid Warrior 10, Hive Tyrant 15, Genestealer 8).
|
||
|
||
## PDF-verified statblocks (authoritative, WS BS S T Ag Int Per WP Fel | W | armour)
|
||
```
|
||
Chaos Space Marine 50 50 65 45 40 35 40 43 17 | 29 | Power Armour H8/A8/B10/L8 CR363
|
||
Tyranid Warrior 55 30 60 50 44 20 35 50 – | 48 | Reinforced Chitin All8 CR370
|
||
Hive Tyrant 78 33 60 53 45 45 49 70 – | 120 | Bonded Exoskeleton All10 CR371
|
||
Hormagaunt 45 20 35 30 55 10 40 30 – | 9 | Chitinous Carapace All3 CR372
|
||
Tau Fire Warrior 25 35 30 30 25 30 20 30 30 | 12 | Fire Warrior Armour All6 CR369
|
||
Tau Stealth Suit 30 42 42 47 38 30 40 40 40 | 30 | XV15 H8/B8/A7/L7 CR368
|
||
Gun Drone 20 25 20 40 40 15 25 20 10 | 15 | (machine) CR368
|
||
Chaos Heretic 35 35 35 35 30 20 30 35 20 | 10 | Flak All4 CR361
|
||
Tau Commander 41 57 50 57 48 45 45 50 50 | 90 | XV-8 Crisis Suit All9 MoX
|
||
Genestealer 55 0 45 40 50 20 40 40 10 | 16 | Reinforced Chitin All4 MoX44
|
||
Apostle Bellephrades75 60 66 45 48 48 48 75 30 | 80 | Armour of Chaos All12 CR363
|
||
```
|
||
|
||
## DONE ✅ (continued)
|
||
- **Sanitizer now also ADDS missing canonical creatures** and mirrors public→DB
|
||
copy. Public + database JSON are both **38 entries** and identical. New entries
|
||
added: **Genestealer, Ork Nob, Apostle Bellephrades, Industrial Servitor**.
|
||
- **Bestiary UI redesign** (`src/components/BestiaryTab.jsx`): colour-coded stat
|
||
chips (WS/BS/S/T/Ag/Int/Per/WP/Fel each with its own colour), tier bar + badge
|
||
(Troops/Elite/Master by wounds), Wounds + TB pills, `verified`/`review` badges
|
||
from `statSource`/`needsReview`, and smart armour rendering (`{all:8}`→"All 8",
|
||
location objects → "Head 8 · Body 10 · …"). Build compiles clean.
|
||
- **Long-entry cleanup**: `scripts/sanitize-bestiary-canonical.js` now strips raw
|
||
`pageText`/`snippet` extraction text, removes copied root-level spillover from
|
||
canonical entries, and trims OCR/structural `skills`/`talents`/`traits`/`gear`
|
||
at section boundaries. Public + database JSON remain identical, 38 entries,
|
||
with no rendered text field over 300 chars.
|
||
- **40k RPG Tools index staging**: `scripts/import-40krpgtools-bestiary-index.js`
|
||
downloads the public master bestiary index from `https://www.40krpgtools.com/bestiary/`
|
||
and writes Deathwatch-only missing references to
|
||
`database/remote-extract/40krpgtools-bestiary-index.json`. This is an index
|
||
only (name/type/affiliation/book/page/source URL), not statblocks, so it does
|
||
not write directly into the live bestiary.
|
||
- **DB-backed live bestiary**: `database/mariadb.js` now creates the `bestiary`
|
||
table and exposes replace/upsert helpers. `scripts/import-bestiary-json-to-db.js`
|
||
imports the sanitized active bestiary into MariaDB. Current DB import: **84
|
||
rows**, **57 PDF-verified**, all 84 are roller-ready because `/api/bestiary/enemies`
|
||
reads the same `bestiary` table before falling back to JSON.
|
||
- **40k RPG Tools hydration**: `scripts/hydrate-bestiary-from-40krpgtools.js`
|
||
uses the staged index as a page map and local PDFs as the stat source. First
|
||
Mark of the Xenos pass added **46** compact statblocks to the active bestiary.
|
||
The remaining index-only rows stay out of the live combat table until a real
|
||
profile and wounds can be extracted.
|
||
- **First Founding pass**: hydrated **4** additional compact statblocks from the
|
||
local `FF.pdf`: The Headsman of Cellebos, Blight-Master Ussax, Atum Sum, and
|
||
Warsmith Koros. Re-imported MariaDB afterward; current live DB count is **88**
|
||
bestiary rows. Verified these four through `/api/bestiary/enemies` with valid
|
||
Wounds/TB/armour output for the combat roller.
|
||
|
||
## TODO / NEXT (in priority order)
|
||
1. **More NEW creatures**: the PDF scan found ~40 more clean profiles whose names
|
||
need pairing via per-page `-layout` extraction (Ork Boy, Broadside XV-88, Kroot
|
||
Carnivore, Bloodletters/greater daemons, more Tyranids). Extend the `CANON` table
|
||
in `scripts/sanitize-bestiary-canonical.js` (auto-adds if missing).
|
||
The 40k RPG Tools staging file currently has **344 Deathwatch index rows not
|
||
already in the active bestiary**; use it to prioritize local PDF page extraction.
|
||
Do not import index-only rows into the live `bestiary` table because live
|
||
bestiary rows automatically appear in the combat roller.
|
||
2. Re-verify canonical-fill entries against PDFs (Termagant, Ork Nob, Daemon Prince,
|
||
servitors, officers) to promote them from `needsReview` to `pdf` (verified).
|
||
3. **ComfyUI (192.168.1.96)** image generation for players/beasts — user asked;
|
||
feasible. Plan: backend `POST /api/imagegen` proxy → ComfyUI `/prompt` API with a
|
||
stored workflow JSON, poll `/history/{id}`, save PNG under `/avatars` or a new
|
||
`/beast-images`; UI "Generate portrait" buttons on PlayerTab + BestiaryTab.
|
||
NOT STARTED (needs the ComfyUI workflow JSON + confirmation the box is reachable).
|
||
|
||
## How to continue
|
||
- Re-run sanitizer: `node scripts/sanitize-bestiary-canonical.js` (idempotent;
|
||
writes fresh backups each run).
|
||
- Extract a creature page: `pdftotext -layout -f <page> -l <page> database/rules/CR.pdf -`
|
||
(or MoX.pdf). Profile = the 9-number row; Wounds/Armour/Weapons lines follow.
|
||
- The DoS/roller conventions and the jsdom-segfault test workarounds are in
|
||
project memory (`~/.claude/projects/-nas-git-dwroller/memory/`).
|
||
```
|
||
```
|