Alex 585584d6c2 Add comprehensive rulebook scraper for all Deathwatch PDFs ...

Extracts from CR, FF, RoB, MoX, HtC:
- 36+ weapons (ranged + melee) from Core Rulebook
- 37 talents, 16 traits, 11 weapon qualities
- 89 psychic powers
- 99+ bestiary entries from Mark of the Xenos
- 1100+ lore/chapter entries across all books
- 123 speciality entries
- Total: 3415 rules, 228 bestiary entries in DB

Uses PyMuPDF (fitz) for text blocks and pdfplumber for table extraction.
PDFs are stored on NFS and not in git.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-03-01 19:37:12 +01:00

..

analyze-bestiary-quality.js

feat: Implement bestiary routes for data retrieval and management

2025-08-17 18:15:39 +02:00

build-shop-db.js

Add migration script to import JSON data into SQLite database

2025-08-17 19:14:29 +02:00

check-shop-stats.js

Add migration script to import JSON data into SQLite database

2025-08-17 19:14:29 +02:00

choose-and-parse-alexei-pdfjs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

clean-armoury.js

fix1

2025-08-15 16:23:34 +02:00

clean-bestiary-fields.js

feat: Implement bestiary routes for data retrieval and management

2025-08-17 18:15:39 +02:00

clean-corrupted-bestiary.js

feat: Implement bestiary routes for data retrieval and management

2025-08-17 18:15:39 +02:00

clean-enemies-raw.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

cleanup-skill-dupes.py

Fix rules search ranking and import clean skills/talents from Core Rulebook

2026-03-01 18:25:08 +01:00

comprehensive-armor.js

Add comprehensive weapons dataset and scraper for 40k RPG Tools

2025-08-15 23:36:06 +02:00

comprehensive-weapons.js

Add comprehensive weapons dataset and scraper for 40k RPG Tools

2025-08-15 23:36:06 +02:00

copy-bestiary-live.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

copy-bestiary-to-public.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

database-cleanup-report.md

feat: Implement bestiary routes for data retrieval and management

2025-08-17 18:15:39 +02:00

dump-pdf-pages.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

extract-alexei-pdfjs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

extract-armoury.js

new version

2025-08-15 10:35:27 +02:00

extract-enemies-from-pdfs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

extract-exact-alexei-from-pdfjs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

extract-pages-for-name.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

extract-rules.js

feat: Add validation and normalization utilities for player records

2025-08-16 12:15:32 +02:00

fast-build.sh

feat: Implement bestiary routes for data retrieval and management

2025-08-17 18:15:39 +02:00

fetch-armoury.js

fix1

2025-08-15 16:23:34 +02:00

fetch-enemies.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

fill-bestiary-placeholders.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

fix-talent-titles.py

Fix rules search ranking and import clean skills/talents from Core Rulebook

2026-03-01 18:25:08 +01:00

harvest-bestiary-from-pdfs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

import-api-search-statblocks.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

import-bestiary-deathwatch.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

import-fandom-skills.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

import-remote-unmatched.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

import-rules-from-ocr-text-refined.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

import-rules-from-ocr-text.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

import-skills-from-pdf-whitelist.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

import-skills-from-pdf.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

import-verified-to-rules.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

local-ci.sh

feat: enhance build process and testing workflow

2025-08-16 15:14:28 +02:00

match-bestiary-to-pdfs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

merge-npcs-from-api.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

merge-validate-upsert-npcs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

move-staging-to-rules.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

normalize-and-merge-bestiary.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

normalize-bestiary.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

ocr-and-rebuild-rules.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

openai-client.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

openai-pdf-validate.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

parse-alexei-pdfjs-results.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

parse-alexei.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

parse-armor.js

fix1

2025-08-15 16:23:34 +02:00

parse-weapons.js

fix1

2025-08-15 16:23:34 +02:00

prune-bestiary.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

prune-rules-by-pdf.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

rebuild-rules-from-skills.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

refine-enemies.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

replace-skills-from-csv.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

sanitize-rules-openai.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

scrape-40k-tools.js

Add comprehensive weapons dataset and scraper for 40k RPG Tools

2025-08-15 23:36:06 +02:00

scrape-rulebooks.py

Add comprehensive rulebook scraper for all Deathwatch PDFs

2026-03-01 19:37:12 +01:00

search-keywords.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

search-pdfs-for-name.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

show-bestiary-entry.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

smoke-openai-charm.js

Refactor code structure for improved readability and maintainability

2025-08-20 15:29:22 +02:00

structure-enemies.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

update-rules-from-sources.py

Fix rules search ranking and import clean skills/talents from Core Rulebook

2026-03-01 18:25:08 +01:00

upsert-alexei-canonical.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

upsert-allewis-from-5001.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

upsert-missing-movement-from-5001.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

upsert-missing-movement-from-pdfs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

upsert-missing-profiles-from-pdfs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

upsert-missing-wounds-from-pdfs.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

upsert-remote-statblocks.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00

validate-bestiary.js

feat: Add scripts for upserting and validating bestiary data

2025-08-17 16:21:00 +02:00