Files

Alex 4c1b19fb8c feat: Implement bestiary routes for data retrieval and management

- Added bestiaryRoutes.js to handle API endpoints for fetching enemies, full bestiary data, and statistics.
- Implemented caching mechanism for bestiary data with a 5-minute expiration.
- Created transformation functions for bestiary entries to standardize data format for the dice roller.
- Added admin-only endpoint to force reload bestiary data with GM authentication.

chore: Create scripts for analyzing and cleaning bestiary data

- Developed analyze-bestiary-quality.js to review data quality and identify duplicates or inconsistencies.
- Implemented clean-bestiary-fields.js to extract and clean specific fields from bestiary entries, reducing text duplication.
- Created clean-corrupted-bestiary.js to remove corrupted entries based on defined patterns.
- Generated a comprehensive database-cleanup-report.md summarizing the cleaning process and results.

build: Add fast build script for streamlined deployment

- Introduced fast-build.sh to facilitate quick builds and PM2 reloads without reinstalling dependencies.

test: Add GM authentication logic test

- Created test-gm-auth.js to verify the correctness of GM authentication logic with various test cases.

2025-08-17 18:15:39 +02:00

3.4 KiB

Raw Blame History

Database Table Size Review Report

Issue Analysis

After reviewing the database structure, I found that the main size issues were in the bestiary data (JSON file), not in SQL database tables. The problematic "tables" you mentioned were actually JSON fields within the bestiary entries:

Problematic Fields in Bestiary Data:

armour - massive text duplication (1775+ chars per entry)
skills - extensive repeated content (2037+ chars per entry)
talents - redundant text blocks (1924+ chars per entry)
traits - duplicated descriptions (1826+ chars per entry)
weapons - oversized weapon data (1718+ chars per entry)
gear - bloated gear descriptions (1615+ chars per entry)
description - contained in various fields with duplication

Root Cause

The bestiary extraction process from PDF files resulted in massive text duplication where each field contained:

The actual relevant data (100-200 characters)
Huge chunks of repeated PDF page content (1500+ characters)

Solution Implemented

Created and executed /scripts/clean-bestiary-fields.js which:

Cleaning Process:

Backup Creation: Automatically backed up original data
Pattern Recognition: Identified corruption markers like "361 X I I I : A d v e r s a r i e s"
Field Extraction: Used regex patterns to extract only relevant data from each field
Data Validation: Preserved essential game statistics while removing duplication

Results:

✅ File Size Reduction: 331KB → 224KB (32% compression)
✅ Storage Savings: 107KB total reduction
✅ Data Quality: Clean, structured fields with only relevant information
✅ Cleaned Entries: 11 out of 24 entries had significant improvements

Before vs After Examples:

Skills Field:

Before: 2037 characters with massive PDF text duplication
After: 102 characters - "Awareness (Per), Climb (S), Speak Language (Low Gothic, Unholy Tongue) (Int), Swim (S), Survival (Int)"

Armour Field:

Before: 1775 characters with repeated content
After: 153 characters - "Flak Robes and Brazen Carapace Armour (Horde 5)"

Database Tables Review:

SQLite Database (`sqlite/deathwatch.db` - 204KB):

✅ players table: Appropriately sized
✅ sessions table: Normal size
✅ shop_items table: 123 items with reasonable stats field sizes (<200 chars each)
✅ player_inventory table: Efficient relational structure
✅ transactions table: Clean transaction logs

No SQL table size issues found

Performance Impact:

API Response Speed: Faster bestiary data loading
Memory Usage: Reduced memory footprint for cached data
Transfer Size: Smaller payloads for web requests
Storage Efficiency: More compact database files

Deployment Status:

✅ Cleaned data deployed to production
✅ API cache reloaded with clean data
✅ PM2 server reloaded successfully
✅ Build system updated with optimized data

Recommendations:

Future PDF Extraction: Improve extraction scripts to prevent text duplication
Data Validation: Add automated checks for oversized fields during import
Regular Cleanup: Run field cleaning script periodically during data updates
Monitoring: Track field sizes during bestiary updates

The "large tables" issue has been resolved through data optimization rather than database restructuring, maintaining full functionality while significantly improving efficiency.

3.4 KiB Raw Blame History