tilbudgivern/docs/api/BYGMA_AUTOMATION_README.md

Bygma Installation Manuals - Auto Scraper

Overview

Automated system for downloading, analyzing, and storing installation manuals from Bygma.

Features

🤖 Automated Scraping

• Login: Automatically logs into Bygma /proff/ section
• Smart Filtering: Only downloads PDFs with "montage" in the filename
• Download Management: Downloads PDFs one at a time to temp location

🧠 AI Analysis

• GPT-4o Integration: Analyzes each PDF with OpenAI GPT-4o
• Data Extraction:
  • Product name and manufacturer
  • All installation steps (Danish)
  • Time estimates per unit (m², pcs, etc.)
  • Required tools
  • Safety requirements
  • Material requirements
  • Skill level (easy/medium/hard)
  • Weather conditions
  • Key points

💾 Database Storage

• Automatic Import: Saves analysis directly to installation_manuals table
• JSON Fields: Structured data for installation steps, tools, materials
• Query Ready: Data immediately available for quote generation

🗑️ Space Efficient

• Auto Cleanup: Deletes PDFs after analysis to save disk space
• Temp Storage: Uses /temp_pdfs/ directory that's cleaned up automatically

Usage

Run Scraper

cd /mnt/HC_Volume_103713257/tilbudgivern
python3 scrape_and_analyze_bygma.py

Configuration

Edit credentials in scrape_and_analyze_bygma.py:

BYGMA_EMAIL = "your-email@example.com"
BYGMA_PASSWORD = "your-password"
PRODUCT_URL = "https://www.bygma.dk/proff/..."

Output Example

🔍 Bygma Montage PDF Scraper + AI Analyzer
1️⃣ Logging in to Bygma...
   ✅ Logged in
2️⃣ Navigating to product page...
   ✅ At product page
3️⃣ Finding montage PDFs...
   ✅ Found 7 montage PDFs
4️⃣ Processing PDFs...
   📄 [1/7] 280_Montagevejledning_Boelgepladetag.pdf
      ⬇️ Downloading...
      ✅ Downloaded (2.5 MB)
      📖 Extracting text...
      ✅ Extracted 38,643 characters
      🤖 Analyzing with GPT-4o...
      ✅ Found 15 installation steps
      💾 Saving to database...
      ✅ Saved to database (ID: 1)
      🗑️ PDF deleted (saved space)

Database Schema

CREATE TABLE installation_manuals (
    id INT PRIMARY KEY AUTO_INCREMENT,
    product_name VARCHAR(255),
    manufacturer VARCHAR(255),
    manual_url TEXT,
    manual_filename VARCHAR(255),
    manual_type ENUM('montage', 'vedligeholdelse', 'datablad'),
    installation_steps JSON,
    time_estimate_per_unit DECIMAL(5,2),
    time_unit ENUM('per_sqm', 'per_unit', 'per_meter', 'per_hour'),
    required_tools JSON,
    safety_requirements JSON,
    material_requirements JSON,
    skill_level ENUM('let', 'medium', 'svær'),
    weather_conditions TEXT,
    key_points JSON,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);

Integration with Quotes

Node.js Example

const { getInstallationTask } = require('./get_installation_tasks');

// Get installation data for a material
const task = await getInstallationTask('Cembrit Bølgeplader', 50); // 50 m²

console.log(task.formatted_description);
// Output:
// OPGAVE: Installation af Cembrit Bølgeplader (50 m²)
// 
// Monteringstrin:
// 1. Forberedelse: Kontroller underlag...
// 2. Montering af bærelægter...
// ...
// 
// Estimeret tid: 125 timer (16 arbejdsdage)
// Værktøj: Savskære, Skruemaskine, ...
// Sikkerhed: Personligt sikkerhedsudstyr påkrævet...

Files

• scrape_and_analyze_bygma.py - Main scraper with AI analysis
• scrape_bygma_with_login.py - Simple login + download (without AI)
• analyze_manual_pdfs.py - Standalone PDF analyzer
• get_installation_tasks.js - Node.js integration for quotes
• BYGMA_SCRAPER_STATUS.md - Detailed technical notes

Dependencies

pip install playwright openai PyPDF2 mysql-connector-python
playwright install chromium

Notes

• Rate Limiting: Adds delays between downloads to avoid overwhelming server
• Error Handling: Continues processing if one PDF fails
• Encoding: Handles Danish characters (æ, ø, å) correctly
• Authentication: Stores session cookies for faster subsequent requests

Future Enhancements

• Support multiple products in one run
• Deduplicate PDFs by content hash
• Add progress bar for long-running operations
• Email notification when scraping completes
• Webhook integration for real-time updates