# Bygma Installation Manuals - Auto Scraper

## Overview
Automated system for downloading, analyzing, and storing installation manuals from Bygma.

## Features

### 🤖 Automated Scraping
- **Login:** Automatically logs into Bygma /proff/ section
- **Smart Filtering:** Only downloads PDFs with "montage" in the filename
- **Download Management:** Downloads PDFs one at a time to temp location

### 🧠 AI Analysis
- **GPT-4o Integration:** Analyzes each PDF with OpenAI GPT-4o
- **Data Extraction:**
  - Product name and manufacturer
  - All installation steps (Danish)
  - Time estimates per unit (m², pcs, etc.)
  - Required tools
  - Safety requirements
  - Material requirements
  - Skill level (easy/medium/hard)
  - Weather conditions
  - Key points

### 💾 Database Storage
- **Automatic Import:** Saves analysis directly to `installation_manuals` table
- **JSON Fields:** Structured data for installation steps, tools, materials
- **Query Ready:** Data immediately available for quote generation

### 🗑️ Space Efficient
- **Auto Cleanup:** Deletes PDFs after analysis to save disk space
- **Temp Storage:** Uses `/temp_pdfs/` directory that's cleaned up automatically

## Usage

### Run Scraper
```bash
cd /mnt/HC_Volume_103713257/tilbudgivern
python3 scrape_and_analyze_bygma.py
```

### Configuration
Edit credentials in `scrape_and_analyze_bygma.py`:
```python
BYGMA_EMAIL = "your-email@example.com"
BYGMA_PASSWORD = "your-password"
PRODUCT_URL = "https://www.bygma.dk/proff/..."
```

### Output Example
```
🔍 Bygma Montage PDF Scraper + AI Analyzer
1️⃣ Logging in to Bygma...
   ✅ Logged in
2️⃣ Navigating to product page...
   ✅ At product page
3️⃣ Finding montage PDFs...
   ✅ Found 7 montage PDFs
4️⃣ Processing PDFs...
   📄 [1/7] 280_Montagevejledning_Boelgepladetag.pdf
      ⬇️ Downloading...
      ✅ Downloaded (2.5 MB)
      📖 Extracting text...
      ✅ Extracted 38,643 characters
      🤖 Analyzing with GPT-4o...
      ✅ Found 15 installation steps
      💾 Saving to database...
      ✅ Saved to database (ID: 1)
      🗑️ PDF deleted (saved space)
```

## Database Schema

```sql
CREATE TABLE installation_manuals (
    id INT PRIMARY KEY AUTO_INCREMENT,
    product_name VARCHAR(255),
    manufacturer VARCHAR(255),
    manual_url TEXT,
    manual_filename VARCHAR(255),
    manual_type ENUM('montage', 'vedligeholdelse', 'datablad'),
    installation_steps JSON,
    time_estimate_per_unit DECIMAL(5,2),
    time_unit ENUM('per_sqm', 'per_unit', 'per_meter', 'per_hour'),
    required_tools JSON,
    safety_requirements JSON,
    material_requirements JSON,
    skill_level ENUM('let', 'medium', 'svær'),
    weather_conditions TEXT,
    key_points JSON,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
);
```

## Integration with Quotes

### Node.js Example
```javascript
const { getInstallationTask } = require('./get_installation_tasks');

// Get installation data for a material
const task = await getInstallationTask('Cembrit Bølgeplader', 50); // 50 m²

console.log(task.formatted_description);
// Output:
// OPGAVE: Installation af Cembrit Bølgeplader (50 m²)
// 
// Monteringstrin:
// 1. Forberedelse: Kontroller underlag...
// 2. Montering af bærelægter...
// ...
// 
// Estimeret tid: 125 timer (16 arbejdsdage)
// Værktøj: Savskære, Skruemaskine, ...
// Sikkerhed: Personligt sikkerhedsudstyr påkrævet...
```

## Files

- `scrape_and_analyze_bygma.py` - Main scraper with AI analysis
- `scrape_bygma_with_login.py` - Simple login + download (without AI)
- `analyze_manual_pdfs.py` - Standalone PDF analyzer
- `get_installation_tasks.js` - Node.js integration for quotes
- `BYGMA_SCRAPER_STATUS.md` - Detailed technical notes

## Dependencies

```bash
pip install playwright openai PyPDF2 mysql-connector-python
playwright install chromium
```

## Notes

- **Rate Limiting:** Adds delays between downloads to avoid overwhelming server
- **Error Handling:** Continues processing if one PDF fails
- **Encoding:** Handles Danish characters (æ, ø, å) correctly
- **Authentication:** Stores session cookies for faster subsequent requests

## Future Enhancements

- [ ] Support multiple products in one run
- [ ] Deduplicate PDFs by content hash
- [ ] Add progress bar for long-running operations
- [ ] Email notification when scraping completes
- [ ] Webhook integration for real-time updates