149 lines
4.3 KiB
Markdown
149 lines
4.3 KiB
Markdown
# Bygma Installation Manuals - Auto Scraper
|
|
|
|
## Overview
|
|
Automated system for downloading, analyzing, and storing installation manuals from Bygma.
|
|
|
|
## Features
|
|
|
|
### 🤖 Automated Scraping
|
|
- **Login:** Automatically logs into Bygma /proff/ section
|
|
- **Smart Filtering:** Only downloads PDFs with "montage" in the filename
|
|
- **Download Management:** Downloads PDFs one at a time to temp location
|
|
|
|
### 🧠 AI Analysis
|
|
- **GPT-4o Integration:** Analyzes each PDF with OpenAI GPT-4o
|
|
- **Data Extraction:**
|
|
- Product name and manufacturer
|
|
- All installation steps (Danish)
|
|
- Time estimates per unit (m², pcs, etc.)
|
|
- Required tools
|
|
- Safety requirements
|
|
- Material requirements
|
|
- Skill level (easy/medium/hard)
|
|
- Weather conditions
|
|
- Key points
|
|
|
|
### 💾 Database Storage
|
|
- **Automatic Import:** Saves analysis directly to `installation_manuals` table
|
|
- **JSON Fields:** Structured data for installation steps, tools, materials
|
|
- **Query Ready:** Data immediately available for quote generation
|
|
|
|
### 🗑️ Space Efficient
|
|
- **Auto Cleanup:** Deletes PDFs after analysis to save disk space
|
|
- **Temp Storage:** Uses `/temp_pdfs/` directory that's cleaned up automatically
|
|
|
|
## Usage
|
|
|
|
### Run Scraper
|
|
```bash
|
|
cd /mnt/HC_Volume_103713257/tilbudgivern
|
|
python3 scrape_and_analyze_bygma.py
|
|
```
|
|
|
|
### Configuration
|
|
Edit credentials in `scrape_and_analyze_bygma.py`:
|
|
```python
|
|
BYGMA_EMAIL = "[email protected]"
|
|
BYGMA_PASSWORD = "your-password"
|
|
PRODUCT_URL = "https://www.bygma.dk/proff/..."
|
|
```
|
|
|
|
### Output Example
|
|
```
|
|
🔍 Bygma Montage PDF Scraper + AI Analyzer
|
|
1️⃣ Logging in to Bygma...
|
|
✅ Logged in
|
|
2️⃣ Navigating to product page...
|
|
✅ At product page
|
|
3️⃣ Finding montage PDFs...
|
|
✅ Found 7 montage PDFs
|
|
4️⃣ Processing PDFs...
|
|
📄 [1/7] 280_Montagevejledning_Boelgepladetag.pdf
|
|
⬇️ Downloading...
|
|
✅ Downloaded (2.5 MB)
|
|
📖 Extracting text...
|
|
✅ Extracted 38,643 characters
|
|
🤖 Analyzing with GPT-4o...
|
|
✅ Found 15 installation steps
|
|
💾 Saving to database...
|
|
✅ Saved to database (ID: 1)
|
|
🗑️ PDF deleted (saved space)
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
```sql
|
|
CREATE TABLE installation_manuals (
|
|
id INT PRIMARY KEY AUTO_INCREMENT,
|
|
product_name VARCHAR(255),
|
|
manufacturer VARCHAR(255),
|
|
manual_url TEXT,
|
|
manual_filename VARCHAR(255),
|
|
manual_type ENUM('montage', 'vedligeholdelse', 'datablad'),
|
|
installation_steps JSON,
|
|
time_estimate_per_unit DECIMAL(5,2),
|
|
time_unit ENUM('per_sqm', 'per_unit', 'per_meter', 'per_hour'),
|
|
required_tools JSON,
|
|
safety_requirements JSON,
|
|
material_requirements JSON,
|
|
skill_level ENUM('let', 'medium', 'svær'),
|
|
weather_conditions TEXT,
|
|
key_points JSON,
|
|
created_at TIMESTAMP,
|
|
updated_at TIMESTAMP
|
|
);
|
|
```
|
|
|
|
## Integration with Quotes
|
|
|
|
### Node.js Example
|
|
```javascript
|
|
const { getInstallationTask } = require('./get_installation_tasks');
|
|
|
|
// Get installation data for a material
|
|
const task = await getInstallationTask('Cembrit Bølgeplader', 50); // 50 m²
|
|
|
|
console.log(task.formatted_description);
|
|
// Output:
|
|
// OPGAVE: Installation af Cembrit Bølgeplader (50 m²)
|
|
//
|
|
// Monteringstrin:
|
|
// 1. Forberedelse: Kontroller underlag...
|
|
// 2. Montering af bærelægter...
|
|
// ...
|
|
//
|
|
// Estimeret tid: 125 timer (16 arbejdsdage)
|
|
// Værktøj: Savskære, Skruemaskine, ...
|
|
// Sikkerhed: Personligt sikkerhedsudstyr påkrævet...
|
|
```
|
|
|
|
## Files
|
|
|
|
- `scrape_and_analyze_bygma.py` - Main scraper with AI analysis
|
|
- `scrape_bygma_with_login.py` - Simple login + download (without AI)
|
|
- `analyze_manual_pdfs.py` - Standalone PDF analyzer
|
|
- `get_installation_tasks.js` - Node.js integration for quotes
|
|
- `BYGMA_SCRAPER_STATUS.md` - Detailed technical notes
|
|
|
|
## Dependencies
|
|
|
|
```bash
|
|
pip install playwright openai PyPDF2 mysql-connector-python
|
|
playwright install chromium
|
|
```
|
|
|
|
## Notes
|
|
|
|
- **Rate Limiting:** Adds delays between downloads to avoid overwhelming server
|
|
- **Error Handling:** Continues processing if one PDF fails
|
|
- **Encoding:** Handles Danish characters (æ, ø, å) correctly
|
|
- **Authentication:** Stores session cookies for faster subsequent requests
|
|
|
|
## Future Enhancements
|
|
|
|
- [ ] Support multiple products in one run
|
|
- [ ] Deduplicate PDFs by content hash
|
|
- [ ] Add progress bar for long-running operations
|
|
- [ ] Email notification when scraping completes
|
|
- [ ] Webhook integration for real-time updates
|