openclaw/tools/SELF-HEALING.md

Self-Healing Agent System

Inspired by: Reddit - Self-healing feedback loop between Claude Code and OpenClaw

Overview

Three-tier self-healing system that automatically detects, diagnoses, and fixes agent task failures:

1. 🔍 Error Daemon (Continuous Monitoring)

Script: tools/jarvis-error-daemon.sh

• Function: Continuously monitors all agent sessions for errors
• Interval: Checks every 30 seconds
• Max Retries: 3 attempts per task
• Healing Agent: Claude Sonnet 4.5 (for troubleshooting)

How it works:

1. Scans ~/.openclaw/agents/free/sessions/*.jsonl for error patterns
2. Extracts error bundle (task context + error details)
3. Spawns healing agent (Sonnet) to diagnose root cause
4. Applies fixes if possible
5. Retries task if healing recommends it

Usage:

# Start daemon (continuous monitoring)
./tools/jarvis-error-daemon.sh monitor

# Test error extraction
./tools/jarvis-error-daemon.sh test /path/to/session.jsonl

Logs:

• Errors: /home/alex/clawd/logs/jarvis-errors.log
• Healing: /home/alex/clawd/logs/jarvis-healing.log

2. 🛠️ Healing Wrapper (Manual Delegation)

Script: tools/delegate-with-healing.sh

• Function: Wrap manual task delegation with automatic retry + healing
• Max Retries: 3 attempts
• Healing Agent: Claude Sonnet 4.5
• Timeout: Configurable (default: 30 min)

How it works:

1. Fetch task from Kanban API
2. Execute with specified agent
3. Monitor for failures (exit code, error patterns)
4. On failure: spawn Sonnet for diagnosis
5. Apply fixes and retry
6. Report success/failure after max retries

Usage:

# Delegate task 335 (auto-detect agent)
./tools/delegate-with-healing.sh 335

# Use specific agent
./tools/delegate-with-healing.sh 335 gemini-flash

# Custom timeout (1 hour = 3600s)
./tools/delegate-with-healing.sh 335 gemini-flash 3600

Logs:

• /home/alex/clawd/logs/healing-wrapper.log

3. 🤖 Jarvis Integration (Built-in Auto-Healing)

Script: tools/jarvis-delegate.sh (enhanced)

• Function: Built-in self-healing in automated delegation
• Triggered by: Jarvis auto-delegation cron (every 30 min)
• Max Retries: 3 attempts per task
• Healing Agent: Claude Sonnet 4.5

How it works:

1. Jarvis scans Kanban for agent:* tagged tasks
2. Delegates to appropriate agent (gemini/nvidia/deepseek)
3. Monitors execution (exit code + completion patterns)
4. On failure: spawns Sonnet for diagnosis
5. Applies fixes and retries automatically
6. Returns task to queue if max retries exceeded

Features:

• Automatic task status updates (queue → in_progress → done)
• Heuristic success detection (keywords: completed, done, success, ✅)
• Failure analysis with context (last 100 lines of output)
• Concurrency limit (max 3 tasks simultaneously)

No manual intervention required - runs autonomously.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     SELF-HEALING SYSTEM                     │
└─────────────────────────────────────────────────────────────┘

┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  Error Daemon    │    │ Healing Wrapper  │    │ Jarvis Delegation│
│  (Continuous)    │    │  (Manual)        │    │  (Automated)     │
├──────────────────┤    ├──────────────────┤    ├──────────────────┤
│ Monitor sessions │    │ Delegate + retry │    │ Auto-delegate    │
│ Extract errors   │    │ Heal failures    │    │ + healing loop   │
│ Spawn healer     │    │ Apply fixes      │    │ Built-in retry   │
│ Retry tasks      │    │ Report status    │    │ Queue on fail    │
└──────────────────┘    └──────────────────┘    └──────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 ▼
                     ┌─────────────────────┐
                     │  Healing Agent      │
                     │  (Claude Sonnet)    │
                     ├─────────────────────┤
                     │ 1. Diagnose error   │
                     │ 2. Propose fix      │
                     │ 3. Apply changes    │
                     │ 4. Recommend retry  │
                     └─────────────────────┘

Healing Agent Responsibilities

Claude Sonnet 4.5 acts as the troubleshooting expert:

1. Diagnose: Analyze error bundle to identify root cause
2. Fix: Apply specific changes:
   • Edit code/config files
   • Install missing dependencies
   • Fix syntax errors
   • Adjust task parameters
   • Recommend different agent/model
3. Decide: Recommend whether to retry or stop

Cost Optimization

Inspired by Reddit:

"My token cost of training has gone down by about 95%, since Claude Code ($200/month) does more of the troubleshooting for OpenClaw"

Our approach:

• Worker agents: Gemini Flash, Nvidia (cheap/fast execution)
• Healing agent: Claude Sonnet ($200/month fixed in Claude Code)
• Savings: Healing only on failures (~5-10% of tasks)

Cost comparison:

• Before: All tasks use Sonnet → expensive
• After: Gemini/Nvidia for work, Sonnet for healing → 95% cheaper

Integration with Jarvis

Self-healing is integrated into Jarvis automation:

Cron jobs:

• Health monitor: Hourly (includes error daemon status)
• Auto-delegation: Every 30 min (built-in healing)
• GitHub scanner: Daily 16:00

Healing triggers:

• Automatic: Jarvis delegation failures
• Manual: delegate-with-healing.sh wrapper
• Continuous: Error daemon monitoring

Testing

Test healing on failed task:

# Manual delegation with healing
./tools/delegate-with-healing.sh 335 gemini-flash

# Watch healing logs
tail -f /home/alex/clawd/logs/healing-wrapper.log

Test error daemon:

# Extract error bundle from session
./tools/jarvis-error-daemon.sh test ~/.openclaw/agents/free/sessions/some-session.jsonl

Test Jarvis integration:

# Trigger auto-delegation manually
./tools/jarvis-delegate.sh

# Watch delegation + healing logs
tail -f /home/alex/clawd/logs/jarvis-delegation.log

Monitoring

Health check:

./tools/jarvis-status.sh

Logs:

• Error daemon: /home/alex/clawd/logs/jarvis-errors.log
• Healing wrapper: /home/alex/clawd/logs/healing-wrapper.log
• Jarvis delegation: /home/alex/clawd/logs/jarvis-delegation.log

Key metrics:

• Tasks attempted
• Tasks failed
• Healing triggered
• Retries successful
• Final success rate

Roadmap

• Error daemon as systemd service (auto-start)
• Healing success metrics dashboard
• Learning: Store successful fixes in knowledge base
• Proactive healing: Predict failures before they happen
• Multi-agent healing: Parallel diagnosis by different models

Credits

Inspired by Reddit user's self-healing loop between Claude Code and OpenClaw for browser automation training.

See Also

• JARVIS.md - Main Jarvis documentation
• tools/jarvis-delegate.sh - Auto-delegation with healing
• tools/delegate-with-healing.sh - Manual wrapper
• tools/jarvis-error-daemon.sh - Continuous monitoring