# Self-Healing Agent System

**Inspired by:** [Reddit - Self-healing feedback loop between Claude Code and OpenClaw](https://www.reddit.com/r/openclaw/comments/1r5t6e9/watching_claude_code_train_my_agent_has_been/)

## Overview

Three-tier self-healing system that automatically detects, diagnoses, and fixes agent task failures:

### 1. 🔍 Error Daemon (Continuous Monitoring)

**Script:** `tools/jarvis-error-daemon.sh`

- **Function:** Continuously monitors all agent sessions for errors
- **Interval:** Checks every 30 seconds
- **Max Retries:** 3 attempts per task
- **Healing Agent:** Claude Sonnet 4.5 (for troubleshooting)

**How it works:**

1. Scans `~/.openclaw/agents/free/sessions/*.jsonl` for error patterns
2. Extracts error bundle (task context + error details)
3. Spawns healing agent (Sonnet) to diagnose root cause
4. Applies fixes if possible
5. Retries task if healing recommends it

**Usage:**

```bash
# Start daemon (continuous monitoring)
./tools/jarvis-error-daemon.sh monitor

# Test error extraction
./tools/jarvis-error-daemon.sh test /path/to/session.jsonl
```

**Logs:**

- Errors: `/home/alex/clawd/logs/jarvis-errors.log`
- Healing: `/home/alex/clawd/logs/jarvis-healing.log`

### 2. 🛠️ Healing Wrapper (Manual Delegation)

**Script:** `tools/delegate-with-healing.sh`

- **Function:** Wrap manual task delegation with automatic retry + healing
- **Max Retries:** 3 attempts
- **Healing Agent:** Claude Sonnet 4.5
- **Timeout:** Configurable (default: 30 min)

**How it works:**

1. Fetch task from Kanban API
2. Execute with specified agent
3. Monitor for failures (exit code, error patterns)
4. On failure: spawn Sonnet for diagnosis
5. Apply fixes and retry
6. Report success/failure after max retries

**Usage:**

```bash
# Delegate task 335 (auto-detect agent)
./tools/delegate-with-healing.sh 335

# Use specific agent
./tools/delegate-with-healing.sh 335 gemini-flash

# Custom timeout (1 hour = 3600s)
./tools/delegate-with-healing.sh 335 gemini-flash 3600
```

**Logs:**

- `/home/alex/clawd/logs/healing-wrapper.log`

### 3. 🤖 Jarvis Integration (Built-in Auto-Healing)

**Script:** `tools/jarvis-delegate.sh` (enhanced)

- **Function:** Built-in self-healing in automated delegation
- **Triggered by:** Jarvis auto-delegation cron (every 30 min)
- **Max Retries:** 3 attempts per task
- **Healing Agent:** Claude Sonnet 4.5

**How it works:**

1. Jarvis scans Kanban for `agent:*` tagged tasks
2. Delegates to appropriate agent (gemini/nvidia/deepseek)
3. Monitors execution (exit code + completion patterns)
4. On failure: spawns Sonnet for diagnosis
5. Applies fixes and retries automatically
6. Returns task to queue if max retries exceeded

**Features:**

- Automatic task status updates (queue → in_progress → done)
- Heuristic success detection (keywords: completed, done, success, ✅)
- Failure analysis with context (last 100 lines of output)
- Concurrency limit (max 3 tasks simultaneously)

**No manual intervention required** - runs autonomously.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     SELF-HEALING SYSTEM                     │
└─────────────────────────────────────────────────────────────┘

┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
│  Error Daemon    │    │ Healing Wrapper  │    │ Jarvis Delegation│
│  (Continuous)    │    │  (Manual)        │    │  (Automated)     │
├──────────────────┤    ├──────────────────┤    ├──────────────────┤
│ Monitor sessions │    │ Delegate + retry │    │ Auto-delegate    │
│ Extract errors   │    │ Heal failures    │    │ + healing loop   │
│ Spawn healer     │    │ Apply fixes      │    │ Built-in retry   │
│ Retry tasks      │    │ Report status    │    │ Queue on fail    │
└──────────────────┘    └──────────────────┘    └──────────────────┘
         │                       │                       │
         └───────────────────────┼───────────────────────┘
                                 ▼
                     ┌─────────────────────┐
                     │  Healing Agent      │
                     │  (Claude Sonnet)    │
                     ├─────────────────────┤
                     │ 1. Diagnose error   │
                     │ 2. Propose fix      │
                     │ 3. Apply changes    │
                     │ 4. Recommend retry  │
                     └─────────────────────┘
```

## Healing Agent Responsibilities

**Claude Sonnet 4.5** acts as the troubleshooting expert:

1. **Diagnose:** Analyze error bundle to identify root cause
2. **Fix:** Apply specific changes:
   - Edit code/config files
   - Install missing dependencies
   - Fix syntax errors
   - Adjust task parameters
   - Recommend different agent/model
3. **Decide:** Recommend whether to retry or stop

## Cost Optimization

**Inspired by Reddit:**

> "My token cost of training has gone down by about 95%, since Claude Code ($200/month) does more of the troubleshooting for OpenClaw"

**Our approach:**

- **Worker agents:** Gemini Flash, Nvidia (cheap/fast execution)
- **Healing agent:** Claude Sonnet ($200/month fixed in Claude Code)
- **Savings:** Healing only on failures (~5-10% of tasks)

**Cost comparison:**

- Before: All tasks use Sonnet → expensive
- After: Gemini/Nvidia for work, Sonnet for healing → 95% cheaper

## Integration with Jarvis

Self-healing is integrated into Jarvis automation:

**Cron jobs:**

- Health monitor: Hourly (includes error daemon status)
- Auto-delegation: Every 30 min (built-in healing)
- GitHub scanner: Daily 16:00

**Healing triggers:**

- Automatic: Jarvis delegation failures
- Manual: `delegate-with-healing.sh` wrapper
- Continuous: Error daemon monitoring

## Testing

**Test healing on failed task:**

```bash
# Manual delegation with healing
./tools/delegate-with-healing.sh 335 gemini-flash

# Watch healing logs
tail -f /home/alex/clawd/logs/healing-wrapper.log
```

**Test error daemon:**

```bash
# Extract error bundle from session
./tools/jarvis-error-daemon.sh test ~/.openclaw/agents/free/sessions/some-session.jsonl
```

**Test Jarvis integration:**

```bash
# Trigger auto-delegation manually
./tools/jarvis-delegate.sh

# Watch delegation + healing logs
tail -f /home/alex/clawd/logs/jarvis-delegation.log
```

## Monitoring

**Health check:**

```bash
./tools/jarvis-status.sh
```

**Logs:**

- Error daemon: `/home/alex/clawd/logs/jarvis-errors.log`
- Healing wrapper: `/home/alex/clawd/logs/healing-wrapper.log`
- Jarvis delegation: `/home/alex/clawd/logs/jarvis-delegation.log`

**Key metrics:**

- Tasks attempted
- Tasks failed
- Healing triggered
- Retries successful
- Final success rate

## Roadmap

- [ ] Error daemon as systemd service (auto-start)
- [ ] Healing success metrics dashboard
- [ ] Learning: Store successful fixes in knowledge base
- [ ] Proactive healing: Predict failures before they happen
- [ ] Multi-agent healing: Parallel diagnosis by different models

## Credits

Inspired by [Reddit user's self-healing loop](https://www.reddit.com/r/openclaw/comments/1r5t6e9/) between Claude Code and OpenClaw for browser automation training.

## See Also

- `JARVIS.md` - Main Jarvis documentation
- `tools/jarvis-delegate.sh` - Auto-delegation with healing
- `tools/delegate-with-healing.sh` - Manual wrapper
- `tools/jarvis-error-daemon.sh` - Continuous monitoring