# OpenClaw Architecture & Troubleshooting Guide

This guide provides comprehensive reference for understanding OpenClaw's internal architecture and troubleshooting common issues.

## System Architecture Overview

### Gateway Hub

The OpenClaw gateway is a WebSocket-based central orchestration hub that manages all agent interactions.

**Key Components:**

- **Location**: `src/gateway/server.impl.ts`
- **Default Port**: 18789
- **Protocol**: HTTP/WebSocket with token or password authentication
- **Entry Point**: `openclaw gateway run` (foreground) or systemd/launchd service (background)

**Gateway Lifecycle:**

- Acquires a lock file to prevent multiple instances: `/tmp/openclaw-{uid}/gateway.{configHash}.lock`
- Handles SIGTERM/SIGINT for graceful shutdown (5s timeout)
- Handles SIGUSR1 for in-process restart (no supervisor required)
- Maintains persistent WebSocket connections to channels (Discord, Slack, Telegram, etc.)

**Gateway RPC Methods:**

- Chat: `chat.send`, `chat.abort`, `chat.stream`
- Agents: `agents.list`, `agent.status`, `agent.job`
- Sessions: `sessions.list`, `sessions.patch`, `sessions.preview`
- Channels: `channels.status`, `channels.list`
- System: `health`, `config.snapshot`, `cron.*`

### Agent System

Agents are personas/configurations that handle conversations. Multiple agents can coexist on the same gateway.

**Agent Structure:**

```
~/.openclaw/agents/{agentId}/
├── agent/               # Agent-specific config
└── sessions/            # Session metadata & transcripts
    ├── sessions.json    # Session metadata (cached 45s)
    └── {uuid}.jsonl     # Message transcripts
```

**Agent Workspace Files:**

```
~/.openclaw/workspace/
├── BOOTSTRAP.md         # Bootstrap instructions (presence = "bootstrapping")
├── AGENTS.md           # Subagent definitions
├── TOOLS.md            # Available tools
├── IDENTITY.md         # Agent personality
├── USER.md             # User context
├── SOUL.md             # Deep system prompt
└── MEMORY.md           # Memory system config
```

### Session Management

Sessions track conversation state and are stored per-agent.

**Session Storage:**

- **Metadata**: `~/.openclaw/agents/{agentId}/sessions/sessions.json`
- **Transcripts**: `~/.openclaw/agents/{agentId}/sessions/{sessionId}.jsonl`
- **Format**: JSON metadata with JSONL transcripts (one message per line)
- **Caching**: 45s TTL (configurable via `OPENCLAW_SESSION_CACHE_TTL_MS`)

**SessionEntry Structure:**

```typescript
{
  sessionId: string; // UUID identifying the session
  updatedAt: number; // Timestamp (ms) of last update
  sessionFile: string; // Path to transcript .jsonl file
  model: string; // Model used (e.g., "claude-3-5-sonnet-20241022")
  modelProvider: string; // Provider (anthropic, google, openai)
  channel: string; // Channel (telegram, discord, etc)
  totalTokens: number; // Token usage tracking
  compactionCount: number; // Context compression count
  // ... 80+ other fields
}
```

**Session Keys:**

- Format: typically `channel:sender` or custom based on `session.scope` config
- Examples: `telegram:12345`, `discord:user_id`, `global`
- Multi-agent format: `{agentId}:{channelSessionKey}`

**Session Lifecycle:**

1. Message arrives → Session resolved via session key
2. Agent loads session context from `.jsonl` transcript
3. Agent executes with model
4. Session metadata updated in `sessions.json`
5. Message appended to `.jsonl` transcript
6. Cache invalidated

### Configuration System

**Primary Config**: `~/.openclaw/openclaw.json` (JSON5 format)

**Key Sections:**

```json5
{
  gateway: {
    mode: "local", // "local" or "remote"
    bind: "loopback", // "loopback", "lan", or "auto"
    port: 18789,
    auth: {
      mode: "token", // "token" or "password"
      token: "...",
    },
  },
  agents: {
    default: "main",
    list: [
      {
        id: "main",
        name: "Main Agent",
        workspace: "~/.openclaw/workspace",
        model: "claude-3-5-sonnet-20241022",
      },
    ],
  },
  session: {
    scope: "per-sender", // "per-sender" or "global"
    reset: {
      idleMinutes: 60,
    },
  },
}
```

**Path Locations:**

- Config: `~/.openclaw/openclaw.json`
- State: `~/.openclaw/`
- Sessions: `~/.openclaw/agents/{agentId}/sessions/`
- Gateway Lock: `/tmp/openclaw-{uid}/gateway.{configHash}.lock`
- Credentials: `~/.openclaw/credentials/`

## Common Troubleshooting Scenarios

### "Gateway Stuck" or Unresponsive

**Symptoms:**

- `openclaw status` hangs or times out
- Commands don't respond
- Gateway service appears running but unreachable

**Diagnostic Steps:**

1. **Check for orphaned processes:**

   ```bash
   ps aux | grep "openclaw.*gateway"
   ```

   Look for multiple gateway processes with different PIDs

2. **Check gateway lock:**

   ```bash
   ls -la /tmp/openclaw-$UID/gateway.*.lock
   cat /tmp/openclaw-$UID/gateway.*.lock
   ```

   Verify the PID in the lock file matches running process

3. **Check systemd service (Linux):**

   ```bash
   systemctl --user status openclaw-gateway
   journalctl --user -u openclaw-gateway -n 50
   ```

4. **Check port binding:**
   ```bash
   ss -ltnp | grep 18789
   ```

**Resolution:**

```bash
# Kill all gateway processes
pkill -9 -f "openclaw.*gateway"

# Remove lock files
rm -f /tmp/openclaw-$UID/gateway.*.lock

# Restart gateway
sudo systemctl restart openclaw-gateway    # systemd
# OR
openclaw gateway run                       # foreground
```

### "1 bootstrapping" - Agent Won't Respond

**Symptoms:**

- `openclaw status` shows "1 bootstrapping"
- Agent seems stuck in initialization

**Cause:**
An agent has a `BOOTSTRAP.md` file present in its workspace, indicating it needs initialization.

**Diagnostic:**

```bash
find ~/.openclaw/agents -name "BOOTSTRAP.md"
```

**Resolution:**

```bash
# Option 1: Delete bootstrap file to mark as initialized
rm ~/.openclaw/agents/main/BOOTSTRAP.md

# Option 2: Or move it to backup
mv ~/.openclaw/agents/main/BOOTSTRAP.md ~/.openclaw/agents/main/BOOTSTRAP.md.bak
```

**Note:** The presence of `BOOTSTRAP.md` is just a marker. Once removed, the agent is considered bootstrapped.

### Session Timeout Loop

**Symptoms:**

- Specific session ID keeps timing out after 10 minutes (600000ms)
- Logs show: `[agent/embedded] embedded run timeout: sessionId=... timeoutMs=600000`
- Session file grows large but conversations don't complete

**Diagnostic:**

```bash
# Check session file size
ls -lh ~/.openclaw/agents/sonnet/sessions/*.jsonl

# Check recent gateway logs for timeout messages
journalctl --user -u openclaw-gateway -n 200 | grep timeout
```

**Resolution:**

```bash
# Backup the problematic session
SESSION_ID="e5a7a8f2-a84d-407b-8e48-fbbd95aead53"
cd ~/.openclaw/agents/sonnet/sessions
mv ${SESSION_ID}.jsonl ${SESSION_ID}.jsonl.bak
rm -f ${SESSION_ID}.jsonl.lock

# Restart gateway
sudo systemctl restart openclaw-gateway
```

**Investigation:**

- Check what triggers the session (cron jobs, automated tasks)
- Review session transcript for clues: `head -n 20 ~/.openclaw/agents/sonnet/sessions/${SESSION_ID}.jsonl.bak`
- Consider if the task is genuinely taking >10min or if it's stuck

### Dual Installation Conflict (User vs System)

**Symptoms:**

- Different versions shown by `openclaw --version` vs systemd service
- Gateway runs old version despite updates
- Config changes not taking effect

**Diagnostic:**

```bash
# Check which openclaw binary is being used
which openclaw
ls -la $(which openclaw)

# Check for user-local installation
ls -la ~/.local/bin/openclaw
ls -la ~/.local/lib/node_modules/openclaw

# Check system installation
ls -la /usr/bin/openclaw
ls -la /usr/lib/node_modules/openclaw

# Check systemd ExecStart path
systemctl --user cat openclaw-gateway | grep ExecStart
```

**Resolution (Remove User-Local Installation):**

```bash
# Remove user-local installation
rm ~/.local/bin/openclaw
rm -rf ~/.local/lib/node_modules/openclaw

# Update system installation
sudo npm install -g openclaw@latest

# Verify single installation
which openclaw
openclaw --version
```

**Important:** If systemd service runs as a specific user (e.g., `User=alex`), ensure config files are accessible:

```bash
ls -la ~/.openclaw/openclaw.json
ls -la ~/.openclaw/.env
```

All should be owned by the service user with appropriate permissions (typically 600).

### Sessions Not Persisting

**Symptoms:**

- Conversations don't remember context
- `openclaw status` shows 0 sessions despite recent conversations
- Session metadata lost after restart

**Diagnostic:**

```bash
# Check session store exists and is writable
ls -la ~/.openclaw/agents/*/sessions/
cat ~/.openclaw/agents/main/sessions/sessions.json | jq '.'

# Check session scope configuration
openclaw config get session.scope
```

**Resolution:**

```bash
# Verify session directory permissions
chmod 700 ~/.openclaw/agents/*/sessions
chmod 600 ~/.openclaw/agents/*/sessions/sessions.json

# Disable session cache temporarily to test
OPENCLAW_SESSION_CACHE_TTL_MS=0 openclaw status

# Check if sessions are configured per-sender or global
openclaw config set session.scope per-sender
```

### Gateway Lock Error

**Symptoms:**

- Error: "Gateway is already running (PID: 12345)"
- Cannot start gateway despite no visible process

**Diagnostic:**

```bash
# Check lock file
cat /tmp/openclaw-$UID/gateway.*.lock

# Check if PID is actually running
ps -p <PID_FROM_LOCK>
```

**Resolution:**

```bash
# If process is dead, remove lock manually
rm /tmp/openclaw-$UID/gateway.*.lock

# If process is alive but should be killed
kill -9 <PID>
rm /tmp/openclaw-$UID/gateway.*.lock

# Then restart
openclaw gateway run
```

### Remote Gateway Not Reachable

**Symptoms:**

- Local gateway works but can't connect remotely
- `openclaw gateway probe` times out

**Diagnostic:**

```bash
# Check gateway mode
openclaw config get gateway.mode

# Check network binding
openclaw config get gateway.bind

# Test local connection
curl -v http://localhost:18789/health

# Check firewall (if LAN mode)
sudo iptables -L -n | grep 18789    # Linux
sudo pfctl -sr | grep 18789         # macOS
```

**Resolution:**

```bash
# For Tailscale access
openclaw config set gateway.mode remote
openclaw config set gateway.tailscale.mode serve

# For LAN access
openclaw config set gateway.bind lan

# Restart gateway
sudo systemctl restart openclaw-gateway
```

## Diagnostic Commands Reference

### Status & Health

```bash
# Full status overview
openclaw status

# Deep status with probes (slower but thorough)
openclaw status --deep

# Channel-specific status
openclaw channels status --probe

# Gateway health check
curl http://localhost:18789/health
```

### Session Management

```bash
# List all sessions
openclaw sessions list

# List sessions for specific agent
openclaw sessions list --agent main

# View session transcript
cat ~/.openclaw/agents/main/sessions/<session-id>.jsonl | jq '.'

# Clear old sessions (interactive)
openclaw sessions clean
```

### Gateway Operations

```bash
# Start gateway (foreground)
openclaw gateway run

# Start with force (frees port)
openclaw gateway run --force

# Install as service
openclaw gateway install

# Service management (systemd)
sudo systemctl start openclaw-gateway
sudo systemctl stop openclaw-gateway
sudo systemctl restart openclaw-gateway
sudo systemctl status openclaw-gateway

# View gateway logs
journalctl --user -u openclaw-gateway -f
```

### Configuration

```bash
# View current config
openclaw config show

# Get specific value
openclaw config get gateway.port

# Set value
openclaw config set gateway.port 18790

# Validate config
openclaw config validate
```

### Debugging

```bash
# Enable debug logging
openclaw config set logging.level debug
openclaw config set logging.gatewayWsLog full

# Check for config issues
openclaw doctor

# View recent logs
journalctl --user -u openclaw-gateway -n 100

# Follow logs live
journalctl --user -u openclaw-gateway -f

# Check process tree
ps auxf | grep openclaw

# Check network connections
ss -tunap | grep openclaw
```

## Key Files & Directories

### Configuration

- `~/.openclaw/openclaw.json` - Primary configuration
- `~/.openclaw/.env` - Environment variables
- `~/.openclaw/identity/device.json` - Device identity

### Sessions & State

- `~/.openclaw/agents/{agentId}/sessions/sessions.json` - Session metadata
- `~/.openclaw/agents/{agentId}/sessions/{sessionId}.jsonl` - Transcripts
- `~/.openclaw/workspace/` - Agent workspace files

### Service & Locks

- `/tmp/openclaw-{uid}/gateway.*.lock` - Gateway lock file
- `/etc/systemd/system/openclaw-gateway.service` - Systemd unit (system)
- `~/.config/systemd/user/openclaw-gateway.service` - Systemd unit (user)

### Logs

- `journalctl --user -u openclaw-gateway` - Systemd logs (Linux)
- `~/Library/Logs/OpenClaw/` - App logs (macOS)
- `/tmp/openclaw-gateway.log` - Custom log location (if configured)

## Understanding Status Output

### Example Status Output

```
Gateway: reachable · local · 18789 · loopback
Agents: 3 · 1 bootstrapping · sessions 42
Channels: 3 configured · 2 connected

Top 5 Sessions:
  telegram:12345  main     claude-3-5-sonnet  4h     125K tokens
  discord:67890   research claude-3-opus      2d     450K tokens
```

**Decoding:**

- **Gateway**: reachable (responds), local (not tunneled), port 18789, loopback binding
- **Agents**: 3 total agents, 1 has `BOOTSTRAP.md` present, 42 sessions across all agents
- **Channels**: 3 configured (in config), 2 actually connected
- **Sessions**: Show channel:sender, agent, model, age, token usage

### Status Flags

- `--deep` - Perform network probes (slower but more thorough)
- `--all` - Show all sessions (not just top 5)
- `--json` - Output in JSON format for scripting

## Advanced Troubleshooting

### Session Cache Issues

Session metadata is cached for 45 seconds by default. This can cause stale data issues.

**Disable cache temporarily:**

```bash
OPENCLAW_SESSION_CACHE_TTL_MS=0 openclaw status
```

**Clear cache (restart gateway):**

```bash
sudo systemctl restart openclaw-gateway
```

### Multiple Gateways on Same Host

It's possible (but unusual) to run multiple gateways with different configs.

**Check for multiple gateways:**

```bash
ps aux | grep "openclaw.*gateway"
ls /tmp/openclaw-$UID/gateway.*.lock
```

**Each gateway needs:**

- Unique config file path (different config hash)
- Unique port
- Different agent IDs to avoid session conflicts

### Memory/Token Debugging

Sessions track token usage and compaction count.

**Check session token usage:**

```bash
cat ~/.openclaw/agents/main/sessions/sessions.json | \
  jq '.[] | {sessionId, totalTokens, compactionCount}'
```

**High compaction count** (>5) suggests:

- Very long conversations
- Context window approaching limit
- Consider resetting session: `/new`

### Bootstrap Hooks & Plugins

Plugins can modify agent bootstrap behavior via `agent.bootstrap` hook.

**Check which plugins are loaded:**

```bash
openclaw plugins list
```

**Disable all plugins temporarily:**

```bash
openclaw config set plugins.enabled false
sudo systemctl restart openclaw-gateway
```

## Best Practices

### Before Making Changes

1. **Check current status:** `openclaw status`
2. **Pull latest config:** `git pull` (if version controlled)
3. **Backup sessions:** `cp -r ~/.openclaw/agents ~/.openclaw/agents.bak`

### After Making Changes

1. **Validate config:** `openclaw config validate`
2. **Restart cleanly:** `sudo systemctl restart openclaw-gateway`
3. **Verify:** `openclaw status --deep`
4. **Check logs:** `journalctl --user -u openclaw-gateway -n 50`

### Multi-Agent Safety

If multiple agents/operators are working on the same system:

- Avoid creating/dropping git stashes without coordination
- Don't switch branches unexpectedly
- Commit your changes before pulling: `git commit -am "wip" && git pull --rebase`
- Keep work scoped to specific files/features

### Regular Maintenance

```bash
# Weekly: Check for updates
openclaw update check

# Monthly: Clean old sessions
openclaw sessions clean --older-than 30d

# Monthly: Check security audit
openclaw doctor

# Quarterly: Review and prune unused agents/channels
openclaw config show
```

## Emergency Recovery

### Complete Reset (Nuclear Option)

**⚠️ WARNING: This deletes all sessions and configuration**

```bash
# Stop gateway
sudo systemctl stop openclaw-gateway

# Backup everything
cp -r ~/.openclaw ~/.openclaw.backup.$(date +%Y%m%d-%H%M%S)

# Remove state
rm -rf ~/.openclaw

# Reinstall
npm install -g openclaw@latest

# Reconfigure
openclaw config wizard
openclaw gateway install
openclaw gateway start
```

### Restore from Backup

```bash
# Stop gateway
sudo systemctl stop openclaw-gateway

# Restore config
cp ~/.openclaw.backup.*/openclaw.json ~/.openclaw/openclaw.json

# Restore sessions
cp -r ~/.openclaw.backup.*/agents ~/.openclaw/agents

# Restart
sudo systemctl start openclaw-gateway
```

## Related Documentation

- Gateway Configuration: `/gateway`
- Session Management: `/sessions`
- Agent System: `/agents`
- Channel Integration: `/channels`
- Security: `/security`

## Quick Reference Card

```bash
# Status
openclaw status                    # Quick overview
openclaw status --deep            # Full diagnostic

# Gateway
sudo systemctl restart openclaw-gateway
journalctl --user -u openclaw-gateway -f

# Sessions
openclaw sessions list
rm ~/.openclaw/agents/*/sessions/<session-id>.jsonl*

# Bootstrap
rm ~/.openclaw/agents/*/BOOTSTRAP.md

# Process Management
ps aux | grep openclaw
pkill -9 -f "openclaw.*gateway"

# Locks
ls -la /tmp/openclaw-$UID/
rm /tmp/openclaw-$UID/gateway.*.lock

# Config
openclaw config show
openclaw config validate
openclaw doctor
```