openclaw/docs/troubleshooting/architecture-guide.md

OpenClaw Architecture & Troubleshooting Guide

This guide provides comprehensive reference for understanding OpenClaw's internal architecture and troubleshooting common issues.

System Architecture Overview

Gateway Hub

The OpenClaw gateway is a WebSocket-based central orchestration hub that manages all agent interactions.

Key Components:

• Location: src/gateway/server.impl.ts
• Default Port: 18789
• Protocol: HTTP/WebSocket with token or password authentication
• Entry Point: openclaw gateway run (foreground) or systemd/launchd service (background)

Gateway Lifecycle:

• Acquires a lock file to prevent multiple instances: /tmp/openclaw-{uid}/gateway.{configHash}.lock
• Handles SIGTERM/SIGINT for graceful shutdown (5s timeout)
• Handles SIGUSR1 for in-process restart (no supervisor required)
• Maintains persistent WebSocket connections to channels (Discord, Slack, Telegram, etc.)

Gateway RPC Methods:

• Chat: chat.send, chat.abort, chat.stream
• Agents: agents.list, agent.status, agent.job
• Sessions: sessions.list, sessions.patch, sessions.preview
• Channels: channels.status, channels.list
• System: health, config.snapshot, cron.*

Agent System

Agents are personas/configurations that handle conversations. Multiple agents can coexist on the same gateway.

Agent Structure:

~/.openclaw/agents/{agentId}/
├── agent/               # Agent-specific config
└── sessions/            # Session metadata & transcripts
    ├── sessions.json    # Session metadata (cached 45s)
    └── {uuid}.jsonl     # Message transcripts

Agent Workspace Files:

~/.openclaw/workspace/
├── BOOTSTRAP.md         # Bootstrap instructions (presence = "bootstrapping")
├── AGENTS.md           # Subagent definitions
├── TOOLS.md            # Available tools
├── IDENTITY.md         # Agent personality
├── USER.md             # User context
├── SOUL.md             # Deep system prompt
└── MEMORY.md           # Memory system config

Session Management

Sessions track conversation state and are stored per-agent.

Session Storage:

• Metadata: ~/.openclaw/agents/{agentId}/sessions/sessions.json
• Transcripts: ~/.openclaw/agents/{agentId}/sessions/{sessionId}.jsonl
• Format: JSON metadata with JSONL transcripts (one message per line)
• Caching: 45s TTL (configurable via OPENCLAW_SESSION_CACHE_TTL_MS)

SessionEntry Structure:

{
  sessionId: string; // UUID identifying the session
  updatedAt: number; // Timestamp (ms) of last update
  sessionFile: string; // Path to transcript .jsonl file
  model: string; // Model used (e.g., "claude-3-5-sonnet-20241022")
  modelProvider: string; // Provider (anthropic, google, openai)
  channel: string; // Channel (telegram, discord, etc)
  totalTokens: number; // Token usage tracking
  compactionCount: number; // Context compression count
  // ... 80+ other fields
}

Session Keys:

• Format: typically channel:sender or custom based on session.scope config
• Examples: telegram:12345, discord:user_id, global
• Multi-agent format: {agentId}:{channelSessionKey}

Session Lifecycle:

1. Message arrives → Session resolved via session key
2. Agent loads session context from .jsonl transcript
3. Agent executes with model
4. Session metadata updated in sessions.json
5. Message appended to .jsonl transcript
6. Cache invalidated

Configuration System

Primary Config: ~/.openclaw/openclaw.json (JSON5 format)

Key Sections:

{
  gateway: {
    mode: "local", // "local" or "remote"
    bind: "loopback", // "loopback", "lan", or "auto"
    port: 18789,
    auth: {
      mode: "token", // "token" or "password"
      token: "...",
    },
  },
  agents: {
    default: "main",
    list: [
      {
        id: "main",
        name: "Main Agent",
        workspace: "~/.openclaw/workspace",
        model: "claude-3-5-sonnet-20241022",
      },
    ],
  },
  session: {
    scope: "per-sender", // "per-sender" or "global"
    reset: {
      idleMinutes: 60,
    },
  },
}

Path Locations:

• Config: ~/.openclaw/openclaw.json
• State: ~/.openclaw/
• Sessions: ~/.openclaw/agents/{agentId}/sessions/
• Gateway Lock: /tmp/openclaw-{uid}/gateway.{configHash}.lock
• Credentials: ~/.openclaw/credentials/

Common Troubleshooting Scenarios

"Gateway Stuck" or Unresponsive

Symptoms:

• openclaw status hangs or times out
• Commands don't respond
• Gateway service appears running but unreachable

Diagnostic Steps:

1. Check for orphaned processes:

   ps aux | grep "openclaw.*gateway"
   

   Look for multiple gateway processes with different PIDs

2. Check gateway lock:

   ls -la /tmp/openclaw-$UID/gateway.*.lock
   cat /tmp/openclaw-$UID/gateway.*.lock
   

   Verify the PID in the lock file matches running process

3. Check systemd service (Linux):

   systemctl --user status openclaw-gateway
   journalctl --user -u openclaw-gateway -n 50
   

4. Check port binding:

   ss -ltnp | grep 18789
   

Resolution:

# Kill all gateway processes
pkill -9 -f "openclaw.*gateway"

# Remove lock files
rm -f /tmp/openclaw-$UID/gateway.*.lock

# Restart gateway
sudo systemctl restart openclaw-gateway    # systemd
# OR
openclaw gateway run                       # foreground

"1 bootstrapping" - Agent Won't Respond

Symptoms:

• openclaw status shows "1 bootstrapping"
• Agent seems stuck in initialization

Cause: An agent has a BOOTSTRAP.md file present in its workspace, indicating it needs initialization.

Diagnostic:

find ~/.openclaw/agents -name "BOOTSTRAP.md"

Resolution:

# Option 1: Delete bootstrap file to mark as initialized
rm ~/.openclaw/agents/main/BOOTSTRAP.md

# Option 2: Or move it to backup
mv ~/.openclaw/agents/main/BOOTSTRAP.md ~/.openclaw/agents/main/BOOTSTRAP.md.bak

Note: The presence of BOOTSTRAP.md is just a marker. Once removed, the agent is considered bootstrapped.

Session Timeout Loop

Symptoms:

• Specific session ID keeps timing out after 10 minutes (600000ms)
• Logs show: [agent/embedded] embedded run timeout: sessionId=... timeoutMs=600000
• Session file grows large but conversations don't complete

Diagnostic:

# Check session file size
ls -lh ~/.openclaw/agents/sonnet/sessions/*.jsonl

# Check recent gateway logs for timeout messages
journalctl --user -u openclaw-gateway -n 200 | grep timeout

Resolution:

# Backup the problematic session
SESSION_ID="e5a7a8f2-a84d-407b-8e48-fbbd95aead53"
cd ~/.openclaw/agents/sonnet/sessions
mv ${SESSION_ID}.jsonl ${SESSION_ID}.jsonl.bak
rm -f ${SESSION_ID}.jsonl.lock

# Restart gateway
sudo systemctl restart openclaw-gateway

Investigation:

• Check what triggers the session (cron jobs, automated tasks)
• Review session transcript for clues: head -n 20 ~/.openclaw/agents/sonnet/sessions/${SESSION_ID}.jsonl.bak
• Consider if the task is genuinely taking >10min or if it's stuck

Dual Installation Conflict (User vs System)

Symptoms:

• Different versions shown by openclaw --version vs systemd service
• Gateway runs old version despite updates
• Config changes not taking effect

Diagnostic:

# Check which openclaw binary is being used
which openclaw
ls -la $(which openclaw)

# Check for user-local installation
ls -la ~/.local/bin/openclaw
ls -la ~/.local/lib/node_modules/openclaw

# Check system installation
ls -la /usr/bin/openclaw
ls -la /usr/lib/node_modules/openclaw

# Check systemd ExecStart path
systemctl --user cat openclaw-gateway | grep ExecStart

Resolution (Remove User-Local Installation):

# Remove user-local installation
rm ~/.local/bin/openclaw
rm -rf ~/.local/lib/node_modules/openclaw

# Update system installation
sudo npm install -g openclaw@latest

# Verify single installation
which openclaw
openclaw --version

Important: If systemd service runs as a specific user (e.g., User=alex), ensure config files are accessible:

ls -la ~/.openclaw/openclaw.json
ls -la ~/.openclaw/.env

All should be owned by the service user with appropriate permissions (typically 600).

Sessions Not Persisting

Symptoms:

• Conversations don't remember context
• openclaw status shows 0 sessions despite recent conversations
• Session metadata lost after restart

Diagnostic:

# Check session store exists and is writable
ls -la ~/.openclaw/agents/*/sessions/
cat ~/.openclaw/agents/main/sessions/sessions.json | jq '.'

# Check session scope configuration
openclaw config get session.scope

Resolution:

# Verify session directory permissions
chmod 700 ~/.openclaw/agents/*/sessions
chmod 600 ~/.openclaw/agents/*/sessions/sessions.json

# Disable session cache temporarily to test
OPENCLAW_SESSION_CACHE_TTL_MS=0 openclaw status

# Check if sessions are configured per-sender or global
openclaw config set session.scope per-sender

Gateway Lock Error

Symptoms:

• Error: "Gateway is already running (PID: 12345)"
• Cannot start gateway despite no visible process

Diagnostic:

# Check lock file
cat /tmp/openclaw-$UID/gateway.*.lock

# Check if PID is actually running
ps -p <PID_FROM_LOCK>

Resolution:

# If process is dead, remove lock manually
rm /tmp/openclaw-$UID/gateway.*.lock

# If process is alive but should be killed
kill -9 <PID>
rm /tmp/openclaw-$UID/gateway.*.lock

# Then restart
openclaw gateway run

Remote Gateway Not Reachable

Symptoms:

• Local gateway works but can't connect remotely
• openclaw gateway probe times out

Diagnostic:

# Check gateway mode
openclaw config get gateway.mode

# Check network binding
openclaw config get gateway.bind

# Test local connection
curl -v http://localhost:18789/health

# Check firewall (if LAN mode)
sudo iptables -L -n | grep 18789    # Linux
sudo pfctl -sr | grep 18789         # macOS

Resolution:

# For Tailscale access
openclaw config set gateway.mode remote
openclaw config set gateway.tailscale.mode serve

# For LAN access
openclaw config set gateway.bind lan

# Restart gateway
sudo systemctl restart openclaw-gateway

Diagnostic Commands Reference

Status & Health

# Full status overview
openclaw status

# Deep status with probes (slower but thorough)
openclaw status --deep

# Channel-specific status
openclaw channels status --probe

# Gateway health check
curl http://localhost:18789/health

Session Management

# List all sessions
openclaw sessions list

# List sessions for specific agent
openclaw sessions list --agent main

# View session transcript
cat ~/.openclaw/agents/main/sessions/<session-id>.jsonl | jq '.'

# Clear old sessions (interactive)
openclaw sessions clean

Gateway Operations

# Start gateway (foreground)
openclaw gateway run

# Start with force (frees port)
openclaw gateway run --force

# Install as service
openclaw gateway install

# Service management (systemd)
sudo systemctl start openclaw-gateway
sudo systemctl stop openclaw-gateway
sudo systemctl restart openclaw-gateway
sudo systemctl status openclaw-gateway

# View gateway logs
journalctl --user -u openclaw-gateway -f

Configuration

# View current config
openclaw config show

# Get specific value
openclaw config get gateway.port

# Set value
openclaw config set gateway.port 18790

# Validate config
openclaw config validate

Debugging

# Enable debug logging
openclaw config set logging.level debug
openclaw config set logging.gatewayWsLog full

# Check for config issues
openclaw doctor

# View recent logs
journalctl --user -u openclaw-gateway -n 100

# Follow logs live
journalctl --user -u openclaw-gateway -f

# Check process tree
ps auxf | grep openclaw

# Check network connections
ss -tunap | grep openclaw

Key Files & Directories

Configuration

• ~/.openclaw/openclaw.json - Primary configuration
• ~/.openclaw/.env - Environment variables
• ~/.openclaw/identity/device.json - Device identity

Sessions & State

• ~/.openclaw/agents/{agentId}/sessions/sessions.json - Session metadata
• ~/.openclaw/agents/{agentId}/sessions/{sessionId}.jsonl - Transcripts
• ~/.openclaw/workspace/ - Agent workspace files

Service & Locks

• /tmp/openclaw-{uid}/gateway.*.lock - Gateway lock file
• /etc/systemd/system/openclaw-gateway.service - Systemd unit (system)
• ~/.config/systemd/user/openclaw-gateway.service - Systemd unit (user)

Logs

• journalctl --user -u openclaw-gateway - Systemd logs (Linux)
• ~/Library/Logs/OpenClaw/ - App logs (macOS)
• /tmp/openclaw-gateway.log - Custom log location (if configured)

Understanding Status Output

Example Status Output

Gateway: reachable · local · 18789 · loopback
Agents: 3 · 1 bootstrapping · sessions 42
Channels: 3 configured · 2 connected

Top 5 Sessions:
  telegram:12345  main     claude-3-5-sonnet  4h     125K tokens
  discord:67890   research claude-3-opus      2d     450K tokens

Decoding:

• Gateway: reachable (responds), local (not tunneled), port 18789, loopback binding
• Agents: 3 total agents, 1 has BOOTSTRAP.md present, 42 sessions across all agents
• Channels: 3 configured (in config), 2 actually connected
• Sessions: Show channel:sender, agent, model, age, token usage

Status Flags

• --deep - Perform network probes (slower but more thorough)
• --all - Show all sessions (not just top 5)
• --json - Output in JSON format for scripting

Advanced Troubleshooting

Session Cache Issues

Session metadata is cached for 45 seconds by default. This can cause stale data issues.

Disable cache temporarily:

OPENCLAW_SESSION_CACHE_TTL_MS=0 openclaw status

Clear cache (restart gateway):

sudo systemctl restart openclaw-gateway

Multiple Gateways on Same Host

It's possible (but unusual) to run multiple gateways with different configs.

Check for multiple gateways:

ps aux | grep "openclaw.*gateway"
ls /tmp/openclaw-$UID/gateway.*.lock

Each gateway needs:

• Unique config file path (different config hash)
• Unique port
• Different agent IDs to avoid session conflicts

Memory/Token Debugging

Sessions track token usage and compaction count.

Check session token usage:

cat ~/.openclaw/agents/main/sessions/sessions.json | \
  jq '.[] | {sessionId, totalTokens, compactionCount}'

High compaction count (>5) suggests:

• Very long conversations
• Context window approaching limit
• Consider resetting session: /new

Bootstrap Hooks & Plugins

Plugins can modify agent bootstrap behavior via agent.bootstrap hook.

Check which plugins are loaded:

openclaw plugins list

Disable all plugins temporarily:

openclaw config set plugins.enabled false
sudo systemctl restart openclaw-gateway

Best Practices

Before Making Changes

1. Check current status: openclaw status
2. Pull latest config: git pull (if version controlled)
3. Backup sessions: cp -r ~/.openclaw/agents ~/.openclaw/agents.bak

After Making Changes

1. Validate config: openclaw config validate
2. Restart cleanly: sudo systemctl restart openclaw-gateway
3. Verify: openclaw status --deep
4. Check logs: journalctl --user -u openclaw-gateway -n 50

Multi-Agent Safety

If multiple agents/operators are working on the same system:

• Avoid creating/dropping git stashes without coordination
• Don't switch branches unexpectedly
• Commit your changes before pulling: git commit -am "wip" && git pull --rebase
• Keep work scoped to specific files/features

Regular Maintenance

# Weekly: Check for updates
openclaw update check

# Monthly: Clean old sessions
openclaw sessions clean --older-than 30d

# Monthly: Check security audit
openclaw doctor

# Quarterly: Review and prune unused agents/channels
openclaw config show

Emergency Recovery

Complete Reset (Nuclear Option)

⚠️ WARNING: This deletes all sessions and configuration

# Stop gateway
sudo systemctl stop openclaw-gateway

# Backup everything
cp -r ~/.openclaw ~/.openclaw.backup.$(date +%Y%m%d-%H%M%S)

# Remove state
rm -rf ~/.openclaw

# Reinstall
npm install -g openclaw@latest

# Reconfigure
openclaw config wizard
openclaw gateway install
openclaw gateway start

Restore from Backup

# Stop gateway
sudo systemctl stop openclaw-gateway

# Restore config
cp ~/.openclaw.backup.*/openclaw.json ~/.openclaw/openclaw.json

# Restore sessions
cp -r ~/.openclaw.backup.*/agents ~/.openclaw/agents

# Restart
sudo systemctl start openclaw-gateway

Related Documentation

• Gateway Configuration: /gateway
• Session Management: /sessions
• Agent System: /agents
• Channel Integration: /channels
• Security: /security

Quick Reference Card

# Status
openclaw status                    # Quick overview
openclaw status --deep            # Full diagnostic

# Gateway
sudo systemctl restart openclaw-gateway
journalctl --user -u openclaw-gateway -f

# Sessions
openclaw sessions list
rm ~/.openclaw/agents/*/sessions/<session-id>.jsonl*

# Bootstrap
rm ~/.openclaw/agents/*/BOOTSTRAP.md

# Process Management
ps aux | grep openclaw
pkill -9 -f "openclaw.*gateway"

# Locks
ls -la /tmp/openclaw-$UID/
rm /tmp/openclaw-$UID/gateway.*.lock

# Config
openclaw config show
openclaw config validate
openclaw doctor