openclaw/tools/voice-surface/README.md

Voice Surface

Local voice interface for OpenClaw - push-to-talk web UI with full speech-to-text and text-to-speech pipeline.

Features

• 🎤 Push-to-talk browser interface (mouse, touch, or spacebar)
• 🔊 Full audio pipeline: STT → Agent → TTS
• 🔒 Owner-only: token-based authentication
• 🚀 Local-first: runs entirely on your machine
• 🎯 Simple & robust: minimal dependencies, straightforward flow

Architecture

1. Frontend: Static HTML page with MediaRecorder + WebSocket
2. Backend: Node.js (tsx) WebSocket server
3. Pipeline:
   • Capture audio from browser
   • Convert to WAV (ffmpeg, if available)
   • Transcribe with whisper-cli or openclaw media audio
   • Process with openclaw agent --local --json
   • Synthesize reply with edge-tts or openclaw tts
   • Stream audio back to browser

Prerequisites

Required

• Node.js 22+
• pnpm (for running OpenClaw commands)

Optional (improves performance)

• whisper.cpp/whisper-cli - for local STT (fallback: openclaw media audio)
• ffmpeg - for audio conversion (fallback: use original format)
• edge-tts - for TTS (fallback: openclaw tts)

Installing Optional Dependencies

whisper.cpp (recommended for best STT performance):

# Clone and build whisper.cpp
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
make

# Download a model (base recommended for speed/quality balance)
bash ./models/download-ggml-model.sh base

# Add to PATH or create symlink
sudo ln -s $(pwd)/main /usr/local/bin/whisper-cli

ffmpeg (for audio conversion):

# Ubuntu/Debian
sudo apt install ffmpeg

# Arch/CachyOS
sudo pacman -S ffmpeg

edge-tts (for TTS):

pip install edge-tts
# or
pipx install edge-tts

Installation

cd tools/voice-surface
pnpm install

Configuration

Environment variables (optional):

# Bind address (default: 127.0.0.1)
export BIND_HOST=192.168.1.220

# Port (default: 3030)
export PORT=3030

# Auth token (auto-generated if not set)
export AUTH_TOKEN=your-secret-token

Usage

Option 1: Run directly

cd tools/voice-surface
pnpm start

Or use the convenience script:

./tools/voice-surface/run.sh

Option 2: Install as systemd service (recommended for persistent use)

cd tools/voice-surface
./install-service.sh

This will:

• Install the service for your user
• Prompt for an auth token (or auto-generate one)
• Set up auto-restart on failure

After installation:

# Start the service
sudo systemctl start voice-surface@$(whoami)

# Enable auto-start on boot
sudo systemctl enable voice-surface@$(whoami)

# Check status
sudo systemctl status voice-surface@$(whoami)

# View logs
sudo journalctl -u voice-surface@$(whoami) -f

Accessing the interface

The server will display the access URL with authentication token:

🚀 Voice Surface running at:
   http://127.0.0.1:3030/?token=abc123xyz789

Access the interface

1. Open the URL in your browser
2. Grant microphone permissions when prompted
3. Hold the button (or spacebar) to talk
4. Release to send and process

Security Notes

• Server binds to 127.0.0.1 by default (localhost only)
• Set BIND_HOST=192.168.1.220 to allow LAN access
• Always use the token - it's required for all connections
• Token is displayed on startup if not explicitly set

Troubleshooting

"Microphone access denied"

• Check browser permissions
• HTTPS not required for localhost

"No speech detected"

• Speak clearly and hold button while talking
• Check microphone levels in system settings
• Ensure you're holding the button long enough

"whisper-cli failed"

• Check whisper-cli is in PATH: which whisper-cli
• Server will auto-fallback to openclaw media audio

"ffmpeg not available"

• Install ffmpeg (see prerequisites)
• Server will work without it (may be less reliable)

WebSocket connection issues

• Verify token in URL
• Check firewall settings if binding to LAN address
• Check server logs for errors

Development

The server uses:

• tsx for TypeScript execution
• ws for WebSocket handling
• fluent-ffmpeg for audio conversion (optional)

File structure:

tools/voice-surface/
├── src/
│   └── server.ts          # WebSocket server + audio pipeline
├── public/
│   └── index.html         # Push-to-talk UI
├── package.json
├── README.md
└── run.sh                 # Convenience launcher

API Protocol

WebSocket messages (JSON):

Client → Server:

{ "type": "audio-chunk", "data": "<base64>" }
{ "type": "audio-end" }

Server → Client:

{ "type": "status", "message": "Processing..." }
{ "type": "transcript", "text": "..." }
{ "type": "reply", "text": "..." }
{ "type": "audio", "data": "<base64>", "mimeType": "audio/mpeg" }
{ "type": "error", "message": "..." }

License

Same as OpenClaw (see repository root).