Troubleshooting Guide

Symptoms:

• Bridge calls timeout
• No response in Slack/WhatsApp
• Watchtower shows no recent activity

Debugging Steps:

1. Check if container is running:

   docker ps | grep <agent-name>

2. Check container logs:

   docker logs --tail 100 <agent-name>

   Look for: crashes, OOM errors, unhandled exceptions
3. Check gateway status:

   docker exec <agent-name> openclaw status

4. Check live observability events:

   curl "<observability-url>/events?service=<agent>&limit=50" \
     -H "Authorization: Bearer $OBS_COLLECTOR_TOKEN"

Common Causes:

• Process crashed (check exit code in docker ps -a)
• Out of memory (Docker OOM killer)
• Deadlock or hung request (restart container)
• Gateway config error (check gateway.toml syntax)

Fix:

docker restart <agent-name>

Symptoms:

• Error: "Bridge timeout after 60s"
• Caller hangs waiting for response

Debugging Steps:

1. Check if target agent is reachable:

   curl http://host.docker.internal:<port>/health

2. Check live bridge telemetry:

   curl "<observability-url>/events?service=bridge&event=call&limit=20" \
     -H "Authorization: Bearer $OBS_COLLECTOR_TOKEN"

   Look for: long duration_ms values, error-level events
3. Test the target agent directly:

   docker exec <target-agent> openclaw status

Common Causes:

• Target agent is processing a long task (LLM call, tool execution)
• Target agent is stuck in a loop
• Network issue (rare on localhost)
• Target agent's gateway is overloaded (too many concurrent requests)

Fix:

• If agent is busy but healthy: wait or retry later
• If agent is stuck: docker restart <target-agent>
• If happens frequently: increase timeout or optimize target agent's performance

Symptoms:

• Scout cycle didn't run at expected time
• No Slack notification from scheduled task
• Watchtower cron shows Cron data is partial
• Summary cards show — plus an all-agents fetch-failed warning instead of counts

Debugging Steps:

1. Read the Watchtower state message before the counts:
   If Watchtower says Cron data is partial, some agents answered and some did not. If summary cards show — with an all-agents failure banner, treat the entire summary as unavailable rather than zero.
2. List all cron jobs:

   curl "<gateway-url>/api/cron/list"

   Check: enabled field, nextRun timestamp
3. Check job run history:

   curl "<gateway-url>/api/cron/runs?jobId=<job-id>"

   Look for: error field, completed field
4. Check gateway logs during scheduled time:

   docker logs scout --since 10m

Common Causes:

• Job is disabled (check enabled: false)
• Gateway was down/restarting during scheduled time
• Previous run still executing (won't start concurrent run)
• Cron expression has wrong timezone (default is UTC)
• One or more agent gateways failed to answer the cron list request, so Watchtower can only show partial coverage

Fix:

• Manually trigger: curl -X POST <gateway-url>/api/cron/run?jobId=<id>
• Re-enable: Update job with enabled: true
• Adjust schedule if timezone is wrong
• If Watchtower coverage is partial or unavailable, restore agent/gateway connectivity first, then refresh before trusting the summary cards again

Symptoms:

• Error: "ECONNREFUSED" or "Connection refused"
• Memory / knowledge search failing
• OpenObserve or Watchtower observability surface not responding
• Watchtower shows Observability data unavailable
• Code execution requests timing out

Debugging Steps:

1. Check if the service process is running:

   ps aux | grep <service-name>

2. Try a direct health check:

   curl http://localhost:<service-port>/health

3. If Watchtower named the failure, trust the category:
   Observability data unavailable means the route loader could not load OpenObserve data on that refresh. Treat it as source-health trouble, not an empty event feed.
4. Check service logs — look for crash messages or startup errors:

   tail -f <service-log-path>

Common Causes:

• Service crashed or was stopped
• Port conflict (another process using the same port)
• Service not restarted after a host reboot
• Upstream OpenObserve connectivity failed, so Watchtower rendered an explicit unavailable state instead of stale empty data

Fix:

Restart the affected service using its startup script. Check the Infrastructure page for each service’s expected port and startup method.

Symptoms:

• Agent responses getting slower
• Container keeps restarting
• Mac mini fans running loud

Debugging Steps:

1. Check container resource usage:

   docker stats --no-stream

   Look for: high CPU%, high MEM%
2. Check host resources:

   top -o cpu # or htop

3. Check disk usage:

   df -h docker system df # Docker disk usage

4. Check session_status for token usage:

   High token counts = expensive operations

Common Causes:

• Memory leak in agent (growing context, cached data)
• Too many concurrent operations
• Large file operations (logs, databases)
• Docker volume running out of space

Fix:

• Restart agent: docker restart <agent>
• Clean up disk: docker system prune -a
• Trim old logs/data from workspaces
• Increase Docker resource limits in Docker Desktop settings

Symptoms:

• Scout cycle ran an hour early or late (DST shift)
• Cron job fires at UTC midnight instead of local midnight
• Two cycles ran close together then a long gap with no runs
• nextRun timestamp doesn't match expected schedule

Root Cause Patterns:

• Missing timezone: Cron expressions default to UTC. A job meant to run at 8 AM CT needs tz: "America/Chicago" — without it, it runs at 8 AM UTC (3 AM CT).
• DST transitions: everyMs-based jobs aren't affected by DST, but cron-based jobs are. "0 8 * * *" with tz: "America/Chicago" will correctly shift with DST.
• Stagger jitter: Jobs with staggerMs add random jitter to the scheduled time — this is intentional to avoid thundering-herd, but makes timing look unpredictable.
• Previous run still executing: If a cycle runs long, the next scheduled fire is skipped (no concurrent runs for the same job).

Fix:

• Always set tz explicitly on cron-scheduled jobs
• Use cron schedule kind (not everyMs) when you need wall-clock alignment
• Check /api/cron/runs?jobId=<id> for skipped runs — they show as skipped: true

Symptoms:

• Agent responses become truncated, generic, or start repeating earlier context
• Agent forgets instructions given earlier in the session
• Tool calls fail with "context length exceeded" or similar errors
• Model switches to a cheaper fallback unexpectedly

Root Cause Patterns:

• Long-running session: Sessions accumulate context with every message and tool response. A charter cycle that reads many files and makes many edits fills the context window fast.
• Large tool outputs: Reading entire files, getting verbose API responses, or listing large directories bloats the context.
• Repeated full-file reads: Reading the same file multiple times in one session duplicates content in context.
• Sub-agent output: Spawned sub-agents return their full output to the parent session, which can be very large.

Debugging Steps:

1. Check session token usage: Use openclaw status or the session status card — look for token counts approaching the model's limit.
2. Review the session transcript: Identify which messages or tool responses are disproportionately large.
3. Check if a fallback model was used: Some gateways auto-fallback when context overflows — look for model switch events in logs.

Fix:

• For operators: Start a fresh session or use limit and offset when reading files to get only the lines you need
• For cycle design: Use lightContext: true on cron jobs to reduce inherited context weight
• For sub-agents: Request concise summaries rather than full output dumps
• For large files: Read targeted sections with offset/limit instead of entire files

Symptoms:

• PR merge blocked by merge conflict
• Git rebase or merge fails during sync-in
• CI fails on a branch that was passing locally
• git status shows "both modified" on files you didn't expect to conflict

Root Cause Patterns:

• Concurrent cycles: Another agent or a manual edit pushed to the same branch or base while this cycle was running.
• Stale working branch: The feature branch fell behind main because it was created hours or days ago and main has moved forward.
• Shared data files: Files like progress logs, scratchpads, or freshness tags are frequently modified by multiple actors — these are the most common conflict points.
• Squash-merge divergence: If main was squash-merged and the branch still has the original commits, git sees them as different and conflicts appear on files touched by both.

Debugging Steps:

1. Identify conflicting files: git diff --name-only --diff-filter=U
2. Check what changed on main: git log --oneline main..origin/main — were the same files modified?
3. Decide conflict strategy: For data files (logs, timestamps), newer usually wins. For code, manual resolution is needed.

Fix:

• Rebase onto current main: git fetch origin && git rebase origin/main
• Resolve conflicts: For data files, accept the newer version; for code, merge both changes carefully
• If rebase is too tangled: Create a fresh branch from current main and cherry-pick the relevant commits
• Prevention: Keep branches short-lived — rebase onto main at the start of each cycle

Symptoms:

• Bridge calls return 502 Bad Gateway immediately after a gateway restart
• Health endpoint returns 200 but bridge calls still fail
• Cron jobs fail with connection errors for the first 30-60 seconds after restart
• openclaw status shows healthy but tool calls hang or error

Root Cause Patterns:

• Cold-start delay: After a gateway restart, the HTTP server binds quickly but the model client, tool registry, and session store take time to initialize. The health endpoint reports "up" before the system is fully ready.
• Session replay: Long sessions need time to load history before they can accept new messages — calls arriving during replay get rejected.
• Dependency warmup: Tools that connect to external services (database, search index) need their own connection pools established.
• Cron race condition: If a cron job fires during the restart window, it hits the gateway before it's ready and fails — and depending on config, may not retry.

Debugging Steps:

1. Check gateway uptime: docker inspect --format={{.State.StartedAt}} <agent> — was it recently restarted?
2. Check gateway logs for "ready" message: docker logs <agent> --tail 30 — look for "gateway ready" or "session loaded" entries
3. Retry after 60 seconds: Most gateways are fully operational within 30-60 seconds of container start

Fix:

• Immediate: Wait 60 seconds and retry — the gateway will be ready
• For cron jobs: Use staggerMs or a startup delay to avoid firing during the cold-start window
• For bridge callers: Implement a retry with backoff (2-3 retries, 10-30 second intervals) for 502 responses
• For operators: Check the gateway's "ready" log line before assuming it's fully operational

Symptoms:

• Bridge call returns 401 Unauthorized
• Bridge call returns 403 Forbidden
• Advisor freshness gate fails with auth error (not timeout)
• Bridge works from one agent but not another

Root Cause Patterns:

• Expired gateway token: Bridge calls authenticate using the gateway token (gatewayToken). If the target gateway rotated its token, the caller still has the old one.
• Missing token in bridge config: The bridge script reads the token from environment or config — if it's unset, the call goes out without auth.
• Wrong gateway URL: If the target gateway moved to a new port or host, the bridge points at the old address.
• Agent allowlist mismatch: Some gateways restrict which agents can bridge in. If the caller's identity isn't in the allowlist, the gateway rejects the connection.

Debugging Steps:

1. Check the exact error: 401 = bad/missing token, 403 = authenticated but not authorized
2. Verify caller's bridge config: Check gatewayToken and gatewayUrl for the target in the caller's environment
3. Test target gateway health: curl http://<target-host>:<port>/health
4. Compare tokens: The caller's configured token must match the target gateway's current token

Fix:

• Update the caller's bridge config with the current gateway token
• If the target gateway was reconfigured, update the URL in the caller's bridge config
• For allowlist issues: update the target gateway's agents.allowedAgents to include the caller