Troubleshooting — Infrastructure¶

Common issues and fixes across the Dashecorp agent fleet, CI/CD pipelines, and developer tooling.

Table of Contents¶

Agent Offline / Not Responding
GitHub App Token Issues
Claude OAuth Token Expiry
Discord Bot Issues
CI / GitHub Actions Failures
Docker Container Issues
Git / Push Errors
Markdown / YAML Lint Failures
Escalation

Agent Offline / Not Responding¶

Pi-E not responding in Discord¶

Symptom: No reply in #pi-e after several minutes.

Diagnose:

ssh -i ~/.ssh/rpi-claude claude@100.107.48.17
docker ps | grep openclaw

Fix:

cd /home/claude/openclaw
docker compose restart

If container won't start, check logs:

docker compose logs --tail=50 openclaw-gateway

Token expired? See Claude OAuth Token Expiry.

Volt-E not responding¶

Symptom: No reply in #volt-e; agent-runner process down.

Diagnose:

ssh -i ~/.ssh/vps-srv1099021 root@100.111.142.118
systemctl status agent-runner
journalctl -u agent-runner -n 50

Fix:

systemctl restart agent-runner

Review-E not responding¶

Symptom: No review posted after >30min on a PR that triggered request-review.yml.

Diagnose:

ssh -i ~/.ssh/rpi-pi4-02 claude@100.77.12.75
systemctl status agent-runner
journalctl -u agent-runner -n 50

Fix:

systemctl restart agent-runner

If stuck mid-review, check /var/log/agent-runner/ for the active session.

iBuild-E not responding¶

Symptom: No build triggered on agent-ready issue; #ibuild-e silent.

Diagnose:

ssh -i ~/.ssh/mac-executor claude@100.92.170.124
launchctl list | grep ibuild
# Check logs
cat ~/Library/Logs/ibuild-e/agent.log | tail -50

Fix:

launchctl stop ai.invotek.ibuild-e
launchctl start ai.invotek.ibuild-e

ATL-E not posting notifications¶

Symptom: No PR/CI notifications from ATL-E in #tasks.

Diagnose:

ssh -i ~/.ssh/dell-stig-1 claude@100.95.212.93
kubectl get cronjob atl-agent -n atl-agent
kubectl get pods -n atl-agent --sort-by=.metadata.creationTimestamp | tail -5
kubectl logs -n atl-agent <most-recent-pod>

Is it suspended?

kubectl get cronjob atl-agent -n atl-agent -o jsonpath='{.spec.suspend}'
# If true: unsuspend
kubectl patch cronjob atl-agent -n atl-agent -p '{"spec":{"suspend":false}}'

ArgoCD out of sync?

kubectl patch application atl-agent -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

GitHub App Token Issues¶

`401 Unauthorized` / `Bad credentials` in agent API calls¶

Symptom: Agent's GitHub API calls fail with 401. Usually happens after 1 hour.

Cause: GitHub App installation tokens expire after 1 hour.

Fix (manual refresh):

# On the agent host — script lives at ~/.config/github-apps/get-token.py
# (On Pi-E the source is /home/node/.openclaw/github-apps/get-token.py;
#  the credential helper always resolves via $HOME/.config/github-apps/)
TOKEN=$(python3 ~/.config/github-apps/get-token.py pie-agent-bot Stig-Johnny)
# Update gh CLI hosts.yml
python3 -c "
import yaml, sys, os
with open(os.path.expanduser('~/.config/gh/hosts.yml')) as f: config = yaml.safe_load(f)
config['github.com']['oauth_token'] = sys.argv[1]
with open(os.path.expanduser('~/.config/gh/hosts.yml'), 'w') as f: yaml.dump(config, f)
" "$TOKEN"

Long-term fix: Container restart re-runs the entrypoint which refreshes tokens automatically.

`git push` fails with `authentication failed`¶

Symptom: fatal: Authentication failed on push.

Cause: The credential helper is using an expired or missing token.

Fix:

# Regenerate token and update remote URL
TOKEN=$(python3 ~/.config/github-apps/get-token.py pie-agent-bot Stig-Johnny)
git remote set-url origin "https://x-access-token:${TOKEN}@github.com/Stig-Johnny/REPO.git"
git push

PEM file not found¶

Symptom: get-token.py exits with FileNotFoundError: pie-agent-bot.pem not found.

Fix: Check PEM paths (resolves via $HOME, so depends on which user the agent runs as):

ls -la ~/.config/github-apps/*.pem
# Pi-E (user: node):   /home/node/.config/github-apps/pie-agent-bot.pem
# Volt-E (user: root): /root/.config/github-apps/volt-e-agent-bot.pem

If missing, search for entry name pie-agent-bot in the Invotek vault — it has the PEM attached. Recover manually or via bw CLI:

bw get item "pie-agent-bot" | jq '.attachments'

`declare -A` error in get-token.sh (Mac only)¶

Symptom: declare: -A: invalid option on macOS.

Cause: macOS ships bash 3.2 which doesn't support associative arrays.

Fix: Use the Python script instead (path is consistent across agents):

python3 ~/.config/github-apps/get-token.py pie-agent-bot Stig-Johnny

Claude OAuth Token Expiry¶

Agent says "Authentication required" or stops responding¶

Symptom: Agent posts an auth error, or docker logs shows No authentication token found.

Background: Remote agents (Pi-E, Volt-E, Review-E) use Claude Max OAuth tokens that self-refresh via a 3-day keepalive cron. Token lifetime is ~90 days but cron extends it automatically.

Primary fix (automatic): Wait for the M4 MacBook's push script (com.invotek.push-claude-creds LaunchAgent) to push a fresh token. It runs every 5 minutes.

Manual fix:

# From MacBook Air M4
~/.local/bin/push-claude-creds.sh

Emergency fix (VNC): 1. Open screen sharing://100.107.48.17 (Pi-E) or 100.111.142.118:PORT (Volt-E) 2. Open terminal inside container 3. Run claude auth login → follow browser flow

Discord Bot Issues¶

Bot offline indicator in server member list¶

Symptom: Bot shows as offline in Discord.

Cause: Discord token invalid or agent container/service not running.

Fix: 1. Check if the agent is running (see Agent Offline) 2. If running but bot still offline, token may have been rotated. Reset token in Discord Developer Portal, update Bitwarden, update OpenClaw config, restart agent.

Bot missing Message Content Intent error in logs¶

Symptom: Bot receives messages but body is empty; message.content is always "".

Fix: 1. Go to Discord Developer Portal → application → Bot tab 2. Enable Message Content Intent under Privileged Gateway Intents 3. Save (no restart needed — intent propagates within seconds)

Mentions not working in workflows¶

Symptom: @Pi-E in a workflow or script doesn't ping the bot.

Fix: Use the Bot User ID format: <@1477270576765735024>. Plain @Pi-E does not work programmatically. See docs/infrastructure/agents.md for all bot IDs.

CI / GitHub Actions Failures¶

YAML Lint — "too many spaces inside braces"¶

Symptom: CI fails with [error] too many spaces inside braces (braces) in a workflow file.

Fix: Change ${{ expression }} → remove extra spaces: ${{ expr }}. The yamllint braces rule requires exactly zero spaces inside { and } in YAML mappings (GitHub Actions template syntax is an exception but yamllint flags it).

To disable for a specific file, add to the YAML lint config:

rules:
  braces:
    forbid-flow-sequences: false
    min-spaces-inside: 0
    max-spaces-inside: 1

Markdown Lint — MD025 multiple H1 headings¶

Symptom: MD025/single-title/single-h1 Multiple top-level headings on a docs file.

Cause: File has a YAML front matter block with a title field AND a # Heading at the top — markdownlint counts both.

Fix: Either remove the front matter title field, or use ## Heading (H2) for the first markdown heading.

Auto-merge PR check fails¶

Symptom: Auto-merge PR check fails, blocking merge.

Cause: This check depends on all other checks passing. It is a downstream indicator, not a root cause.

Fix: Look at the other failing checks first (lint, tests, security scan). Auto-merge will succeed once all required checks pass.

iOS Unit Tests fail with "Package.resolved out of date"¶

Symptom: CI fails: Package.resolved is out of date! Xcode Cloud will fail.

Fix:

cd ios
xcodegen generate
xcodebuild -resolvePackageDependencies -project RewardE.xcodeproj -scheme RewardE \
  -skipPackagePluginValidation CODE_SIGNING_ALLOWED=NO
cp RewardE.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved Package.resolved
git add Package.resolved
git commit -m "chore: update Package.resolved"

`release-please` PR stuck / not updating¶

Symptom: Release PR (#N) was merged but a new one wasn't created, or the version wasn't bumped.

Fix: 1. Check release-please workflow run logs for the last push to main 2. Ensure release-please-manifest.json and release-please-config.json exist and are valid 3. If releases are stuck at an old version, manually push a chore: release-please trigger commit

Docker Container Issues¶

Container won't start — `Permission denied` on workspace files¶

Symptom: docker compose up fails; logs show permission denied accessing /home/node/.openclaw/workspace/.

Cause: Files in the workspace are owned by root (from a previous run as root inside the container).

Fix:

# On the Pi host (as claude or root)
sudo chown -R 1000:1000 /home/claude/openclaw/workspace/
docker compose restart

`git: detected dubious ownership` errors¶

Symptom: git commands fail with detected dubious ownership in repository.

Fix:

git config --global --add safe.directory /path/to/repo

Add the specific repo path. This happens when the repo was cloned as root but git runs as node (1000).

Git / Push Errors¶

Force push protection error¶

Symptom: remote: error: GH006: Protected branch update failed, force push is not allowed.

Fix: Never force-push. Rebase or merge instead. If you've accidentally amend-committed a pushed branch, create a new commit with the correction.

`fatal: unable to access` / TLS handshake timeout¶

Symptom: Git operations time out on the Pi.

Cause: Network interruption, or Tailscale not connected.

Fix:

# Check Tailscale
tailscale status
# If down:
sudo tailscale up

Markdown / YAML Lint Failures¶

Quick fix for common markdownlint errors¶

Error	Fix
MD025: Multiple H1	Use `##` for first body heading if front matter has `title:`
MD012: Multiple blank lines	Remove extra blank lines (max 1 consecutive)
MD009: Trailing spaces	`sed -i 's/ \+$//' file.md`
MD013: Line too long	Wrap at 120 chars or add `<!-- markdownlint-disable MD013 -->`
MD041: First line not H1	Add front matter `title:` or make first line `# Title`

Disable markdownlint for a block¶

<!-- markdownlint-disable MD025 -->
# Second heading that would trigger MD025
<!-- markdownlint-enable MD025 -->

Escalation¶

Issue	Escalate to
Claude OAuth token can't be refreshed	Stig-Johnny (VNC login or new token push)
GitHub App PEM lost	Stig-Johnny (Bitwarden recovery)
Tailscale node disconnected (Pi offline)	Stig-Johnny (physical access or reboot)
Production data concern	Stig-Johnny immediately
Bitwarden vault access lost	Stig-Johnny (recovery codes)
App Store review rejected	Stig-Johnny (human decision required)

Post in #admin with: what's broken, what you tried, error message/logs.

Troubleshooting — Infrastructure¶

Table of Contents¶

Agent Offline / Not Responding¶

Pi-E not responding in Discord¶

Volt-E not responding¶

Review-E not responding¶

iBuild-E not responding¶

ATL-E not posting notifications¶

GitHub App Token Issues¶

401 Unauthorized / Bad credentials in agent API calls¶

git push fails with authentication failed¶

PEM file not found¶

declare -A error in get-token.sh (Mac only)¶

Claude OAuth Token Expiry¶

Agent says "Authentication required" or stops responding¶

Discord Bot Issues¶

Bot offline indicator in server member list¶

Bot missing Message Content Intent error in logs¶

Mentions not working in workflows¶

CI / GitHub Actions Failures¶

YAML Lint — "too many spaces inside braces"¶

Markdown Lint — MD025 multiple H1 headings¶

Auto-merge PR check fails¶

iOS Unit Tests fail with "Package.resolved out of date"¶

release-please PR stuck / not updating¶

Docker Container Issues¶

Container won't start — Permission denied on workspace files¶

git: detected dubious ownership errors¶

Git / Push Errors¶

Force push protection error¶

fatal: unable to access / TLS handshake timeout¶

Markdown / YAML Lint Failures¶

Quick fix for common markdownlint errors¶

Disable markdownlint for a block¶

Escalation¶

`401 Unauthorized` / `Bad credentials` in agent API calls¶

`git push` fails with `authentication failed`¶

`declare -A` error in get-token.sh (Mac only)¶

`release-please` PR stuck / not updating¶

Container won't start — `Permission denied` on workspace files¶

`git: detected dubious ownership` errors¶

`fatal: unable to access` / TLS handshake timeout¶