Skip to content

Troubleshooting — Infrastructure

Common issues and fixes across the Dashecorp agent fleet, CI/CD pipelines, and developer tooling.

Table of Contents

Agent Offline / Not Responding

Pi-E not responding in Discord

Symptom: No reply in #pi-e after several minutes.

Diagnose: 

ssh -i ~/.ssh/rpi-claude claude@100.107.48.17
systemctl status agent-runner
journalctl -u agent-runner -n 50

Fix: 

sudo systemctl restart agent-runner

If the service won't start, check logs: 

journalctl -u agent-runner -n 100 --no-pager

Token expired? See Claude OAuth Token Expiry.

Volt-E not responding

Symptom: No reply in #volt-e; agent-runner process down.

Diagnose: 

ssh -i ~/.ssh/vps-srv1099021 root@100.111.142.118
systemctl status agent-runner
journalctl -u agent-runner -n 50

Fix: 

systemctl restart agent-runner

Review-E not responding

Symptom: No review posted on a dashecorp/* PR (rig-dispatched) or a Stig-Johnny/claude-3 PR (via request-review.yml).

Note: Review-E no longer runs on the Pi4-02 host (retired). It runs in the GCP k3s cluster (europe-north1-b) as a consumer of the Valkey assignments:review-e stream (KEDA-scaled), posting as the review-e-bot[bot] GitHub App.

Diagnose: 

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra
kubectl get pods -n review-e
kubectl logs -n review-e review-e-rig-agent-runtime-0 --tail=200
# Check the assignment stream depth (queue should be ≤ 1 on a healthy rig)
kubectl -n rig-conductor exec valkey-primary-0 -- valkey-cli XLEN assignments:review-e

If the stream is backing up or pods are unhealthy, see the rig-dispatch sections below.

iBuild-E not responding

Symptom: No build triggered on agent-ready issue; #ibuild-e silent.

Diagnose: 

ssh -i ~/.ssh/mac-executor claude@100.92.170.124
launchctl list | grep ibuild
# Check logs
cat ~/Library/Logs/ibuild-e/agent.log | tail -50

Fix: 

launchctl stop ai.invotek.ibuild-e
launchctl start ai.invotek.ibuild-e

rig-conductor not dispatching reviews / assignments

Symptom: No PR/CI notifications in Discord; agents not picking up issues. (Pre-2026-05-20 this was the "ATL-E not posting" section — ATL-E was retired and replaced by rig-conductor + rig-planner.)

Diagnose: 

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra
kubectl get pods -n rig-conductor
kubectl logs -n rig-conductor deploy/rig-conductor-api --tail=200
# Health probe via cluster DNS (ClusterIPs aren't stable across recreates):
kubectl run -n rig-conductor probe --rm -it --image=curlimages/curl -- \
  curl -sS http://rig-conductor-api.rig-conductor.svc.cluster.local:8080/api/health

Webhook delivery failing?

Check the org-level webhook deliveries for https://rig-conductor.dashecorp.com/api/webhook/github (requires admin:org_hook scope on the gh CLI token): 

gh api /orgs/dashecorp/hooks/<id>/deliveries

Argo CD out of sync? 

kubectl patch application rig-conductor -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

GitHub App Token Issues

401 Unauthorized / Bad credentials in agent API calls

Symptom: Agent's GitHub API calls fail with 401. Usually happens after 1 hour.

Cause: GitHub App installation tokens expire after 1 hour.

Fix (manual refresh): 

# On the agent host — script lives at ~/.config/github-apps/get-token.py
# (the credential helper always resolves via $HOME/.config/github-apps/)
TOKEN=$(python3 ~/.config/github-apps/get-token.py pie-agent-bot Stig-Johnny)
# Update gh CLI hosts.yml
python3 -c "
import yaml, sys, os
with open(os.path.expanduser('~/.config/gh/hosts.yml')) as f: config = yaml.safe_load(f)
config['github.com']['oauth_token'] = sys.argv[1]
with open(os.path.expanduser('~/.config/gh/hosts.yml'), 'w') as f: yaml.dump(config, f)
" "$TOKEN"

Long-term fix: Container restart re-runs the entrypoint which refreshes tokens automatically.

git push fails with authentication failed

Symptom: fatal: Authentication failed on push.

Cause: The credential helper is using an expired or missing token.

Fix: 

# Regenerate token and update remote URL
TOKEN=$(python3 ~/.config/github-apps/get-token.py pie-agent-bot Stig-Johnny)
git remote set-url origin "https://x-access-token:${TOKEN}@github.com/Stig-Johnny/REPO.git"
git push

PEM file not found

Symptom: get-token.py exits with FileNotFoundError: pie-agent-bot.pem not found.

Fix: Check PEM paths (resolves via $HOME, so depends on which user the agent runs as): 

ls -la ~/.config/github-apps/*.pem
# Pi-E (user: claude):   /home/claude/.config/github-apps/pie-agent-bot.pem
# Volt-E (user: root): /root/.config/github-apps/volt-e-agent-bot.pem

If missing, search for entry name pie-agent-bot in the Invotek vault — it has the PEM attached. Recover manually or via bw CLI: 

bw get item "pie-agent-bot" | jq '.attachments'

declare -A error in get-token.sh (Mac only)

Symptom: declare: -A: invalid option on macOS.

Cause: macOS ships bash 3.2 which doesn't support associative arrays.

Fix: Use the Python script instead (path is consistent across agents): 

python3 ~/.config/github-apps/get-token.py pie-agent-bot Stig-Johnny

Claude OAuth Token Expiry

Agent says "Authentication required" or stops responding

Symptom: Agent posts an auth error, or the agent log shows No authentication token found.

Background: Agents authenticate with a single 1-year Claude Max OAuth token (generated via claude setup-token, stored in Bitwarden as "Claude Code OAuth Token (1-Year, Mac Mini)", expires 2027-03-26, deadline tracker #17). There is no automated push/sync or keepalive cron. A monitoring-only watchdog on Pi-E (token-watchdog.sh, cron every 5 min) only alerts on expiry; it never writes tokens.

Primary fix: The token rarely expires within its 1-year life. If an agent reports an auth error, rotate the 1-year token per the deploy-agent-runner skill (Phase 8: Claude Auth) and reload the service.

Manual fix: rotate the single 1-year OAuth token per the deploy-agent-runner skill (Phase 8: Claude Auth) — regenerate with claude setup-token, store it in Bitwarden + the agent's env, then reload the service (Linux: sudo systemctl restart agent-runner; Mac: launchctl unload/load ~/Library/LaunchAgents/ai.invotek.<agent>.plist).

Emergency fix (VNC): 1. Open screen sharing://100.107.48.17 (Pi-E) or 100.111.142.118:PORT (Volt-E) 2. Open a terminal on the host (agent-runner runs as a host process, not a container) 3. Run claude setup-token → follow the browser flow, then reload the service

Discord Bot Issues

Bot offline indicator in server member list

Symptom: Bot shows as offline in Discord.

Cause: Discord token invalid or agent container/service not running.

Fix: 1. Check if the agent is running (see Agent Offline) 2. If running but bot still offline, token may have been rotated. Reset token in Discord Developer Portal, update Bitwarden, then update the token on the agent host and restart the agent — see How to Recreate a Bot Token for the exact per-runtime steps (agent-runner .env/.env.<agent>, or rig-gitops SOPS for Review-E).

Bot missing Message Content Intent error in logs

Symptom: Bot receives messages but body is empty; message.content is always "".

Fix: 1. Go to Discord Developer Portal → application → Bot tab 2. Enable Message Content Intent under Privileged Gateway Intents 3. Save (no restart needed — intent propagates within seconds)

Mentions not working in workflows

Symptom: @Pi-E in a workflow or script doesn't ping the bot.

Fix: Use the Bot User ID format: <@1477270576765735024>. Plain @Pi-E does not work programmatically. See docs/infrastructure/agents.md for all bot IDs.

CI / GitHub Actions Failures

YAML Lint — "too many spaces inside braces"

Symptom: CI fails with [error] too many spaces inside braces (braces) in a workflow file.

Fix: Change ${{ expression }} → remove extra spaces: ${{ expr }}. The yamllint braces rule requires exactly zero spaces inside { and } in YAML mappings (GitHub Actions template syntax is an exception but yamllint flags it).

To disable for a specific file, add to the YAML lint config: 

rules:
  braces:
    forbid-flow-sequences: false
    min-spaces-inside: 0
    max-spaces-inside: 1

Markdown Lint — MD025 multiple H1 headings

Symptom: MD025/single-title/single-h1 Multiple top-level headings on a docs file.

Cause: File has a YAML front matter block with a title field AND a # Heading at the top — markdownlint counts both.

Fix: Either remove the front matter title field, or use ## Heading (H2) for the first markdown heading.

Auto-merge PR check fails

Symptom: Auto-merge PR check fails, blocking merge.

Cause: This check depends on all other checks passing. It is a downstream indicator, not a root cause.

Fix: Look at the other failing checks first (lint, tests, security scan). Auto-merge will succeed once all required checks pass.

iOS Unit Tests fail with "Package.resolved out of date"

Symptom: CI fails: Package.resolved is out of date! Xcode Cloud will fail.

Fix: 

cd ios
xcodegen generate
xcodebuild -resolvePackageDependencies -project RewardE.xcodeproj -scheme RewardE \
  -skipPackagePluginValidation CODE_SIGNING_ALLOWED=NO
cp RewardE.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved Package.resolved
git add Package.resolved
git commit -m "chore: update Package.resolved"

release-please PR stuck / not updating

Symptom: Release PR (#N) was merged but a new one wasn't created, or the version wasn't bumped.

Fix: 1. Check release-please workflow run logs for the last push to main 2. Ensure release-please-manifest.json and release-please-config.json exist and are valid 3. If releases are stuck at an old version, manually push a chore: release-please trigger commit

Docker Container Issues

Container won't start — Permission denied on workspace files

Symptom: An agent fails to write to its workspace; logs show permission denied accessing files under ~/repos/claude-3/ (e.g. /home/claude/repos/claude-3/).

Cause: Workspace files are owned by a different user than the one the agent-runner process runs as (e.g. the repo was cloned or a command was run as root, but the service runs as claude). Under agent-runner there is no container — the agent runs as a host process, so files must be owned by the agent's host user.

Fix: 

# On the host (as the agent user or root). Pi-E runs as `claude`:
sudo chown -R claude:claude /home/claude/repos/claude-3
sudo systemctl restart agent-runner

git: detected dubious ownership errors

Symptom: git commands fail with detected dubious ownership in repository.

Fix: 

git config --global --add safe.directory /path/to/repo

Add the specific repo path. This happens when the repo was cloned as root but git runs as node (1000).

Git / Push Errors

Force push protection error

Symptom: remote: error: GH006: Protected branch update failed, force push is not allowed.

Fix: Never force-push. Rebase or merge instead. If you've accidentally amend-committed a pushed branch, create a new commit with the correction.

fatal: unable to access / TLS handshake timeout

Symptom: Git operations time out on the Pi.

Cause: Network interruption, or Tailscale not connected.

Fix: 

# Check Tailscale
tailscale status
# If down:
sudo tailscale up

Markdown / YAML Lint Failures

Quick fix for common markdownlint errors

Error Fix
MD025: Multiple H1 Use ## for first body heading if front matter has title:
MD012: Multiple blank lines Remove extra blank lines (max 1 consecutive)
MD009: Trailing spaces sed -i 's/ \+$//' file.md
MD013: Line too long Wrap at 120 chars or add <!-- markdownlint-disable MD013 -->
MD041: First line not H1 Add front matter title: or make first line # Title

Disable markdownlint for a block

<!-- markdownlint-disable MD025 -->
# Second heading that would trigger MD025
<!-- markdownlint-enable MD025 -->

Escalation

Issue Escalate to
Claude OAuth token can't be refreshed Stig-Johnny (VNC login or new token push)
GitHub App PEM lost Stig-Johnny (Bitwarden recovery)
Tailscale node disconnected (Pi offline) Stig-Johnny (physical access or reboot)
Production data concern Stig-Johnny immediately
Bitwarden vault access lost Stig-Johnny (recovery codes)
App Store review rejected Stig-Johnny (human decision required)

Post in #admin with: what's broken, what you tried, error message/logs.

Rig dispatch silently broken — symptoms & cleanup (2026-05-09)

The rig can look healthy (pods running, fresh heartbeats, queues empty) while actually being dead at the dispatch layer — agents sit idle, agent-ready issues never reach an executor, no PRs ship for days.

Quick triage (under 2 minutes)

# 1) How many agent-ready issues across the rig?
for repo in dashe-reward-e dashe-fast-e dashe-nutri-e dashe-count-e dashe-drink-e dashe-heart-e \
            rig-agent-runtime rig-gitops rig-conductor rig-docs infra; do
  hits=$(gh issue list --repo dashecorp/$repo --state open --label "agent-ready" --json number --jq 'length')
  [ "$hits" -gt 0 ] && echo "  $repo: $hits"
done

# 2) Last agent-authored PR — should be < 24h on a healthy rig
gh pr list --repo dashecorp/dashe-reward-e --state all --limit 5 \
  --json author,createdAt,title \
  --jq '.[] | select(.author.login | test("dev-e|review-e|ibuild-e|planner")) | "\(.createdAt[0:10]) \(.title[0:60])"' | head -3

# 3) Stream depths + stuck consumers
gcloud compute ssh invotek-k3s --zone=europe-north1-b --command="
  for s in assignments:dev-e-node assignments:dev-e-dotnet assignments:dev-e-python assignments:review-e assignments:rig-planner assignments:ibuild-e; do
    len=\$(kubectl -n rig-conductor exec valkey-primary-0 -- valkey-cli XLEN \$s)
    pen=\$(kubectl -n rig-conductor exec valkey-primary-0 -- valkey-cli XPENDING \$s agents 2>&1 | head -1)
    echo \"\$s len=\$len pending=\$pen\"
  done
"

A healthy rig: agent-ready issues processed within minutes, recent agent PRs visible, queue depth ≤ 1, pending = 0 on every stream.

Symptom: stuck consumers (XPENDING > 0 with idle in hours/days)

Cause: an agent consumer claimed a stream message and was killed/restarted before ACK. Typical culprit is slo-reclaimer — a watchdog that reclaims stuck messages but can itself get stuck.

Fix:

gcloud compute ssh invotek-k3s --zone=europe-north1-b --command="
  for s in assignments:dev-e-node assignments:dev-e-dotnet assignments:dev-e-python \
            assignments:review-e assignments:rig-planner assignments:ibuild-e; do
    kubectl -n rig-conductor exec valkey-primary-0 -- valkey-cli XPENDING \$s agents - + 50 2>&1 \
      | awk 'NR%4==1 && /^[0-9]/{print \$1}' \
      | while read id; do
          [ -n \"\$id\" ] && kubectl -n rig-conductor exec valkey-primary-0 -- valkey-cli XACK \$s agents \$id
        done
  done
"

Symptom: empty-payload assignments published in a tight loop

Stream entries with repo="", issueNumber=0, title="", body="". Agents pull them, claude-cli returns category=idle, $0.05–0.08 burned per turn.

Cause: corrupt mt_doc_issuestatus rows in Postgres — ReconciliationService and SloEnforcerService keep republishing them every tick. Common origin: an event lands on an issue stream before any ISSUE_APPROVED (e.g., an EpicDecomposed event or webhook race), Marten's projection auto-creates a default-empty IssueStatus and Apply methods set state=queued without populating Repo/IssueNumber.

Find them:

gcloud compute ssh invotek-k3s --zone=europe-north1-b --command="
  kubectl -n rig-conductor exec postgres-0 -- psql -U conductor -d rig_conductor -c \"
    SELECT id, data->>'State' as state, COUNT(*) over() as total
    FROM mt_doc_issuestatus
    WHERE length(COALESCE(data->>'Repo','')) < 5 OR (data->>'IssueNumber')::int = 0;
  \"
"

Delete them (safe — they have no real Repo/IssueNumber to link back to GitHub):

gcloud compute ssh invotek-k3s --zone=europe-north1-b --command="
  kubectl -n rig-conductor exec postgres-0 -- bash -c \"psql -U conductor -d rig_conductor -c \\\"
    DELETE FROM mt_doc_issuestatus
    WHERE length(COALESCE(data->>'Repo','')) < 5 OR (data->>'IssueNumber')::int = 0;
  \\\"\"
"

Then trim the assignment streams to clear the published backlog:

gcloud compute ssh invotek-k3s --zone=europe-north1-b --command="
  for s in assignments:dev-e-node assignments:dev-e-dotnet assignments:dev-e-python \
            assignments:review-e assignments:rig-planner; do
    kubectl -n rig-conductor exec valkey-primary-0 -- valkey-cli XTRIM \$s MAXLEN 1
  done
"

Defensive fix tracked in dashecorp/rig-conductor#716.

Symptom: KEDA ScaledObject READY: False

Look at kubectl get scaledobject -A. If READY=False with event message ScaledObject doesn't have correct Idle/Min/Max Replica Counts specification, KEDA stops managing the StatefulSet — replicas revert to chart default (often 1) — pods you thought were paused stay running.

KEDA validation requires maxReplicaCount >= 1. To pause a deployment, don't set max=0 — use the paused-replicas annotation:

metadata:
  name: <so-name>
  annotations:
    autoscaling.keda.sh/paused-replicas: "0"
spec:
  minReplicaCount: 0
  maxReplicaCount: 1   # KEDA-valid; effective replicas pinned to 0 by annotation

To unpause: remove the annotation.

Symptom: planner files children that get re-decomposed forever

Pattern: planner files parent epic + N children correctly. Then planner re-runs on each child and files M sub-children of those, ad infinitum. Caused by:

  1. Conductor dispatches every agent-ready issue to planner first. Children get re-dispatched.
  2. Planner's "epic intent" detector fires on child bodies (multiple acceptance criteria, mention of "epic" in the parent reference).
  3. No self-author guard.

Fixed by rig-agent-runtime#275 (self-author short-circuit) + #278 (child-labeling fires issues.labeled webhook so dispatch goes to dev-e not back to planner). If symptom recurs, verify the planner image SHA pinned in rig-gitops includes both PRs.

Symptom: TestFlight shows wrong build (e.g. operator's last seen build is older than several uploaded ones)

Several causes today:

Visible Real
TestFlight defaults to highest build number The highest may be a stale Xcode Cloud build (pre-migration). Tap version → pick the build that came from build-and-upload.yml
Builds uploaded but not in Testers group Apple's auto-distribution from hasAccessToAllBuilds=True is best-effort, not real-time. Verify with: gh api ... (see appstoreconnect skill)
Tester state=INVITED not INSTALLED Their invite was sent but never accepted. Re-send via API (see appstoreconnect).

Disable any monitor-xcode-cloud.yml workflow on a repo migrated to GitHub-Actions iOS CI — it'll keep alarming on absent Xcode Cloud builds.

Symptom: build-and-upload.yml succeeded but TestFlight build never appeared

altool returns exit 0 even on ENTITY_ERROR.ATTRIBUTE.INVALID.DUPLICATE (build number collides with an existing build). The fix landed 2026-05-09: dashe-reward-e's workflow now greps altool output for ENTITY_ERROR / UPLOAD FAILED / Validation failed and exits non-zero. Build number is auto-incremented from the latest in ASC instead of relying on project.yml's CURRENT_PROJECT_VERSION (which drifts).

If you see a duplicate-build error today: it's almost always two concurrent workflow runs (auto-trigger from merge + manual workflow run) racing. Cancel one.

Symptom: iBuild-E "completed" with 0 turns and no PR

Two failure modes today, both same outward signature:

  1. Auth failure (claude-cli reports Invalid API key then exits clean). Plist's ANTHROPIC_API_KEY is the leaked/revoked one. See ibuild-e skill — fix is sudo -n launchctl bootout/bootstrap after replacing the key.

  2. Workspace fallback (claude-cli runs many turns in ~/repos/star-rewards/ instead of dashecorp/dashe-reward-e, ends without ability to push). task-workspace script missing or WORKSPACE_ROOT not in plist. See ibuild-e skill for required setup.

gcloud compute ssh to GCP rig host

If a runbook tells you to SSH to 100.126.108.70 and you don't have a key — use:

gcloud compute ssh invotek-k3s --zone=europe-north1-b --command="<your command>"

The operator's gcloud OAuth is configured against project invotek-github-infra. No SSH key needed; gcloud injects a temporary key. Don't waste time on ssh root@100.126.108.70 (no authorized key) or tailscale ssh (not enabled on the node).

Long-term cleanup tracked in dashecorp/infra#219.