Operations¶

Deployment¶

Cluster migrated: The rig was on the Dell k3s vCluster (Oslo) pre-2026-03-27 and migrated to GCP k3s (europe-north1-b). Most operational commands below now target the GCP cluster; legacy Dell vCluster bootstrap is kept as historical reference but is no longer the deployment target.

Prerequisites (current — GCP k3s)¶

GCP k3s control plane reachable via gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra
~/.kube/k3s-gcp-config kubeconfig configured locally
FluxCD installed on the cluster (root Kustomization in flux-system namespace)
SOPS + GCP KMS for secret decryption (no SealedSecrets on GCP)

Legacy: Create Rig vCluster on Dell k3s (pre-2026-03-27)¶

# SSH to Dell
ssh -i ~/.ssh/dell-stig-1 claude@100.95.212.93

# Create vCluster via vCluster Platform CLI or API
vcluster create rig --namespace rig-vcluster

# Get kubeconfig
vcluster connect rig --namespace rig-vcluster -- kubectl get ns

Bootstrap FluxCD¶

# In dashecorp/rig-gitops repo (org owner — drop --personal)
flux bootstrap github \
  --owner=dashecorp \
  --repository=rig-gitops \
  --branch=main \
  --path=clusters/rig

Deploy Components¶

FluxCD reconciles from rig-gitops:

rig-gitops/
├── clusters/rig/
│   └── flux-system/          # FluxCD bootstrap
├── base/
│   ├── namespaces.yaml       # conductor, dev-agents
│   └── postgres/             # PostgreSQL StatefulSet
├── apps/
│   ├── rig-conductor/        # rig-conductor Deployment + Service (was atl-e/ pre-2026-05-20)
│   ├── rig-planner/          # rig-planner Deployment + Service
│   └── dev-e/                # Dev-E Deployment
└── secrets/
    ├── conductor-secrets.yaml  # SealedSecret
    └── dev-e-secrets.yaml      # SealedSecret
    └── ghcr-pull.yaml          # Image pull secret

Monitoring¶

Health Checks¶

# Conductor health
curl https://rig.dashecorp.com/health

# Agent status
curl https://rig.dashecorp.com/api/agents

# Queue status
curl https://rig.dashecorp.com/api/queue

Logs¶

# SSH to the GCP k3s control plane (was Dell + vCluster pre-2026-03-27)
gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra

# rig-conductor logs (was app=atl-e-conductor pre-2026-05-20)
kubectl logs -n rig-conductor -l app=rig-conductor-api --tail=50

# Dev-E logs
kubectl logs -n dev-e -l app=dev-e --tail=50

# PostgreSQL logs (lives in rig-conductor namespace alongside the API)
kubectl logs -n rig-conductor -l app=postgres --tail=30

Discord Notifications¶

rig-conductor posts to these channels:

Channel	What
#tasks	Assignments, PR created, merged
#dev-e	Dev-E progress, errors
#admin	Escalations, human gates, agent offline

Troubleshooting¶

Dev-E Not Picking Up Work¶

Check agent health: curl rig.dashecorp.com/api/agents
Check pod status: kubectl get pods -n dev-agents
Check logs: kubectl logs -n dev-agents -l app=dev-e --tail=20
Check Claude OAuth token expiry (deadline-tracker #17)

Assignment Stuck¶

Check Conductor queue: curl rig.dashecorp.com/api/queue
Check if issue has unmet dependencies
Check if issue is human-gated
Check assignment status in Conductor logs

Claude Code Hangs¶

Dev-E has a per-issue timeout (default: 30 minutes). If Claude Code hangs:

Pod restarts via liveness probe failure
rig-conductor marks assignment as failed
On second failure: escalates to human

Review Loop¶

If Dev-E keeps iterating on review feedback without converging:

Max review iterations: 3
After 3 iterations without approval: mark as failed, escalate

Database Recovery¶

PostgreSQL uses a PVC. If data is lost:

# Connect to PostgreSQL
kubectl exec -it -n conductor postgres-0 -- psql -U conductor

# Check state
SELECT * FROM assignments WHERE status = 'assigned';
SELECT * FROM agents;

Updating Components¶

Update rig-conductor¶

Push changes to dashecorp/rig-conductor → CI builds image (Stig-Johnny/atl-agent was the pre-2026-05-20 source repo)
Update image tag in dashecorp/rig-gitops/apps/rig-conductor/deployment.yaml (was apps/atl-e/ pre-2026-05-20)
Push to dashecorp/rig-gitops → FluxCD reconciles

Update Dev-E¶

Push changes to Stig-Johnny/dev-e → CI builds image
Update image tag in rig-gitops/apps/dev-e/deployment.yaml
Push to rig-gitops → FluxCD reconciles

Rotate Secrets¶

# Generate new SealedSecret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# Commit to rig-gitops
git add secrets/ && git commit -m "chore: rotate secrets" && git push

Cloudflare Access¶

Dashboard at rig.dashecorp.com is protected by Cloudflare Access:

Policy: Allow invotekas@gmail.com
Tunnel: Configured via dashecorp/rig-gitops Cloudflare Tunnel config (Stig-Johnny/cluster-gitops archived 2026-05-04)
Zero Trust: No public access without authentication