Kubernetes

Two separate k8s environments managed by GitOps.

There are TWO clusters reconciling this GitOps repo

Read this before adding a manifest under apps/. dashecorp/rig-gitops is reconciled by two Kubernetes clusters, and clusters/ in that repo is the whole check:

Cluster	Where	Reconciles	Nodes
GCP rig (invotek-k3s)	GCE, europe-north1-b	clusters/gcp/ — the production rig	3 (1 control plane + 2 workers)
Dell (dell-stig)	Ålesund, no hands-on access	clusters/dell/: apps/buzz, apps/finance-dashboard (plus Stig-Johnny/finance-brain) and an image-cleanup CronJob	1

The two clusters share no apps/ path since 2026-10-02 (rig-gitops#888, #892). Until then clusters/dell/ also pointed at the production apps/rig-conductor, apps/dev-e, apps/review-e and apps/keda, so anything added there deployed to both. A CronJob written for the GCP rig on 2026-09-23 was running in Ålesund minutes later, alerting into Discord #admin, with nobody intending it. Check clusters/ before pointing a second cluster at an existing path. Scheduler messages are the quickest tell of which cluster spoke: GCP says 0/3 nodes are available, the Dell says 0/1.

Dell k3s cluster (Ålesund)

dell-stig is a Dell Precision 5520 laptop: single node, k3s v1.36.5+k3s1 (since 2026-10-02) on Ubuntu 24.04.5, kernel 6.8.0-146. ssh -i ~/.ssh/dell-stig-1 claude@100.95.212.93.

What it runs (after the 2026-10-02 cleanup): - Flux, with 4 Kustomizations from clusters/dell/: flux-system, buzz, finance-dashboard-config and finance-dashboard. The last one builds Stig-Johnny/finance-brain. - Apps: buzz (relay, MinIO, Postgres, Redis), finance-dashboard, and the image-cleanup CronJob. - kube-system: CoreDNS, Traefik, metrics-server, local-path and nfs-subdir-external-provisioner. - On the host: k3s, tailscaled, ssh, and buzz-relay-port-forward.service, a kubectl port-forward that publishes buzz-relay on 100.95.212.93:13000. - Size: 8 namespaces and 23 pods. - Keys: it still holds production's sops-age key, because buzz and finance-dashboard-config decrypt with it.

The 2026-10-02 cleanup. The operator approved it group by group, one question at a time. - The stale rig. clusters/dell/ reconciled production's apps/rig-conductor, apps/dev-e, apps/review-e and apps/keda, plus a hand-applied planner on the stale branch fix/planner-switch-to-codex. - None of it worked, but it posted its own Pending pods to #admin and was one image or volume away from a second set of agents on the shared queues (rig-gitops#816). - rig-gitops#888 and #892 removed it, and the planner was deleted by hand. The record is rig-gitops docs/2026-10-02-dell-drops-the-rig.md. - Dead workloads, removed by hand because they were never in Git: - vCluster Platform (Loft), and the tablez, runthedocs and team-x vclusters; - ArgoCD, Vault, kubernetes-dashboard, Metabase and ai-accountant; - the invotek tunnel pod, and sealed-secrets with its 17 keys; - empty namespaces and leftover RBAC.

Their NAS data stays: nfs-csi archives on delete, and the static PVs were Retain. - Host: code-server and the host cloudflared.service, with both packages purged. The 2 Actions runners for Stig-Johnny/mcp-server-dotnet were deregistered through the API, which settles the open infra#389 decision. - infra: - The never-applied k8s/dell Synology module is retired (dashecorp/infra#547). - The Cloudflare objects of the removed workloads are adopted into OpenTofu and deleted (dashecorp/infra#548). - The stigjohnny.no ones stay, by operator decision; see Networking.

A stale APIService blocks every namespace deletion. Loft's v1.cluster.loft.sh and v1.management.loft.sh had no endpoints once Loft was scaled to 0. The namespace controller's discovery failed (NamespaceDeletionDiscoveryFailure), so conductor-e stayed in Terminating from the spring until they were deleted. Run kubectl get apiservices | grep -v Local before deleting namespaces.

Open risk: buzz's MinIO image has no upstream. - The image: Docker Hub no longer has the minio/minio or minio/mc repositories (both 404 on 2026-10-02). minio/minio:RELEASE.2025-09-07T16-13-09Z therefore exists only in the Dell's containerd cache. - minio-init: buzz/minio-init has been in ImagePullBackOff since 2026-09-14, on a tag that never existed. - Backup: /var/lib/rancher/image-backup/minio-RELEASE.2025-09-07T16-13-09Z.tar (OCI archive, sha256 fcaf11d2…). Restore it with sudo k3s ctr -n k8s.io images import <tar>. - Decision needed: the image source is the operator's call (rig-gitops#884).

NAS volumes, and their 2026-10-01 fix (rig-gitops#882). - Setup (since 2026-10-14, rig-gitops#988): buzz uses the nfs-subdir-external-provisioner (storage class nfs-csi) against the Synology DS412plus. - Flux owns all of it. The provisioner and nfs-csi are in apps/nfs-subdir-external-provisioner/. buzz's four volumes are static PVs on Retain, and each claim names its PV. A reinstall of the Dell (the drive swap) therefore brings buzz back on its existing data. Every adopted object carries prune: disabled, so a revert leaves them in place. - Where it came from: an Argo CD render of chart 4.0.18 on 2025-06-04, on the tailnet address. On 2026-08-01 at 20:56Z it was moved by hand to the LAN address, 192.168.1.116:/volume1/docker, because NFS over the tailnet timed out during buzz's rollout (rig-gitops#695). That timeout was the outage described below. - It stays on the LAN address. The tailnet path works again since infra#525, but from the Dell on 2026-10-14 it read 4–11× slower: a 64 MiB read ran at 7.6/21.0 MB/s (cold/warm), against 82.1/92.2 MB/s on the LAN. Details are in rig-gitops docs/2026-10-14-dell-nas-provisioner-under-flux.md. - Both LAN addresses come from DHCP: the NAS's 192.168.1.116, which buzz's PVs name (and spec.nfs is immutable), and the Dell's 192.168.1.173, on a USB Ethernet adapter. - The export is open: /volume1/docker *, so any device on the home LAN can mount it. In operator-inbox#40 the operator reserves both addresses in the router first, then restricts the export in DSM to the Dell (192.168.1.173). - The outage: the Tailscale policy that went into OpenTofu on 2026-04-13 lets tag:server reach only tag:server, and the NAS is untagged. From then on every NFS mount hung: load about 9,000, 321 processes in state D, the provisioner 0/1. - The fix: dashecorp/infra#525 added host nas and one rule, dell → nas:2049. The mounts resumed by themselves, with no reboot. - Small fix pending: both local-path and nfs-csi are marked as the default storage class. Kubernetes gives a claim with no class the newer of the two, nfs-csi.

Operator plan (2026-10-02). It supersedes rig-gitops#807's "dev/test rig, later". claude-5 owns this thread.

1. Clean up this cluster. Done 2026-10-02, except the MinIO decision.
2. Update the OS and reboot. Done 2026-10-02. apt full-upgrade, then a reboot into kernel 6.8.0-146, booted once through grub-reboot. GRUB's saved default stayed on the old kernel, and a trial-boot check would have rebooted back to it if the tailnet was not online within 10 minutes. It wasn't needed, and it has been removed.
3. The Dell had been up 26 weeks with a kernel reboot pending (/var/run/reboot-required) and 69 packages upgradable. Security updates already come in through unattended-upgrades.
4. There is no disk encryption (/etc/crypttab is empty, root is plain ext4), so a reboot cannot stall on a passphrase. tailscaled, ssh and k3s start at boot.
5. Upgrade k3s one minor version at a time. Done 2026-10-02: v1.32.5 → v1.32.13 (the etcd 3.5.26 bridge) → v1.33.13 → v1.34.12 → v1.35.9 → v1.36.5, with Flux moved to v2.9.3 before v1.36 (dashecorp/rig-gitops#894). Each step was a PR bumping var.k3s_version (dashecorp/infra#554, #556, #558, #560 and #562), applied by apply (hosts) and verified. The path and its sources are in dashecorp/infra docs/2026-10-02-dell-k3s-upgrade-path.md. The upgrade research was delegated to the Mac Mini Claude.
6. Move The Rig here. Re-scoped by the operator on 2026-10-02 to a multi-location rig: "Keep GCP on until move is complete. We need to support that we have multiple rig clusters in different locations. It should be possible to choose in rig-cockpit what cluster or rig location we are using." The design and every decision are on dashecorp/infra#508.
7. Model: each tenant lives in exactly one location, and can be moved.
8. Where the map lives: the tenant → location map is in Cloudflare D1, owned by rig-cockpit-worker. Every conductor fences its GitHub writes on it and fails closed.
9. Moves: manual only, from Rig Cockpit. A move is a fresh start: nothing is copied, and the tenant's work state is rebuilt from GitHub.
10. Webhooks: they go through a Worker with a Queue buffer on rig-conductor.dashecorp.com.
11. Images and secrets: GHCR only, and one shared SOPS age key for every location.
12. Before any tenant moves to the Dell: off-site backups.
13. Drain timeout on moves: 15 min, then force (decided 2026-10-02, posted on infra#508).

14. Implementation, for the rig, in order:

    1. rig-gitops#895 (base plus a GCP overlay). Merged as rig-gitops#898; all 17 GCP Flux Kustomizations render byte-identical.
    2. rig-conductor#2561 (fencing). Merged in 8 stacked parts (#2570, #2572, #2575, #2576, #2578, #2580, #2582, #2588).
    3. rig-cockpit-worker#250 (D1 map and moves). Merged: schema #252, read APIs #257, move APIs #259, cron #261, Locations page #263.
    4. rig-cockpit-worker#251 (webhook router). Merged: the shadow router #264 (deployed, 760741d) and the routing policy #267. Live Queue routing itself is not built (checked 2026-10-06):
       • The live PR, rig-cockpit-worker#266, closed unmerged.

       • 267 merged only "part A: the pure routing policy" and closed the live-routing issue #265 with it.

       • On main, src/handlers/webhook-router.js is shadow-only, nothing reads ROUTING_MODE, and wrangler.toml has no queue binding for it.
       • No open issue tracks the build.
    5. dashecorp/infra#565 (Cloudflare pieces). Applied: the queues and rig-conductor-gcp (#567), and the webhook-path route (#569). See Networking → Rig webhook router.
    6. rig-conductor#2562 (reconcile sweep). Merged: #2590, #2592, #2595 and #2597, plus the fix rig-conductor#2601 (merged as #2602): the sweep was marked done before its append, so a failed write was never retried.
    7. rig-gitops#896 (GHCR, and the Dell overlay in standby). Merged so far: PR-1 #901, PR-2 #902 and PR-2b #904 (the CI parity proof, with curl), infra#571 (the Artifact Registry reader for rig-gitops CI), the C2 workflow rig-gitops#906 with its selftest #908, PR-3 rig-gitops#917, and rig-gitops#919 (the parity proof via the workflow token). Plan: docs/2026-10-03-ghcr-and-dell-standby-plan.md.

    8. rig-gitops#897 (the Dell's off-site backups). Merged so far:

       • the backup tools, reviewed and whitelisted: claude-3#1024, #1026, #1030 (sops) and #1034 (openssl), and rig-docs#343, #345, #347 and #349;
       • P1a, the R2 bucket (infra#573, lifecycle fix infra#577; see Networking → Cloudflare R2);
       • P3, the backup image (rig-conductor#2605), and P3b, openssl pinned (#2622). See CI/CD for the image reference;
       • P4 (#2608, #2610), P5 (#2613, #2616) and P6 (#2620, #2626, #2630, #2633): the signed runs with the D5 not_after rule, fetch-run.sh, the drill kit and the runbooks;
       • the image digest bump, rig-conductor#2637.

       All of it is suspended and inert until P9. signing.pub is still an .invalid placeholder until O2b. - Live state (2026-10-03): - Rig Cockpit: the D1 tenant → location map and the Locations page are live in rig-cockpit-worker. - Conductor: fencing and the reconcile sweep, with its fix, are merged but inert until LOCATION is set. - Webhooks: the router runs in shadow mode on rig-conductor.dashecorp.com/api/webhook/github (infra#569, operator-approved, checked with a 200 ping). Live Queue routing is not built yet (step 4 above), so there is no ROUTING_MODE to flip. Building it is agent work; switching it to live is a production change and needs the operator. Superseded: live since 2026-10-15 (rig-cockpit-worker#304). - Phone C2: the credential-ops environment (infra#575) and the merged C2 workflow. See CI/CD → credential-ops. - Approvals MVP (rig-cockpit-worker#268): phase 1 is live, with the waiting-run banner and the operator inbox on the web and on iOS build 15; push waits on the APNs secrets (operator-inbox #5). Phase 2, Face ID approvals, is merged on rig-approve but not deployed. See Operator inbox. - Current focus (since 2026-10-04): the operator asked to "focus on getting everything running on the Dell cluster". The Dell standby overlay (#896 Phase B) is open on rig-gitops as a stack: - #911 (PR-10): merged on 2026-10-04. - #913 (PR-11): a draft, still waiting on the C6 pre-checks (operator-inbox #7). - The conflict: by 2026-10-06 it conflicted with main, because it still carried PR-10's commit from before PR-10 was squash-merged. - The update: it was brought up to date by merging main. A rebase would have needed a force-push, which the auto-mode classifier blocks. D7 was settled in the same PR (below). - Review: the new head 11fd790 is green but needs a fresh Review-E approval, because MergeGate only counts an approval of the current head. The 2026-10-04 approval no longer counts. - #915 (PR-19): a draft stacked on #913. - Update: on 2026-10-06 the new #913 head was merged into it (03bb609), so its dell render invariants CI runs against the KEDA pin. On the old head it rendered PR-11's keda and egress-gw clean: 30 objects, all 10 rules OK. - Next: once #913 merges and #915 retargets to main, it goes ready and needs a fresh approval. - Order: PR-19 can't go ahead of PR-11, because it edits a doc that PR-11 creates. - PR-12 needs only PR-11 now: PR-3, PR-4, D3 and rc#2601 (merged as #2602) are done. PR-13 waits for PR-6 to PR-9 and PR-12's live checks (the plan's dependency list). - D7, settled 2026-10-06 in #913: the Dell's KEDA chart is frozen to 2.21.*. - keda.sh's compatibility table lists KEDA 2.21 for Kubernetes 1.34–1.36. It is the only release that covers the Dell's v1.36.5, and chart 2.21.0 is in the kedacore Helm index. - GCP stays on 2.16. - The GCP render is unchanged apart from a render-harness stub name. - Phase A, #896 (2026-10-06): - C2 ran on 2026-10-06, approved by the operator through credential-ops, and wrote six SOPS-encrypted ghcr-pull-secret files with 7 ENC[ markers each. The token is in Credentials. - PR-3, rig-gitops#917, merged: the six pull secrets are wired into the dashecorp, dev-e, review-e, rig-conductor, rig-planner and stigjohnny namespaces. - rig-gitops#919, merged: the parity proof reads GHCR with the workflow token. A parity run on main proved every own tag has the same digest in Artifact Registry and GHCR (CI/CD). - PR-4, rig-conductor#2688, open (opened with the operator's go): the conductor pulls ghcr.io/dashecorp/rig-conductor with ghcr-pull-secret, and its pin writes GHCR. Publishing still pushes to both registries, so a rollback is a revert, and if the pull failed, the rolling update would keep the old pod serving. - Next, #896: - PR-5 to PR-9: the rest of the GHCR switches, after PR-4. They are production changes, gated by the CI parity proof. - PR-10 to PR-15: the Dell overlay. It needs the Dell tunnel token (C4) and the tenant-map token (C5), and it must drop GCP's GCS backup CronJob. The Dell already holds production's sops-age key (see above), so it must confirm that the Dell's Flux decrypts the rig's secrets with it. - PR-16 to PR-18: retire Artifact Registry after a 7-day soak. - Next, #897: - O1 (operator-inbox #2), then P1b, the R2 tokens, then P2, GitHub Actions secrets from remote state. - P7: the rig-gitops apps/rig-backup-r2 Dell overlay. It needs #896's overlay, plus ghcr-pull-secret in the rig-backup-drill namespace. - O2 and O2b: the age keypair and the signing key, each made offline. - P8: the R2 writer token into SOPS. - The proving runs, drill A, then P9, the go-live. - Go-live, each needing the operator: ROUTING_MODE=live (done 2026-10-15, rig-cockpit-worker#304), an infra route for /api/merge*, and LOCATION=gcp with the TENANT_MAP_* secrets in rig-gitops (TENANT_MAP_BASE_URL, TENANT_MAP_ACCESS_*; done 2026-10-15, rig-gitops#1027). - Status, 2026-10-06 (evening): - Triage: every open issue in rig-conductor, rig-agent-runtime, rig-gitops, infra, rig-docs, operator-inbox and claude-3 (222) was read and sorted by what it means for the Dell: - 7 block it; - 29 are Dell-specific; - 69 are rig bugs at any location; - 43 look stale; - 74 are unrelated.

    The register, the fix list and the 14-step path are on the private page Dell Rig Readiness. - No Dell access from the orchestrator. The claude-3 workspace runs on the Mac Mini, which has no Dell key (see Hosts), so it can't run C6 or the live checks. dashecorp/infra#601 proposes one OpenTofu-managed authorized_keys line, applied by apply (hosts). The auto-mode classifier refused to generate the key, so this waits on the operator. - Gaps found, now tracked: - live webhook routing: rig-cockpit-worker#265, reopened (step 4 above); - the Dell tunnel, its hostname and its Access app (PR-14's infra step): dashecorp/infra#602; - a reader token for the Dell's conductor on the tenant map (PR-15's infra and worker change): dashecorp/infra#603; - etcd snapshots and a NAS copy, which #897's body promises but nothing builds: noted on rig-gitops#897. - PR-12 is drafted: rig-gitops#925 (issue #924), stacked on PR-19. - What it does: the Dell's conductor and data plane, fenced, with the conductor at 0. dell render invariants passed with flux 2.9.3 (53 objects, every rule OK), and Review-E approved it. - What CI can't check: CI doesn't build the nested rig-conductor-api Kustomization, which comes from the rig-conductor repo. A local kustomize build showed exactly the five intended fields differ from GCP's. - Before it merges: C9, the Dell's flux-system source reading dashecorp/rig-conductor. - The stack PR-11 → PR-19 → PR-12 is approved but held in draft. All three wait on C6. - Why draft: MergeGate squash-merges an approved stacked PR into its parent branch, so #915 and #925 stay drafts until they are retargeted to main. - Approvals given in draft: these don't trigger MergeGate when the PR goes ready, so each one needs a plain squash merge, or a fresh approval after it goes ready. - Merge path fixed: rig-agent-runtime#775 merged at 17:27Z and closed #773. The runtime now reads Review-E's verdict from GitHub, not from its reply text, and it takes effect with the next runtime pin. BRAIN's description is corrected in rig-docs#352. rig-conductor's comment above /api/merge still describes the text match, and gets fixed with the next conductor change. - GHCR Phase A, PR-5 merged: rig-gitops#927, merged by MergeGate at 17:25Z. It moves the dormant stigjohnny fleet (4 releases at 0 and paused) and the unrendered staging and Job files to GHCR. The live proof showed 25 tags matching and none mismatched. Next is PR-6, which restarts production Dev-E pods and needs the operator's go and C0. - Runtime pin: rig-gitops#921 (sha-8d0c6e0) had stalled after its approval. Its only dispatch check had finished as cancelled, and MergeGate's guard reads the latest run of each check. Re-running that check fired the check-completed webhook, and MergeGate merged it at 17:07Z. - CI: infra main is green again; infra#600 was a GitHub 500 while fetching the Cloudflare provider, and a re-run passed. rig-agent-runtime's Build & Push for 8d0c6e0, which failed on 2026-10-02 with "unknown blob", passed on a re-run. - Deadline, 2026-10-14: GPT-5.5 leaves Codex for ChatGPT sign-in plans, and the rig's Codex lanes sign in with ChatGPT Business. 13 HelmReleases in rig-gitops set it, 6 of them as the main model. - The fix, superseded the same evening (see below): rig-gitops#923 (issue #922) moves all 17 model: lines to gpt-6-sol. That is the replacement OpenAI names for Business plans, and openai/codex's model catalog maps gpt-5.5 to it. The rig's codex CLI, 0.160.0, supports it. - Held in draft: OpenAI makes GPT-6 Sol available in a workspace "once an administrator has enabled it". Merging before the operator confirms that would break the Codex lanes now instead of on 10-14. Its approval came in draft, so it needs a plain merge. - Status, 2026-10-06 (late evening): - Dell access from the Mac Mini: dashecorp/infra#604 (closes #601) adds an OpenTofu-managed key for the orchestrator, and apply (hosts) installed it (Hosts). - C6: all seven pre-checks passed (operator-inbox#7, closed). - KEDA's CRDs and stale APIServices: none. - kube-dns sits on 10.43.0.10, and 10.43.200.53 was free. - Capacity: 8 CPUs, 29 GiB of RAM available, 177 GB of disk free. - Both local-path and nfs-csi are marked as the default storage class; clearing one belongs to rig-gitops#696. - PR-11 is live on the Dell. rig-gitops#913 merged at 20:17Z, after gh pr ready and a fresh Review-E approval. The live checks pass: - the keda HelmRelease is Ready on chart 2.21.0; - v1beta1.external.metrics.k8s.io is Available; - egress-dns is on 10.43.200.53; - buzz and finance-dashboard are unaffected. - Next on the Dell: PR-19 (#915), then PR-12 (#925). Each needs retargeting to main, an update from main, gh pr ready, and its live checks. PR-12 also needs C9, the Dell's Flux GitRepository for rig-conductor reporting Ready. PR-13 waits on PR-6 to PR-9. - The Codex lanes run gpt-6.1-sol. - The first switch: #923 merged at 20:25Z with gpt-6-sol. - The correction: the operator then confirmed only that GPT-6.1 Sol is enabled in the Business workspace. rig-gitops#933 (closes #932) moved the same 17 lines to gpt-6.1-sol and merged at 20:47Z. codex 0.160.0's catalog lists that model. - To do: it defaults to low reasoning effort, where gpt-5.5 used medium, so raising it to medium is a follow-up. - Stale issues: 37 were closed after a re-check, each with an evidence comment. Five stay open: rig-conductor #1991, #2091 and #2320, and infra #365 and #366. - Status, 2026-10-07: - PR-19 and PR-12 are live on the Dell. - PR-19: rig-gitops#915 merged at 09:54Z. Every rig-gitops PR that touches apps/ or clusters/ now runs the dell render invariants check. - PR-12: rig-gitops#925 merged at 10:01Z. Its live checks pass: - Valkey and Postgres are Running, with PVCs on local-path. - The conductor sits at 0 replicas with LOCATION=dell and no RUNPOD_API_KEY. - The backup and the daily cost report are suspended, and there is no NodePort. - C9 was not needed: the Dell's Flux credential reads rig-conductor. - Merge trick: a stacked PR whose parent was squash-merged gets the parent's final head merged in first, then main. Merging main directly hits add/add conflicts. - k3s exited on big applies, now fixed (infra#606, infra#607). - What happened: after PR-11 and again after PR-12, etcd stalled for up to 5 s. The embedded controller-manager missed its 10 s lease renew deadline, k3s exited, and the API was down about 40 s. The containers kept running. - The fix: the k3s unit now passes leader-elect-lease-duration=60s and leader-elect-renew-deadline=40s to the controller-manager, scheduler and cloud-controller-manager, through var.install_exec. - Verification: apply (hosts) applied it at 10:41Z after an etcd snapshot. The leases show 60 s. - Still to prove: whether the next big apply (PR-13) passes without an exit. k3s has since exited once at idle, on a lease this fix doesn't cover (2026-10-13 below). - Pending gaps for PR-13: - The Dell's Postgres has no pgvector extension, because Flux never creates the one-time Job (reconcile: disabled). PR-13 must enable it before the conductor starts. - Both StorageClasses are marked default, so nfs-csi, the newer one, wins. Nothing relies on the default. ArgoCD is gone from the Dell, and the NFS provisioner is unowned. rig-gitops#696 has the plan to adopt it and clear its default, which waits for the operator. - GHCR PR-6 (rig-gitops#939, the Dev-E fleet): every check is green, including the live parity proof. It is held as a draft for C0 (operator-inbox#24), because this Mac Mini cannot read the GCP cluster. operator-inbox#25 asks whether the orchestrator should get read-only access. - The claimed-issue opt-out is live (rig-agent-runtime#778): - The rule: an issue with any GitHub assignee is claimed, and no rig agent works on it. - The two halves: - rig-conductor#2691 skips the planner signal, the agent-ready dispatch and the 10-minute scan. - rig-agent-runtime#780, pinned by rig-gitops#941, adds a planner stop rule and a runtime guard that records ISSUE_CANCELLED with reason=claimed_by_assignee. - Existing bug found on the way: ASSIGNMENT_SKIPPED_CLOSED has always been rejected with a 400 (rig-agent-runtime#779). Fixed that evening, below. - Evening of 2026-10-07 (operator AFK): the rig's event contract and MergeGate. - Six runtime events were silently rejected. rig-conductor's POST /api/events answers 400 Unknown event type to any type its MapToEvent lacks, and six that rig-agent-runtime posts got that answer, five of them since April or May. Some posts didn't log the answer, so nobody saw it: - ASSIGNMENT_SKIPPED_CLOSED is stored as an AssignmentSuppressed with reason agent_skipped_closed, on the issue stream. It changes no row state (rig-conductor#2694). - STREAM_CONSUMER_DOWN and CODEX_LOGIN_REQUIRED are stored as AgentDegraded and alert #admin, deduplicated per pod for 6 h. Both meant an agent that heartbeats but does no work. The Codex alert carries the CLI's details and points at rig-gitops docs/codex-auth-bootstrap.md (rig-conductor#2698). - COI_BLOCKED, SESSION_CWD_INVALIDATED and AGENT_SESSION_INITIALIZED are no longer posted; their log lines stay (rig-agent-runtime#783). - The guard: every rejected post is now logged. rig-agent-runtime's src/conductor-event-types.test.js pins the set of types the runtime posts, so a new type fails CI until rig-conductor has an arm for it (rar#783, #785, #787). rig-conductor#2367 is closed. - Pins: rig-gitops#947, #949 and #951. A pin whose runtime change alters agent behaviour needs a Rollout log entry in rig-gitops docs/image-pinning.md, and Review-E asked for one even on a docs-only pin. - MergeGate sweeps for lost triggers (rig-conductor#2365, via #2700 and #2696). - How it works: every 5 min, MergeGateSweepService re-runs the guarded Merge() for a Valkey hash with review-approved and no merged: key. ci-passed is not required, because the guard reads the checks. - Bounds: each trigger gets 12 attempts, backing off from 5 min to 2 h, kept in merge-gate-sweep:{repo}#{pr}. Only a newer condition resets that count. - Effect: a merge trigger lost to a pod rollover or a dropped webhook no longer strands an approved PR. BRAIN says so since rig-docs#372. - Supply-chain pins from the Structurizr review: - rig-docs loads mermaid@11.17.2 exactly, at securityLevel: 'antiscript' (rig-docs#370). Review-E rejected 'strict', which Mermaid documents as encoding the <br/> and <small> labels 16 pages use. - claude-3's C4-PlantUML includes pin the v2.14.0 commit (claude-3#1069). - Mermaid is still unreviewed and unregistered: operator-inbox#27. - TAGS GONE CRAZY playtest server (claude-4): it runs on the GCP cluster in namespace tgc-play (rig-gitops#937). It is reached at tgc-play.dashecorp.com behind an owner-only Access app (infra#609). Its pull secret came from a C2 run with the new namespaces input (rig-gitops#935). - Status, 2026-10-13: - GHCR PR-6 is live. rig-gitops#939 merged at 13:10:41Z on the operator's "yes PR-6", and Flux applied it by 13:11:24Z. - The rollout: the four running Dev-E lanes restarted on ghcr.io/dashecorp/rig-agent-runtime with ghcr-pull-secret, and the pull events show ghcr.io. The digests are unchanged, and the agents are idle with fresh heartbeats. - One change from the plan: the new Job dev-e-codex-home-permissions-v2 mounts agent-home on GCP, not the Filestore claim rig-shared-claude, because the agent-home soak had passed. It completed on a worker, and the old Job was pruned. The plan now says the same for PR-7's two Jobs. - PR-6 broke dev-e-bootstrap from 13:18:49Z to 13:55:12Z (rig-gitops#969). - What happened: PR-6 renamed the Job and, in the same commit, gave the GCP overlay a control-plane patch for the new name. The child Kustomization built the new revision before its parent flux-system had applied the new spec.patches, so -v2 was created without the patch. Every later reconcile then failed with spec.template: field is immutable, and Flux applied nothing in dev-e. The running agents were unaffected. - The fix: rig-gitops#970 dropped the patch, so the desired Job equals the live one. - Lesson: a PR that renames an immutable object can rely only on the base. Check a Kustomization's Ready again after its next interval, not only at the first apply. The 13:12Z rollout check passed six minutes before the failure. - C0: run read-only by the claude-3 session on the MacBook Air, which has the GCP access (operator-inbox#24, closed). Pull-secret names came from the Flux inventory, so no Secret was read. - Review-E: editing a PR's description doesn't trigger a re-review. An empty commit did. - Next: PR-7, after a day's soak. It went the same day instead (below). - The Dell's k3s exited at idle (dashecorp/infra#622). - What happened: on 2026-10-08 at 02:18:05Z, k3s logged leaderelection lost for k3s-etcd and was down about 43 s. etcd had stalled for 3 minutes from 02:15:03Z, with nothing being deployed. - Why the 10-07 fix missed it: infra#607 relaxed only the controller-manager, scheduler and cloud-controller-manager leases. k3s's own k3s and k3s-etcd leases are still 45 s. - Since then: a milder stall on 2026-10-12, 2.3 s at worst, passed without an exit. The cause is unknown. - The agent-home soak passed (checked 2026-10-09). - Filestore has had no writes since the switch, and 7 daily snapshots of the new disk exist. - Cleanup step (0) is done (2026-10-13, on the operator's "yes step 0"). - What it did: one forced upgrade per release (spec.upgrade.force from rig-gitops#968, then flux reconcile hr --force while each agent was idle). review-e went to v517 at 13:55:46Z and dev-e at 13:56:46Z. This dropped the leftover shared-claude volume from both StatefulSets, and rig-gitops#972 removed the flag. - What's left on old claims: only the review-e and rig-planner permission Jobs. GHCR PR-7 moves them to agent-home when it renames them. - Then: the rest of rig-gitops#883 (prune the old PVs and PVCs, docs), and the Filestore deletion (infra#526, #527), each with the operator's yes. - GHCR PR-7 is live. rig-gitops#974 merged at 16:40:54Z, so Review-E and the planner pull from GHCR. - The soak waiver: the operator approved the merge ("yes PR-7", 14:53:28Z) and waived the rest of PR-6's day of soak ("Yes now", 16:37:25Z), because Dev-E had already pulled the same image tag from GHCR. Review-E had held its approval to the soak date written in the PR, which MergeGate does not check. The waiver went into the PR body, and Review-E approved the new head. - The rollout: both Review-E pods and the planner restarted at 16:41:10–11Z, and the pull events show ghcr.io. The -v2 permission Jobs completed on a worker with claim agent-home, and the old Jobs were pruned. - The agent-home cleanup is done (rig-gitops#883, closed). - Adoption: rig-gitops#976 put the hand-applied rig-shared-claude PVs and PVCs into a legacy-filestore Kustomization. A server-side dry run passed before the merge. - Pruning: rig-gitops#978 (merged 16:47:57Z) deleted that Kustomization, and Flux garbage-collected the four objects, together with the planner's rig-planner-filestore PV and PVC. - The result: only the three agent-home PVs remain, all on 10.166.0.33, and no workload names an old claim. The infra docs update is dashecorp/infra#629. - Left: unmount /mnt/fs-* on invotek-k3s, then infra#526 and #527, each with the operator's yes. Deleting the two instances saves about $368 a month. - Filestore is deleted. On the operator's "yes delete filestore" (20:46:18Z) I unmounted the read-only mounts on invotek-k3s. Then dashecorp/infra#631 lifted rig-planner's deletion protection (step 5a), and dashecorp/infra#632 destroyed both instances (step 5b). The apply finished at 21:05Z with 0 added, 0 changed, 3 destroyed, and gcloud filestore instances list is empty. The Filestore API stays enabled, infra#508 is closed, and the saving is about $368 a month. - Review-E misrouting (rig-conductor#2713). The review of rig-gitops#976 arrived keyed to its issue #975, with an implementation template appended. The claude-tmux session stalled until the 360 s fallback, which approved it correctly. - The Codex temp-folder leak is fixed: rig-agent-runtime#763 was closed by #790. - Status, 2026-10-14: - GHCR PR-8 is live. rig-gitops#980 merged at 12:11:19Z on the operator's "yes PR-8" (12:09:34Z), with the dashecorp lanes idle. Nothing under apps/ or clusters/ references Artifact Registry any more. - The rollout: the four dashecorp HelmReleases (Review-E, Dev-E, the planner and Rig-E) upgraded, and they pull ghcr.io/dashecorp/rig-agent-runtime with ghcr-pull-secret; the pull events show ghcr.io. dashecorp-bootstrap was still Ready after its next interval (12:21:49Z). - The codex Job: v3 became dashecorp-codex-home-permissions-v4 and completed on worker-2. v1 (Failed since it was created) and v3 were pruned. The v3 Job's GCP placement patch was dropped, not re-targeted, per the PR-6 lesson: the claim is already bound on worker-2, so its volume places the Job there. - A pre-approved draft merges when it is marked ready. Review-E approved the draft before the operator's yes, so MergeGate merged it 19 s after it left draft. Mark a production PR ready only once the lanes are idle. - Rig-E was down about 3 minutes, because its new pod landed on worker-2, which had to pull the image. Its unauthenticated probes answer as before: /api/health 200, POST /api/chat without a token 401, /api/state 404. - PR-9 merged at 12:35:12Z (rig-gitops#982), so GHCR Phase A is complete. The image-pin gate now rejects any Artifact Registry reference. Review-E caught a hole in the first version: an explicit registry port (us-docker.pkg.dev:443/…) got past the check. The gate now parses the registry host. The D8 soak before retiring Artifact Registry (at least 7 days) ends no earlier than 2026-10-21. - Worker disks are too small for bursts (dashecorp/infra#633). - What happened: the two fresh pulls on worker-2 hit no space left on device, and the node was under DiskPressure from 12:14:51 to 12:19:57Z. The retry succeeded, and nothing was evicted. - Why: the workers' 30 GB disks idle at 80–84%, and the kubelet's image GC starts only at 85%. The daily image-prune CronJob covers only the control-plane node, though the agents moved to the workers on 2026-09-25. - Fixed the same afternoon with no spend; dashecorp/infra#633 is closed. initialize_params.size is ForceNew, so growing the disks would have needed each one adopted as a google_compute_disk (draft infra#640). The operator chose a cleanup instead ("yes cleanup to all workers and nodes"). - Over SSH: the journals were trimmed to 500M, and only images that no workload references were deleted. worker-1 went from 79% to 57%, worker-2 from 84% to 71%, and the control plane from 37% to 30%. A blind crictl rmi --prune would have deleted worker-2's current agent images. crictl rmi returned non-zero for deletes that worked; a listing confirmed the images were gone. - rig-gitops#986 (14:22Z): the node-config DaemonSet caps journald at 500M on the workers. It also runs growpart and resize2fs, which do nothing until a disk grows. - rig-gitops#985 (14:27Z): image-prune is now a DaemonSet that prunes every node daily in the 04:xx UTC hour. The CronJob is gone. - The disk grow (infra#640/#639) is closed. Reopen it if a worker passes 80%. - The Dell's NAS provisioner and buzz's volumes are in Git (rig-gitops#988, Closes #987). It merged at 15:59:45Z on the operator's "Yes 988" (15:57:50Z). Review-E approved the head 9 s earlier. - Before the merge: a server-side dry run on the Dell, under Flux's field manager (kubectl diff --server-side --field-manager=kustomize-controller --force-conflicts), showed the exact changes and no immutable-field errors. - The result at 16:00:30Z: the provisioner restarted once, onto v4.0.2@sha256:63d5e045…, and took its lease at 16:00:41. buzz's four PVs are on Retain, and its pods were not restarted. The Dell has the same five NFS mounts, all on the LAN address. - The render check: the Dell's no-nfs-volume rule now allows a location's own NFS server (LOCAL_NFS). Two selftest mutations prove that every other server still fails. - A test trap: an NFSv4.1 mount of a second address of the same server reuses the existing client and its connection. A tailnet test mount next to LAN mounts therefore measures the LAN. Test with vers=4.0. - 2026-10-15 morning: the Dell's Cloudflare side, capacity guardrails, and a tunnel-secret leak. - Merged and verified live: - rig-gitops#999: image-prune at 512Mi. The first daily run had been OOMKilled on 2 of 3 GCP nodes by k3s's 253 MB in-container unpack. - rig-gitops#1001: the temporary upgrade.force removed from 13 releases, with no release version change and no restart on either cluster. - infra#650: the reinstall runbook, now with an NFS-client step and max-pods=200 in install_exec. - infra#652: the Dell tunnel dashecorp-dell, the hostname rig-conductor-dell (403 without a token) and its Access app. - infra#653: the Dell conductor's own tenant-map reader token. - rig-gitops#1004: capacity guardrails on the Dell. There are 6 agent slots plus Rig-E in a 13.5 GiB budget, with a quota and LimitRange per agent namespace and KEDA capped at 1. At the cap, work waits instead of running out of memory. Kubelet reservations for the reinstall are on infra#648. - Security, infra#654: OpenTofu prints a resource's id, and a random_id's id is its value. - GCP's live tunnel secret has been in every OpenTofu CI log that refreshes cloudflare/dashecorp.com, and in PR plan comments. - The new Dell secret was printed once at creation, and the orchestrator printed it again into its session output. - Fix draft infra#655 moves the Dell secret to random_bytes, takes GCP's out of the printed resource with no tunnel change, and adds a module-lint guard. Review-E asked for an explicit first Dell replacement. - Rotating GCP's secret (a no-downtime swap to a new tunnel, or at GCP's retirement) is the operator's decision. Rule: mask [id=...] in every CI-log grep. - Open: - rig-gitops#1005: the Dell disk and memory alert to #admin. - rig-cockpit-worker#300: the worker accepts one tenant-map reader token per location, synced by deploy.yml from an OpenTofu-written secret. - rig-gitops#1002: C4/C5 through credential-ops, so no one copies a Cloudflare-made token. The operator approves the encrypt run on the phone. The workflow is drafted on the Mac Mini. - The move is still blocked on the worn drive: the external SSD decision is the operator's. - 2026-10-15 midday: the Dell's alert and reader set are live, the C4/C5 route is merged, and the drive's write load is measured. - Merged and verified live: - rig-gitops#1005: node-pressure-watch on the Dell. Its first run (08:00Z) read disk 18% used and memory 88% available, so no alert. - rig-cockpit-worker#300: deployed with no extra readers (TENANT_MAP_READERS is unset), and /api/tenants still refuses a request with no token. GCP's conductor never reads the tenant map, because its fence is off. Only the Dell's fenced conductor will, after PR-15. - rig-gitops#1008, #1009 and #1010: C4/C5 through credential-ops. They add a shared recipient-pinning library, the encrypt helper with its selftest (175 checks, green on its first CI run) and the encrypt job. The operator approves each run on the phone. - infra#655, the tunnel-secret fix: OpenTofu 1.9's replace_triggered_by ignores a create of its trigger, so the Dell's new secret goes on a new tunnel, dashecorp-dell-v2. Its name is new too, because Cloudflare answers HTTP 409 for a name a live tunnel has. The operator said yes at 09:25:42Z. It merged at 09:35:58Z and applied as planned: 3 added, 1 changed, 3 destroyed, and nothing on GCP. The new tunnel's ID (f3b364b4…) differs from GCP's and from the old one, and rig-conductor-dell still answers 403 without a token. - infra#658, OpenTofu writes the C4/C5 secrets and the worker's TENANT_MAP_READERS: - terraform_remote_state drops the sensitive mark of outputs, so every value is wrapped in sensitive() again. The older tgc-docs and tgc-media consumers are tracked in infra#659. - A precondition fails the plan while the old Dell tunnel's ID is still current. - Review-E asked for a switch that writes [], to remove the Dell's reader. - The worn drive's write load, measured at idle: about 30 GB a day and 600 flushes a minute. - About 60% comes from Flux re-extracting artifacts onto disk-backed emptyDirs, and etcd comes next. - No stall over 100 ms has happened since the PR-13 install. - The I/O scheduler is none, so I/O priority has no effect. - The fixes need no new hardware: - rig-gitops#1014: Flux's working space in RAM, a 5m Git fetch, 120s/100s/20s leader leases for Flux and KEDA, images kept until the disk is 70% full, and the alert moved off the busy minutes. - infra#661: apt at 02:41 and 02:56 UTC, with man-db off. - The operator approved both at 09:25:42Z. #1014 was live on the Dell by 09:42:42Z: flags, RAM volumes and 120s leases checked, and all 17 Kustomizations and 19 HelmReleases Ready. #661 was live by 09:47Z, with next runs at 02:41:00 and 02:56:00 UTC. The write rate is re-measured on 2026-10-16 against a baseline taken at 09:42:23Z. Neither covers a sudden drive failure; the off-site backups (rig-gitops#897) and the SSD do. - PR-14 and PR-15 (rig-gitops#1011, #1012) are drafted and verified as branches. Their PRs open after the C4/C5 commits land. - Still the operator's: rotating GCP's tunnel secret, and the SSD. - 2026-10-15 afternoon: the Dell's own tunnel is live, the tenant map is wired, and two format bugs are fixed. - infra#658: Review-E's one blocker, a removal path with no code behind it, was fixed with var.dell_tenant_map_reader_enabled (default true; false writes [] and keeps it written after the token is gone). Approved 09:58:38Z, operator yes 10:08:05Z, merged 10:08:37Z. The apply wrote 4 secrets with no value printed: DELL_TUNNEL_TOKEN, DELL_TENANT_MAP_CLIENT_ID and DELL_TENANT_MAP_CLIENT_SECRET on the rig-gitops environment credential-ops (none at repo level), and TENANT_MAP_READERS on rig-cockpit-worker. Follow-ups: infra#659 (tgc remote-state sensitivity), infra#660 (Access tokens expire after 1 year by default), infra#662 (plaintext_value → value). - Worker deploy (operator yes 10:08:05Z): deploy.yml run 38718557371 on main 9d3444b put TENANT_MAP_READERS on the Worker at about 10:13Z. Unauthenticated /api/tenants, /api/locations and /api/health still return 401. - C4 (run 38718895703, approved by the operator on the phone) pushed apps/cloudflared/dell/tunnel-token.sops.yaml to PR-14's branch at 10:27:37Z. - PR-14 = rig-gitops#1015, the Dell's own cloudflared: Review-E approved 10:42:29Z, operator yes 10:44:27Z, merged 10:45:30Z. - Live on the Dell by 10:50:31Z: one pod, tunnel dashecorp-dell-v2 (f3b364b4…), 4 registered QUIC connections (osl02 ×2, arn08, arn09), no errors. - GCP's cloudflared is untouched (same pod, 0 restarts). rig-conductor-dell.dashecorp.com still returns 403 without a token. - C5's first run (38718908545) stopped safely and wrote nothing: ABORT: DELL_TENANT_MAP_CLIENT_SECRET is not an Access client secret (64 hex). - Cloudflare changed the format: Access service-token secrets made on or after 2026-08-26 are cfast_[40 alphanumeric][8-character checksum]. Older 64-hex secrets still work. - Fix: rig-gitops#1017 (issue #1016). Its test-only commit went red in CI with exactly the 5 predicted failures, and the fix went green. MergeGate merged it at 10:44:25Z. - The re-run 38722557796, approved by the operator, pushed the file at 10:55:00Z. - PR-15 = rig-gitops#1018, the Dell's conductor reads the tenant map with its own token: - Review-E first requested changes: Flux cannot decrypt a Secret whose kind and metadata are encrypted. The orchestrator answered with evidence, and Review-E withdrew it at 11:09:52Z: kustomize-controller v1.9.4 decryptor.go:639 decrypts any resource that carries sops.mac; the same Kustomization already reconciles four such files on both clusters; and no Kustomize transformer touches an encrypted resource. Docs: rig-gitops docs/sops.md, "Whole-file encryption, and the one rule it needs". Guard: rig-gitops#1019. - Operator "yes PR-15, once Review-E approves" at 11:02:28Z. Merged 11:11:02Z. - On the Dell the conductor rolled, mapConfigured: true, and the worker accepts its token. But every fetch fails to parse: the worker sends generatedAt as epoch milliseconds (a number), and the conductor's DTO expects a string. - The fence stays fail-closed (map_unreachable). GCP is untouched, because its fence is off. - The fix, rig-conductor#2717 (issue #2716): EpochMsOrStringConverter on GeneratedAt. It never throws: an integer becomes ISO-8601 UTC, and a string is kept as sent. - The new test reproduced the Dell's exact error, and with the fix the whole API test project passes (2381). - Its merge auto-pins the image, so the conductor restarts on GCP and the Dell. - Operator yes at 11:36:45Z. Merged at 11:55:22Z (89b6787) once Review-E approved, and the image pin #2719 auto-merged at 11:57:30Z. - Live on both clusters by 12:01:33Z, 0 restarts. - On the Dell the map loads (mapFetchedAt 12:01:06Z, lastFetchError null). It lists 3 tenants, dashecorp, invotek and runthedocs, all at gcp, so the standby refuses their work with mapped_elsewhere. - GCP's conductor restarted onto the same image; its fence is off, and its public probes are unchanged. - The SOPS render guard, rig-gitops#1020 (issue #1019): a render rule, sops-untouched, fails when a Kustomize transformer would rewrite a whole-file SOPS resource before Flux decrypts it. CI also runs it on GCP. - Flux's commonMetadata is safe: kustomize-controller applies it after decryption. - Review-E found two ways around the first version: a patch that replaces a stub's metadata, and an annotation key that commonMetadata repeats. - The fix walks each build chain and expects exactly one untouched stub for every SOPS file. - Testing it on real files found a third gap: Flux appends spec.components to spec.path's own kustomization file. That is fixed too. Review-E approved the fixed head, and MergeGate merged it at 12:33:47Z (f50c92c). The Dell re-applied it with 18/18 Kustomizations and 19/19 HelmReleases Ready. - Live webhook routing, steps 2–5 done: no code gave the router its HMAC secret (GITHUB_WEBHOOK_SECRET), so: - Step 2: OpenTofu writes it to rig-cockpit-worker as ROUTER_WEBHOOK_SECRET, because GitHub refuses Actions secret names that start with GITHUB_. - infra#664 merged at 12:44:05Z on the operator's "yes 663" (12:43:31Z). Its apply added exactly 1 resource; every other module had no changes. - rig-cockpit-worker#302 merged at 12:47:54Z on "yes rcw-301" (12:46:46Z). Its deploy logged set worker secret GITHUB_WEBHOOK_SECRET. - The probes are unchanged. The GCP conductor kept receiving deliveries through the shadow router (27 in 4 minutes) with no signature rejects. - Step 4: the operator checked the Router shadow: would route logs and replied "shadow ok" at 12:51:07Z: every delivery on the right tenant, hmacValid: true. The orchestrator could not read them: the operator's browser Cloudflare login has only other accounts. - Step 4b, rig-cockpit-worker#306 (issue #305), the gcp repoint: - The hold: before asking for "yes rcw-303", the orchestrator re-checked the live path. In D1, gcp's base_url was still migration 0009's seed, the stable host https://rig-conductor.dashecorp.com ("Repoint … once #565 lands"), which carries the router's own route. - The queue consumer is invoked by no request, so it is not on the route's zone. Cloudflare documents the same-zone origin bypass only for "a Worker's own zone". A public case, QNFO/qnfo-workers#635, shows a scheduled handler's fetch to same-account Workers on workers.dev getting error 1042. - So a live forward would most likely have re-entered the router and got 508. GitHub would see 202 for every delivery, and none would reach the conductor. #304 was held. - The fix: migration 0014 repoints gcp to https://rig-conductor-gcp.dashecorp.com (infra#566: same tunnel and origin, Access admits only the Worker's service token, no Worker route). It runs only while the row still holds the stable host, and it clears the four health columns. A test checks that no location's base_url host is WEBHOOK_ROUTER_HOST. - A Mac Mini job wrote it: red 4/4, then green, npm test 1123/1123. The orchestrator softened one unverified claim. - Review-E approved 28db2a7 at 13:53:02Z, and the operator said "yes rcw-305" at 13:53:01Z. Merged at 13:53:39Z (ee6502e); the migration applied at 13:53:59Z. - /api/locations at 13:55:50Z: gcp at rig-conductor-gcp, healthy, last probe OK at 13:55:25.748Z, no error. The cron probe is the same kind of zone-less fetch as the consumer, through Access with the Worker's token. - Step 5, rig-cockpit-worker#304 (issue #303), done: it sets ROUTING_MODE = "live". Review-E approved head 91870f9 at 13:01:33Z (1119/1119); main plus #304 passed 1123/1123 locally before the merge. - The operator said "yes rcw-303" at 14:02:43Z. Merged at 14:04:01Z (d88f250); the deploy put ROUTING_MODE "live" at 14:04:33Z. - Verified: repo hook 639292554 deliveries answer 202 (200 before), and 4 of 4 reached the conductor by GUID, 2–8 s later:

         | Delivery | `202` at | Conductor |
         |---|---|---|
         | workflow_run.completed | 14:05:05.1Z | 14:05:09.8Z |
         | check_run.completed | 14:05:04.9Z | 14:05:13.2Z |
         | workflow_run.completed | 14:05:48.3Z | 14:05:50.7Z |
         | check_run.completed | 14:05:48.3Z | 14:05:51.5Z |
    
         - Org-hook deliveries from `dashecorp/rig-docs` reached the conductor at 14:06:03–13Z.
         - The conductor log since the switch has 0 signature lines and 0 exceptions.
       - **Before the live version deployed:** 2 deliveries in the merge burst (14:04:16–17Z, shadow pass-through) got `500`, while the conductor's Kestrel logged a slow-request-body timeout.
         - One of them (check_run.created) was still processed. The other (workflow_run.requested, informational) was not; GitHub can redeliver it.
         - They are the only non-2xx answers in the hook's last 100 deliveries. In live mode the queue absorbs such a burst and retries a 5xx.
       - The router doc in rig-cockpit-worker records steps 4b and 5 in #308 (issue #307), a docs-only PR.
    

    • Afternoon:
      • infra#666 (issue #665), merged by MergeGate at 14:24:40Z: the Dell reinstall runbook now stages the four rig-gitops#989 kubelet args (reservations, eviction, serialized pulls). Docs only; no apply ran.
      • < and % are safe in install_exec: the installer's escape() backslash-escapes < before eval (get.k3s.io line 1203), and on the Dell's systemd 255 a test user unit kept %, and a trailing % literal.
      • The new check expects allocatable 7 CPU and 28870Mi. The measured capacity is 8 CPU and 32708648Ki, and today allocatable equals it.
      • The router since it went live (14:04:33Z): all 28 deliveries on hook 639292554 up to 17:04Z answered 202. The GCP conductor processed 132 deliveries by GUID between 15:15:33Z and 17:04:17Z (its log window), with 0 signature lines.
    • The goal changed: the Dell primary, GCP a cold standby. The operator at 14:52:29Z: "I want dell too be the main rig. I want the GCP to run on as low cost as possible. But if Dell goes down it should be ready to start up gcp again and take over".
      • GCP is not deleted. #990's step 14 becomes "GCP to cold standby, with failover and failback", tracked in rig-gitops#1021; a comment on #990 records it.
      • The design, rig-gitops#1023 (issue #1022): merged at 15:32:52Z (b9f233a) after Review-E approved it. It is rig-gitops docs/2026-10-15-gcp-cold-standby.md (status: proposed), written by Mac Mini job standby-1021.
      • Cost: about $179/month today, about $8 in standby, about $7 once Artifact Registry and the 3 orphan PVC disks are gone. Prices checked on gcloud-compute.com on 2026-10-15: e2-standard-2 $53.86 a month; pd-standard $0.044 and pd-balanced $0.11 per GB-month (europe-north1).
      • Standby: OpenTofu stops the VMs through a stored desired-state object, not a -var, because infra's workflow applies every module on every merge and its dispatch takes no inputs. The disks keep etcd, the Secrets and the images.
      • Failover: GCP starts itself after the Dell has been down 10 min, which needs a narrow infra-dispatch token in the Worker. The tenants move on the operator's tap, which needs a skipDrain change (the drain is clock-only today).
        • The default is a fresh start on GCP's own database with the reconcile sweep. ReconcileSweep__DryRun is still true at both locations and must be turned off.
        • RTO is about 25–40 min. Restoring R2 first takes about 2.5 h: the R2 backups are suspended, and a restore decrypts by hand with the offline key.
      • Before GCP can sleep: a role swap (GCP runs the planner, Rig-E and the dotnet-codex canary, and only GCP holds RUNPOD_API_KEY); the stable host must follow the primary; GCP's tunnel secret must be rotated, now that there is no retirement.
      • The plan: "now" items first (fence GCP with LOCATION=gcp and a tenant-map reader; the Worker's down alert, Fail-over button and drain skip; the infra state object and the stable-host CNAME), then the rest after #990 Phase 3.
    • Evening: GCP is fenced, standby steps 2–5 are live, and the approve-rule is merged.
      • GCP's own tenant-map token, infra#668: merged 18:08:52Z. The second apply (18:14:43Z) wrote GCP_TENANT_MAP_CLIENT_ID / _SECRET to rig-gitops' credential-ops and made TENANT_MAP_READERS [dell, gcp]. The Worker deploy put it live at 18:17:52Z; the Dell's map reads stayed clean.
      • The c5-gcp step, rig-gitops#1025: merged 17:55:26Z. Run 38784866311 pushed the encrypted file to the fence branch. The operator approved it in Safari: GitHub Mobile shows no Approve for runs triggered under the operator's own login, because no notification is sent.
      • GCP's conductor is fenced, rig-gitops#1027: LOCATION=gcp with its own token. Merged 18:57:52Z; the new pod was Ready at 18:58:37Z.
      • All 10 runbook checks passed. A 30-min capture was clean: 0 stale maps, 0 errors, 0 refusals, 3 tenants gcp/active.
      • Finding: dashecorp's dry-run activation sweep stopped on a 403 on dashecorp/dashe-website check-runs, with 14714 rate limit remaining. It must be fixed before the sweep goes live (design step 9).
      • rig-cockpit-worker, the failover side:
      • #310 (19:42:10Z): migration 0015 adds the dell location row, a standby flag and health_notified. Down/up health alerts go to the operator by APNs push, claimed with a compare-and-swap. There is no #admin fallback, because DISCORD_ADMIN_WEBHOOK_URL is not provisioned. The operator confirmed that pushes arrive.
      • #312 (20:20:21Z): skipDrain, only while the source is unhealthy; POST /api/locations/{from}/evacuate; a failed notice is retried.
      • #314 (21:15:21Z): the Fail over… action in the Locations tab, with Review-E's fix for a stale-response race.
      • Operator decisions, 21:36:16Z ("yes standby defaults", "yes rotate-gcp"): a fresh start on GCP's own DB at failover; every GCP-only app moves to the Dell in Phase 3; a narrow INFRA_DISPATCH_TOKEN, created by the operator, lets the Worker start GCP automatically (step 7); the GCP tunnel-secret rotation runs right after step 6.
      • The design is accepted (rig-gitops#1028). Step 6, the rig-location switch, is in progress as infra#669.
      • The approve-rule, claude-3#1191 (merged 21:42:24Z, 34061e0): a standing exception in CLAUDE.md. The orchestrator may approve a waiting credential-ops run in rig-gitops (C2, C4/C5) only if it dispatched the run itself from main, and the operator replied with a literal yes naming that step after being told the run number, step and target branch. The approval goes through POST /repos/dashecorp/rig-gitops/actions/runs/<id>/pending_deployments with a comment quoting the yes and its timestamp. Never a rejection, never another environment (CI/CD).
    • 2026-10-18: runthedocs watch day 2 is green. One check at 04:14Z (deadline #48, ends 2026-10-19):
      • runthedocs is active at the Dell (epoch 2), and dashecorp and invotek stay at GCP (epoch 1). The Dell's fence map was 50 s old, with no fetch error.
      • k3s has not restarted since the planned 2026-10-16 13:53Z restart.
      • The drive matches its baseline: smart_ok, critical-warning bit 0x4 and 107% used as before, 100% spare, no media errors, worst await 2 ms.
      • The Dell's conductor (pod k8mlz) had run 19 h with 0 restarts. Its log had 0 fail/crit lines in 11 h; the warnings are the expected mapped_elsewhere fence lines for the two GCP tenants.
    • 2026-10-17: runthedocs holds on the Dell (watch day 1 green), and the re-dispatch hold is enforced.
      • Watch day 1 (deadline #48, ends 2026-10-19): three checks, all green, at 07:32Z, 14:19Z and 19:53Z.
      • runthedocs stays active at the Dell (epoch 2). Both fences are fresh: map age under 40 s, no fetch error.
      • k3s has not restarted since the planned 2026-10-16 13:53Z restart.
      • The drive matches its baseline: smart_ok, no media errors, await 1–4 ms.
      • The Dell's conductor runs with 0 restarts and no fail/crit lines.
      • rig-conductor#2723 ("yes 2723", merged 08:58:34Z, bac1e72): REDISPATCH_HOLD_MODE=enforce at both locations.
      • The two swift issues that kept looping to the stopped iBuild-E (dashe-nutri-e#63, rig-cockpit-ios#116) were held at 09:06Z and set to failed. That ended the #admin spam.
      • Since then, every StaleClaimWatcher tick at GCP has released 0 claims.
      • False "review-e DOWN (crashloop)" alerts (21:14:56Z and 22:02:56Z).
      • Review-E is KEDA-scaled from zero, and one activation starts two StatefulSets that share the agent id.
      • So two reviews within 30 minutes trip AgentCrashLoopPolicy's 3-starts rule, even though the heartbeat is 0–1 s old.
      • Filed as rig-conductor#2725.
      • Open, deferred by the operator: at 22:12:06Z the Dell's ReviewScan routed run-the-docs/engine#95, labelled needs-review since 2026-07-02, to assignments:review-e@runthedocs. runthedocs has no Review-E fleet, so nothing will take it.
    • 2026-10-16 evening: #990 re-planned, three gaps closed, and runthedocs is the first tenant on the Dell.
      • The re-plan (rig-gitops#990): the drive stays.
      • The tenants move one at a time, lightest first, and GCP keeps running until the last one is stable.
      • runthedocs goes first, before the backups, because its history stays on GCP. invotek waits for the backups (#897, which starts with O2). dashecorp waits for a clean week with invotek.
      • Stop rule: a drive alert, or an unplanned k3s restart with tenants on the Dell.
      • Measured: GCP's pods write about 34–42 GB/day, so the Dell would go from 13 to about 45–55 GB/day, which is 1% of rated wear every 3 weeks (110.7 TB written of about 103 TBW).
      • Three gaps that would have broken the first move:
      • rig-gitops#1045 ("yes 1045"): the Dell's conductor knew only invotek, under the wrong org (dashecorp). A one-shot Job registered all three tenants with GCP's identities. The finished Job was removed in #1047.
      • rig-cockpit-worker#320 ("yes 320"): the webhook router now checks repo overrides before the installation, as the conductor's TenantMatch does. The six rig-internal repos are pinned to invotek, so invotek can move apart from dashecorp.
        • Its deploy failed, because D1 caps a compound SELECT at 5 terms (stock SQLite allows 500). The pipeline stopped before "Deploy worker", so nothing live changed.
        • #323 ("yes 323") rewrote migration 0017 as six single inserts and added test/utils/migrations-d1-limits.test.js. It deployed at 20:05:49Z (version 9fff35ce).
      • infra#690 ("yes 690"): /etc/rancher/k3s/config.yaml, hand-written since March with max-pods=200, is now owned by OpenTofu (null_resource.kubelet_config). It adds rig-gitops#989's reservations, the eviction thresholds and serialize-image-pulls.
        • Not through install_exec: the k3s docs say a CLI list replaces the file's list, and % is a systemd specifier.
        • Applied at 13:53Z with one k3s restart: allocatable 7 CPU and 29562920Ki, worst drive await 5 ms.
        • Review-E caught that an interrupted restart would never recover. A pending-restart marker fixes it.
      • runthedocs moved to the Dell (#990 step 12). The operator tapped Move in Rig Cockpit at about 21:11Z.
      • Timeline: draining from 21:11:49Z, switched at 21:27:44Z, active at dell (epoch 2) by 21:28:55Z.
      • The Dell's dry-run sweep completed with nothing set. GCP now refuses runthedocs as mapped_elsewhere.
      • Two test webhooks reached the Dell's conductor about 3 s after GitHub sent them, and GCP got neither.
      • A 3-day watch runs until 2026-10-19 (deadline #48).
      • Not proven: an agent run on the Dell. runthedocs has no agent fleet anywhere: assignments:*@runthedocs has no consumer, and run-the-docs PRs have not been reviewed since at least June.
      • Gotchas from the day:
      • An operator-authored PR outside the six rig-internal repos gets Review-E only with the needs-review label. rcw#320 waited 1h45m without it.
      • In portrait on an iPhone, the Rig Cockpit Locations tab does not scroll; landscape works (rcw#325). The iOS app has no Locations tab.
      • RepoDiscoveryService is not fenced (rig-conductor#2721, low).
      • The GCP conductor's log buffer holds only about 30 minutes.
    • 2026-10-16 midday: standby steps 7 and 8, and an early warning on the Dell's drive.
      • Step 7 is live: the Worker starts GCP itself. The operator said "yes rcw-318, yes rcw-317" at 09:54:19Z.
      • rig-cockpit-worker#318 (deployed 09:55Z, migration 0016_location_auto_starts): when every location holding an active tenant has been unhealthy for 10 min or more and gcp is marked standby and not healthy, the cron dispatches infra's rig-location.yml with gcp=running primary=keep, once per incident, and tells the operator. It never stops GCP, moves a tenant or changes primary.

      • 317 (deployed 10:00Z) put INFRA_DISPATCH_TOKEN on the Worker; the deploy logged "set worker secret".

      • It is inert today, because every tenant is on GCP. Drill 1 is the first real dispatch.
      • Step 8: infra#688 merged with no changes (metadata_startup_script is in ignore_changes). The startup scripts skip tailscale up when already logged in, and never use --force-reauth. The live VMs change only through scripts/gcp-startup-script-push.sh, which the operator runs, and only if drill 1 shows the problem.
      • The Dell keeps its worn drive (107% used, critical_warning 0x4). The operator declined a replacement, because it sits at their parents' house and they live in Gran Canaria. The 24 h re-measure after the write-load cut: 13.4 GB/day, down from about 30, and no stalls (max 34 ms).
      • The early warning is live instead. The operator said "yes 687, yes 1043" at 11:15:40Z.
      • infra#687: nvme-diag writes /run/nvme-diag-health/health every 30 s.
      • rig-gitops#1043: node-pressure-watch posts to #admin on media errors, spare below 100, a new warning bit, a 30 s average await over 200 ms, a failed read, or a missing or stale file.
      • The stall check is sustained latency only. Review-E showed that one slow I/O among many fast ones averages away, and /sys/block keeps no per-I/O latency. The first run, at 11:37Z, was quiet.
    • 2026-10-16 morning: the GCP tunnel rotation is complete, and the step-7 token exists.
      • The rotation (infra docs/2026-10-15-gcp-tunnel-rotation.md):
      • R2: rig-gitops#1034, the c4-gcp step, merged at 00:32:18Z. A cancelled notifier check had held it, approved, for 37 minutes. MergeGate accepts only success, skipped and neutral, and its "Notify" skip token never matched the job name dispatch. rig-gitops#1037 renamed the job.
      • The c4-gcp run: on the operator's "yes c4-gcp" (08:03:57Z), the orchestrator approved run #16 under the approve-rule. It pushed the encrypted v2 token to #1035, which then merged at 08:15:22Z. cloudflared-v2 registered 4 connections to dashecorp-gcp-v2 (9b7a1aa0…).
      • R3: infra#682 (applied 08:23:12Z) moved all 7 GCP CNAMEs to v2: 0 added, 7 changed, 0 destroyed. The old connector's request counter, read from its metrics port by pod IP, went flat.
      • R4a: rig-gitops#1039 (08:44:24Z) deleted GCP's base cloudflared with a Component $patch: delete, and Flux pruned it and its Secret. The Dell keeps the base Deployment. tunnel-id now names v2. c4-gcp refuses the old tunnel's fixed ID and requires tunnel-id, so it can still re-encrypt v2's token.
      • R4b: infra#684 (applied 08:53Z) destroyed dashecorp-gcp and its config (2 destroyed).
      • Afterwards: all 8 hostnames answered as at the 08:13Z baseline, webhooks arrived with 0 signature rejects, and v2 had served 306 requests with 0 errors.
      • Step 7: the operator created INFRA_DISPATCH_TOKEN (credentials; deadline #47). The Worker change that uses it is rig-cockpit-worker#315.
      • invotek.no: the operator renewed it on 2026-10-16. Deadline #5 has moved to 2027-10-16. Norid's public WHOIS shows no expiry date, so this rests on the operator's word.
      • Still the operator's:
      • the offline backup key O2 (operator-inbox#43);
      • the SSD decision, after the write re-measure;
      • the literal yes that deploys #315.
    • Late evening: step 6 is complete, and the GCP tunnel rotation has started.
      • The rig-location switch (infra#669, step 6): four PRs, all merged and verified.
      • infra#673 (22:28:29Z): the state object gs://tablez-terraform-state/rig-location/desired.json ({"gcp":"running","primary":"gcp"}), infra's rig-standby environment (the operator is the only reviewer, main only, no admin bypass), and an inert stable-host rule on the Dell tunnel.
      • infra#676 (23:17:02Z): gcp/dashecorp and cloudflare/dashecorp.com read the object at plan time. The apply changed nothing.
      • infra#680 (23:30:27Z), Review-E's P1: a stop is published only while the stable host's live CNAME names dashecorp-dell-v2, and primary=gcp only while GCP's three VMs are live RUNNING, re-checked at write time. A 41-case offline selftest runs in CI.
      • infra#675 (23:41:03Z): rig-location.yml, dispatch only, not yet dispatched.
      • Change from the design: primary=dell also waits for the operator in rig-standby, so the step-7 token must have no Deployments permission. Failover and failback are two dispatches each.
      • The GCP tunnel rotation (design step 13, moved earlier):
      • R1, infra#678 (23:21:59Z): dashecorp-gcp-v2 with a random_bytes secret that OpenTofu never prints. A second run wrote GCP_TUNNEL_TOKEN to rig-gitops' credential-ops (23:27:55Z). Nothing moved.
      • R2, in review: rig-gitops#1034 (the c4-gcp step) and #1035 (a second GCP connector, cloudflared-v2, a draft until the run). #1035 also widens tgc-play's NetworkPolicy, which admitted only app: cloudflared.
      • Next: the c4-gcp run on the operator's literal yes, then R3 (every GCP CNAME → v2) and R4 (the old tunnel deleted).
      • Found: after a force-push, GitHub reported earlier reviews against the new head, in REST and GraphQL. MergeGate matches approvals by commit_id, so a stale approval might satisfy it. To be tested: dashecorp/rig-conductor#2720.
    • Backups (rig-gitops#897): O3, the R2 writer key pair, now goes through credential-ops as C4 and C5 did, so no person handles it. O2, the offline age key (operator-inbox#43), stays the operator's.
    • Still the operator's (both standby yeses came at 21:36:16Z, see Evening):
      • the literal yes for the c4-gcp credential-ops run (rotation R2);
      • creating the narrow INFRA_DISPATCH_TOKEN for GCP's auto-start (step 7), with Actions only and no Deployments;
      • renewing invotek.no, due 2026-10-16;
      • the offline backup key O2 for rig-gitops#897, needed before the first tenant moves to the Dell;
      • the SSD decision, after the 2026-10-16 write re-measure.
    • Evening: the Dell standby is complete, and the operator wants everything moved there.
    • PR-13 is live (rig-gitops#992, merged 18:07Z). Every agent fleet is defined on the Dell at 0 replicas. The agent-home claims are on local-path, and the fenced conductor runs at 1 (LOCATION=dell, no tenant map, every write fails closed). By 20:05Z all 19 HelmReleases were Ready, with 0 agents running.
    • The install burst stalled the worn drive: 1.7–6.7 s per I/O with ps=0, so wear, not APST (posted on infra#622). k3s restarted once, and the API was down about 2 minutes. Apps were unaffected.
    • Live webhook routing (rig-cockpit-worker#296) is deployed in shadow mode. Review-E caught a bug in it: the Queue's default json content type emptied every webhook body. It was fixed with contentType: 'v8'.
    • Incident, 18:54–20:03Z, the GCP planner was down (rig-gitops docs/2026-10-14-agent-env-duplicates-incident.md). #994 removed duplicate extraEnv entries. Helm 3's strategic merge then deleted both copies of each duplicated key from the live StatefulSets: the planner lost DISCORD_BOT_TOKEN, and four agents lost OPENAI_API_KEY. The revert (#996) could not re-add them ($setElementOrder). The fix was #997: the dedupe plus spec.upgrade.force (replace, not patch). It was verified on the live objects.
    • The goal now: everything on the Dell, GCP to zero, then deleted (plan rig-gitops#990). The Dell has more CPU and RAM than the whole GCP rig. The blocker is the worn drive. The recommendation is an external USB SSD at the house to hold k3s's data, awaiting the operator's decision. The other steps are the off-site backups (the O2 key, operator-inbox#43), the Dell tunnel and tenant map (PR-14 and PR-15), live routing, capacity guardrails (#989), and the tenant moves. Changed on 2026-10-15: the Dell becomes the primary and GCP a cold standby (rig-gitops#1021).
    • The Dell's etcd stalls have a cause (dashecorp/infra#622). A Mac Mini job analysed a read-only evidence bundle from the Dell, and the orchestrator spot-checked its findings.
    • The cause: for 2–5 hours at a time, the NVMe drive gets 15–21 times slower at the same load. There have been five such episodes since 10-07. Inside one, any write burst stalls etcd's WAL fdatasync:
      • The */15 CronJob unschedulable-pod-watch causes 90% of the warnings.
      • On 10-08, apt-daily caused a 25.7 s fsync, after which k3s lost its k3s-etcd lease and exited.

    • What slows the drive: it is worn out. The capture below found it at 13:47Z. smart-log reads:

      • critical_warning 0x4 (reliability degraded) and percentage_used 107%;
      • 110.63 TB written in 63,270 power-on hours;
      • spare at 100% and no media errors.

      Thermal throttling is ruled out: the drive has never had a thermal-management transition. APST is on, and not yet ruled out. The rig writes about 13 GB a day to the drive. fwupd lists no NVMe firmware update; the BIOS has one, 1.28.0 to 1.40.0. - Next: replace the drive (any M.2 2280 NVMe) at the operator's next visit, along with the BIOS update. No tenant moves to the Dell before then. - Data at risk is small. buzz's volumes are on the NAS, and the rest (the standby Postgres and Valkey, and a 550 MB etcd) rebuilds from GitOps. - Backups: etcd stores Secrets unencrypted here (secrets-encrypt is disabled), and the NAS export is open. So the NAS copy planned in rig-gitops#897 must be age-encrypted, which waits on the operator's O2 key. - Cleanup (the operator's yes): used space went from 45.0 to 30.3 GiB, by removing the two dead runner directories, 89 stale buildah images and two old k3s data dirs. Two more k3s data dirs stay, because running container shims still use them. A cleanup does not undo wear. - Corrections: my first read on the issue had two errors. Two Job pods did start in the 10-08 window, and sar does record nvme0n1. - Fixes, both live, through OpenTofu k8s/dell/k3s: - dashecorp/infra#637 (Closes #634): all five leases go to 120 s with a 90 s renew deadline. The controller-manager, scheduler and cloud-controller-manager are set through install_exec. k3s's own k3s and k3s-etcd are set through a systemd drop-in with CATTLE_ELECTION_*, which k3s v1.36.5 reads through rancher/wrangler v3.7.0. - Applied at 13:12:09Z: all five have read 120 s since 13:13:10Z. k3s restarted twice, about 15 s down in total. - k3s and k3s-etcd kept their holder. The three kube controllers had no leader for about 82 s while their old 60 s leases expired. - NRestarts reads 0, because systemd resets it on a manual restart. - dashecorp/infra#638 (Closes #635), live at 13:46:53Z: a drive capture. - nvme-cli 2.8-1ubuntu0.1. The operator approved it ("yes nvme-cli", 2026-10-14T12:40:55Z); it is recorded in approved-tools.md, and in infra's own registry with a security review. - A 30 s journal line, tag nvme-diag, and per-minute sar. - A one-shot read, tag nvme-diag-once. It stays in the journal only, because id-ctrl prints the serial. - The Dell's agents (PR-13) wait for the new drive. - Kept on purpose: the Dell's automatic security updates. It holds the shared SOPS key, so surviving the stall beats removing the trigger. - Operator decisions (2026-10-03): - #896 D1: GHCR pulls use a read-only classic PAT. - Merging: the orchestrator merges a PR once Review-E or an independent subagent review has approved it. Production changes, credentials and spend still go to the operator. - Approvals: every operator approval is to become a Rig Cockpit push notification with an Approve tap. GitHub Mobile is the stopgap until then. A GitHub App user token would approve deployments as the operator, so the design adds a passkey step-up.

GCP k3s Cluster (Dashecorp)

Host: GCP Compute Engine, europe-north1-b (Finland) Nodes: 3 Standard (non-preemptible) VMs, all e2-standard-2 (2 vCPU, 8 GB; ~7.75 GiB allocatable): invotek-k3s control plane + invotek-k3s-worker-1/2. The workers were e2-medium (4 GB) until infra#461 resized them on 2026-09-26 (see the 2026-09-25/26 incidents below). The workers were Spot until infra#385 merged 2026-07-02; verified PROVISIONING_MODEL=STANDARD, all RUNNING/Ready on 2026-09-08. GitOps repo: dashecorp/rig-gitops (Stig-Johnny/cluster-gitops archived 2026-06-03) Orchestrator: FluxCD Secrets: SOPS + GCP KMS IaC: OpenTofu (dashecorp/infra/gcp/dashecorp/, tofu CLI) Storage: local-path (aliased as nfs-csi). The agents' home is NFS. Since 2026-10-01 13:23Z every lane mounts PVC agent-home, an NFSv4.2 export of the 10 GB pd-balanced disk agent-home on invotek-k3s (10.166.0.33:/srv/agent-home, daily snapshots kept 7 days; dashecorp/infra#508, rig-gitops#885). The two 1 TiB Filestore instances it replaced were destroyed on 2026-10-13 (dashecorp/infra#632, on the operator's approval) after the 7-day soak and the rig-gitops cleanup (#883). Inode budget: Codex left one empty tmp/arg0/codex-arg0* directory per run, about 2,175 a day on this 655,360-inode disk. rig-agent-runtime#790 now prunes them after 2 hours (#763 closed), and the inode count stays flat. Cost: compute at europe-north1 on-demand list price is 3 × $53.86 = ~$162/month since 2026-09-26 (was ~$108 with e2-medium workers), before sustained-use discounts; disks extra. The startup credit is spent: the $2,000 Google for Startups credit ran out in August 2026, so the bill is paid in real money. September cost $542.80 gross: two 1 TiB Filestore volumes $366.24 (replaced on 2026-10-01 by the ~$1.25/month agent-home disk, and deleted on 2026-10-13, infra#632), E2 compute $112.74, Artifact Registry storage $43.78, the rest under $15. Read the current figures from the billing export (invotek-github-infra.billing_export) with bq query, not from this page. Billing account: since 2026-10-01 10:10:59Z the project bills to Dashecorp Billing (0151A6-EDAFD0-29E42A), no longer 015BBC-422A59-EB7AF4 (dashecorp/infra#530). Until Dashecorp Billing's BigQuery export is on, the table above stops at that time.

Migrated 2026-03-27: Dell k3s (Oslo) → GKE Autopilot → GCP spot VMs + FluxCD

Node service account and metadata access (since 2026-10-02)

All three VMs run as the project's default Compute account, 1090472687120-compute@. Until 2026-10-02 it held project Editor. Pods can reach the GCE metadata server, which the backup CronJob relies on. So any pod could take an Editor token and mint a key for github-actions@, the OpenTofu CI account that decrypts all infra state; the project has no organisation, so no policy blocked that.

Two layers close it: - Agent namespaces cannot reach 169.254.169.254 (rig-gitops#887, apps/metadata-guard). This covers dashecorp, dev-e, review-e, rig-planner, stigjohnny, rig-assistant and rig-lint. gce-pd-csi-driver and rig-conductor keep access. - The account holds exactly roles/compute.storageAdmin plus the custom role gcePdCsiAttach (instance get, attach and detach), and nobody holds Editor (dashecorp/infra BOOTSTRAP §24). gcp/dashecorp/node-sa.tf fails the plan if that drifts (infra#539). Changing those roles takes an operator command, because CI cannot change project IAM.

What Runs on It

Service	Namespace	Endpoint
rig-conductor API + dashboard	rig-conductor	rig-conductor.dashecorp.com via CF tunnel + Access; also NodePort 30080, no auth (in-VPC/tailnet only; networking.md, dashecorp/infra#448)
rig-planner	rig-planner	(cluster-internal)
Dev-E agent pods	dev-e	—
Review-E agent pods	review-e	—
Postgres (Marten event store) + Valkey (streams)	rig-conductor	Postgres cluster-internal, on the control node's local-path boot disk. Valkey is also on NodePort 30079 with auth off (in-VPC/tailnet only, dashecorp/infra#448), and its data is on a 2Gi local-path PVC (data-valkey-upstream-0)
Rig-E — dashecorp tenant chat agent (rig-agent-runtime, read-only, brain = operator vLLM pod)	dashecorp	rig-e.dashecorp.com via CF tunnel → Rig Cockpit picker
dashecorp tenant fleet (review-e / dev-e / planner @dashecorp, KEDA 0→1)	dashecorp	—
stigjohnny tenant fleet	stigjohnny	Pending — waits on the missing stigjohnny-shared-claude PVC (parked), not on node capacity

RunPod brain (conductor-owned)

The rig-conductor pod on this cluster owns the operator's on-demand vLLM GPU pod on RunPod (dashecorp/rig-conductor → docs/runpod-pod-control.md is canonical): POST /api/pods/qwen/start runs a volume-first wake — reuse the pod attached to volume qwen-weights (bjn7d94ej9, EU-RO-1) if it can get its GPU back, otherwise create a replacement from template cnz5fxyadi on the cheapest acceptable GPU (≥ 32 GB, ≤ 1.00 USD/h) and terminate the stale pod; POST …/stop; GET … reports podId + proxyUrl. RunPodIdleMonitor stops the pod after 15 idle minutes counted from model-ready — the first successful vLLM /metrics scrape, not from RUNNING (rig-conductor#2056, live 2026-09-09; the flat window used to spend the ~6-minute model load as idle). A pod that never becomes ready is stopped after RUNPOD_BOOT_GRACE_MINUTES (default 20), and the 6 h wall-clock cap is unchanged. GET … also returns readySince — isRunning: true with readySince: null means "waking, not usable yet". All on RunPod REST v2 (v1 retired, rig-conductor#2043). Key: rig-conductor-secrets/runpod-api-key (see credentials.md). Rig-E no longer pins a literal pod URL: llm.baseUrlFrom resolves it from the conductor's GET /api/pods/qwen at boot and per turn, re-resolving once on a 404/502/503/504/network failure (rig-agent-runtime#677, shipped 2026-09-09), so a RunPod migration or a replacement pod costs no gitops PR. A chat turn against a stopped pod also wakes it (rig-agent-runtime#682/#689). Proven unattended on 2026-09-08 (idle stop fired; volume resolution without a pin); first live wake-on-chat 2026-09-09.

Operator runbook: gpu-brain.md — wake it, tell whether it is actually usable, stop it, and what the waking replies mean.

Three-node placement — the 2026-09-07 outage and its fixes

What happened: invotek-k3s-worker-2 went into a memory-pressure livelock (no OOM-kill, kubelet stopped answering) and took the conductor and Rig-E down with it — a full rig outage surfacing as a cockpit 502. Root cause: every runtime and the conductor still carried requests.memory: 1Mi from the single-node era, so the scheduler had bin-packed 7 runtimes + the conductor + Rig-E onto one 3.9 GiB worker. Post-mortem: rig-conductor#2003.

Fixes (verified in production the same night):

PR	Change	Effect
rig-gitops#725	13 HelmReleases: requests.memory 1Mi → 640Mi (7 × 640 Mi > one worker's 3.9 GiB)	scheduler is forced to spread the fleet — now 4 / 4 across the workers
rig-conductor#2005	conductor requests.memory 512Mi → 1536Mi; preferred nodeAffinity for the control plane (weight 100); preferred podAntiAffinity vs app.kubernetes.io/name=rig-agent-runtime with namespaceSelector: {} (weight 50)	conductor lands on the 8 GiB control node next to Postgres, never packed in with runtimes
rig-cockpit-worker#77	external node-liveness check (open)	the one post-mortem item not yet built — nothing outside the cluster notices a hung node today

Placement on 2026-09-08 (live):

Node	Allocatable	Runs
invotek-k3s (control plane)	8 GiB	postgres-0, rig-conductor-api
invotek-k3s-worker-1	4 GiB	rig-e, dev-e-codex, review-e-codex, rig-planner
invotek-k3s-worker-2	4 GiB	dev-e, dev-e-dotnet, dev-e-dotnet-codex, review-e

Rules that came out of it: memory requests are what the scheduler bin-packs on, so they must be honest on a multi-node cluster (the old "1Mi so it always schedules" rule is single-node-only — Dell, not the rig). Pod (anti-)affinity is namespace-scoped by default; a rule meant to keep the conductor away from runtimes in other namespaces is a silent no-op without namespaceSelector: {} (Review-E caught this on #2005). Canonical write-up of the memory-request change lives in dashecorp/rig-gitops → docs/2026-06-18-single-node-cluster-resilience.md (amended by #725).

The 2026-09-25/26 incidents — two nodes overloaded by the heaviest agent

Control plane, 2026-09-25 from ~19:52Z. The dashecorp Dev-E pod landed on the control-plane node by chance: no agent had node affinity, and its local-path PVC dashecorp-shared-claude had bound there. It ran a full vitest/workerd suite (~1.07 GB cgroup) next to k3s-server's SQLite, Postgres and the conductor, and the boot disk saturated: I/O PSI ~90%, load 20 on 2 CPUs. That caused 67 Postgres read timeouts an hour, a conductor liveness restart, ~25% failed heartbeats, k3s API timeouts and Flux/KEDA crash-loops. Full event-table scans in four pollers made it worse (rig-conductor#2476).

Worker-2, 2026-09-26 from 12:10Z. After the fix below moved the dashecorp agents to invotek-k3s-worker-2, that node (then e2-medium, 3.8 GiB allocatable) ran five agent pods, each requesting 640Mi with a 2Gi limit. Dev-E alone peaked at 1516Mi. The node livelocked on memory: kubelet and sshd dead, the VM still RUNNING. Both invotek Review-E pods were stuck Terminating on it, so no PR could be reviewed or merged.

Change	What it did
rig-gitops#865	Dashecorp agents get required anti-control-plane affinity (a Flux postRenderers patch, because the chart has no affinity value) and a new claim, dashecorp-shared-claude-v2, which bound on a worker. The v1 claim is kept, so Flux does not prune it.
rig-gitops#867	The v2 permissions Job gets an imagePullSecrets it lacked. v1's copy had been Failed for 88 days for the same reason.
infra#461	Workers e2-medium → e2-standard-2 (8 GB), +$53.86/month for both. allow_stopping_for_update made the CI apply a stop/resize/start, which power-cycled the hung worker-2 through IaC instead of a manual reset.
rig-gitops#869	The dashecorp Dev-E requests 1536Mi, its measured peak, instead of 640Mi.
rig-conductor#2481/#2489/#2495	The conductor builds a (type, seq_id) index on mt_events per tenant with CREATE INDEX CONCURRENTLY (invotek took 17.7 s). There were 0 Postgres read timeouts in the first 70 minutes after it went live.

Still open: - The other agent fleets still request 640Mi. Review-E peaks at ~860Mi. - There is no external node-liveness alert (rig-cockpit-worker#77). - Three Cloudflare modules fail every infra apply on a metadata-only diff (infra#463).

The anti-control-plane rule for apps/dev-e and apps/review-e, listed here before, landed in rig-gitops#880: all 10 base HelmReleases in apps/dev-e, apps/review-e and apps/rig-planner carry it (checked 2026-10-02).

Single-node-era resilience (historical, 2026-06; tolerations still in force)

Written when the workers were Spot-preempted and everything ran on the control node. That premise ended 2026-07-02, but the tolerationSeconds: 3600 mitigation and the boot-disk hygiene below remain in force and still matter — Postgres and Valkey still live on the control node's local-path boot disk.

Everything used to run on the one on-demand e2-standard-2 control node (the two spot workers stay preempted), with Postgres + Valkey on its local-path boot disk. That makes the node I/O-bound: a burst of disk I/O (a ~1.3 GB agent image-roll, even a backup pg_dump) can saturate the boot disk, stall the kubelet heartbeat → NodeNotReady → the TaintManager mass-evicts every pod (a full cascade outage hit 2026-06-18).

Eviction mitigation (complete 2026-06-18): long not-ready/unreachable NoExecute tolerations (tolerationSeconds: 3600) now ride every GitOps-managed pod through a transient NotReady instead of evicting it — data plane (rc#1842), agent fleet (rig-agent-runtime#633), Flux + KEDA controllers (rig-gitops#640/#642), Valkey (rig-gitops#645). Only k3s-managed coredns/metrics-server keep the default 300 s. Canonical write-up: dashecorp/rig-gitops → docs/2026-06-18-single-node-cluster-resilience.md.

Boot-disk hygiene: the image-prune CronJob (kube-system, 04:17 UTC; since 2026-10-14 a DaemonSet that prunes every node daily in the 04:xx UTC hour, rig-gitops#985) was fixed so it actually reclaims the ~27 GB containerd image store (rig-gitops#638); the rig-conductor Postgres backup runs daily 02:00 UTC (rig-conductor#1846/#1849/#1852).

Backups: running, more complete since 2026-09-23, and still never restored

The daily 02:00 dump succeeds and has for months. Measured on 2026-09-23, four separate things were wrong with what that success meant — read this before trusting "there are backups":

finding	state
Nothing had ever restored a dump. The restore-test CronJob was written in June, declared shipped in rc#1781, and was never in the cluster — it sat only in the unused deploy/k8s/kustomization.yaml while Flux applies deploy/. It also still carried the bookworm-pgdg apt repo that rc#1848 fixed in its sibling, so its first run would have died at setup	rc#2310 fixed. Now deployed as rig-conductor-restore-test (0 4 * * 1) but suspend: true — it needs one manual proving run before it means anything: kubectl -n rig-conductor create job --from=cronjob/rig-conductor-restore-test restore-test-manual-1, read the log for ok tenant=… mt_events=<n>, then unsuspend
It dumped 5 of 7 non-system databases. rig_control — the tenant-to-database map the job reads to decide what to dump — was never dumped, so a restore gave per-tenant objects with no record of whose they were	rc#2316 fixed (rc#2318). rig_control is dumped first, and the job now enumerates pg_database and names anything unaccounted. A control-dump failure alerts and fails the run but no longer skips the tenant dumps (rc#2322). First control dump: backups/_control/2026-09-23/control.dump, 40,760 B, 13:56:22Z
invotek's memory DB has never been dumped, once. mem is derived by the convention rig_t_<id>_mem, and the primary tenant predates TenantProvisioner (its event DB is still rig_conductor), so that name has never existed. The run still printed "all tenants ok (evt + mem)"	reported per-tenant now; resolving where invotek's memory actually lives is still open
No retry could succeed. The backup SA is roles/storage.objectCreator (create, not delete), so a second attempt 403'd on everything the first had uploaded (a GCS overwrite is a delete plus a create), and backoffLimit: 2 was decorative	rc#2325 shipped create-only uploads (ifGenerationMatch=0). A 412 keeps the existing object. A 403 falls back once to the exact old request, so a retry is never worse than before. Whether GCS returns 412 before the IAM 403 for an objectCreator-only SA is unproven, and rc#2320 stays OPEN until a real backup log shows it. Until then, never re-run the backup on a day it already ran

Every third-party image that ran on a floating tag in the GCP rig's manifests is now digest-pinned (2026-09-23). CI keeps only the rig-gitops half of them that way. cloudflare/cloudflared (the public tunnel), envoyproxy/envoy, nginx (cost-dashboard), the alpine:3.20/3.21 CronJobs and alpine/git (rig-gitops#822); google/cloud-sdk:slim in the backup and restore-test CronJobs (rc#2330; this tag had already moved Debian under the backup once, which is rc#1848); pgvector/pgvector:pg16 for postgres-0 (rc#2332, PostgreSQL 16.13). rig-gitops image-pin-check.yml (rig-gitops#826) fails any image: in apps/ or clusters/ that is not a digest, one of our own sha- builds or a full X.Y.Z tag. Only three Dell-only images are allowlisted (rig-gitops#825). The rig-conductor repo's deploy/ has no such gate (rc#2339): Flux applies it from that repo, so the google/cloud-sdk and postgres-0 pgvector pins would survive a PR only by review. That is not "every image is digest-pinned". The Flux controllers and coredns/coredns:1.11.3 run on upstream semver release tags, which the gate accepts by policy, so they are immutable by convention only. And the gate cannot see an image that is not written in the repo: KEDA (2.16.*) and Weave GitOps (4.*) render theirs from chart defaults and set no image:, and the k3s-bundled rancher/* images are not in the manifests at all. KEDA and the rancher/* images run on upstream release tags too. A pin makes an image immutable. It does not review it (see approved-tools.md).

Valkey is upstream docker.io/valkey/valkey:9.0.6, pinned by index digest sha256:63346cb2… (the same on ghcr.io/valkey-io/valkey). It is a plain StatefulSet valkey-upstream in rig-conductor, and clients reach it as valkey-primary (and valkey-external NodePort 30079). It replaced Bitnami 9.0.3 on 2026-09-23 at 18:34Z (rig-gitops#819): first a replica (#834), then a prep step that keeps the Service across helm uninstall and holds the replica until Bitnami stops (#836), then the cutover (#837). The client gap was about 10 s. Runbook: dashecorp/rig-gitops → docs/2026-09-23-valkey-switchover.md. Auth is off. RESTORE, FLUSHDB and FLUSHALL are disabled in its valkey.conf (upstream fixed CVE-2026-25243 in 9.0.4 and CVE-2026-63639 in 9.0.5; the security review still requires the command off). Data lives on PVC data-valkey-upstream-0 (local-path, 2Gi).

A few things are left over (rig-gitops#838):

• Bitnami's PVC valkey-data-valkey-primary-0 is kept as the rollback until 2026-09-30.
• Suspended staging still renders the Bitnami chart.
• Never roll back to bitnamilegacy/valkey:8.0.2. It would downgrade onto a 9.x AOF.

The pin gate does not stop this digest being deleted on its own: a bare :9.0.6 passes as a full release tag (rig-gitops#839).

Durable fix (operator-gated): move Postgres off the boot disk onto an SSD pd-balanced PVC — dashecorp/rig-conductor → docs/2026-06-11-postgres-pd-balanced-migration.md (needs the #admin-approved GCE PD CSI driver + the node-SA IAM grant in dashecorp/infra BOOTSTRAP step 20). Tolerations only bound the blast radius; they do not stop the node going NotReady. Open conductor finding: rig-conductor#1854 (agent heartbeats event-sourced → unbounded mt_events growth).

Worker nodes — Spot preemption → Standard migration — RESOLVED 2026-07-02

Resolved: infra#385 merged 2026-07-02; both workers are Standard and Running (verified 2026-09-08), and e2-standard-2 since infra#461 (2026-09-26). The stigjohnny fleet is still Pending, but on the missing stigjohnny-shared-claude PVC (parked), not on worker capacity. The gcp/dashecorp apply on main is still RED, for an unrelated pre-existing KMS permission issue (infra#440). The original text is kept below as history.

The two workers (invotek-k3s-worker-1/2) are Spot e2-medium VMs, so GCP preempts them and they sit TERMINATED (last preempted 2026-06-10). While they're down, anything that can't fit on the single control node stays Pending — e.g. the per-tenant stigjohnny agent fleet (dev-e/review-e/planner StatefulSets) has been Pending since the workers died. The preempted VMs also leave orphan NotReady node objects (Kubelet stopped posting node status) that must be deleted as part of the fix. The shared dev-e/review-e/rig-planner fleets run fine on the control node; only workloads needing worker capacity are affected.

Durable fix — infra#382 / #385 (DRAFT, operator-gated): switch the workers to Standard (non-preemptible) VMs that self-join via a k3s node-token fetched from GCP Secret Manager by the instance SA (no token in TF state). The CI WIF SA has no Secret Manager role, so the secret + IAM are operator-owned: rotate the node-token, gcloud secrets create k3s-node-token + add the version + grant secretAccessor to the compute SA, delete the two terminated VMs, then mark infra#385 ready → apply creates the Standard workers. Runbook: dashecorp/infra → docs/2026-06-27-k3s-worker-secret-manager.md. Until then gcp/dashecorp apply on main is RED (benign — only the worker resources fail; all other infra applies are green).

ARC Runners — RETIRED

ARC k3s runners were retired 2026-05-20 (operator-confirmed). The arc-systems and arc-runners namespaces, the 25 HelmReleases formerly in dashecorp/runners/, and the arc-linux-{repo} / arc-linux-docker labels are no longer in use. All Linux CI moved to GitHub-hosted ubuntu-latest.

Access

# kubectl via kubeconfig
export KUBECONFIG=~/.kube/k3s-gcp-config
kubectl get pods -A

# SSH — BOTH flags are mandatory (see the note below)
gcloud compute ssh invotek-k3s \
  --account stig@dashecorp.com --project invotek-github-infra --zone europe-north1-b
# ^ RESOLVED 2026-09-21 (claude-3#901). The 2026-09-10 failure
#   ("Required 'compute.instances.get' permission for projects/invotek-github-infra",
#   which blocked every cockpit change for hours during a Review-E stall —
#   dashecorp/rig-conductor#2072) was a WRONG-ACCOUNT error wearing a lost-access
#   costume. The project name is NOT stale. Listing both accounts directly:
#     post@nutrieapp.com  -> includes invotek-github-infra (1090472687120)   YES
#     invotekas@gmail.com -> sunlit-mantra-476909-r4, and nothing else        NO
#   invotekas@gmail.com is the Invotek OWNER account (Play Store owner actions,
#   Cloudflare + Tailscale login, Cloudflare Access, GitHub enterprise billing) and
#   has never had GCP compute access. stig@dashecorp.com is the rig account (post@nutrieapp.com until
#   2026-09-29; that domain was to lapse 2026-10-05 (claude-3#975) but auto-renewed to 2027-10-05, refund
#   requested); it is
#   also what every `gcloud compute ssh --command=` line in gpu-brain.md uses.
#   The trap: the only gcloud config is `default` -> sjs@veriton.io / verirton-dev,
#   so an unflagged command runs as the wrong identity and fails as above. Pass
#   --account AND --project every time. Do NOT `gcloud config set account` — that
#   would disturb unrelated Veriton work on the same machine.
#   Still true: the local kubeconfig holds only unrelated clusters, so reach the
#   cluster over SSH (or `--command=...`) rather than expecting kubectl to work here.
#   Not declared anywhere: human IAM on invotek-github-infra has no
#   google_project_iam_member/_binding in dashecorp/infra, so there is no
#   authoritative record of who can reach the rig. That is why this took an outage
#   plus eleven days to answer.

# FluxCD status
flux get kustomizations
flux get helmreleases -A

Why agents have no read access to the rig cluster (2026-10-04). The only route is gcloud compute ssh plus sudo k3s kubectl, which is root on the control plane, and the auto-mode classifier blocks it as a "Production Read". The MacBook also has no rig kubeconfig context and no Tailscale. So reading the cluster, for example the logs behind rig-conductor#2639, is the operator's job.

Proposed fix, NOT approved: a GET-only "rig status" endpoint on rig-cockpit-worker for the agent identity (agents online, stream pending). The operator hasn't answered, so don't build it without their OK.

Dell k3s

Its current state, the 2026-10-02 cleanup and the operator's plan are in Dell k3s cluster (Ålesund) above. vCluster Platform and the starc leftovers listed here before were removed that day.

Access

ssh -i ~/.ssh/dell-stig-1 claude@100.95.212.93