Kubernetes¶
Two separate k8s environments managed by GitOps.
GCP k3s Cluster (Dashecorp)¶
Host: GCP Compute Engine, europe-north1-b (Finland)
Nodes: 1 e2-standard-2 (on-demand, control plane) + 2 e2-medium workers (spot, TERMINATED — Spot-preempted; migrating to Standard via infra#385, see below)
GitOps repo: dashecorp/rig-gitops (Stig-Johnny/cluster-gitops archived 2026-06-03)
Orchestrator: FluxCD
Secrets: SOPS + GCP KMS
IaC: OpenTofu (dashecorp/infra/gcp/dashecorp/, tofu CLI)
Storage: local-path (aliased as nfs-csi)
Cost: ~$44/month (GCP startup credits, expires Nov 2027)
Migrated 2026-03-27: Dell k3s (Oslo) → GKE Autopilot → GCP spot VMs + FluxCD
What Runs on It¶
| Service | Namespace | Endpoint |
|---|---|---|
| rig-conductor API + dashboard | rig-conductor | (cluster-internal) |
| rig-planner | rig-planner | (cluster-internal) |
| Dev-E agent pods | dev-e | — |
| Review-E agent pods | review-e | — |
Single-node resilience (invotek-k3s)¶
Everything runs on the one on-demand e2-standard-2 control node (the two spot workers stay preempted), with Postgres + Valkey on its local-path boot disk. That makes the node I/O-bound: a burst of disk I/O (a ~1.3 GB agent image-roll, even a backup pg_dump) can saturate the boot disk, stall the kubelet heartbeat → NodeNotReady → the TaintManager mass-evicts every pod (a full cascade outage hit 2026-06-18).
Eviction mitigation (complete 2026-06-18): long not-ready/unreachable NoExecute tolerations (tolerationSeconds: 3600) now ride every GitOps-managed pod through a transient NotReady instead of evicting it — data plane (rc#1842), agent fleet (rig-agent-runtime#633), Flux + KEDA controllers (rig-gitops#640/#642), Valkey (rig-gitops#645). Only k3s-managed coredns/metrics-server keep the default 300 s. Canonical write-up: dashecorp/rig-gitops → docs/2026-06-18-single-node-cluster-resilience.md.
Boot-disk hygiene: the image-prune CronJob (kube-system, 04:17 UTC) was fixed so it actually reclaims the ~27 GB containerd image store (rig-gitops#638); the rig-conductor Postgres backup runs daily 02:00 UTC (rig-conductor#1846/#1849/#1852).
Durable fix (operator-gated): move Postgres off the boot disk onto an SSD pd-balanced PVC — dashecorp/rig-conductor → docs/2026-06-11-postgres-pd-balanced-migration.md (needs the #admin-approved GCE PD CSI driver + the node-SA IAM grant in dashecorp/infra BOOTSTRAP step 20). Tolerations only bound the blast radius; they do not stop the node going NotReady. Open conductor finding: rig-conductor#1854 (agent heartbeats event-sourced → unbounded mt_events growth).
Worker nodes — Spot preemption → Standard migration (operator-gated)¶
The two workers (invotek-k3s-worker-1/2) are Spot e2-medium VMs, so GCP preempts them and they sit TERMINATED (last preempted 2026-06-10). While they're down, anything that can't fit on the single control node stays Pending — e.g. the per-tenant stigjohnny agent fleet (dev-e/review-e/planner StatefulSets) has been Pending since the workers died. The preempted VMs also leave orphan NotReady node objects (Kubelet stopped posting node status) that must be deleted as part of the fix. The shared dev-e/review-e/rig-planner fleets run fine on the control node; only workloads needing worker capacity are affected.
Durable fix — infra#382 / #385 (DRAFT, operator-gated): switch the workers to Standard (non-preemptible) VMs that self-join via a k3s node-token fetched from GCP Secret Manager by the instance SA (no token in TF state). The CI WIF SA has no Secret Manager role, so the secret + IAM are operator-owned: rotate the node-token, gcloud secrets create k3s-node-token + add the version + grant secretAccessor to the compute SA, delete the two terminated VMs, then mark infra#385 ready → apply creates the Standard workers. Runbook: dashecorp/infra → docs/2026-06-27-k3s-worker-secret-manager.md. Until then gcp/dashecorp apply on main is RED (benign — only the worker resources fail; all other infra applies are green).
ARC Runners — RETIRED¶
ARC k3s runners were retired 2026-05-20 (operator-confirmed). The arc-systems and arc-runners namespaces, the 25 HelmReleases formerly in dashecorp/runners/, and the arc-linux-{repo} / arc-linux-docker labels are no longer in use. All Linux CI moved to GitHub-hosted ubuntu-latest.
Access¶
# kubectl via kubeconfig
export KUBECONFIG=~/.kube/k3s-gcp-config
kubectl get pods -A
# SSH
gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra
# FluxCD status
flux get kustomizations
flux get helmreleases -A
Dell k3s (Decommissioned)¶
The Dell cluster (100.95.212.93) previously ran all Dashecorp workloads. ARC runners and ArgoCD auto-sync have been disabled. The Dell still hosts:
- vCluster Platform (vcluster.invotek.no)
- Legacy starc project