Skip to content

CI/CD

Workflows, runner routing, Discord notifications, auto-merge, and release pipeline.

Standard Workflow Set (legacy Stig-Johnny app repos — ARCHIVED)

The per-repo workflow set below is the legacy Stig-Johnny pattern and still lives in this orchestrator repo (Stig-Johnny/claude-3). The 6 consumer apps moved to dashecorp/dashe-* on 2026-06-09 and are now rig-managed: they do not use request-review.yml (review is dispatched push-based by rig-conductor via the Valkey assignments:review-e stream) and carry a different, larger workflow set (e.g. build-and-upload.yml, android-release.yml). Do not add these per-repo workflows to dashecorp repos. Operator-authored PRs, including those from claude-N sessions (which use the operator's account), are not routed to Review-E automatically. Add the needs-review label, and ReviewScanService hands the PR to Review-E within about 5 minutes. MergeGate then squash-merges about 30 s after an APPROVED verdict on the current head with every check green. Seen on dashe-cutie#41 and #43 on 2026-09-29; #14 sat unreviewed for 19 days without the label. A PR over 500 added+deleted lines or 20 files gets CHANGES_REQUESTED asking for a split. Tests and docs count toward that, and large-pr-ok is only for migrations, codemods, dependency bumps and generated code. Four more behaviours, found on 2026-09-29: (1) On a repo whose conductor webhook delivers, rig-planner adds agent-ready to every T1 issue within about 1–3 minutes, even when an open PR already closes it, so Dev-E duplicates the work. For an issue you implement yourself, remove the label and comment (dashecorp/rig-agent-runtime#756). (2) Don't re-run failed checks on a PR Review-E has already approved: MergeGate's guard can merge in the window before the new check runs exist (dashecorp/rig-conductor#2513, which merged dashecorp/infra#475 with a failed plan). (3) merged_by=Stig-Johnny doesn't mean a human merged, because MergeGate's merges show that account too. (4) A closing verb before #N anywhere in a PR body (close, fix or resolve, in any tense), even in prose about a later step, closes that issue on merge. Write "stays open until step 4" instead. (5) Found on 2026-10-18: a Review-E run that takes longer than about 20 minutes is lost without a trace. Review-E's KEDA trigger is a Redis list that the agent pops when it wakes, so cooldownPeriod: 1200 scales the pod to 0 twenty minutes later, even mid-review, and the 20-minute CLI timeout (dashecorp/rig-agent-runtime#727) ends at the same moment. No review is posted, no failure is recorded, and ReviewScanService does not route the same head again. Keep PRs well under the 500-line limit so a review fits, and re-dispatch a silent PR with an empty commit (dashecorp/rig-gitops#1050).

The legacy app repos had these 7 workflows:

Workflow Trigger Purpose
request-review.yml pull_request opened/reopened/ready_for_review/synchronize/closed Request Review-E + Discord #tasks thread; re-request review when a stale CHANGES_REQUESTED is pushed over; notify on close/merge
pr-comments-to-discord.yml issue_comment Forward PR comments to Discord thread
forward-copilot-reviews.yml pull_request_review Forward Copilot review to Discord
build-status-to-discord.yml check_suite completed Post build status to Discord thread
auto-resolve-copilot-conversations.yml check_suite completed Resolve Copilot conversation threads
notify-failure.yml workflow_run completed Discord alert on CI failure
release-please.yml push to main Version bump, changelog, GitHub Release, auto-merge release PR

Known gaps: none actionable. The old "Nutri-E auto-merge on older version" gap is moot — Stig-Johnny/nutri-e is archived read-only and its tracking issue #309 is closed.

Review-E identity: Review-E reviews as the review-e-bot[bot] GitHub App; the legacy review-e-dashecorp user account is being retired. Workflows that key off Review-E's reviews must match both logins. request-review.yml was missed in the App migration — its synchronize re-review check only matched review-e-dashecorp, so a stale CHANGES_REQUESTED pushed over by the bot was never re-requested and the PR stalled (observed on #733; tracked in #745, fixed in #746).

Runner Routing

Job Type Runner Label Host
iOS SDK build (Xcode-required) macos-26 GitHub-hosted. cuti-e/ios-sdk CI runs on the org-standard macos-26 label — the same runner as the dashe-* app builds (build-and-upload.yml). Its release-please / pin-expiry-check workflows run on ubuntu-latest. (The self-hosted macOS runner was retired 2026-06-29 — see note below.)
Android build / publish ubuntu-latest GitHub-hosted. cuti-e/android-sdk publishes on ubuntu-latest. (The old [self-hosted, macOS, ios] MacBook Air "AAPT2 x86 via Rosetta" routing was retired 2026-06-29.)
Linux CI (linting, scripts, release-please, Docker actions) ubuntu-latest GitHub-hosted
Linux self-hosted none The last 2, Dell systemd runners scoped to Stig-Johnny/mcp-server-dotnet (orphaned, infra#389), were deregistered and removed on 2026-10-02. There is no self-hosted runner anywhere.

ARC k3s runners were retired 2026-05-20 (operator-confirmed). All Linux CI uses GitHub-hosted ubuntu-latest. The cost is acceptable at current rig volume.

The self-hosted macOS CI runner was retired 2026-06-29 — there is no self-hosted macOS runner anywhere now (the Mac Mini runner stood up earlier that day was decommissioned the same day). Moving iOS/Android CI to GitHub-hosted runners removes the self-hosted-runner maintenance burden that caused a 24h queue-cancel outage. The Mac Mini M4 (Tailscale 100.92.170.124, user claude) hosts iBuild-E only (iClaw-E retired 2026-09-25, see hosts.md) — no CI runner; the MacBook Air M4 hosts dev + Claude Code only — no runner. The only self-hosted runners are the 2 Dell Linux systemd runners (below) — and they are repo-scoped to Stig-Johnny/mcp-server-dotnet (not a general fallback), currently orphaned and flagged for decommission (infra#389).

PR Merge Flow

Note: There is no per-PR auto-merge.yml workflow in the app repos. Regular PRs are merged manually via gh pr merge once CI + Copilot pass. Automatic merging applies only to release PRs (see Release Pipeline below, handled by release-please.yml). Per-PR auto-merge is potential future work for the app repos, not a current implementation. One live exception: cuti-e/ios-sdk has an auto-merge.yml gated to Dependabot only.

cuti-e/ios-sdk gets no automatic review (checked 2026-09-29). Copilot reviewed none of the last four PRs (#102, #103, #105, #106), and requesting it through the API or gh pr edit --add-reviewer @copilot does nothing. request-review.yml still requests the retired review-e-dashecorp user. Before merging, get an independent review: #106 had a consent bug that only a review agent caught, where a retry could send a ping after opt-out. Then squash-merge by hand, and merge the release-please PR by hand too, as with #89 and #108. Do not hand-edit .release-please-manifest.json, CHANGELOG.md or the x-release-please-version line in a feature PR. release-please takes the last version from the last merged release PR, so a hand edit gives a duplicate CHANGELOG section.

claude-3 (orchestrator repo): its own per-PR auto-merge.yml was removed 2026-06-23 (#812). It had relied on the legacy classic GH_PAT, which expired and failed every merge with 401 Bad credentials. Per operator decision, claude-3 PRs now merge manually (orchestrator squash via gh pr merge --squash) — which is also rig-conductor's own documented fallback. The target is for rig-conductor (conductor-e) to own claude-3's merge gate, but per BRAIN the conductor MergeGate is scoped to dashecorp-org repos only, so that requires migrating claude-3 into dashecorp first (not yet scheduled).

Orchestrator cross-org reads (monitor-infra.yml, build-unified-docs.yml): the same expired GH_PAT was replaced by the codi-e-bot GitHub App (App ID 4127706, @dashecorp-owned, read-only). These workflows mint short-lived per-org tokens via actions/create-github-app-token from the ORCHESTRATOR_BOT_PEM repo secret. The action requires the key in PKCS#8. The secret was first uploaded as PKCS#1, which failed both workflows with error:1E08010C:DECODER routines::unsupported from June until the operator re-set it on 2026-09-30 (#765). Both have minted tokens since (runs 36696733943 and 36696738684). Runbook: docs/runbooks/ops-e-reset-orchestrator-bot-pem.md. App details + PKCS#8 conversion: .claude/skills/github-apps/SKILL.md.

  1. PR opened → request-review.yml creates Discord thread, tags Review-E
  2. CI runs on GitHub-hosted ubuntu-latest
  3. Copilot reviews automatically
  4. Maintainer checks CI status, Copilot review, and unresolved conversations
  5. Squash merge via gh pr merge
  6. Merge notification posted to Discord thread

dashe-cutie production deploys

Since 2026-09-29 (claude-3#976), dashe-cutie deploys to production automatically. The operator approves only changes that cannot be undone. Two GitHub environments, declared in OpenTofu in dashecorp/infra github/dashecorp/dashe-cutie-environments.tf, carry the rule:

Environment Reviewer Used by
production-auto none; only main may deploy deploy-worker.yml when every pending D1 migration is additive, or none is pending; the Pages deploys on main
production the operator deploy-worker.yml when a pending migration is destructive or cannot be classified; any run not on main; the dispatch-only deploys (deploy-production.yml, deploy-all.yml's deploy-worker, deploy-admin-dashboard-production.yml)

The Worker path (deploy-worker.yml, dashe-cutie#48): runs are serialized by concurrency: deploy-worker, and each Pages workflow has its own group.

  1. A classify-migrations job reads the applied names from production's d1_migrations (read-only) and classifies each pending file with cloudflare-worker/scripts/migration-classifier.js. The classifier is an allowlist: CREATE TABLE, CREATE [UNIQUE] INDEX, CREATE VIEW, ALTER TABLE … ADD COLUMN, and INSERT [OR IGNORE] INTO without an upsert. Anything else, a failed lookup, or a failed classify job (dashe-cutie#53) sends the deploy to production.
  2. The production job records the live Worker version, applies migrations, deploys, and runs the smoke tests.
  3. On the automatic path, a failure after the live version changed rolls the Worker back to the recorded version, and the run fails.

Why the line sits at migrations: wrangler rollback does not undo D1 changes. The decision note is dashecorp/infra docs/2026-09-29-dashe-cutie-autonomous-deploys.md, which supersedes docs/2026-09-29-dashe-cutie-production-gate.md.

Alerts: a failed run reaches #admin through rig-conductor's CI-failure alerter. That depends on dashe-cutie's conductor webhook being signed and subscribed to workflow_run, which it has been since dashecorp/infra#475. dashe-reward-e's hook was signed the same way in dashecorp/infra#482. The other five dashe-* app hooks are still unsigned, so the conductor rejects their deliveries and posts "webhook signature mismatch" to #admin (dashecorp/infra#474).

Rollback drill: gh workflow run deploy-worker.yml --repo dashecorp/dashe-cutie --ref main -f rollback_drill=true redeploys main, fails the smoke test on purpose, and rolls back to the version that was live, so users see no change. It was verified on 2026-09-29 in run 36578355346 (dashe-cutie#70/#71):

  1. It recorded 127aa255, deployed ca389004, and failed on purpose.
  2. wrangler rollback restored 127aa255 at 100% of traffic, and health was ok.

  3. admin got the CI-failure alert.

Agents never approve a run waiting on production, even though their gh identity is the operator's account and could technically do it. Report the run link and stop.

rig-gitops: credential-ops and the phone-only C2 path (since 2026-10-03)

The operator runs credential steps from the phone: a pre-reviewed workflow waits in a gated environment, and the operator approves it in GitHub Mobile until Rig Cockpit approvals exist (claude-3#1027, see Credentials → Credential automation).

The credential-ops environment (dashecorp/infra#575), verified: - the operator (Stig-Johnny) is the only reviewer; - admins cannot bypass it (can_admins_bypass=false); - it deploys from main only.

Agents never approve a run waiting on credential-ops. Report the run link and stop. The one exception is the standing rule in CLAUDE.md (credential-ops approvals, claude-3#1190), in effect since the operator merged it on 2026-10-15 (#1191). Under it, the orchestrator may approve a dashecorp/rig-gitops run (C2, C4/C5) when all of these hold: - it dispatched that run itself, from main; - the operator was told the run number, the step and the target branch, and replied with a literal yes naming that step; - it approves through POST /repos/dashecorp/rig-gitops/actions/runs/<id>/pending_deployments, with a comment quoting that yes and its jsonl timestamp.

Never a rejection. Every other environment (production, apply, dashe-cutie's production) stays the operator's. GitHub Mobile shows no Approve for runs triggered under the operator's own login, because no notification is sent; without the rule, the operator approves those in a browser (Safari, run 38784866311).

The C2 workflow is c2-ghcr-pull-secrets.yml, with the helper scripts/ci-c2-encrypt.sh. It is merged: rig-gitops#906, plus its selftest #908. It uses sops 3.13.3, reviewed in the sops CLI review (claude-3#1030). First run: 2026-10-06 (run 37474036869, about 13:54Z), approved by the operator through credential-ops. It wrote the six pull secrets for PR-3. C2 runs only through credential-ops; there is no laptop fallback, so if Actions is down, wait. A run goes like this: 1. Verify the names on the environment: gh api repos/dashecorp/rig-gitops/environments/credential-ops/secrets, and …/variables for GHCR_READER_USER. 2. The orchestrator dispatches it: gh workflow run c2-ghcr-pull-secrets.yml --repo dashecorp/rig-gitops --ref main -f target_branch=<branch>. 3. The operator approves it, in GitHub Mobile until Rig Cockpit approval pushes work, or the orchestrator does under the exception above. 4. It writes the six ghcr-pull-secret.sops.yaml files to that branch. Each must have 7 ENC[ markers. 5. A bot push triggers no checks, so push a commit before opening the PR.

C4/C5 is c4-c5-cloudflare-secrets.yml (rig-gitops#1008, #1009, #1010). It encrypts a Cloudflare-made secret that OpenTofu wrote to credential-ops (dashecorp/infra#658, Credentials), so no one copies it by hand. C4 writes the Dell's tunnel token (apps/cloudflared/dell/tunnel-token.sops.yaml); C5 writes the Dell conductor's tenant-map Access token; c5-gcp (rig-gitops#1025) writes GCP's own (apps/rig-conductor/gcp/), which fenced GCP's conductor (rig-gitops#1027). c4-gcp (rig-gitops#1034, in review) will write the token of GCP's rotated tunnel dashecorp-gcp-v2 (apps/cloudflared/gcp/tunnel-token-v2.sops.yaml, dashecorp/infra#677). First runs: 2026-10-15, C4 run 38718895703, C5 run 38722557796 and c5-gcp run 38784866311. The render check's sops-untouched rule (rig-gitops#1020) fails any Kustomize transformer that would touch such a whole-file encrypted file before Flux decrypts it, on both clusters. A run goes like this: 1. The orchestrator dispatches it on main: gh workflow run c4-c5-cloudflare-secrets.yml --repo dashecorp/rig-gitops --ref main -f step=c4|c5|c5-gcp -f target_branch=<branch>. 2. A selftest job runs, then the encrypt job, which waits on credential-ops. 3. The operator approves it, or the orchestrator does under the exception above. No other agent approves. 4. It pushes exactly one encrypted file to that branch, as github-actions[bot].

  • The helper accepts 64-hex or cfast_ secrets (rig-gitops#1017). Cloudflare Access service-token secrets made on or after 2026-08-26 are cfast_[40 alphanumeric][8-character checksum]. Before #1017, C5's first run (38718908545) stopped on the new format and wrote nothing.
  • The encrypted files encrypt every field, kind and metadata included. Flux decrypts them, since kustomize-controller decrypts any resource carrying sops.mac. See rig-gitops docs/sops.md, "Whole-file encryption, and the one rule it needs" (guard: rig-gitops#1019).

GHCR parity CI (rig-gitops#896 PR-2b, #904) proves with curl that rig-gitops CI reaches GHCR, and Artifact Registry through WIF, using the reader binding from dashecorp/infra#571. On #908, parity selftest, live parity proof and c2 helper selftest are all green. The first two are not required yet; an infra PR makes them required. - Since rig-gitops#919 (2026-10-06), live parity proof reads GHCR with the run's own GITHUB_TOKEN (packages: read), not a stored token. - That needs a one-time manual step, which neither the API nor OpenTofu manages: the operator gave rig-gitops Read under "Manage Actions access" on the packages rig-agent-runtime and rig-conductor (operator-inbox #10). - Proven on main (run 37479832627): every own tag has the same digest in Artifact Registry and GHCR.

The rig-backup image (rig-gitops#897 P3, rig-conductor#2605) is published to GHCR by a workflow whose actions are all SHA-pinned. P3b pins openssl (rig-conductor#2622, reviewed in claude-3#1034). The image is now ghcr.io/dashecorp/rig-backup:sha-8941b2c2055af9aed28e201d11b328e5a3cc9d24@sha256:080333cff10553f09415d918cdeecb598fa9d53f9594fadf5cb126b5d9a420f8 (digest bump rig-conductor#2637). It produces no attestations yet; attestation tooling is an open operator decision (claude-3#1028).

rig-approve: merged without Review-E (since 2026-10-03)

The private rig-approve repo (dashecorp/infra#583) is where phase 2 of the Rig Cockpit approvals MVP (rig-cockpit-worker#268) lands. Review-E doesn't gate it: since infra#587 it needs 0 approvals and no code-owner review, and CODEOWNERS is * @Stig-Johnny.

  • To merge: a green npm test plus an independent review, then gh pr merge --squash. Never --admin.
  • Phase 2 is merged: 13 PRs, the page track (#8, #9, #10, #17, #18) and the updater track (#19–#26), after five rounds of independent review. See Operator inbox → Face ID approvals.
  • It has no rig-conductor webhook. The planner still triages its issues, because rig-ignore isn't honored (rig-conductor#2628): remove agent-ready with a comment.

dashecorp/infra: modules that act on a host (since 2026-10-01)

On 2026-09-30 the operator ruled that nobody runs tofu apply by hand. dashecorp/infra's terraform.yml applies the matrix modules after merge. One module acts on a single machine: k8s/dell/k3s, the Dell's k3s version. It is planned in PR plans, which need no connection to the node. After merge, a separate job applies it.

The apply (hosts) job: - runs after the matrix, on push or dispatch to main, in the main-only hosts environment; - installs the official Tailscale client from Tailscale's apt repository, with the signing key pinned to fingerprint 2596A99E…F5868; - joins as an ephemeral tag:ci node. The OAuth secret goes in through --auth-key=file:; - SSHes to the Dell with the OpenTofu-generated key and pinned host keys, then runs tofu apply; - always logs out.

A guard in the module skips the k3s installer when the node already runs the pinned version with the same flags, so only a real version bump restarts k3s.

  • Design: infra docs/2026-09-30-host-modules-from-ci.md.
  • Upgrades: bump var.k3s_version in a PR (infra docs/2026-06-03-dell-k3s-version-control.md).

State locks: PR plans and main applies share each stack's GCS state lock. Since dashecorp/infra#533 (2026-10-01, infra#529), tofu plan and the matrix tofu apply wait up to 10 minutes for a held lock (-lock-timeout=10m) instead of failing at once with Error 412. apply (hosts) waits 5 minutes, so the wait fits in its 15-minute job timeout. Before that change, overlapping runs failed seven times on 2026-10-01 alone. A lock still held after the wait is a stuck one: find the run that holds it before re-running anything, and clear it with the TF Force Unlock workflow, never by deleting the lock object.

The rig-location switch (dashecorp/infra#669, since 2026-10-15): rig-location.yml is dispatch only, and not yet dispatched. - What it does: writes gs://tablez-terraform-state/rig-location/desired.json with GCP's power (gcp) and the stable host's target (primary), which gcp/dashecorp and cloudflare/dashecorp.com read at plan time. Then it runs one locked, targeted tofu apply per module. - The gate: gcp=standby and primary=dell wait for the operator in infra's rig-standby environment. Agents never approve it: the approve-rule covers only rig-gitops' credential-ops. - The ordering rule: the helper scripts/rig-location.sh publishes a stop only while the stable host's live CNAME names the Dell tunnel, and primary=gcp only while GCP's three VMs are live RUNNING. - Tests: rig-location-selftest.yml runs the helper's offline selftest on PRs. - Design: infra docs/2026-10-15-rig-location-switch.md.

github/stigjohnny is skipped while STIGJOHNNY_GITHUB_TOKEN is unset. The module is inert (managed_repos = []), but its provider still calls GET /orgs/{owner}, anonymously without a token. On 2026-10-01 one main apply of it sat silent for 29m45s, most likely waiting out the runner's anonymous rate limit, and every later main run queued behind it. Each matrix leg now sets SKIP_MODULE; a skipped leg prints a notice and passes. Setting the secret brings the legs back.

rig-docs: the C4 model, rendered by Structurizr (since 2026-10-07)

  • Source: the rig's C4 model lives in one place, dashecorp/rig-docs public/diagrams/rig-c4.dsl (Structurizr DSL). The reference page reference/rig-c4-model holds only prose and links to the views.
  • Render: .github/workflows/c4.yml runs whenever the DSL or the workflow changes, and weekly.
  • It runs the DSL gate, then validate and export (mermaid, static), and uploads the c4-site artifact.
  • The deploy workflow copies the newest artifact into public/c4/ with gh run download, which needs actions: read, so the hourly deploy never runs Structurizr.
  • Live at https://research.rig.dashecorp.com/c4/.
  • Approval: Structurizr 2026.09.19 is approved for this CI job only, as the Homebrew-built bottle with the bottle and its WAR SHA-256-pinned (review, rig-docs facts homebrew/structurizr@2026.09.19). structurizr local (unauthenticated, all interfaces), server, the vendor WAR and the Docker image are not approved.
  • DSL rules:
  • The gate allows no ! except !identifiers, no ${, no ://, and no image, theme or workspace extends.
  • Every deployment environment needs one deploymentGroup per site, or relationships are copied across sites.
  • Every infrastructureNode needs an identifier and explicit -> relationships.
  • Public: the docs site, /c4/ included, is public until rig-docs#358 puts it behind Cloudflare Access after a brain index exists.

Release Pipeline

  1. Conventional commits on main trigger release-please
  2. Release PR opened automatically with version bump + changelog
  3. Release PR auto-approved and merged with a short-lived review-e-bot App installation token, minted per run by actions/create-github-app-token. The PEM must be PKCS#8 — that action rejects PKCS#1.

    RELEASE_PAT is RETIRED (2026-07-03, infra#397 step 4) and must NOT be rotated — deadline #23 is closed. All seven dashecorp/dashe-* release workflows migrated off it. An org-level RELEASE_PAT still exists and returns 401; dashecorp/rig-memory-mcp still probes it and silently degrades to GITHUB_TOKEN, so it stops opening release PRs rather than failing loudly. Historic symptom, for recognition only: an expired PAT surfaced as Release Please Bad credentials.

  4. Release PR auto-merged
  5. GitHub Release created with tag
  6. release-notes-to-discord.yml posts release notes to Discord

All actions SHA-pinned for security.

Discord Integration Flow

PR Created
  → request-review.yml → Discord #tasks thread (stores message ID in PR comment)
  → Review-E picks up from Discord notification

CI Runs
  → build-status-to-discord.yml → Updates Discord thread

Copilot Reviews
  → forward-copilot-reviews.yml → Posts to Discord thread

PR Merged/Closed
  → notify-closed job → Posts resolution to Discord thread (using stored message ID)

Key pattern: Discord message ID stored as HTML comment in PR (<!-- discord-review-msg-id:ID -->). This ID doubles as thread ID for webhook posts (?thread_id=ID).

iOS Build Pipeline (GitHub Actions)

Xcode Cloud was retired (claude-3 PR #776, merged 2026-06-16). iOS apps now build and ship to TestFlight via GitHub Actions: each dashecorp/dashe-* app repo has a build-and-upload.yml workflow on the macos-26 GitHub-hosted runner (fastlane match signing via dashecorp/ios-certs, ASC API upload to TestFlight internal; App Store submission stays human-gated). See iOS Build & Distribution for the full pipeline. The legacy account-wide Xcode Cloud GitHub App is no longer triggered.

dashecorp/rig-cockpit-ios runs the same workflow with three deviations: its default branch is feature/fork-cutie-admin-cockpit, not main (main is a stale Cuti-E fork, so target PRs at the default branch; a merge there auto-builds to TestFlight Internal Testers, which is allowed); it also builds on a v* tag push; and its entitlement check derives the expected Universal Link domain from the app's own entitlements file. Uploading is not releasing — a separate release-to-testers job attaches the build to the Internal Testers group, because that group has hasAccessToAllBuilds: false and Apple refuses to flip the flag via the API. See Rig Cockpit surfaces.

Rig-internal repo CI

The four repos the orchestrator authors PRs in itself (rig-conductor, rig-agent-runtime, rig-cockpit-worker, rig-gitops). Their CI shape matters when a chain of locally verified slices is published one PR at a time. Verified 2026-09-19/20 while publishing dashecorp/rig-conductor#2104.

Repo Gate Runner Notes
rig-conductor test.yml (dotnet test) + contract.yml + docs / docs-check ubuntu-latest Runs two structural guard scripts before the SDK setup — scripts/check-tenant-scope-guard.sh and scripts/check-onboarding-gate.sh. They fail the build before any test executes, so a new Marten session without a tenant, or an ungated tenant activation, never reaches the suite.
rig-agent-runtime test.yml (npm test) + docs-check + build.yml ubuntu-latest, Node 20 The dev laptop runs Node 24/25, so a timing-sensitive test can pass locally and fail in CI.
rig-cockpit-worker test.yml (npm test) ubuntu-latest No docs-check in this repo — doc frontmatter is enforced only by Review-E's prose review.
rig-gitops docs-check, mermaid-check, dispatch, rig-e-persona-check ubuntu-latest Also hosts the reusable docs-check.yml that the other repos call.

Reading a red run in rig-agent-runtime. A cancelled node:test reports fail 0. A run can read # tests 835, # pass 823, # fail 0, # cancelled 12 and be red with nothing failing: node:test cancels the pending test and everything queued behind it in that file, so the culprit PR named by main-guard is often not where the fault is. Read # cancelled before believing # fail 0. The usual cause is a promise whose only settle path is an AbortSignal.timeout() abort — that timer is unref'd and does not keep the event loop alive (rig-agent-runtime#691, #724).

Rig-E operator-task test surfaces (2026-09-21)

What gates the feature after its 47 PRs merged, and where each thing runs:

Surface Where Runs
route e2e per stage (23 cases) + the lifecycle walk (one task proposed→verified, all 15 status keys asserted at every stage) rig-conductor/tests/ConductorE.Api.Tests/ test.yml, every PR and push to main
cross-repo contract — the Worker's real signer and client against the conductor's real routes, 4 + 66 checks rig-conductor/tests/contract/ contract.yml, every PR; checks rig-cockpit-worker out with a short-lived review-e-bot App token (org secret RELEASE_APP_PEM, org var RELEASE_APP_ID)
persona propose-only contract + a selftest that mutates the live persona five ways rig-gitops/scripts/check-rig-e-persona.py rig-e-persona-check.yml, path-triggered on the helmrelease
live drift + health — running persona vs rig-gitops main and deployed image vs conductor main (DRIFT, exit 1); pod readiness, retry budget and the task signing-key state (UNHEALTHY, exit 3) (Configured; anything else means every task route answers 503 while the Cockpit looks alive — the state enum is read, never the key) scripts/check-rig-e-live.py (this repo) selftest only in CI; the live half on demand (see below)
the feature actually running (propose → Confirm → issue → PR → merge → deploy → verified) — RAN TWICE. 2026-09-21 17:38:45Z — task f62ed96a, confirm→live 12m20s, dashe-website#203 → PR #204 → b705b98b, merged on the operator's release click. 2026-09-22 10:11:29Z — the first falsifiable run: task ff18ab60, dashe-website#207 → PR #208 → 3d1c159, verified 12:01 local, machine-judged merge, no release click. Operator-gated (wakes the GPU, files a real issue).

The 2026-09-22 run — what it proved, and what it cost

The 2026-09-21 run passed under an assertion that could not fail: OperatorTaskStageProjector returned (Verified, "Live on dashecorp.com") whenever the probe read a hard-coded const, and dashecorp.com already served it. rig-conductor#2253 replaced that with an expectation derived from the template's own anchors, so the check can now be wrong. The run below is the first one whose green means something.

Arming it took a content change, not a conductor change. The propose gate reads the served page, not the repo, so putting the App Store link back (dashe-website#205 → PR #206 → 09f93a3, operator-approved: "we can live with … the app store link until we have fixed what needs fixing") re-armed the one pinned template with zero code. rig-conductor#2238 (a second template) was not a prerequisite, and the earlier claim here that the template was "spent" named the wrong refusal branch: RefusePreState tests IsExpected first, so a satisfied page answers 422 intent_already_satisfied — recoverable — not pre_state_unexpected.
Sent (new conversation — the persona proposes at most once per conversation) 11:01 local
GPU brain ready 11:08:58 — pod 3nmdwxoks8thh0, RTX PRO 4500 Blackwell, $0.72/h
Rig-E proposed ff18ab60 — the propose gate accepted a real request, RefusePreState reading the live page 11:09
Operator Confirm → issue dashe-website#207 filed by dev-e-bot 11:11
implementing 11:14
in_review pr#208, card showing Diff: Matches / matches_intent + the +1/−1 patch 11:58
Review-E approved, MergeGate merged server-side 3d1c159 12:01
deployed (Cloudflare Pages success) · verified https://reward-e.com · Live 12:01

approval_needed never appeared, and that is the result. rig-conductor#2245's machine-judged diff_matches_intent merge had never executed on a real PR; the 2026-09-21 run merged on a human click (MergeGate's Operator release approved in Rig Cockpit for 547c3e4a… comment on PR #204 is posted only for a head-pinned operator release). This time Dev-E's diff was byte-identical to the pinned OldAnchor/NewAnchor (OperatorTaskIssueTemplate.cs:17-18), OperatorTaskDiffPolicy.Decide returned Matches, and the card offered the operator nothing to click. Total human actions: one sentence and one Confirm tap.

Cost: ~23 min of GPU at $0.72/h ≈ $0.28. The old pod could not start ("not enough free GPUs on the host machine"), so RunPodWakeService created a replacement, terminated the stale one, and RunPodIdleMonitor stopped the new one at 15:00 idle — the whole wake/replace/stop path ran unattended.

Three defects the run exposed, none of which the test suite can see:

  1. One Dev-E slot starves the tenant. assignments:dev-e-node@dashecorp is min 0 / max 1 for every dashecorp repo. rig-cockpit-worker#222, filed 27 s before the operator's request, took the slot at 10:13:29Z; #207 was published at 10:15:00Z and sat unread for 43 minutes while #222 burned three 20-minute attempts (rig-agent-runtime#727 CLI timeout → #726 Codex-on-stdin fallback → retry). Once it had the slot, Dev-E went from branch to open PR in 62 seconds. Do not file agent-ready dashecorp issues while a cockpit run is in flight.
  2. stage: implementing cannot distinguish queued from working. It is derived from agent-ready + no PR, so the card read implementing for 44 minutes, 42 of them queue. A "still implementing" card is not evidence of progress — the distinction rig-cockpit-worker#223's diagram has to carry rather than inherit.
  3. rig-conductor#2227 dropped an event from this very run. Planner-E's CLI_COMPLETED for #207 exhausted 3 retries on pk_mt_events_stream_and_version and was lost. #207's timeline is permanently missing it, which matters because a diagram node is to be marked reached only from the timeline.

Free pre-flight before ever spending a wake: GET /v1/tasks/health proves the task signing key, the Cloudflare Access service token, tenant resolution and the store in one read, with no GPU and no issue filed (it asserts a 404 carrying hostingScope + repoTenant). The Cockpit renders it as the Task link chip — once, and only while Rig-E is the selected target. Make that chip green the go/no-go.

The live half is deliberately not scheduled. No workflow here has GCP auth, and reading the cluster needs the CI service account to hold IAP tunnel + OS Login on invotek-k3s — an IAM change (OpenTofu) plus a judgement about whether a scheduled job should hold cluster read access. Options and the decision are in claude-3#903; the cheapest is a CronJob inside the cluster, where kubectl access already exists. Run it by hand meanwhile:

python3 scripts/check-rig-e-live.py            # 0 clean · 1 DRIFT · 2 inconclusive · 3 UNHEALTHY
python3 scripts/check-rig-e-live.py --selftest # no cluster, no credentials

DRIFT and UNHEALTHY are separate verdicts (claude-3#912). 1 means running != main — the persona, or an image that is not a commit on conductor main. 3 means a live symptom of a cluster that is running main: a pod not Ready, the signing key, or the rc#2227 retry budget. They were one list and one exit code, which made this check red by default, because the retry budget stays over its threshold until rc#2227 is fixed and its headline then read "1 difference between what is running and what main says" when the running build was exactly main. A check that is permanently red for a known, unrelated reason is one you learn to skip. Drift outranks health when both are present: a cluster running something other than main makes every health reading a reading of the wrong build. 2 still means something could not be read, and a dead tunnel is never a drift.

The signing-key item currently answers 2, for a reason worth knowing: rig-conductor-api logs ~45 lines/sec (Npgsql command logging plus four lines per GitHub call), so the container log rotates past the pod's own startup line in under 36 minutes and the state is unreadable — rig-conductor#2249. Until that is turned down, the most load-bearing fact about the feature cannot be verified from outside the cluster.

It is read-only by construction: kubectl get and kubectl logs through the documented IAP tunnel, never exec, and it prints counts and shapes rather than persona text. When it does get scheduled, report on a change of state only — never a green heartbeat, because a diary of "still fine" is how you stop reading the diary (a 20-day Mac Mini outage happened exactly that way).

Image pins reach the pods by one path. rig-conductor ImagePinDispatchService runs after each runtime image build and discovers its targets from the rig-gitops tree: every apps/** YAML with a standalone tag: <stack> line (rc#2305/#2308, closing rc#2303). It refuses to pin a subset: a truncated tree or zero releases found fails the run. The first discovered pin, rig-gitops#814, touched 17 files. They included apps/dashecorp/ and apps/stigjohnny/, which the old ten hard-coded paths had skipped (rig-gitops#792). Two releases opt out with a standalone # pin-rig-agent-runtime: manual comment line and are bumped by hand: Rig-E (apps/dashecorp/rig-e-helmrelease.yaml, whose tag moves with llm.baseUrlFrom) and apps/dev-e-staging/. The manual pin-rig-agent-runtime.yml workflow is deleted (rig-gitops#800/#803). It could never open its own PR, and its whole-file marker grep opted Rig-E out by accident (rig-gitops#780). Nothing pins third-party images. rig-gitops image-pin-check.yml (rig-gitops#826) fails CI on a floating one, so a third-party bump is a hand-written digest PR.

The rule that survives: verify the running pod, not the manifest. After a pin, the StatefulSet spec changes minutes before the container is pulled and Ready, and KEDA may scale the pod straight back to zero once it is idle —

kubectl get pod -n dashecorp dev-e-rig-agent-runtime-0 \
  -o jsonpath='{.status.containerStatuses[0].image}{" ready="}{.status.containerStatuses[0].ready}'

Dispatch and review traps — five ways the rig looks idle when it is stuck (2026-09-22)

Found while shipping the Rig-E sandbox template pair. Each one produces no error anywhere the operator looks: the issue keeps its agent-ready label, KEDA reports the queue inactive with 0 replicas, and the PR sits green. Two of them make Review-E look wrong when it is not.

# Trap The signal The lever
1 A dependency on a pull request number [Webhook] …#233 queued — blocked by [#231], then [ExecutionLogExpiry] … EnvelopeTimedOut and a ❌ reaction Remove the dependency phrasing from the issue body — write "pull request 231", never #231 under a Depends on heading — then toggle agent-ready off and on. A correct re-dispatch logs → dev-e-node (normal) and Published assignment to stream …
2 A push does not re-request review on my own rig-internal PR [Webhook] review-disputed … orchestrator-authored rig-internal PR, skipping re-dispatch (rc#960) Dismiss the stale review (gh api -X PUT repos/O/R/pulls/N/reviews/<id>/dismissals -f message=… -f event=DISMISS) naming the commit that addressed it. In rig-conductor that alone did not route it — the needs-review label did, via ReviewScanService's 5-minute sweep: [ReviewScan] Routed …#2283@<sha> (issue #2279) to review-e (missed by webhook)
3 The PR lags its own branch gh api repos/O/R/pulls/N --jq '.head.sha, .commits' disagrees with gh api repos/O/R/branches/<branch> --jq .commit.sha Push an empty commit. On #2283 the PR held 3959a6a (4 commits) for 50 min while the branch was e414720, so Review-E twice blocked on an already-fixed finding. A review's .commit.oid says which head it judged — compare it to .headRefOid before disputing anything
4 The 24 h assignment key [Webhook] …#234 already assigned to dev-e-node — skipping Nothing clears it in-band: assigned:<repo>#<n> is a SETNX with a 24 h TTL (Program.cs, the dedup block). File a fresh issue carrying the same body and close the old one as superseded
5 MergeGate never retries a PR that Review-E approved while its checks were still running (2026-09-29) The PR is APPROVED at its head, CLEAN, every check green, and still open long after. MergeGate tries only when Review-E's approval calls /api/merge; a retry when the checks finish needs the repo's rig-conductor webhook to deliver them, and dashe-nutri-e's hook is one of the dashe-* hooks dashecorp/infra#474 has not adopted yet. dashe-nutri-e#60 was approved at 14:31Z while checks ran and was still open at 14:55Z The orchestrator merges it under the operator's standing grant for approved and clean dashecorp PRs, and comments on the PR why. The lasting fix is adopting the webhook (infra#474) Since 2026-10-07 the MergeGate sweep retries it within minutes (rig-conductor#2365), as long as Valkey still holds Review-E's approval

Review-E dispatch for operator PRs outside the rig-internal repos (2026-10-04). Per the BRAIN, the only trigger is the 5-minute ReviewScanService sweep, which picks up needs-review. Webhooks don't dispatch these PRs, so a label toggle, an empty commit or closing and reopening the PR doesn't help; all three were tried on rig-cockpit-worker#285. Review-E was down from 23:13Z on 2026-10-03 to about 10:03Z on 2026-10-04, and still skipped #285 after that, until it approved it at 12:20Z (rig-conductor#2639).

Operator PRs on a rig repo outside RigBoundary's list get Dev-E (2026-10-21). The rig-internal list is hard-coded in rig-conductor's RigBoundary.RigInternalRepos, and it decides every gate: Dev-E re-dispatch on CHANGES_REQUESTED, Review-E on PR open, main-guard, and the planner skip. On a repo missing from it, a CHANGES_REQUESTED on the orchestrator's PR sends Dev-E to push a fix commit to that PR's branch, rewrite its body and open a sub-issue. This happened on rig-events#23 and #24, a day after the repo was created; rig-conductor#2731 added it to the list. Two rules follow. Add a new rig repo to RigBoundary when it is created. After a CHANGES_REQUESTED, git fetch and look for foreign commits before pushing. Dev-E stays on rig-cockpit-worker and rig-cockpit-ios by the operator's choice. In the same series, Review-E blocked "Part of #19" as a missing issue link, so give each PR of a multi-part change its own issue to close.

When Review-E is down, agents can't stand in for it. On 2026-10-04 the auto-mode classifier denied a substitute review on the Mac Mini followed by a merge past the review gate, as "CI Bypass". The denial covers the outcome, so don't retry it by any route. rig-cockpit-worker's CODEOWNERS is * @review-e-bot, so only the real bot satisfies its protection, and agents never use --admin. The options are: - the operator merges with the bypass, or lifts the denial with a permission rule; - Review-E gets fixed (rig-conductor#2639).

More auto-mode denials (2026-10-06), and what worked instead. A denial covers the outcome, so none of these was retried by another route. - A force-push of a rebased rig PR branch ("Git Destructive", rig-gitops#913). Bring a stale PR up to date by merging main into its branch instead. That is a plain fast-forward push, and because the PR is squash-merged, the merge commit costs nothing. - Generating an SSH key to reach a host ("Unauthorized Persistence", dashecorp/infra#601). New access to a host waits on the operator. - A subagent asked to close 43 stale issues ("External System Writes"). Bulk issue closing goes to the operator. - An unrelated read-only gh run view, right after one denial. It was refused once, and the same command passed a few minutes later.

MergeGate and stacked PRs (2026-10-04). MergeGate squash-merges an approved stacked PR into its parent branch, not only into main. On 2026-10-04 it collapsed the slider's A2, A3 and A5 into A1's branch, and the result was too big to review. So open PRs one at a time on main, or open only the bottom of a stack as ready and keep the rest as drafts. Mark each one ready after its parent has merged to main and GitHub has retargeted it.

Size: Review-E requests changes on a PR over 500 lines or 20 files and asks for a split. The collapsed A1 (22 files, +1484) was re-split rather than labelled: large-pr-ok is not for decomposable feature work.

Infra PRs: before merging, check the plan comment's add, change and destroy counts against the counts you expected (for example, 8,1,0 on infra#589). Explain any extra change in a comment first. A plan can carry drift that was already there, so check which resources change, not just the counts. The rig-backups-dell R2 lifecycle reorder that used to show in every cloudflare/dashecorp.com plan (infra#594) is fixed by infra#597: R2 returns the rules sorted by id, so the config now builds them keyed by id. Wait for terraform.yml to apply on main before opening the next infra PR.

An approved draft can need a plain squash merge. Until 2026-10-07 MergeGate was event-driven and didn't retry, so a PR that Review-E approved while it was a draft could sit unmerged (rig-cockpit-worker#287). Since rig-conductor#2696, a sweep re-runs the guarded merge every 5 minutes for a PR whose approval is recorded and that has no merge recorded, at most 12 times per trigger, backing off to 2 hours (rig-conductor#2365). A recorded approval only makes the PR a candidate: the guard still checks that GitHub shows the approval on the current head. Merge it with gh pr merge --squash once the approval is on the head, every check is green and the state is CLEAN, after waiting 5 minutes or more. Never --admin.

Try gh pr ready first (2026-10-06). rig-conductor routes pull_request ready_for_review to Review-E. On rig-gitops#913 and #923, marking an approved draft ready brought a fresh Review-E approval 2.5 and 6 minutes later, and MergeGate merged 7 to 8 seconds after it. Merge by hand only if nothing happens within about 10 minutes.

An approval that mentions "requesting changes" never merges (2026-10-06). The Review-E runtime decides whether to nudge MergeGate (POST /api/merge) by matching its own reply text, not by its review state. reviewVerdict() in rig-agent-runtime's src/review-follow-up.js treats any reply containing changes requested, changes_requested or requesting changes as a change request, even "Not requesting changes for this". dashe-tags-gone-crazy#2 was approved at its head at 15:10Z, was CLEAN with no required checks, and never merged; the same day, other approvals merged 6–8 s after Review-E approved. The signal is an APPROVED review whose body contains the phrase, on a PR still open minutes later. Fixed by rig-agent-runtime#775, which merged on 2026-10-06 at 17:27Z and closed #773. The runtime now reads review-e-bot[bot]'s latest verdict on the PR from GitHub, so the wording of an approval no longer matters. An unreadable review list sends no nudge and logs why. Until a runtime pin with the fix is deployed, the operator merges a stuck approval with gh pr merge N --repo O/R --squash --match-head-commit <approved sha>. The auto-mode classifier blocks agents from running that, even when the operator asks, so give the operator the command to run with !.

Merging a PR can reopen its issue (2026-10-06). When a PR that closes an issue merges, rig-conductor reopens the issue if its body still has unticked - [ ] boxes. It comments "⏳ Umbrella partial completion. PR #N merged: 0/4 tasks complete" (dashe-tags-gone-crazy#1). Tick the boxes the PR completed (gh issue edit N --body-file …), then close the issue again with a comment saying where each was done; it stays closed. To avoid this, write the acceptance criteria of a single-PR issue as plain bullets, not checkboxes.

Telling a stalled lane from an idle one. KEDA looks identical either way, so read the conductor instead:

kubectl logs -n rig-conductor <rig-conductor-api-pod> --since=30m \
  | grep -E 'queued — blocked by|EnvelopeTimedOut|already assigned|Published assignment'

If an agent-ready issue has no pod and no PR after ~15 minutes, that grep names the reason before any capacity theory does.

Decide who builds before filing in a planned repo. An issue opened in rig-cockpit-worker, dashe-website or a dashe-* app is planner-queued on the issues.opened webhook, TaskSpec'd and dispatched to Dev-E within minutes (→ planner queue, then Published assignment to stream assignments:dev-e-node@dashecorp). rig-conductor and rig-gitops issues are not auto-planned. So either put every exact value the agent needs in the body and let it build (it cannot read an unpushed branch, and a truncated digest is not a value), or push the PR first and file the issue pointing at it. Doing both wastes the single-slot lane twice: rig-cockpit-worker#229 was filed by the orchestrator, built locally as #230, and simultaneously split and built by Dev-E as #231/#235/#236 — #230 was closed as a duplicate.

Triage a red run before re-running it. Two timing flakes blocked merges today, both unrelated to the PRs they blocked and both green 5/5 in isolation: CockpitTaskLifecycleWalkE2eTests.A_release_deferred_… racing MergeGate.LastAudit (rig-conductor#2273, it also fails on main) and RunPodIdleMonitorTests.Idle_window_starts_at_ready_not_at_running counting an extra tick (rig-conductor#2284). Check the failing test name against those two before assuming the PR broke something.

A third known flake, rig-conductor#2479: a concurrent Marten schema apply fails a dotnet test test with "DDL Execution … Failed". It failed 2 of the last 12 main runs (2026-10-05). A single re-run is the accepted response.

A job that never got a runner is not a code failure. During a GitHub Actions incident on 2026-10-05 (about 19:11 to 21:56Z), hosted jobs sat queued, and GitHub cancelled them after 15 minutes with "The job was not acquired by Runner of type hosted even after multiple attempts". Such a job has no runner and no steps: re-run it once Actions is back.