Hosts¶
Every physical and virtual host in the Dashecorp/Invotek infrastructure.
Active Hosts¶
Every host marked Ålesund is at the operator's parents' house outside Ålesund (Møre og Romsdal) — remote access only, no hands-on physical access (see Physical Locations).
| Host | Tailscale IP | OS | Location | Role | SSH Access |
|---|---|---|---|---|---|
| MacBook Air M4 24GB | 100.126.195.52 | macOS 15 (Darwin 25.1) | Las Palmas (with operator) | Dev machine, Claude Code (workspaces 3-6) | Local |
| Dell Precision 5520 (dell-stig-1) | 100.95.212.93 | Ubuntu 24.04 LTS | Ålesund | Second k3s cluster. Since the 2026-10-02 cleanup it runs only buzz and finance-dashboard, reconciled from dashecorp/rig-gitops clusters/dell/. No rig paths, no tunnels, no CI runners, no code-server. The operator plans to move The Rig here (dashecorp/infra#508). Its 256 GB NVMe drive is worn out (107% of rated endurance, critical_warning 0x4; dashecorp/infra#622). Replace it before any tenant moves here. At idle it writes about 30 GB a day (measured 2026-10-15). rig-gitops#1014 and infra#661 cut that without new hardware See Kubernetes → Dell cluster |
ssh -i ~/.ssh/dell-stig-1 claude@100.95.212.93 |
| VPS srv1099021 (Volt-E) | 100.111.142.118 | Linux (x86_64) | Boston, USA | Volt-E (agent-runner, systemd) | ssh -i ~/.ssh/vps-srv1099021 root@100.111.142.118 |
| Pi4-02 8GB | 100.77.12.75 | Linux (ARM64) | Ålesund | Spare/legacy node — former Review-E host (agent-runner retired; Review-E now runs in the GCP k3s cluster as rig-agent-runtime pods) | ssh -i ~/.ssh/rpi-pi4-02 claude@100.77.12.75 |
| Pi4-03 8GB (Pi-E) | 100.107.48.17 | Linux (ARM64) | Ålesund | Pi-E agent (agent-runner, systemd) | ssh -i ~/.ssh/rpi-claude claude@100.107.48.17 |
| Mac Mini M4 | 100.92.170.124 | macOS (Apple Silicon) | Ålesund | Delegated headless Claude jobs. iBuild-E (rig-agent-runtime, system LaunchDaemon ai.invotek.ibuild-e) is stopped since 2026-10-14 (claude-3#1206); iClaw-E, Veribox and Vox-E retired 2026-09-25 (see What runs on the Mac Mini). Boots unattended since 2026-09-25: FileVault off, auto-login post, restarts after a power cut (see below) — no CI runner (self-hosted macOS runner retired 2026-06-29; ios-sdk CI now runs on GitHub-hosted macos-26) |
ssh -i ~/.ssh/mac-executor claude@100.92.170.124 |
| DS412+ (NAS) | 100.95.36.122 | Linux | Ålesund | Synology NAS (~9TB): NFS for the Dell's k3s (/volume1/docker, exported to *, so any home-LAN device can mount it; operator-inbox#40). buzz mounts it by the LAN address 192.168.1.116, which comes from DHCP, through a provisioner and PVs that Flux owns (dashecorp/rig-gitops#988). Over the tailnet, only the Dell may reach it, and only on tcp/2049 (dashecorp/infra#525). That path read 4–11× slower, so it is only the fallback |
— |
| Pi4-01 | 100.73.159.126 | Linux (ARM64) | Ålesund | Tailscale exit node | ssh -i ~/.ssh/rpi-pi4-01 claude@100.73.159.126 |
The MacBook's Docker is Colima (a Lima VM; its data disk is ~/.colima/_lima/_disks/colima/datadisk). docker … prune frees space only inside the VM. The Mac gets it back after colima ssh -- sudo fstrim -av. On 2026-10-16 the disk was 99% full; pruning 36 throwaway containers, 638 orphaned anonymous volumes and unused images, then the trim, took it from 8.1 to 107 GiB free (the data disk went from 96 to 3 GB). Docker 28's docker volume prune keeps named volumes. Never prune the running Postgres containers: other workspaces use them.
CI access to the Dell (since 2026-10-01)¶
Since dashecorp/infra#503, nobody applies k8s/dell/k3s by hand. The post-merge apply (hosts) job in dashecorp/infra's terraform.yml reaches the Dell as an ephemeral Tailscale tag:ci node.
- Network: the ACL allows
tag:cionly port 22 on the Dell (host aliasdell). Tagged nodes are not inautogroup:member. PR plans never join the tailnet. - Key: an ED25519 key that OpenTofu generates (
github/dashecorp/infra-hosts-ci.tf). It lives only in that module's encrypted state and in the main-onlyhostsenvironment secretHOSTS_SSH_PRIVATE_KEY, and no person handles it. - On the Dell: one line in
claude's~/.ssh/authorized_keys, ending ininfra-hosts-ciand limited tofrom="100.64.0.0/10"with forwarding off. FingerprintSHA256:PnmBXez0VC7Umqr42lcJ2ZYWKD6+gjmCUYcHWAuwp64. To revoke it, delete that line. The bootstrap and rotation are in infraBOOTSTRAP.md§23. - Host keys: the job pins the Dell's host keys. OpenSSH uses the ED25519 key, and OpenTofu's provisioner the ECDSA key, because Go's SSH client negotiates ECDSA first (infra#518). Re-pin both if the Dell is reinstalled.
- The Mac Mini gets no CI access. Its two OpenTofu modules were never applied and were retired (dashecorp/infra#512).
Mobile Devices (Tailscale)¶
| Device | Tailscale IP | OS | Notes |
|---|---|---|---|
| Samsung Galaxy A17 | 100.90.74.87 | Android | Pushbullet SMS forwarding |
| iPhone 15 Pro Max | 100.110.193.124 | iOS | Primary phone |
| iPad (10th gen) | 100.75.46.53 | iOS | — |
Retired/Offline Nodes¶
| Node | Tailscale IP | Last seen | Notes |
|---|---|---|---|
| 9f71f0bfa6c6 | 100.93.47.98 | 131d ago | Unknown container |
| google-cloud-debian-1 | 100.95.117.82 | 42d ago | GCP instance |
| kubectl-jumpbox | 100.75.171.111 | 271d ago | Retired |
| win-pw0edb77-1 | 100.124.218.17 | — | Private machine (not part of infra) |
| iPad Pro 12.9 6th Gen | 100.74.168.115 | 20d ago | — |
VNC Access¶
| Host | Protocol | Address | Port | Client | When needed |
|---|---|---|---|---|---|
| Mac Mini M4 | VNC (macOS Screen Sharing) | 100.92.170.124 |
5900 | Screens app | Maintenance, claude login |
| Mac Mini M4 | noVNC (browser) | https://100.92.170.124:6080/vnc.html |
6080 | Browser (HTTPS required) | Not running: its launcher was one of claude's LaunchAgents retired 2026-09-25 (see What runs on the Mac Mini) |
| Volt-E (VPS) | VNC (x11vnc) | 100.111.142.118 |
5900 | Screens app | claude login for OAuth refresh |
| Volt-E (VPS) | noVNC (websockify) | http://100.111.142.118:6080 |
6080 | Browser | Fallback |
| Pi4-02 (spare) | VNC (x11vnc) | 100.77.12.75 |
5900 | Screens app | Former Review-E host — Review-E now in GCP k3s; node is spare |
| Pi-E (Pi4-03) | VNC (x11vnc in Docker) | 100.107.48.17 |
5900 | Screens app | Normally not needed (token pushed) |
| Pi-E (Pi4-03) | noVNC (Docker) | http://100.107.48.17:6080 |
6080 | Browser | Fallback |
VNC Notes¶
- All connections require Tailscale — connect via 100.x.x.x addresses
- Mac Mini M4 noVNC: Must use HTTPS (self-signed cert) —
crypto.subtlerequires secure context for ARD auth - Mac Mini M4 Screen Recording: If VNC shows black screen, check System Settings → Privacy & Security → Screen Recording is enabled. Disabling Remote Management first may be needed to unlock the toggle (macOS Sequoia requirement).
- Volt-E: Runs LXDE desktop with x11vnc (systemd service
xvfb-vnc.service). Pi4-02 (the former Review-E host) has the same setup but is now a spare; Review-E runs in GCP k3s. - Pi-E: VNC runs inside Docker container (ports 5900 + 6080 exposed)
- Recommended client: Screens app (iOS/macOS) — works with all hosts
Troubleshooting: claude CLI hangs on the Mac Mini (keychain wedge)¶
Symptom: claude (interactive, -p, or --debug) hangs at startup with zero output; the debug log (~/.claude/debug/*.txt) stops at CA certs: Config fallback; security commands — even security list-keychains — hang and pile up /usr/bin/security procs (PPID 1).
Cause: the macOS keychain daemon (securityd) is wedged (observed at ~17-day uptime) and/or a stuck keychain-authorization dialog is sitting unanswered on the headless GUI — a claude CLI version bump re-signs the binary, so macOS re-prompts "allow access to keychain item 'Claude Code'?" and nobody can click it. claude reads its OAuth credential via security find-generic-password -a claude -s "Claude Code", which then blocks. Token/network/trust are fine (the OAuth token returns HTTP 200 via curl), and the hang is identical over Tailscale SSH and LAN SSH — so it is not a network or session-context issue.
Cannot be fixed from SSH: killall secd doesn't clear it, and securityd is SIP-protected — sudo killall securityd is a no-op (same PID), and sudo launchctl kickstart -k system/com.apple.securityd returns 150: Operation not permitted while System Integrity Protection is engaged.
Fix:
1. Preferred (no reboot): VNC to the Mac screen over Tailscale (Screens → 100.92.170.124:5900; the noVNC fallback is retired) and click Always Allow on the keychain dialog (enter the login password). Releases securityd; the ACL update stops future prompts.
2. Fallback: sudo reboot — gives a fresh securityd, and the Mac comes back unattended (see Unattended boot). Auto-login is post since 2026-09-25, not claude, so auto-login no longer unlocks claude's login keychain. Headless runs are unaffected: after the 2026-09-25 restart, with claude's keychain still locked, zsh -ic "claude -p …" over SSH ran normally, because it authenticates with CLAUDE_CODE_OAUTH_TOKEN from ~/.zshrc. An interactive claude that reads the keychain item may still prompt.
Both fixes assume the Mac is powered on and on the network. There is no hands-on access to it (see Physical Locations) — if the box is off or off the network, recovery means phoning the operator's parents. Outages like that have gone unnoticed for ~20 days and, in 2026-08/09, ~50 days.
Verify: cd /tmp/rtd-demo/cc-demo-repo && claude -p "Reply PONG" --strict-mcp-config returns a reply within seconds (instead of hanging).
SSH to the Mac Mini via the Pi-E LAN jump¶
Pi-E (sjs-pi4-03: eth0 192.168.1.105, wlan0 192.168.68.130 on the Mac's segment) reaches the Mac Mini's native sshd on its LAN IP (DHCP: 192.168.68.150 on 2026-09-24, .131 before; port 22, re-verified 2026-09-24). Useful as an alternate SSH path — note that VNC ports 5900/6080 are filtered on the Mac's LAN IP, so use the Tailscale IP for VNC:
ssh -i ~/.ssh/mac-executor \
-o ProxyCommand="ssh -i ~/.ssh/rpi-claude -W %h:%p claude@100.107.48.17" \
claude@192.168.68.150
The Mac's Wi-Fi uses a private MAC and DHCP, so its address changes (.131 in June, .150 since 2026-09-24). Find the current one from Pi-E with sudo /usr/local/sbin/find-macmini.sh, which scans 192.168.68.0/24 for the Mac's SSH host key: ed25519 SHA256:vFxeMSgMLKyWOBJkgCdse5LizYKlQKO0kPJd7DCN/9o (verified 2026-09-24). If the script finds nothing, the Mac is off, hung, or off the Wi-Fi (see the Wi-Fi hang).
Remote Claude on the Mac Mini (delegated work, since 2026-09-24)¶
The operator's standing instruction: run Claude work on the Mac Mini, because its subscription has more tokens (CLAUDE.md, credential exceptions).
The day-to-day process (one fenced job per task via ~/bin/run-job.sh, pinned model and effort, the standing job preamble, the verdict ledger, context limits and how the orchestrator re-verifies each result) is in Mac Mini delegation. The scripts are in scripts/mac-mini/. The one-off launcher shown below is the older form.
# Write the prompt and a launcher script locally, scp both to ~, then:
ssh -i ~/.ssh/mac-executor claude@100.92.170.124 'zsh -ic ~/run-<task>.sh'
# where run-<task>.sh does:
cd ~/repos/<repo> && claude -p "$(cat ~/<task>-prompt.md)" --model claude-opus-5-5 --output-format json \
--setting-sources project --allowedTools "Read(./**)" "Edit(./src/**)" "Bash(dotnet test:*)" ...
Always pass --setting-sources project. The claude user's ~/.claude/settings.json allows Bash(*), Read, Edit and
Write, and --allowedTools only adds allow rules, so without that flag a tight allowlist restricts nothing. The first
delegated run (claude-3#944) had unrestricted Bash as a result, and a subagent's gh auth status printed most of a 1-hour
ibuild-e-bot token into its transcript (claude-3#947). With the flag, -p mode denies anything not allowlisted except
what Claude Code approves on its own: its read-only Bash commands, the Agent and ToolSearch tools, and reads of the
session's own tool results. Subagents inherit the allowlist (claude-3#1159, the
gotchas).
Run the SSH command with run_in_background so it notifies on completion, then audit the session's tool calls before
trusting the output: the main transcript ~/.claude/projects/*/<session_id>.jsonl and every subagent transcript
~/.claude/projects/*/<session_id>/subagents/*.jsonl (python3 ~/bin/audit-job.py <job> does this for a run-job.sh
job).
Research-only runs (2026-10-02, used for the Dell's k3s path and the multi-location rig design):
- Tools: --allowedTools WebFetch,WebSearch and nothing else.
- Input: append < /dev/null to the remote command. Otherwise claude -p waits 3 s for stdin and warns.
- Output: read only the result field of the JSON.
- Cost: each run reported $0.24 to $0.79 (simulated, see below).
- Sources that fail or need care (job invotek-review, 2026-10-14):
- Google Trends returns HTTP 429 to WebFetch.
- https://api.seodata.dev/v1/keyword?q=<kw>&country=no gives Google Ads search volumes. A 404 means no data, and
spelling matters: ai-agenter returns 1,000, ai agenter returns 404.
- gartner.com and digdir.no return 403.
- WebFetch shows only part of a page's <head>, so don't conclude a meta tag is missing from its output.
- The Mac Mini and the Dell (since 2026-10-06): the Mac Mini's post user holds a Dell key, ~/.ssh/dell-stig-1, with the comment claude-3@mac-executor. Its authorized_keys line on the Dell is managed in OpenTofu, k8s/dell/k3s/orchestrator-access.tf, and CI's apply (hosts) installed it (dashecorp/infra#601, #604).
- Restrictions: the line has from= the Mac Mini's tailnet address, and forwarding is off.
- Use: the claude-3 workspace runs Dell checks from here with ssh -i ~/.ssh/dell-stig-1 claude@100.95.212.93, then sudo -n k3s kubectl ….
- The MacBook: its hand-added key still works.
- Rotate or revoke through var.orchestrator_ssh_public_key; "" removes the line.
- Why it's safe here: the Mac Mini's agent user (claude) has no passwordless sudo, so it can't read post's key.
Implementation runs (2026-10-02/03, the multi-location rig):
- Billing: it runs on its own Max subscription. The USD figure in the JSON output is simulated, not billed.
- Flags: use --resume for continuity, and always pass --effort explicitly.
- Code in and out: work goes in as a git bundle and comes back out as git format-patch; the orchestrator applies and pushes it. ibuild-e-bot cannot access rig repos.
- Usage stats: in tmux, start claude, run /usage, then open the Stats tab. Plan percentages aren't available, because it authenticates with CLAUDE_CODE_OAUTH_TOKEN.
- Effort: the interactive default is medium.
- Headless sandbox limits seen so far:
- it cannot execute bash, node, python or age;
- reads and writes outside the working directory are blocked, so put inputs inside the clone;
- tofu init is blocked;
- a job never starts Docker itself: a job that runs the Testcontainers E2E tests is launched inside ~/bin/with-rigbuild.sh, which starts the rigbuild VM for it (see Mac Mini delegation);
- WebFetch works only if it is allowed.
- Launch gotchas (2026-10-03/04):
- Launch from a script file, started with bash <script> and run_in_background. Never type the scp … && ssh … 'zsh -ic "claude -p …"' launch inline in a Bash call: inline launches stalled before running, three times, and blocked every call queued behind them. The same command in a script started at once.
- Sessions can't rm. Empty the export dir (out/) over ssh before each run.
- A -p run can end "waiting" on a trace that never reports. Resume the session with "finish now".
| Fact | Detail |
|---|---|
| CLI | Claude Code 2.1.282, updated 2026-09-24 from 2.1.211, which cannot run claude-opus-5-5 (needs 2.1.280+). autoUpdates is false in ~/.claude.json, so update it deliberately with claude update. Older versions stay in ~/.local/share/claude/versions; roll back with claude install <version>. iBuild-E spawns the same claude binary for its tasks. |
| Auth | CLAUDE_CODE_OAUTH_TOKEN in ~/.zshrc (Bitwarden "Claude Code OAuth Token (1-Year, Mac Mini)"); claude auth status → authMethod: oauth_token. Only zsh -ic loads it. A plain SSH command says "Not logged in", and a GUI /login saves to the keychain, which SSH cannot read. |
| Model | Always pass --model claude-opus-5-5; the CLI's default there is older. |
| GitHub identity | gh is logged in as ibuild-e-bot[bot], so commits and PRs pushed from there are authored by that App. That App is also the one agent App with workflow-write permission. Since 2026-10-14 the old ibuild-e-bot key files are deleted, and ~claude/.config/github-apps/git-credential-github-app.sh uses the first *.pem in its folder, so git in ~/repos/automate-e and ~/repos/dashecorp-rig-tools authenticates as iclaw-e-bot (recorded by the 2026-10-14 rotation session). |
| Repos | ~/repos/ holds clones of claude-3, rig-gitops and dashecorp-rig-{conductor,agent-runtime,docs,tools}, among others, many stale. ibuild-e-bot cannot read dashecorp/rig-conductor (404). Keep it that way: push code over SSH from the MacBook (GIT_SSH_COMMAND="ssh -i ~/.ssh/mac-executor" git push claude@100.92.170.124:repos/rig-conductor-rigbuild <ref>:refs/heads/<branch>), fetch the results back the same way, and push to GitHub from the MacBook. The Mini's ~/repos/claude-3 checkout carries a live local edit to agent-runner/agents/ibuild-e.json; never touch it, and work in a git worktree instead. |
| Admin | The claude user is in the admin group (corrected 2026-09-24, claude-3#947) and has passwordless sudo (/etc/sudoers.d: claude ALL=(ALL) NOPASSWD: ALL, verified 2026-09-29, claude-3#972). Any process running as claude, including iBuild-E and delegated claude -p jobs, can become root. What still needs the operator is post's password. post is the only volume owner and secure-token user, so a macOS update (softwareupdate --user post --stdinpass) and FileVault operations (fdesetup enable / authrestart) need it. |
| Build toolchain | dotnet 10.0.401, colima 0.10.3, lima 2.2.0 and the docker CLI 29.8.1, pinned. The review, conditions and install plan are in docs/ai-tooling-registry/build-toolchain/2026-09-24-security-review-mac-mini-dotnet-colima.md (claude-3#944). Source ~/.config/rigbuild/env.sh for every build. The colima profile is rigbuild: no mounts, loopback-only ports, stopped after each run. ~/.colima/_lima/_config/override.yaml stops lima forwarding privileged guest ports to * (claude-3#947). Every run that needs the VM goes through ~/bin/with-rigbuild.sh (source): one holder at a time, and it fails closed unless DOCKER_HOST answers as colima-rigbuild and nothing answers on /var/run/docker.sock. OrbStack 2.0.5 is still installed and is not approved. The 2026-10-15 test runs used it (operator-inbox#26); its stop and removal plan is in the review's 2026-10-20 amendment. |
Unattended boot after a power cut (Mac Mini)¶
Since 2026-09-25 the Mac Mini comes back by itself after a restart or power cut. The operator set this up (verified over SSH that day):
| Setting | Value | Check over SSH |
|---|---|---|
| FileVault | Off | fdesetup status → FileVault is Off. |
| Auto-login | post (UI name "Stig-Johnny Stobakk") |
defaults read /Library/Preferences/com.apple.loginwindow autoLoginUser → post |
| Power | sleep 0 (never), autorestart 1 (powers on after a power cut), displaysleep 10 |
pmset -g |
| Lock | Display off after 10 min; password required immediately after screen saver or display off | System Settings → Lock Screen |
Why post, not claude. SSH (claude@100.92.170.124) and the iBuild-E LaunchDaemon (ai.invotek.ibuild-e, runs as claude) need no GUI login once the disk is unlocked at boot. Tailscale does: Tailscale.app runs as post (its network extension runs as root), and every remote path goes over Tailscale. So auto-login has to be the account whose session starts Tailscale. Logging in claude would also have started its dormant RunAtLoad LaunchAgents; those were retired later that day (see below, claude-3#946).
Lock. The Mini is headless, so display-off is what locks it: 10 minutes after boot or the last activity, the console session locks.
Restart test, 2026-09-25. The operator restarted it at 08:24:23Z. It booted at 08:24:29Z and was back over Tailscale by itself at 08:24:53Z (about 30 s), with FileVault off, post at the console, Tailscale.app running as post, and iBuild-E running (node src/index.js in /Users/claude/repos/automate-e). None of claude's LaunchAgents started. The screen locked by itself: locked=True at 08:42:12Z (from ioreg IOConsoleUsers).
Trade-offs (accepted by the operator on 2026-09-24, in exchange for unattended recovery after power cuts):
- With FileVault off, the disk is not encrypted at rest.
- Auto-login stores post's password obfuscated, not encrypted, in /etc/kcpassword.
- Anyone with physical access at the site gets an unlocked admin session for up to 10 minutes after boot or the last activity.
Watch item: macOS updates. An update can re-enable FileVault or reset auto-login. After every macOS update, check over SSH that fdesetup status says Off and autoLoginUser is post. If either has changed, the next restart stops at the FileVault unlock screen or the login window, with no Tailscale either way. For FileVault, see the fallback below.
Installing a macOS update remotely. State on 2026-09-29:
- Version: macOS 26.1 (25B78). Offered: 26.7.1 (macOS Tahoe 26.7.1-25G241) and 27.0.1.
- Automatic installs are off. AutomaticDownload, AutomaticallyInstallMacOSUpdates, CriticalUpdateInstall and ConfigDataInstall are all 0 in /Library/Preferences/com.apple.SoftwareUpdate, so nothing installs or restarts on its own.
- Background Security Improvements auto-install is on (SplatEnabled = 1). They only target the latest minor release, so 26.1 gets none. With FileVault off, a restart from one would come back by itself anyway.
The operator runs the install, because only post can authorize it:
ssh -t -i ~/.ssh/mac-executor claude@100.92.170.124
softwareupdate --list --no-scan # copy the exact label; never use -a or -r (macOS 27 is also offered)
read -rs PW # the OPERATOR types post's password; nothing is echoed
printf '%s\n' "$PW" | sudo nohup softwareupdate --install '<label>' --restart --agree-to-license \
--no-scan --user post --stdinpass >/tmp/su-install.log 2>&1 &
unset PW
The password must come on stdin. --stdinpass <password> as an argument has been broken since macOS 13: the value is parsed as an update label.
After it comes back:
- Check sw_vers and the watch item above.
- Check that the four update keys are still 0.
- If Setup Assistant shows Update Mac Automatically in post's session (reach it over Screen Sharing), choose Only Download Automatically. Continue turns automatic installs back on.
History (claude-3#878, #972). The Mac was unreachable for about 50 days, in two stages:
1. About 2026-08-04 to 08-24: up, but no network. Its Wi-Fi daemon had hung. In the 5 minutes before the power-off below, airportd repeated [BUG] ORPHANED REQUEST every minute and mDNSResponder logged 4,111 Query suppressed … (no DNS service). Tailscale was only the symptom. Per the 2026-09-24 forensic pass, the hang began when the Deco mesh dropped on 2026-08-04; those logs have since rotated out.
2. 2026-08-24 to 09-24: the FileVault unlock screen. At 15:05:07 CEST on 08-24 someone on site held the power button. loginwindow logged the press, the session ended 4 s later with no clean shutdown, and the Mac booted at 15:05:47. It was not a power cut: the Dell and the Pis have been up since 2026-03-30. The Mac stayed at the unlock screen, off the network, until the operator's father typed the password at the console on 09-24.
The second stage is why FileVault was turned off.
Not covered by FileVault-off: a Wi-Fi hang. Stage 1 can happen again. The Mac and iBuild-E stay up, but nothing remote can reach them, and Pi-E sees the Mac missing from 192.168.68.0/24.
- Fix: a power-cycle on site, after which it comes back by itself.
- Prevention, option 1: an Ethernet cable. Ethernet is already first in the service order, and en0's hardware MAC 1c:f6:4c:55:31:09 does not rotate, so a Deco address reservation will hold.
- Prevention, option 2: a watchdog that restarts Wi-Fi when the network disappears.
Neither prevention exists yet.
Fallback: pre-boot SSH unlock (only if FileVault is back on)¶
macOS 26 on Apple silicon runs a limited pre-boot SSH when Remote Login is on (it is): a password accepted there unlocks the disk and boot continues (Apple apple_ssh_and_filevault(7)). There is no Tailscale before unlock, so go through Pi-E on the same LAN:
ssh -i ~/.ssh/rpi-claude claude@100.107.48.17 # Pi-E, over Tailscale
sudo /usr/local/sbin/find-macmini.sh # from Pi-E: the Mac's current IP
ssh -o PubkeyAuthentication=no post@<ip> # the OPERATOR types post's password
SSH drops once the disk unlocks.
Proven on this Mac over Wi-Fi on 26.1. On 2026-05-30 and 2026-06-09, after planned restarts, ssh post@<ip> from Pi-E answered "This system is locked…" and then "System successfully unlocked".
It failed on 2026-08-24 because pre-boot auto-join had no known networks. Every attempt ran with scanChannelCount=0, and en1: no SSID was logged for 30 days. Per the forensic pass, pre-boot reads its Wi-Fi list from NVRAM (current-network, preferred-networks). macOS writes those whenever it joins a network, and nothing had been joined since the Wi-Fi hang.
Two more pre-boot limits:
- Sleep. Pre-boot uses default power settings (Sleep timer 1 display timer 10). On 08-24 it slept about 20 min after boot and afterwards woke only briefly, so unlock within ~10 min.
- Host key. Accept only the fingerprint in the Pi-E section above. Apple does not document whether pre-boot presents the same key. If find-macmini.sh finds nothing, look for a new host with port 22 open on 192.168.68.0/24. Treat a different key as a reason to stop and check (for example in the Deco client list), not to accept blindly.
Other notes:
- Ethernet works for pre-boot unlock on any 26.x.
- For a planned restart with FileVault on, use sudo fdesetup authrestart (needs post's password), which unlocks once automatically.
- The Dell (192.168.1.173) cannot reach the Mac's segment; Pi-E can.
What runs on the Mac Mini (2026-09-25)¶
Cleaned up on 2026-09-25 and verified over SSH that day (claude-3#946). At boot the Mini then ran only two system LaunchDaemons: iBuild-E and the pf anchor loader. Since 2026-10-14 only the pf anchor loader is left (iBuild-E stopped, see the table). The macOS application firewall is disabled.
| Item | State | Where it went / how to undo |
|---|---|---|
ai.invotek.ibuild-e (iBuild-E, runs as claude from ~/repos/automate-e, which is really a rig-agent-runtime checkout) |
Stopped 2026-10-14 (claude-3#1206): the operator's "go" during the ibuild-e-bot key rotation; launchctl bootout, then the plist was moved |
/var/root/ibuild-e.plist.disabled (root, 600). Revive: put a new ibuild-e-bot App key in the plist and in ~claude/.config/github-apps/, move the plist back to /Library/LaunchDaemons/ai.invotek.ibuild-e.plist (600), then sudo launchctl bootstrap system it |
ai.invotek.mac-executor-pf (loads the mac-executor pf anchor) |
Kept, system LaunchDaemon | /Library/LaunchDaemons/ |
All 28 of claude's LaunchAgents: iClaw-E (ai.invotek.iclaw-e), a duplicate ai.invotek.ibuild-e user agent, the retired ai.invotek.mac-executor, the OpenClaw gateway, cloudflared, noVNC and Ollama launchers, Postgres@17, the docs checkers and Google updater shims |
Retired (moved, not deleted). None had run since the 2026-08-24 restart, because claude never logs in at the GUI (auto-login is post) |
~claude/Library/LaunchAgents.retired/, listed in MANIFEST-2026-09-25.txt. Undo: move the plist back to ~claude/Library/LaunchAgents/ |
Veribox: LaunchDaemons ai.invotek.veribox-docs.{serve,rebuild} (installed 2026-07-10; served client docs unauthenticated on 100.92.170.124:8099) |
Removed. Daemons stopped. The 5.1 GB of Veriton-Technologies client clones in ~/veriton (all clean, nothing unpushed) and ~/veribox-docs were deleted; two loose files were saved to the operator's MacBook. Veribox is not rig |
Plists in /Library/LaunchDaemons.retired/ |
Vox-E: LaunchDaemons ai.invotek.voxe-{llm,tts,web} (voice/ops prototype from 2026-06-19, design). Ran as claude, the same user as the autonomous iBuild-E, which its own design forbids; no use in its logs after 2026-06-19 |
Removed. Daemons stopped. ~/voiceops (code, keys, venv) deleted after its code was archived to the operator's MacBook |
Plists in /Library/LaunchDaemons.retired/ |
Vox-E, MacBook side: LaunchAgents com.invotek.voxe-tunnel (ssh -R to the Mini's 127.0.0.1:8769) and com.invotek.voxe-bridge (~/.claude/mcp-servers/mcp-read-bridge.py: read tools of six MCP servers behind a bearer key) |
Disabled on the operator's MacBook | — |
So iClaw-E is no longer on the Mac Mini. Its GitHub App (iclaw-e-bot) and Discord channels still exist.
Known remaining exposure: iBuild-E's runtime listens on *:3000 (LAN + tailnet). Its dashboard routes /, /api/state and /health return 404.
Physical Locations¶
Ålesund (Møre og Romsdal) — operator's parents' house, no hands-on access:
├── Dell Precision 5520 (i7, 32GB, 233GB NVMe, worn out: replace) — k3s cluster
├── Mac Mini M4 — iBuild-E (stopped 2026-10-14)
├── Raspberry Pi 4-01 — exit node
├── Raspberry Pi 4-02 — spare/legacy (former Review-E host; Review-E now in GCP k3s)
├── Raspberry Pi 4-03 — Pi-E
└── Synology DS412+ (~9TB) — NFS for k3s
Las Palmas de Gran Canaria — with the operator:
├── MacBook Air M4 — dev + orchestrator
└── Mobile devices
Boston (Hostinger DC):
└── VPS srv1099021 — Volt-E
Ålesund site notes¶
- No hands-on physical access. The hardware is in a third party's home in a different part of the country. Nobody on the team can touch it. Anything needing a power cycle, a cable, or a screen — a box that is off, a wedged console — means phoning the operator's parents. The Mac Mini was unreachable for ~50 days (2026-08-04 to 09-24) before anyone noticed: first a Wi-Fi hang, then the FileVault unlock screen (see History). Treat "unreachable" as potentially unrecoverable-without-a-phone-call, not as a reboot away.
- Exit nodes: pi4-01 is the documented exit node (see the host table and
networking.md) and pi4-02 also advertises the capability. Measured 2026-08-24: the operator's MacBook is configured to exit via pi4-02. Both sit at this site, so either way the egress is the Ålesund WAN. - Measured WAN egress:
83.243.161.154— ISP "Tafjord Get", the Ålesund-region cable operator. pi4-02, pi4-03 and the Dell each resolved to it independently (2026-08-24). - Re-derive the site location from any Linux host there:
curl -s --interface $(ip route | awk '/^default/{print $5; exit}') https://ipinfo.io/json. You must bind to the physical interface: a host using a Tailscale exit node reports the exit node's egress, not its own. This bites in both directions — the operator's MacBook exits via pi4-02, which is itself at this site, so an unbound check from Gran Canaria returns the Ålesund WAN and looks correct while telling you nothing.