GPU brain¶

Rig-E's language model does not run on the rig. It runs on a rented RunPod GPU pod serving Qwen/Qwen2.5-32B-Instruct-AWQ through vLLM. The pod is off almost all the time.

Sending Rig-E a message in the Rig Cockpit wakes the pod and starts billing. That is the intended behaviour, not an accident — but it is the one thing to know before you type.

Where do I talk to it¶

The web cockpit — rig-cockpit.dashecorp.com — and nowhere else. Manager tab, then pick rig-e in the "Talking to" selector.

"Rig Cockpit" names three different clients and only that one can reach the GPU. The iOS app cannot: it has five read-only tabs and no chat surface at all, so it can neither wake the pod nor ask it anything. The Capacitor shell would inherit the web cockpit's chat, but it has not shipped. See cockpit.md for what each surface can and cannot do.

1. What it is and what it costs¶


GPU	Chosen per wake, cheapest eligible in datacenter `EU-RO-1` on RunPod secure cloud — not a fixed card. An RTX PRO 4500 32 GB when one is free; the 2026-09-12 wake got an A100 80GB PCIe
Price	Whatever card the pod lands on, under the `RUNPOD_MAX_PRICE_PER_HOUR=2.00` cap (`rig-conductor/deploy/k8s/deployment.yaml`); the walk still ranks cheapest-first. Last observed: A100 80GB PCIe at 1.59 USD/h (2026-09-12, the first wake after rc#2092). Billed only while the pod is up — a stopped pod costs nothing. The old flat $0.72/h in this row predated the wake fix
Model / window	`Qwen/Qwen2.5-32B-Instruct-AWQ`, 16,384-token context (input + output). The runtime reserves 2,048 for output and ~930 for tool schemas, leaving ≈13,400 for one prompt (`rig-agent-runtime/src/agent/providers/context-budget.js`, measured twice against the live pod's own 400 reply)
Weights	network volume `qwen-weights` (`bjn7d94ej9`). The volume, not the pod id, is the identity: a pod that loses its GPU is replaced on the same volume automatically
Time to usable	≈3 minutes observed on the first wake after rc#2092 (2026-09-12). The older ≈9-minute figure (wake → `RUNNING` ≈3 min, then vLLM loading ≈6 min) is the slow case and is still what Rig-E's own reply quotes. `RUNNING` never means usable
Typical cycle	wake → answer → automatic idle stop 15 min after ready ≈ 20 min billed, so ≈ $0.24 on a 0.72/h card and ≈ $0.53 on the 1.59/h A100 — the cost of one conversation is set by which card the wake landed
Hard ceiling	6 h wall clock (`MAX_RUNTIME_MINUTES=360`) — a wedged pod cannot bill overnight
Who owns on/off	`rig-conductor` on the GCP k3s cluster. Nothing else talks to RunPod; the key lives in `rig-conductor-secrets/runpod-api-key` (see credentials.md)

Timings and cost are the first live operator run, 2026-09-09 12:26Z. That run also exercised the replacement path: RunPod refused to start the old pod (its GPU was gone) and the conductor created a new one on the same volume without anyone touching a config file.

2. How to wake it¶

The normal way: just chat. Open Rig Cockpit, pick Rig-E, send a message. If the pod is asleep, Rig-E asks the conductor to wake it and tells you so in the reply (§5). Nothing else to do — wait a few minutes and ask again. Asking again inside 15 minutes does not fire a second wake.

The manual way (cockpit unavailable, or you want it warm before you start). start is in-cluster only. The cockpit worker proxies GET /api/pods/{name}, and POST …/stop since rig-cockpit-worker#84, but never start. Both writes carry the conductor's operator-token gate (rig-conductor#2369), enforced since 2026-09-25 (rig-conductor#2436): without X-Rig-Operator-Token they answer 403. The token lives in the rig-conductor-gate-secrets Secret. The commands below read it on the node and hand it to curl on stdin (-H @-), so it never appears in an argument, the output or shell history. Operator only (credential policy): an agent never runs these.

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=stig@dashecorp.com --command='
    IP=$(kubectl -n rig-conductor get svc rig-conductor-api -o jsonpath="{.spec.clusterIP}")
    kubectl -n rig-conductor get secret rig-conductor-gate-secrets -o jsonpath="{.data.rig-operator-token}" | base64 -d \
      | sed "s/^/X-Rig-Operator-Token: /" | curl -s -H @- -X POST "http://$IP:8080/api/pods/qwen/start"; echo
  '

Status	Meaning
`200`	outcome `AlreadyRunning` / `Started` / `Created` — a pod is up (still minutes from usable if it was just started)
`202`	outcome `Waiting` — a start or create is still in flight. Poll the GET, do not re-POST expecting a different answer
`409`	the wake failed; `detail` says why
`503`	RunPod not configured (`RUNPOD_API_KEY` / `RUNPOD_VOLUME_ID`)
`404`	unknown pod name — the only name is `qwen`
`403`	missing or wrong `X-Rig-Operator-Token` (the gate refuses before anything else is checked)

The wake runs on the conductor's application lifetime, not on your HTTP request, so a dropped connection (Cloudflare Access cuts the origin at 100 s) never leaves a half-started pod. Retrying joins the same in-flight wake.

3. How to see whether it is up¶

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=stig@dashecorp.com --command='
    IP=$(kubectl -n rig-conductor get svc rig-conductor-api -o jsonpath="{.spec.clusterIP}")
    curl -s "http://$IP:8080/api/pods/qwen"; echo
  '

A real answer (2026-09-09, pod asleep):

{"name":"qwen","configured":true,"status":"EXITED","isRunning":false,"podId":"bsyo7ucc9d6nkm",
 "proxyUrl":"https://bsyo7ucc9d6nkm-8000.proxy.runpod.net","podsOnVolume":2,
 "datacenter":"EU-RO-1","lookup":"FOUND","readySince":null}

Usable = isRunning: true AND readySince non-null. Nothing else counts.

Field	How to read it
`status`	RunPod's own state: `PROVISIONING` `STARTING` `RUNNING` `EXITED` `ERROR` `TERMINATED`, or `UNKNOWN`. `RUNNING` does not mean usable — vLLM then loads the model for ~6 minutes
`isRunning`	`RUNNING` with at least one GPU attached — the only state that can serve. Still not "ready"
`readySince`	The one that matters: when the idle monitor first saw vLLM answer `/metrics` since this boot. `null` while booting, loading, or down
`proxyUrl`	Set whenever any pod is named, including a dead one. Never read it without `isRunning`
`podsOnVolume`	Should be `1`. `2` means a stale pod and its replacement both exist — see §6
`lookup`	`FOUND` the volume was read · `UNREADABLE` RunPod's API did not answer, so a neighbouring `UNKNOWN` status is a blip and not a verdict · `NOT_FOUND` no pod on the volume at all · `UNCONFIGURED` key or volume env missing

Conductor log lines to grep¶

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=stig@dashecorp.com --command='
    kubectl -n rig-conductor logs deploy/rig-conductor-api --since=2h | grep RunPodIdleMonitor
  '

Line	Means
`watching pod qwen on volume …: stop after 00:15:00 idle once ready …, boot grace 00:20:00, hard cap 06:00:00`	the monitor armed at conductor start
`pod qwen (<id>) ready after <elapsed> (boot grace 00:20:00)`	model ready — the idle clock starts here, and `<elapsed>` is the `RUNNING`→ready model load
`pod qwen (<id>) idle 00:15:00 (Idle) -> stop requested (ok=True)`	the normal automatic stop
`pod qwen (<id>) not ready within boot grace … -> stop requested`	vLLM never came up; stopped as a dead model
`pod qwen (<id>) exceeded runtime cap 06:00:00 -> stop requested`	the 6 h backstop fired
`N pods on volume … — classifying every one` (warn, every tick)	more than one pod on the volume — §6

Rig-E's own side is kubectl -n dashecorp logs rig-e-rig-agent-runtime-0; grep [chat] provider not ready, [chat] wake requested for pod qwen, STUCK: up but not answering, and the boot line openai-api endpoint:.

4. How to stop it¶

Usually you don't. RunPodIdleMonitor in the conductor stops the pod after 15 idle minutes counted from model-ready (rig-conductor#2056 — before 2026-09-09 the window started at RUNNING, so the model-load minutes were spent as idle and a woken pod stopped almost immediately). A pod that never becomes ready is stopped after RUNPOD_BOOT_GRACE_MINUTES (20). A pod up 6 hours is stopped regardless of what the idle streak can see.

Manual stop:

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=stig@dashecorp.com --command='
    IP=$(kubectl -n rig-conductor get svc rig-conductor-api -o jsonpath="{.spec.clusterIP}")
    kubectl -n rig-conductor get secret rig-conductor-gate-secrets -o jsonpath="{.data.rig-operator-token}" | base64 -d \
      | sed "s/^/X-Rig-Operator-Token: /" | curl -s -H @- -X POST "http://$IP:8080/api/pods/qwen/stop"; echo
  '

It stops every pod attached to the volume, which is what you want when a stale pod and its replacement are both billing. 200 with a per-pod ok list (ok is the conjunction) · 404 no pod on the volume · 503 the volume lookup was unreadable and nothing was stopped — retry. 403 means the token was missing or wrong, and nothing was stopped.

A "Stop GPU" button on the cockpit Manager tab is open at dashecorp/rig-cockpit-worker#85, and a status tile at #87. Until they merge the cockpit can only read pod status (#86 allowlisted GET /api/pods/{name}), and the iOS app has no chat and no pod surface at all.

5. What you will see in chat while it wakes¶

Verbatim, from rig-agent-runtime/src/chat/wake-policy.js. All of them come back as normal assistant messages with HTTP 200 — a red error bubble is never expected here.

Reply	State	Normal?
"Waking the GPU brain — that takes about 9 minutes. Ask me again in a few minutes."	pod was asleep; exactly one wake fired	✅ the expected first message
"Still waking — N min elapsed, usually ready by ~9. Ask again shortly."	inside the 15-minute window; no second wake	✅
"The GPU is up and the model is still loading — that takes a few more minutes. Ask again shortly."	`RUNNING`, vLLM loading, no wake on this conductor's clock	✅
"Still waking — N min elapsed; the GPU is up and the model is loading. Ask again shortly."	same, with a wake on the clock	✅
"Still not up after N minutes — I've asked the conductor to wake it again. Try again in a few minutes."	past the 15-minute window; a second wake fired	⚠️ slow, not yet broken. If it repeats, §6
"The model server has been up for N minutes and still is not answering — it may be stuck. Someone needs to look at the pod."	`RUNNING` for over 20 minutes without answering	❌ a real fault; also logged at error level
"I can't reach the rig conductor right now — try again in a minute."	the conductor lookup was `UNREADABLE` — nothing was woken	❌ conductor or RunPod-API problem

6. When something is wrong¶

Symptom	What to check	Where
Raw `HTTP 502` / a generic provider-failure bubble instead of one of the §5 replies	Rig-E must be on image tag `node-sha-418a948…` or later — wake-on-chat (#682) and the model-loading window (#689) are in that tag. An older tag also ignores `llm.baseUrlFrom` and, having no `baseUrl`, would talk to `api.openai.com`	`kubectl -n dashecorp get pod rig-e-rig-agent-runtime-0 -o jsonpath='{.spec.containers[0].image}'` vs `image.tag` in `rig-gitops/apps/dashecorp/rig-e-helmrelease.yaml`; and the boot log `grep "openai-api endpoint:"`
"still loading" past 20 minutes	the reply flips to the stuck wording and logs at error level; vLLM crashed or OOM'd coming up. The conductor's 20-minute boot grace should stop the pod — confirm it did, or you are paying for a dead model	`kubectl -n dashecorp logs rig-e-rig-agent-runtime-0 \\| grep STUCK`; conductor `grep "not ready within boot grace"`; then the §3 GET
"I can't reach the rig conductor right now"	is the conductor pod up, and is RunPod's API answering? `lookup: UNREADABLE` on the GET is the same signal. Nothing was woken, so nothing new is billing	`kubectl -n rig-conductor get pods`; the §3 GET
`podsOnVolume: 2`, warn every tick	a stale pod and its replacement are both attached — a conductor rollout mid-wake loses the sibling-termination tail (`rig-conductor#2058`, open). Harmless while both are `EXITED`; expensive if either is `RUNNING`	the §3 GET; conductor `grep "classifying every one"`. The §4 manual stop stops every pod on the volume
The pod is not stopping	the monitor only stops from the `Ready` phase after a full idle streak, and any running or waiting vLLM request resets it. A stop RunPod refused keeps the phase and retries a full window later. A conductor restart forgets `readySince` until the next tick re-observes the pod. Backstop is the 6 h cap	conductor `grep -E "stop requested\\|ready after"`; then the §4 manual stop

Owning docs¶

This page is the runbook; these are the sources of truth.

Doc	Covers
rig-conductor `docs/runpod-pod-control.md`	Canonical. The three routes with full response shapes, the volume-first wake and GPU selection, the idle phase machine, every env var and default
rig-agent-runtime `docs/2026-09-09-wake-on-chat.md`	The asleep → waking → loading → ready state table, every reply, and the live 2026-09-09 timeline
rig-agent-runtime `docs/2026-09-09-openai-endpoint-from-conductor.md`	`llm.baseUrlFrom`, the `proxyUrl` trust rules, caching and the once-per-turn re-resolve
rig-gitops `docs/2026-09-03-rig-e-deploy.md`	Rig-E's HelmRelease, its capability boundary, and the activation order

See also kubernetes.md for where the conductor runs, and agents.md for Rig-E's place in the agent fleet.