GPU brain¶

Rig-E's language model does not run on the rig. It runs on a rented RunPod GPU pod serving Qwen/Qwen2.5-32B-Instruct-AWQ through vLLM. The pod is off almost all the time.

Sending Rig-E a message in the Rig Cockpit wakes the pod and starts billing. That is the intended behaviour, not an accident — but it is the one thing to know before you type.

1. What it is and what it costs¶


GPU	RTX PRO 4500 32 GB, RunPod secure cloud, datacenter `EU-RO-1`
Price	$0.72 / hour, billed only while the pod is up — a stopped pod costs nothing
Weights	network volume `qwen-weights` (`bjn7d94ej9`). The volume, not the pod id, is the identity: a pod that loses its GPU is replaced on the same volume automatically
Time to usable	≈ 9 minutes (wake → `RUNNING` ≈ 3 min, `RUNNING` → vLLM answering ≈ 6 min)
Typical cycle	wake → answer → automatic idle stop 15 min after ready ≈ 20 min billed ≈ $0.22
Hard ceiling	6 h wall clock (`MAX_RUNTIME_MINUTES=360`) — a wedged pod cannot bill overnight
Who owns on/off	`rig-conductor` on the GCP k3s cluster. Nothing else talks to RunPod; the key lives in `rig-conductor-secrets/runpod-api-key` (see credentials.md)

Timings and cost are the first live operator run, 2026-09-09 12:26Z. That run also exercised the replacement path: RunPod refused to start the old pod (its GPU was gone) and the conductor created a new one on the same volume without anyone touching a config file.

2. How to wake it¶

The normal way: just chat. Open Rig Cockpit, pick Rig-E, send a message. If the pod is asleep, Rig-E asks the conductor to wake it and tells you so in the reply (§5). Nothing else to do — wait a few minutes and ask again. Asking again inside 15 minutes does not fire a second wake.

The manual way (cockpit unavailable, or you want it warm before you start). The start/stop routes are in-cluster only — the cockpit worker proxies GET /api/pods/{name} but not the writes:

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=post@nutrieapp.com --command='
    IP=$(kubectl -n rig-conductor get svc rig-conductor-api -o jsonpath="{.spec.clusterIP}")
    curl -s -X POST "http://$IP:8080/api/pods/qwen/start"; echo
  '

Status	Meaning
`200`	outcome `AlreadyRunning` / `Started` / `Created` — a pod is up (still minutes from usable if it was just started)
`202`	outcome `Waiting` — a start or create is still in flight. Poll the GET, do not re-POST expecting a different answer
`409`	the wake failed; `detail` says why
`503`	RunPod not configured (`RUNPOD_API_KEY` / `RUNPOD_VOLUME_ID`)
`404`	unknown pod name — the only name is `qwen`

The wake runs on the conductor's application lifetime, not on your HTTP request, so a dropped connection (Cloudflare Access cuts the origin at 100 s) never leaves a half-started pod. Retrying joins the same in-flight wake.

3. How to see whether it is up¶

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=post@nutrieapp.com --command='
    IP=$(kubectl -n rig-conductor get svc rig-conductor-api -o jsonpath="{.spec.clusterIP}")
    curl -s "http://$IP:8080/api/pods/qwen"; echo
  '

A real answer (2026-09-09, pod asleep):

{"name":"qwen","configured":true,"status":"EXITED","isRunning":false,"podId":"bsyo7ucc9d6nkm",
 "proxyUrl":"https://bsyo7ucc9d6nkm-8000.proxy.runpod.net","podsOnVolume":2,
 "datacenter":"EU-RO-1","lookup":"FOUND","readySince":null}

Usable = isRunning: true AND readySince non-null. Nothing else counts.

Field	How to read it
`status`	RunPod's own state: `PROVISIONING` `STARTING` `RUNNING` `EXITED` `ERROR` `TERMINATED`, or `UNKNOWN`. `RUNNING` does not mean usable — vLLM then loads the model for ~6 minutes
`isRunning`	`RUNNING` with at least one GPU attached — the only state that can serve. Still not "ready"
`readySince`	The one that matters: when the idle monitor first saw vLLM answer `/metrics` since this boot. `null` while booting, loading, or down
`proxyUrl`	Set whenever any pod is named, including a dead one. Never read it without `isRunning`
`podsOnVolume`	Should be `1`. `2` means a stale pod and its replacement both exist — see §6
`lookup`	`FOUND` the volume was read · `UNREADABLE` RunPod's API did not answer, so a neighbouring `UNKNOWN` status is a blip and not a verdict · `NOT_FOUND` no pod on the volume at all · `UNCONFIGURED` key or volume env missing

Conductor log lines to grep¶

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=post@nutrieapp.com --command='
    kubectl -n rig-conductor logs deploy/rig-conductor-api --since=2h | grep RunPodIdleMonitor
  '

Line	Means
`watching pod qwen on volume …: stop after 00:15:00 idle once ready …, boot grace 00:20:00, hard cap 06:00:00`	the monitor armed at conductor start
`pod qwen (<id>) ready after <elapsed> (boot grace 00:20:00)`	model ready — the idle clock starts here, and `<elapsed>` is the `RUNNING`→ready model load
`pod qwen (<id>) idle 00:15:00 (Idle) -> stop requested (ok=True)`	the normal automatic stop
`pod qwen (<id>) not ready within boot grace … -> stop requested`	vLLM never came up; stopped as a dead model
`pod qwen (<id>) exceeded runtime cap 06:00:00 -> stop requested`	the 6 h backstop fired
`N pods on volume … — classifying every one` (warn, every tick)	more than one pod on the volume — §6

Rig-E's own side is kubectl -n dashecorp logs rig-e-rig-agent-runtime-0; grep [chat] provider not ready, [chat] wake requested for pod qwen, STUCK: up but not answering, and the boot line openai-api endpoint:.

4. How to stop it¶

Usually you don't. RunPodIdleMonitor in the conductor stops the pod after 15 idle minutes counted from model-ready (rig-conductor#2056 — before 2026-09-09 the window started at RUNNING, so the model-load minutes were spent as idle and a woken pod stopped almost immediately). A pod that never becomes ready is stopped after RUNPOD_BOOT_GRACE_MINUTES (20). A pod up 6 hours is stopped regardless of what the idle streak can see.

Manual stop:

gcloud compute ssh invotek-k3s --zone=europe-north1-b --project=invotek-github-infra \
  --account=post@nutrieapp.com --command='
    IP=$(kubectl -n rig-conductor get svc rig-conductor-api -o jsonpath="{.spec.clusterIP}")
    curl -s -X POST "http://$IP:8080/api/pods/qwen/stop"; echo
  '

It stops every pod attached to the volume, which is what you want when a stale pod and its replacement are both billing. 200 with a per-pod ok list (ok is the conjunction) · 404 no pod on the volume · 503 the volume lookup was unreadable and nothing was stopped — retry.

A "Stop GPU" button on the cockpit Manager tab is open at dashecorp/rig-cockpit-worker#85, and a status tile at #87. Until they merge the cockpit can only read pod status (#86 allowlisted GET /api/pods/{name}), and the iOS app has no chat and no pod surface at all.

5. What you will see in chat while it wakes¶

Verbatim, from rig-agent-runtime/src/chat/wake-policy.js. All of them come back as normal assistant messages with HTTP 200 — a red error bubble is never expected here.

Reply	State	Normal?
"Waking the GPU brain — that takes about 9 minutes. Ask me again in a few minutes."	pod was asleep; exactly one wake fired	✅ the expected first message
"Still waking — N min elapsed, usually ready by ~9. Ask again shortly."	inside the 15-minute window; no second wake	✅
"The GPU is up and the model is still loading — that takes a few more minutes. Ask again shortly."	`RUNNING`, vLLM loading, no wake on this conductor's clock	✅
"Still waking — N min elapsed; the GPU is up and the model is loading. Ask again shortly."	same, with a wake on the clock	✅
"Still not up after N minutes — I've asked the conductor to wake it again. Try again in a few minutes."	past the 15-minute window; a second wake fired	⚠️ slow, not yet broken. If it repeats, §6
"The model server has been up for N minutes and still is not answering — it may be stuck. Someone needs to look at the pod."	`RUNNING` for over 20 minutes without answering	❌ a real fault; also logged at error level
"I can't reach the rig conductor right now — try again in a minute."	the conductor lookup was `UNREADABLE` — nothing was woken	❌ conductor or RunPod-API problem

6. When something is wrong¶

Symptom	What to check	Where
Raw `HTTP 502` / a generic provider-failure bubble instead of one of the §5 replies	Rig-E must be on image tag `node-sha-418a948…` or later — wake-on-chat (#682) and the model-loading window (#689) are in that tag. An older tag also ignores `llm.baseUrlFrom` and, having no `baseUrl`, would talk to `api.openai.com`	`kubectl -n dashecorp get pod rig-e-rig-agent-runtime-0 -o jsonpath='{.spec.containers[0].image}'` vs `image.tag` in `rig-gitops/apps/dashecorp/rig-e-helmrelease.yaml`; and the boot log `grep "openai-api endpoint:"`
"still loading" past 20 minutes	the reply flips to the stuck wording and logs at error level; vLLM crashed or OOM'd coming up. The conductor's 20-minute boot grace should stop the pod — confirm it did, or you are paying for a dead model	`kubectl -n dashecorp logs rig-e-rig-agent-runtime-0 \\| grep STUCK`; conductor `grep "not ready within boot grace"`; then the §3 GET
"I can't reach the rig conductor right now"	is the conductor pod up, and is RunPod's API answering? `lookup: UNREADABLE` on the GET is the same signal. Nothing was woken, so nothing new is billing	`kubectl -n rig-conductor get pods`; the §3 GET
`podsOnVolume: 2`, warn every tick	a stale pod and its replacement are both attached — a conductor rollout mid-wake loses the sibling-termination tail (`rig-conductor#2058`, open). Harmless while both are `EXITED`; expensive if either is `RUNNING`	the §3 GET; conductor `grep "classifying every one"`. The §4 manual stop stops every pod on the volume
The pod is not stopping	the monitor only stops from the `Ready` phase after a full idle streak, and any running or waiting vLLM request resets it. A stop RunPod refused keeps the phase and retries a full window later. A conductor restart forgets `readySince` until the next tick re-observes the pod. Backstop is the 6 h cap	conductor `grep -E "stop requested\\|ready after"`; then the §4 manual stop

Owning docs¶

This page is the runbook; these are the sources of truth.

Doc	Covers
rig-conductor `docs/runpod-pod-control.md`	Canonical. The three routes with full response shapes, the volume-first wake and GPU selection, the idle phase machine, every env var and default
rig-agent-runtime `docs/2026-09-09-wake-on-chat.md`	The asleep → waking → loading → ready state table, every reply, and the live 2026-09-09 timeline
rig-agent-runtime `docs/2026-09-09-openai-endpoint-from-conductor.md`	`llm.baseUrlFrom`, the `proxyUrl` trust rules, caching and the once-per-turn re-resolve
rig-gitops `docs/2026-09-03-rig-e-deploy.md`	Rig-E's HelmRelease, its capability boundary, and the activation order

See also kubernetes.md for where the conductor runs, and agents.md for Rig-E's place in the agent fleet.