The runner reports itself healthy, connects, leases a job, and only then discovers that its container runtime cannot create a container — by failing the job.
ci/container/container.go:64 validates the runtime once, before any work starts, but what it
checks is the binary: unsupported name, or not on PATH. It does not check that the runtime
works, so a machine in a state like this one comes up active and takes work:
level=warning msg="RunRoot is pointing to a path (/run/vole-runner/containers) which is not writable. Most likely podman will fail."
Error: default OCI runtime "crun" not found: invalid argument
building image-file "verify" exited 125
Observed on a runner whose podman run root was unusable; the job it leased was failed (see the companion issue about how that reads on the run page). The operator’s only signal was a repository’s check going red.
Why it matters
The runner is the party that decides whether it can do the work, and it is the only party
that can know. Discovering it after leasing means: an attempt spent out of MaxAttempts, a
commit blamed for it, and an error that reads like the repository’s fault in a job log nobody
opened yet. A runner that cannot run a container is not degraded — it is not a runner.
What to decide
- Probe once at startup and refuse to serve: a cheap
podman info(or the runtime’s equivalent) catches a missingcrun, an unwritable run root, a bad storage driver, and aXDG_RUNTIME_DIRmismatch, which is most of the states that produce exit 125. - Where the failure should surface: the unit failing so systemd’s
Restart=on-failureandjournalctl -u vole-runnershow it, versus a live runner that refuses leases. The unit already hasExecStartPrefor the arguments, so a preflight check has a home. - Whether the probe repeats after a restart of the runtime, or whether a runner that
lost its runtime mid-life (disk full,
/runcleared) should stop serving on the next job instead of failing it. - The message must name the cause and the runtime’s own words: “cannot run a container with podman: default OCI runtime “crun” not found”.
Acceptance criteria
- A runner whose runtime cannot create a container fails before it leases anything, with the
runtime’s message in
journalctl -u vole-runner. - No job is consumed by a runner in that state.
- A healthy runner is unaffected: no extra image pull, no measurable delay at startup.
Related: the companion issue about attempts that never ran a step, and #241 (runtime configuration the runner’s setup does not cover).