Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#247

P2: the runner should probe its container runtime at startup instead of failing the first job

closed · opened by keenan 21h ago

keenan opened this 21h ago

The runner reports itself healthy, connects, leases a job, and only then discovers that its container runtime cannot create a container — by failing the job.

ci/container/container.go:64 validates the runtime once, before any work starts, but what it checks is the binary: unsupported name, or not on PATH. It does not check that the runtime works, so a machine in a state like this one comes up active and takes work:

level=warning msg="RunRoot is pointing to a path (/run/vole-runner/containers) which is not writable. Most likely podman will fail."
Error: default OCI runtime "crun" not found: invalid argument
building image-file "verify" exited 125

Observed on a runner whose podman run root was unusable; the job it leased was failed (see the companion issue about how that reads on the run page). The operator’s only signal was a repository’s check going red.

Why it matters

The runner is the party that decides whether it can do the work, and it is the only party that can know. Discovering it after leasing means: an attempt spent out of MaxAttempts, a commit blamed for it, and an error that reads like the repository’s fault in a job log nobody opened yet. A runner that cannot run a container is not degraded — it is not a runner.

What to decide

  • Probe once at startup and refuse to serve: a cheap podman info (or the runtime’s equivalent) catches a missing crun, an unwritable run root, a bad storage driver, and a XDG_RUNTIME_DIR mismatch, which is most of the states that produce exit 125.
  • Where the failure should surface: the unit failing so systemd’s Restart=on-failure and journalctl -u vole-runner show it, versus a live runner that refuses leases. The unit already has ExecStartPre for the arguments, so a preflight check has a home.
  • Whether the probe repeats after a restart of the runtime, or whether a runner that lost its runtime mid-life (disk full, /run cleared) should stop serving on the next job instead of failing it.
  • The message must name the cause and the runtime’s own words: “cannot run a container with podman: default OCI runtime “crun” not found”.

Acceptance criteria

  • A runner whose runtime cannot create a container fails before it leases anything, with the runtime’s message in journalctl -u vole-runner.
  • No job is consumed by a runner in that state.
  • A healthy runner is unaffected: no extra image pull, no measurable delay at startup.

Related: the companion issue about attempts that never ran a step, and #241 (runtime configuration the runner’s setup does not cover).

Comments (0)

No comments yet.