Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Probe the container runtime at startup instead of on the first job

f28097af06382d809c64e8ce93e5838e465b58af · Keenan Nemetz · 2h ago · parent 9704d735

Probe the container runtime at startup instead of on the first job

Validating the runtime checked the binary: an unsupported name, or not on PATH.
That is not the same as the runtime being able to create a container. An
unwritable run root, a storage driver the kernel will not load, an OCI runtime
that is not installed, and an XDG_RUNTIME_DIR mismatch all pass a LookPath check
and none of them can start a container, so a runner came up active, connected,
leased a job, and failed it before a step ran.

The only party that can know is the machine, and the only cheap way to ask is the
runtime itself: `info` fails for each of those states, pulls no image, and answers
in milliseconds on a runner that has run a job. Probe once at startup and refuse
to serve, so the unit fails, Restart=on-failure retries it, and journalctl carries
the reason instead of a job log carrying an exit code.

The message names the runtime and repeats its own last line — podman's warnings
about the run root are not the cause, and "exited 125" is what made a machine's
problem read as a repository whose build broke:

  cannot run a container with podman: Error: default OCI runtime "crun" not found: invalid argument

vole workflow run makes the same check before a local run, so a developer with a
stopped daemon hears about it up front rather than at the first step.

The probe runs once: a runtime that breaks while a runner is serving fails the job
it was building, which is a record the run page has to be able to describe (#246),
not something a second probe would have prevented.

Closes 01M6Z05PXY0HN6B7Q1XYYH3MQC

docs/runner.md +21

@@ -35,6 +35,27 @@ vole runner --remote ssh://git@host:2222 --repos demo --repos docs
35 35 - `--once` leases one job and exits, which is useful for a runner that is
36 36 scheduled per job.
37 37
38 ## Before it takes a job
39
40 The runtime is probed once at startup: the runner runs `<runtime> info` and refuses
41 to serve if it fails, before it connects and before anything is leased. A runtime
42 whose binary is installed but which cannot create a container — a missing OCI
43 runtime, a run root it cannot write, a storage driver the kernel will not load — is
44 the machine's problem, and a runner that took a job in that state would spend an
45 attempt and put the machine's failure on a commit that never ran a step.
46
47 The message repeats the runtime's own last line:
48
49 ```
50 vole runner: cannot run a container with podman: Error: default OCI runtime "crun" not found: invalid argument
51 ```
52
53 The probe is one query and pulls no image. It runs once: a runtime that breaks while
54 the runner is serving fails the job it was building, and restarting the runner probes
55 again. As a system service the unit exits on this error, so `Restart=on-failure`
56 retries it and `journalctl -u vole-runner` carries the message. `vole workflow run`
57 makes the same check before a local run.
58
38 59 ## Running several jobs at once
39 60
40 61 A build is usually one step waiting on one thing at a time, so most of a big machine