Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Probe the container runtime at startup instead of on the first job

f28097af06382d809c64e8ce93e5838e465b58af · Keenan Nemetz · 4h ago · parent 9704d735

Probe the container runtime at startup instead of on the first job

Validating the runtime checked the binary: an unsupported name, or not on PATH.
That is not the same as the runtime being able to create a container. An
unwritable run root, a storage driver the kernel will not load, an OCI runtime
that is not installed, and an XDG_RUNTIME_DIR mismatch all pass a LookPath check
and none of them can start a container, so a runner came up active, connected,
leased a job, and failed it before a step ran.

The only party that can know is the machine, and the only cheap way to ask is the
runtime itself: `info` fails for each of those states, pulls no image, and answers
in milliseconds on a runner that has run a job. Probe once at startup and refuse
to serve, so the unit fails, Restart=on-failure retries it, and journalctl carries
the reason instead of a job log carrying an exit code.

The message names the runtime and repeats its own last line — podman's warnings
about the run root are not the cause, and "exited 125" is what made a machine's
problem read as a repository whose build broke:

  cannot run a container with podman: Error: default OCI runtime "crun" not found: invalid argument

vole workflow run makes the same check before a local run, so a developer with a
stopped daemon hears about it up front rather than at the first step.

The probe runs once: a runtime that breaks while a runner is serving fails the job
it was building, which is a record the run page has to be able to describe (#246),
not something a second probe would have prevented.

Closes 01M6Z05PXY0HN6B7Q1XYYH3MQC

ci/agent/agent.go +7

@@ -340,6 +340,13 @@ func Run(ctx context.Context, opts Options) error {
340 340 if err != nil {
341 341 return err
342 342 }
343 // The binary being on PATH is not the runtime being able to create a container.
344 // Probe before anything is leased: a machine that cannot run a container is not
345 // a degraded runner, and an attempt spent proving it is a commit blamed for the
346 // machine's problem.
347 if err := container.Probe(ctx, box.Config); err != nil {
348 return err
349 }
343 350 cleanupJobTrees(cache)
344 351 cleanupStaleContainers(box)
345 352 cleanupStaleNetworks(box)