Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

#256

Probe the container runtime at startup instead of on the first job

merged · by keenan 48m ago · merged into main at f28097af by keenan 44m ago

keenan opened this 48m ago

Closes #247. A runner whose runtime could not create a container came up active, connected, leased a job, and failed it before a step ran — spending an attempt and putting the machine’s problem on a commit that never ran anything.

What changed

  • ci/container/container.go: Probe(ctx, cfg) runs <runtime> info with a 60-second bound, captures both streams, and on a non-zero exit returns cannot run a container with <runtime>: <the runtime's last line>. The warnings above podman’s fatal line are not repeated; they are not the cause, and the last line is the runtime naming it.
  • ci/agent/agent.go: Run probes after the settings are validated and before anything is leased, so the unit fails, Restart=on-failure retries it, and journalctl -u vole-runner carries the reason.
  • cmd/vole/workflowrun.go: vole workflow run makes the same check before a local run, so a stopped daemon or a broken run root is named up front instead of appearing as a step that exited 125.
  • cmd/vole/runner.go, docs/runner.md: what the probe is, the message it produces, that it runs once, and that a runtime which breaks while serving fails the job it was building.

Decisions in the issue’s checklist

  • Probe once at startup, and refuse to serve. Not a live runner that declines leases: a runner that cannot run a container is not degraded, and the unit failing is what makes systemd and journalctl do the reporting.
  • The process probes, not ExecStartPre. A pre-start shell line would need the runtime name and the timeout a second time, outside the code that already knows what a valid configuration is.
  • No re-probe mid-life. A runtime that breaks while a runner is serving fails the job it was building; that record is #246’s subject, and a probe that stopped serving would turn one bad job into a dead runner for the rest of the process’s life. Restarting the runner probes again.
  • Nothing in the probe pulls an image, and on a runner that has run a job it is a sub-second info.

Testing

New tests: a fake runtime that answers info the way podman does with no OCI runtime (warning, then fatal) and shows the message names the cause; a working runtime passing; lastLine skipping blank lines and warnings; and agent.Run returning the probe error with no instance reachable, so nothing is leased.

vole workflow run verify passes (format, vet, build, licenses, govulncheck, packages, deployment files, links, race suite).

Not verified here: no podman on this machine, so the exact wording of a real podman info failure is from the issue’s log rather than a fresh reproduction. The tests fake the output, so the shape is pinned; the wording comes from podman.

Discussion (0)

No discussion yet.