Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#246

P1: a job that never ran a step is recorded as a failed attempt, so a broken runner reds a passing commit

closed · opened by keenan 21h ago

keenan opened this 21h ago

A run that never executed a step is recorded as a failed attempt, and a required check on main then reds a commit whose tests pass. This happened to e6f0d8b947d985c8615e1172aef06cee8fd3729e right after it landed.

Run 01M6YY97B5V966TVRED4QA2643 (ref refs/heads/main, sha e6f0d8b9):

attempt outcome what ran
1 lease lost mid-test, no result recorded the whole suite up to metadata, all ok, no FAIL
2 failed, exit=1, steps=nil no step at all — the image build failed
3 success, exit=0 the whole suite, green

Attempt 2’s entire log is three lines:

=== vole · verify · main @ e6f0d8b9
environment: file:verify via podman, network bridge
level=warning msg="RunRoot is pointing to a path (/run/vole-runner/containers) which is not writable..."
Error: default OCI runtime "crun" not found: invalid argument
building image-file "verify" exited 125

The runner could not run the job, and the run page said the same thing it says when a test fails. With checks required on a protected branch, the commit looked broken to everyone looking at the gate.

Why it matters

Two different facts are collapsed into one:

  • the job ran and something failed — the repository’s problem, and the check should say so;
  • the job never ran — the machine’s problem, and the commit should not be blamed for it.

Today fail() in ci/agent/agent.go records both the same way (exit=1, steps=nil), and the gate reads the newest run per (workflow, job) for the sha, so any such attempt reds the check until somebody re-runs it by hand.

The same shape comes out of the lease bug in #243: an attempt that dies after its last heartbeat records no result at all, and the run is failed once the retries are spent.

What to decide

  • A record whose steps are empty and whose failure came from the runner’s own setup should be distinguishable from a step exiting non-zero — a separate status, or a field naming where it failed.
  • What the gate does with it: does “no attempt produced a result” count as failed, as no-opinion, or does the run stay queued for another runner?
  • Whether the runner retries such a job on another machine instead of spending the attempt, given MaxAttempts is about flaky tests, not broken runners.
  • The run page should say which of the two happened without reading the log.

Acceptance criteria

  • A commit whose tests pass is not red-gated by a runner that could not start the job.
  • The run page distinguishes “the job failed” from “the runner could not run it”.
  • A step that genuinely exits non-zero still fails the run and still reds the check.

Related: #243 and #245 (the lease loss that produced attempt 1’s no-result shape), and the new issue about probing the runtime at startup.

Comments (0)

No comments yet.