An attempt loses its lease in the gap between two steps, and the retry that follows dies the
same way until the queue gives up. On a runner with the default lease_seconds = 60, a job
whose steps each take a few seconds but together take more than a minute dies during the
first long step:
01:26:49 execStep "test"
01:27:09 heartbeat sent, took 7ms, ok=false <- the first heartbeat, and the lease is gone
vole runner: lease lost while running "test"; stopping
Twelve steps (format, vet, build, licenses, govulncheck, packages, deployment files, markdown links) ran between the lease being granted and test starting, in about
60 seconds. None of them took longer than a heartbeat interval.
Why
runCommand heartbeats while a command runs, and the ticker is created when that command
starts (ci/agent/agent.go:1598, interval lease/3). A step shorter than the interval
therefore never heartbeats at all, and nothing heartbeats between steps. Renewal only
happens inside a step that outlives one interval, so the lease is not renewed by the step
that needs it — it is renewed by whichever step happens to be running when a tick is due.
The queue’s reaper (server/ci.go watchLeases, every 5 s) takes any job whose
lease_until has passed, requeues it, and the runner’s next heartbeat is answered
ok=false. Every step boundary is a window where the lease can expire, so whether a job
survives depends on how long its short steps happen to take: the same workflow passed on one
attempt (15 minutes, finished recorded with a real result) and died at 61 s, 65 s and 77 s
on the next four.
What it costs
Every death is a lost attempt: the retry re-runs the whole job from the start, and five of
them exhaust MaxAttempts and fail the run. A long go test step at the end of a workflow
hits this reliably, because the cheap steps in front of it are exactly the ones that consume
the lease without renewing it.
Workaround, not a fix
Raising lease_seconds to 300 makes a two-and-a-half-minute job fit inside a single lease,
so it passes — because it no longer needs a heartbeat at all. Any job longer than the lease
still dies, and a genuinely dead runner now holds its job for five minutes instead of one.
Acceptance criteria
- An attempt keeps its lease for as long as it is working, including the time spent between
steps: a job of twelve one-second steps and one five-minute step completes at
lease_seconds = 60. - The lease is renewed at every step boundary, not only inside a step longer than one interval.
- A lease that is genuinely lost is still reported and still stops the attempt.
- A test covers a lease held across steps, so the next change to the step loop cannot silently drop the renewal.