Why
Two recent runs on this instance were retried even though nothing in the code had failed:
01M67PXW98Q2TQ2FHAWYJWXXG6(mainat1265ea55): the server was restarted while attempt 1 was runninggovulncheck. The runner’s channel died, the lease expired, and the job was queued again at 17:50:36.01M6780Q4APDAKB4TJDT3MQ3RF(index/searchat41dd571f): attempt 1 ran every step successfully, but its lease expired before it could report. The runner loggedlease lost while running "test"; stopping, and the whole job ran again as attempt 2.
The queue treated every expired lease as a dead runner. It cleared the owner, so a runner that had actually finished could not report, and a clean server stop closed the connection immediately. A deploy therefore cost a second attempt, and a successful attempt could be thrown away.
What changed
- A requeued run keeps the owner of the attempt that was running. A late record and late log chunks are accepted while that attempt is still the current one, and refused as soon as another runner leases the job.
- The runner sends its record when the heartbeat says the lease is gone. The server accepts it only if no new attempt has taken over.
- A clean
vole serveshutdown stops handing out new jobs and waits up to a minute for already-leased jobs to report. The packaged systemd unit leaves room for that wait. - A job killed mid-step still runs again. That is the honest outcome; this change removes the pointless repeats.
Tests
- Unit tests cover a late finish before another lease, a late finish after another lease, a late finish from another account, late log chunks, and drain behaviour.
./vole workflow checkand./vole workflow run verifypass.