Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#137

Keep a finished CI attempt and drain running jobs on restart

merged · by keenan 9d ago · merged into main at 73755a56 by keenan 9d ago

keenan opened this 9d ago

Why

Two recent runs on this instance were retried even though nothing in the code had failed:

  • 01M67PXW98Q2TQ2FHAWYJWXXG6 (main at 1265ea55): the server was restarted while attempt 1 was running govulncheck. The runner’s channel died, the lease expired, and the job was queued again at 17:50:36.
  • 01M6780Q4APDAKB4TJDT3MQ3RF (index/search at 41dd571f): attempt 1 ran every step successfully, but its lease expired before it could report. The runner logged lease lost while running "test"; stopping, and the whole job ran again as attempt 2.

The queue treated every expired lease as a dead runner. It cleared the owner, so a runner that had actually finished could not report, and a clean server stop closed the connection immediately. A deploy therefore cost a second attempt, and a successful attempt could be thrown away.

What changed

  • A requeued run keeps the owner of the attempt that was running. A late record and late log chunks are accepted while that attempt is still the current one, and refused as soon as another runner leases the job.
  • The runner sends its record when the heartbeat says the lease is gone. The server accepts it only if no new attempt has taken over.
  • A clean vole serve shutdown stops handing out new jobs and waits up to a minute for already-leased jobs to report. The packaged systemd unit leaves room for that wait.
  • A job killed mid-step still runs again. That is the honest outcome; this change removes the pointless repeats.

Tests

  • Unit tests cover a late finish before another lease, a late finish after another lease, a late finish from another account, late log chunks, and drain behaviour.
  • ./vole workflow check and ./vole workflow run verify pass.

Discussion (0)

No discussion yet.