Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Keep a finished CI attempt and drain before shutting down

73755a56550dc6819b47469fbb08e85e7c7c5d04 · Keenan Nemetz · 9d ago · parent 1265ea55

Keep a finished CI attempt and drain before shutting down

A job whose lease expired was requeued even when its last step had already succeeded. The queue cleared the owner and the runner refused to report, so the same work ran again as attempt 2 for no failing step.

Keep the owner on a requeued row and accept a late record and log until another runner leases the job. The runner now sends a late record when the heartbeat says the lease is gone, and the server accepts it only for the attempt that is still current.

A clean server stop also drained nothing: closing SSH killed the runner and threw the job away. Stop leasing new work on shutdown and wait up to a minute for the jobs already running to report.

ci/agent/agent.go +12−2

@@ -678,11 +678,21 @@ func (s *runnerSession) finish(remote string, co checkout, job *runnerJob, log *
678 678 // Keep the lease through the report: the record is checked against it, and a
679 679 // slow upload must not turn a finished job into one that expired mid-report.
680 680 if rep, err := s.call(runnerRequest{Op: "heartbeat", Run: job.ID}); err != nil {
681 // The channel is gone, so there is no way to send the record. The queue
682 // will hand the job to another runner after the lease expires.
681 683 fmt.Fprintf(os.Stderr, "vole runner: lost the connection while finishing %s: %v\n", job.ID, err)
682 684 return
683 } else if !rep.OK {
684 fmt.Fprintf(os.Stderr, "vole runner: the lease for %s is gone; the result was not published\n", job.ID)
685 } else if rep.Error != "" {
686 // The server could not answer for the lease, so there is no evidence that
687 // it is still ours. A record sent now would be a guess.
688 fmt.Fprintf(os.Stderr, "vole runner: cannot renew the lease for %s: %s\n", job.ID, rep.Error)
685 689 return
690 } else if !rep.OK {
691 // The lease expired while the last step was running. The queue may not
692 // have given the job to anyone else yet, and this attempt has a real
693 // result. Send it: the server accepts it only while this is still the
694 // attempt the run is on, and refuses it once another runner leases the job.
695 fmt.Fprintf(os.Stderr, "vole runner: the lease for %s expired before the result was reported; sending it anyway\n", job.ID)
686 696 }
687 697
688 698 // The record goes to the server on the same channel that holds the lease. If