vole
Clone
Read only, no credentials:
https://rodent.software/vole.git
Write over SSH with a registered key:
git clone ssh://git@rodent.software:2222/vole.git
A free and open-source forge that lives in your Git repo.
Stop a running job at its runner's next heartbeat
0c14f0789af1ab0c43f99043acd7507eec4956e4 · Keenan Nemetz · 6d ago · parent 7e0b5950
Stop a running job at its runner's next heartbeat A maintainer could cancel a queued run and nothing else: CancelRun only retired waiting work, so a build that was clearly wrong or expensive held its runner until its budget ran out, or until somebody killed the container by hand. The queue cannot kill a process it does not own, but it can ask. Canceling a running run now marks the queue row and leaves the lease alone, because the runner still holds it and its partial attempt has to be acceptable. The runner sees the request on the heartbeat it already sends while a step or an image build runs, stops the container, writes one line to the log, and reports what it did. RecordAttempt and Finish then keep the canceled status instead of mapping the nonzero exit code to failed, so the partial log and step timings survive and the run is not a red build. A stop request whose runner dies first becomes canceled on the expiry sweep rather than being leased again, and a rerun clears the request: it is a new decision. The run page says who asked, distinguishes a run stopped while running from one canceled in the queue, and shows the interrupted step as stopped rather than failed. Closes #162.
server/ci.go +4−4
| @@ -456,14 +456,14 @@ func (s *Server) watchLeases(ctx context.Context) { | |||
| 456 | 456 | case <-ctx.Done(): | |
| 457 | 457 | return | |
| 458 | 458 | case <-t.C: | |
| 459 | back, failed, err := s.Runs.RequeueExpired(time.Now().UTC()) | ||
| 459 | back, stopped, failed, err := s.Runs.RequeueExpired(time.Now().UTC()) | ||
| 460 | 460 | if err != nil { | |
| 461 | 461 | s.Log.Warn("ci: cannot requeue expired leases", "err", err) | |
| 462 | 462 | continue | |
| 463 | 463 | } | |
| 464 | if back > 0 || failed > 0 { | ||
| 465 | s.Log.Warn(fmt.Sprintf("ci: %d job(s) returned to the queue, %d failed after %d attempts", | ||
| 466 | back, failed, runs.MaxAttempts)) | ||
| 464 | if back > 0 || stopped > 0 || failed > 0 { | ||
| 465 | s.Log.Warn(fmt.Sprintf("ci: %d job(s) returned to the queue, %d canceled while stopped, %d failed after %d attempts", | ||
| 466 | back, stopped, failed, runs.MaxAttempts)) | ||
| 467 | 467 | } | |
| 468 | 468 | // A run whose runner disappeared never sends a finish, so the key it was | |
| 469 | 469 | // given is ended by its expiry. Sweeping here keeps the rows bounded and | |