Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Stop a running job at its runner's next heartbeat

0c14f0789af1ab0c43f99043acd7507eec4956e4 · Keenan Nemetz · 6d ago · parent 7e0b5950

Stop a running job at its runner's next heartbeat

A maintainer could cancel a queued run and nothing else: CancelRun only retired
waiting work, so a build that was clearly wrong or expensive held its runner
until its budget ran out, or until somebody killed the container by hand. The
queue cannot kill a process it does not own, but it can ask.

Canceling a running run now marks the queue row and leaves the lease alone,
because the runner still holds it and its partial attempt has to be acceptable.
The runner sees the request on the heartbeat it already sends while a step or an
image build runs, stops the container, writes one line to the log, and reports
what it did. RecordAttempt and Finish then keep the canceled status instead of
mapping the nonzero exit code to failed, so the partial log and step timings
survive and the run is not a red build.

A stop request whose runner dies first becomes canceled on the expiry sweep
rather than being leased again, and a rerun clears the request: it is a new
decision. The run page says who asked, distinguishes a run stopped while running
from one canceled in the queue, and shows the interrupted step as stopped rather
than failed.

Closes #162.

server/ssh/runner.go +10−3

@@ -243,6 +243,10 @@ type runnerReply struct {
243 243 Repos []string `json:"repos,omitempty"`
244 244 // LogSize is how much of the log the server has after a log chunk.
245 245 LogSize int64 `json:"log_size,omitempty"`
246 // Cancel names the account that asked this running job to stop. It rides the
247 // heartbeat reply, so stopping a run needs no new traffic and a runner too old to
248 // know the field ignores it. Empty means nobody has asked.
249 Cancel string `json:"cancel,omitempty"`
246 250 }
247 251
248 252 // serveRunner answers one runner session until it disconnects.
@@ -437,14 +441,17 @@ func (s *Server) serveRunner(ch xssh.Channel, user string) {
437 441
438 442 case "heartbeat":
439 443 // Liveness only: the lease is what a heartbeat buys. Presence is refreshed
440 // here too, so a long build does not make its runner look gone.
444 // here too, so a long build does not make its runner look gone. A stop request
445 // rides the same reply rather than a new message: the runner already asks this
446 // question every few seconds, and a runner that does not know the field keeps
447 // working exactly as before.
441 448 _ = cfg.Runs.TouchSession(sessionID, user, nil, time.Now().UTC())
442 ok, err := cfg.Runs.Heartbeat(req.Run, owner, cfg.Lease, time.Now().UTC())
449 held, cancelBy, err := cfg.Runs.Heartbeat(req.Run, owner, cfg.Lease, time.Now().UTC())
443 450 if err != nil {
444 451 _ = enc.Encode(runnerReply{Op: "heartbeat", Error: err.Error()})
445 452 continue
446 453 }
447 _ = enc.Encode(runnerReply{Op: "heartbeat", OK: ok})
454 _ = enc.Encode(runnerReply{Op: "heartbeat", OK: held, Cancel: cancelBy})
448 455
449 456 case "finish":
450 457 // The result arrives here, not on the metadata branch. The record is