Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 4: a new tip cancels the old build, and the lease serves the quiet repo

760b523a7dbb6fc1c1210a16bf44232a7c592b1b · Keenan Nemetz · 16d ago · parent 34afc569

Phase 4: a new tip cancels the old build, and the lease serves the quiet repo

Two ways a single repository could monopolise the instance's runner pool. A push
series queued a build per commit, so ten pushes in a minute ran ten builds of nine
dead commits — `StatusCanceled` existed, the gate knew how to report it, and nothing
ever wrote it. And the lease ordered by age across every eligible repository, so the
oldest job won wherever it lived: one busy repository's queue was served before a
quiet repository's, no matter how many runners were busy with the first.

Cancellation is now decided where the workflow's own trigger is decided: a push that
owes at least one run retires the *queued* runs of that repo+ref whose sha it
replaces, and nothing else — a running job is never interrupted, since a build
canceled mid-flight is a different promise from one nobody started, and a push that
queues nothing (no matching workflow) cancels nothing. The count is logged, because a
run that stops for no findable reason reads exactly like a run that vanished.

The lease orders by in-flight count per repository first and age within a repository
second, which needs no new config key and no operator tuning: the burst still runs,
it just cannot occupy every runner while another repository waits.

Measured: TestCancelSupersededOnlyTouchesWaitingRunsOfTheSameRef (1 canceled, another
ref untouched, the running job still running), TestLeaseServesTheLeastBusyRepoFirst
(the busy repository's older second job is passed over for the quiet repository), and
TestANewerTipCancelsTheQueuedBuildOfTheOlderOne end to end through the reconciler.

Canceled runs are queue state, not history — they never happened, so they have no
record and `vole index rebuild` drops them rather than inventing one.

internal/runs/runs.go +29−10

@@ -268,6 +268,21 @@ func (s *Store) Enqueue(r Run) (bool, error) {
268 268 return n > 0, nil
269 269 }
270 270
271 // CancelSuperseded retires the queued runs of a ref that a newer tip replaced. Only
272 // waiting rows are touched: a job that is already running is not interrupted, because
273 // a canceled build is a different promise from a build nobody started. The caller
274 // decides when a push supersedes anything at all.
275 func (s *Store) CancelSuperseded(repo, ref, keep string, now time.Time) (int, error) {
276 res, err := s.db.Exec(`UPDATE runs SET status = ?, finished = ?, lease_owner = '', lease_until = 0
277 WHERE repo = ? AND ref = ? AND sha <> ? AND status = ?`,
278 StatusCanceled, now.Unix(), repo, ref, keep, StatusQueued)
279 if err != nil {
280 return 0, err
281 }
282 n, err := res.RowsAffected()
283 return int(n), err
284 }
285
271 286 // Cursors returns the tip each of a repository's refs had when its work was last
272 287 // queued. A ref that is missing has never been reconciled, which is treated as work
273 288 // owed rather than as "nothing to do": the refs are the only durable statement that
@@ -329,21 +344,25 @@ func (s *Store) Lease(owner string, repos []string, ttl time.Duration, now time.
329 344 return nil, nil
330 345 }
331 346 placeholders := strings.TrimSuffix(strings.Repeat("?,", len(repos)), ",")
332 args := make([]any, 0, len(repos)+2)
333 for _, r := range repos {
334 args = append(args, r)
335 }
336 args = append(args, now.Unix())
337
338 347 // Two runners may pick the same candidate; the conditional update is what
339 348 // decides, so the loser simply retries against the next candidate.
340 349 for attempt := 0; attempt < 5; attempt++ {
341 350 var id string
342 // id breaks ties: ULIDs sort by creation, so two pushes in the same second
351 // Least busy repository first, then oldest run within it. Ordering by age alone
352 // lets one repository's push burst occupy every runner on the instance; ordering
353 // by in-flight count means a quiet repository is served while the busy one
354 // works. id breaks ties: ULIDs sort by creation, so two pushes in the same second
343 355 // are still handled oldest first.
344 q := `SELECT id FROM runs WHERE status = ? AND repo IN (` + placeholders + `)
345 ORDER BY created, id LIMIT 1`
346 sel := append([]any{StatusQueued}, args...)
356 q := `SELECT r.id FROM runs r
357 LEFT JOIN (SELECT repo, COUNT(*) AS n FROM runs WHERE status = ? GROUP BY repo) busy
358 ON busy.repo = r.repo
359 WHERE r.status = ? AND r.repo IN (` + placeholders + `)
360 ORDER BY COALESCE(busy.n, 0), r.created, r.id LIMIT 1`
361 sel := make([]any, 0, len(repos)+2)
362 sel = append(sel, StatusRunning, StatusQueued)
363 for _, r := range repos {
364 sel = append(sel, r)
365 }
347 366 if err := s.db.QueryRow(q, sel...).Scan(&id); err != nil {
348 367 if err == sql.ErrNoRows {
349 368 return nil, nil