Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Run several jobs at once with --jobs

18c559ea0010dc22db13c889afda4fe9af3c15d7 · Keenan Nemetz · 2d ago · parent 91b47a22

Run several jobs at once with --jobs

A runner process leased one job and blocked on it, so the only way to get a
second job onto a machine was a second process: another host, another key,
another allowlist entry, and another --dir that learns the repositories cold.
The server was never the limit — a lease is taken per session and claimed by a
conditional update — so this is the runner catching up.

`--jobs N` is N slots in one process. Each slot owns a session and runs at most
one job, and everything that belongs to the process happens once, before the
slots start: the directories, the cache bounds, the leftover sweep, and the
connections. A slot that refuses a job stops every slot, because reconnecting
would ask the same instance the same question. A slot that loses its connection
reconnects with a doubling delay instead, so one flaky link does not take the
other slots' jobs with it; the job it was running is requeued when its lease
expires.

The log stream moves from the session to the job, which is what makes two jobs
on one session impossible to confuse: a slot has one, and it is the stream for
the job in front of it.

Deployment: `RUNNER_JOBS` in the first-boot renderer, the packaged argument
file, and the docs. The container limits are per job, so the machine has to
carry --jobs times what they allow; max_jobs_per_repo is what stops one
repository claiming every slot.

Refs: 01M6SCWCSX39QXS38QAHQSMVGX (#216)

server/ssh/lease_wait_test.go +64

@@ -6,6 +6,7 @@ package ssh
6 6 // what is measured here is latency, not permission.
7 7
8 8 import (
9 "fmt"
9 10 "strings"
10 11 "testing"
11 12 "time"
@@ -140,3 +141,66 @@ func TestALeaseWithoutAWaitAnswersAtOnce(t *testing.T) {
140 141 t.Fatalf("a lease with no wait blocked for %s", waited)
141 142 }
142 143 }
144
145 // A runner with --jobs N is N sessions, and the queue has to hand each one a
146 // different job: two sessions that lease the same run would run it twice and race
147 // each other to report it.
148 func TestSeveralSessionsLeaseDifferentJobs(t *testing.T) {
149 t.Parallel()
150 h := newHarness(t)
151 const slots = 4
152 // A distinct job name per run: enqueue is idempotent per (repo, ref, sha,
153 // workflow, job), so four identical enqueues are one run.
154 for i := 0; i < slots; i++ {
155 h.enqueueJob(t, fmt.Sprintf("slot-%d", i))
156 }
157
158 seen := make(map[string]bool, slots)
159 for i := 0; i < slots; i++ {
160 c := h.runner(t, h.keyPath)
161 rep := c.call(runnerRequest{Op: "lease"})
162 if rep.Run == nil {
163 t.Fatalf("slot %d leased nothing: %+v", i, rep)
164 }
165 if seen[rep.Run.ID] {
166 t.Fatalf("two sessions leased the same run %s", rep.Run.ID)
167 }
168 seen[rep.Run.ID] = true
169 }
170 if len(seen) != slots {
171 t.Fatalf("%d distinct runs leased across %d sessions", len(seen), slots)
172 }
173
174 // One job, one attempt: each session's lease is the attempt it reports under, and
175 // they cannot share one.
176 for id := range seen {
177 run, err := h.runs.Get(id)
178 if err != nil || run == nil {
179 t.Fatalf("leased run %s is not in the queue: %v", id, err)
180 }
181 if run.Attempts != 1 {
182 t.Fatalf("run %s is on attempt %d after one lease", id, run.Attempts)
183 }
184 }
185 }
186
187 // max_jobs_per_repo is what stops one repository's burst from taking every slot on
188 // the instance. A repository at its cap is skipped rather than blocking the lease.
189 func TestARepositoryAtItsCapLeavesTheSlotsForOthers(t *testing.T) {
190 t.Parallel()
191 h := newHarnessWith(t, func(c *Config) { c.MaxJobsPerRepo = 2 })
192 for i := 0; i < 3; i++ {
193 h.enqueueJob(t, fmt.Sprintf("burst-%d", i))
194 }
195
196 leased := 0
197 for i := 0; i < 3; i++ {
198 c := h.runner(t, h.keyPath)
199 if c.call(runnerRequest{Op: "lease"}).Run != nil {
200 leased++
201 }
202 }
203 if leased != 2 {
204 t.Fatalf("%d job(s) leased from a repository capped at 2", leased)
205 }
206 }