Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 4: an idle runner parks on the queue instead of polling it

34afc5691bf2d6c808f905935cc95ffba34f09a7 · Keenan Nemetz · 16d ago · parent a945ae55 

Phase 4: an idle runner parks on the queue instead of polling it

Two costs made a runner pool more expensive than it needed to be. Every lease
request re-derived the account's eligible repositories by opening every repository
and reading its rules — O(repos) per idle runner, every two seconds, to be told
there was nothing to do. And the "long poll" was the client's own timer: the queue
answered at once and the runner slept.

The lease now takes a `wait`: the session holds the request on a wake-up channel
that a committed enqueue closes (and RequeueExpired closes when it puts a job back),
wakes and asks once more, because the wake-up may belong to a job another runner
took first. The channel is taken *before* the lease attempt, so a run queued between
the two cannot be missed. It is in-process on purpose: one `vole serve` owns this
database, and the wait is a courtesy, never a durability mechanism — a lost wake-up
costs one poll interval and nothing else.

The eligibility list is now reused for the roster's five-second window instead of
being re-derived per request, so an allowlist change still takes effect without a
reconnect but a quiet pool makes no queries at all. `--once` sends no wait: a
one-shot runner is here for the job it was started for.

Measured: TestALeaseWaitsForWorkInsteadOfPolling parks a lease for 30s and gets the
job inside 5s of it being queued; TestALeaseWithoutAWaitAnswersAtOnce returns from an
empty queue immediately; TestQueuingWorkWakesTheParked and
TestADuplicateEnqueueDoesNotWakeTheParked pin that a queued run wakes waiters and a
duplicate does not.

Also corrected: `vole runner --help` claimed jobs come from repositories the account
administers, which stopped being true when `acl runners` became the allowlist.

internal/sshserver/runner.go +53−8

@@ -33,9 +33,23 @@ import (
33 33 // "ok":false means the lease is gone — another runner has the job — and the runner
34 34 // must stop rather than report a result nobody will record.
35 35
36 const (
37 // eligibleFor is how long a session reuses its repository list. Access rules live
38 // in the instance database and only change when an admin edits them, so a runner
39 // re-derives its list on the roster's five-second beat instead of on every poll.
40 eligibleFor = 5 * time.Second
41 // maxLeaseWait caps what a runner may ask for. The wait is a hint — the lease
42 // remains the database's decision — so all a long wait holds is one SSH channel.
43 maxLeaseWait = 60 * time.Second
44 )
45
36 46 type runnerRequest struct {
37 47 Op string `json:"op"`
38 48 Run string `json:"run"`
49 // Wait is how many seconds an idle lease may park before answering "nothing to
50 // do". Zero answers at once, which is what a one-shot runner wants: it is here to
51 // run the job it was started for, not to wait for the next one.
52 Wait int `json:"wait,omitempty"`
39 53 }
40 54
41 55 type runnerStep struct {
@@ -108,6 +122,12 @@ func (s *Server) serveRunner(ch ssh.Channel, user, accountID string) {
108 122 dec := json.NewDecoder(ch)
109 123 enc := json.NewEncoder(ch)
110 124 owner := accountID
125 // repos is this session's eligibility list, reused for one poll window. Reading
126 // every repository's rules on every lease request is O(repos) per idle runner, and
127 // the answer changes only when an admin edits an allowlist — so it is refreshed on
128 // the same five-second beat the roster is, and not once per request.
129 var repos []string
130 var reposAt time.Time
111 131 cfg.Log.Info("runner connected", "user", user)
112 132
113 133 for {
@@ -117,19 +137,44 @@ func (s *Server) serveRunner(ch ssh.Channel, user, accountID string) {
117 137 }
118 138 switch req.Op {
119 139 case "lease":
120 // A runner may only run jobs for repositories it administers: running a
121 // job executes that repository's code, which is a bigger power than
122 // writing to it.
123 repos, err := s.runnableRepos(accountID)
124 if err != nil {
125 _ = enc.Encode(runnerReply{Op: "lease", Error: err.Error()})
126 continue
140 now := time.Now().UTC()
141 // A runner may only run jobs for repositories whose runner allowlist it is
142 // on: running a job executes that repository's code, which is a bigger
143 // power than writing to it.
144 if reposAt.IsZero() || now.Sub(reposAt) > eligibleFor {
145 list, err := s.runnableRepos(accountID)
146 if err != nil {
147 _ = enc.Encode(runnerReply{Op: "lease", Error: err.Error()})
148 continue
149 }
150 repos, reposAt = list, now
127 151 }
128 run, err := cfg.Runs.Lease(owner, repos, cfg.Lease, time.Now().UTC())
152 // Take the wake-up before asking, so a run queued between the lease and the
153 // wait closes a channel somebody is holding. Asking first and waiting second
154 // loses exactly that race.
155 wake := cfg.Runs.Work()
156 run, err := cfg.Runs.Lease(owner, repos, cfg.Lease, now)
129 157 if err != nil {
130 158 _ = enc.Encode(runnerReply{Op: "lease", Error: err.Error()})
131 159 continue
132 160 }
161 if run == nil && req.Wait > 0 {
162 wait := time.Duration(req.Wait) * time.Second
163 if wait > maxLeaseWait {
164 wait = maxLeaseWait
165 }
166 select {
167 case <-wake:
168 case <-time.After(wait):
169 }
170 // Ask again rather than assume: the wake-up may belong to a job another
171 // runner took first, and a lease is the only thing that hands work out.
172 run, err = cfg.Runs.Lease(owner, repos, cfg.Lease, time.Now().UTC())
173 if err != nil {
174 _ = enc.Encode(runnerReply{Op: "lease", Error: err.Error()})
175 continue
176 }
177 }
133 178 if run == nil {
134 179 _ = enc.Encode(runnerReply{Op: "lease", OK: true})
135 180 continue