Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 4 slice 2: the queue and the runner

b44cbfaf873fda01a458b527e5684b2386266117 · Keenan Nemetz · 17d ago · parent 8fe238d4

Phase 4 slice 2: the queue and the runner

A push now runs a pipeline. post-receive spools one file per accepted ref update —
the hook must never write to a database or fail a commit — and the server ingests
those into data/runs.db, pinning each run to the commit and to the workflow blob it
was queued from. The unique index on (repo, ref, sha, workflow, job) is what makes
ingestion idempotent, so a spool file read twice after a crash does not run twice.

vole runner connects over SSH with the key it would push with: no token, no stored
secret, and the same identity the rest of the system uses. It may only run jobs for
repositories where its account is an admin, because running a job executes that
repository's code — a bigger power than writing to it.

Delivery is at-least-once by construction:

- A lease hands out a job for a fixed window; the runner heartbeats while it works.
- An expired lease means the runner died, and the janitor returns the job to the
  queue for somebody else.
- A runner whose lease is gone stops rather than reporting a result nobody will
  record, and a job that kills five runners in a row is failed instead of cycling.

Three bugs found while building it, each worth the test that now covers it:

- Background loops lived in Run(), so the tests — which serve the SSH listener
  themselves — never had ingestion or lease expiry. They start in New() now.
- post-receive derived the repository name from GIT_DIR, which git sets to "." for
  hooks, so every event was queued for a repository called ".". The same trap had
  already appeared in a pre-receive message; both now resolve the absolute path.
- Leases ordered only by creation time, so two pushes in the same second could be
  handed out in any order; the id breaks the tie, and ULIDs sort by time.

Verified end to end on a live instance: a push queues a job, a runner leases it,
clones the commit, prints "hi", and the run closes success with the log attached.
The integration tests cover the two exit criteria that matter — a push runs a
pipeline, and killing a runner mid-job requeues the work onto a second runner
(attempts=2, success) — plus the property that editing a workflow after a push does
not change what the already-queued run executes.

Not built yet: statuses on change requests and required-check gating (slice 3), and
signed run records (Phase 7). Runs live in data/runs.db and are visible there, not
on the web.

internal/sshserver/sshserver.go +17

@@ -28,6 +28,7 @@ import (
28 28
29 29 "github.com/rodent-software/vole/internal/identity"
30 30 "github.com/rodent-software/vole/internal/repo"
31 "github.com/rodent-software/vole/internal/runs"
31 32 )
32 33
33 34 const serverVersion = "SSH-2.0-vole"
@@ -40,6 +41,12 @@ type Config struct {
40 41 MaxBlobBytes int64
41 42 // AuditPath is handed to hooks so privileged operations can be recorded.
42 43 AuditPath string
44 // SpoolDir is where post-receive drops accepted push events.
45 SpoolDir string
46 // Runs is the CI queue, served to runners over the vole-runner verb.
47 Runs *runs.Store
48 // Lease is how long a runner may hold a job without a heartbeat.
49 Lease time.Duration
43 50 }
44 51
45 52 type Server struct {
@@ -182,6 +189,15 @@ func (s *Server) exec(ch ssh.Channel, req *ssh.Request, command, user, accountID
182 189 refuse("malformed command")
183 190 return
184 191 }
192 // The CI runner is the one non-git verb. It reuses the same key identity rather
193 // than introducing a token, and it is a stream: the runner leases jobs and
194 // reports results over this channel.
195 if len(argv) == 1 && argv[0] == "vole-runner" {
196 req.Reply(true, nil)
197 s.serveRunner(ch, user, accountID)
198 exit(ch, 0)
199 return
200 }
185 201 if len(argv) != 2 {
186 202 refuse("only git-upload-pack and git-receive-pack are available")
187 203 return
@@ -251,6 +267,7 @@ func (s *Server) exec(ch ssh.Channel, req *ssh.Request, command, user, accountID
251 267 "VOLE_MAX_BLOB="+strconv.FormatInt(s.cfg.MaxBlobBytes, 10),
252 268 "VOLE_AUDIT="+s.cfg.AuditPath,
253 269 "VOLE_SYSTEM_REPO="+s.cfg.Identity.Dir(),
270 "VOLE_SPOOL="+s.cfg.SpoolDir,
254 271 )
255 272 cmd.Stdin = struct{ io.Reader }{ch}
256 273 cmd.Stdout = struct{ io.Writer }{ch}