Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 4 slice 2: the queue and the runner

b44cbfaf873fda01a458b527e5684b2386266117 · Keenan Nemetz · 17d ago · parent 8fe238d4

Phase 4 slice 2: the queue and the runner

A push now runs a pipeline. post-receive spools one file per accepted ref update —
the hook must never write to a database or fail a commit — and the server ingests
those into data/runs.db, pinning each run to the commit and to the workflow blob it
was queued from. The unique index on (repo, ref, sha, workflow, job) is what makes
ingestion idempotent, so a spool file read twice after a crash does not run twice.

vole runner connects over SSH with the key it would push with: no token, no stored
secret, and the same identity the rest of the system uses. It may only run jobs for
repositories where its account is an admin, because running a job executes that
repository's code — a bigger power than writing to it.

Delivery is at-least-once by construction:

- A lease hands out a job for a fixed window; the runner heartbeats while it works.
- An expired lease means the runner died, and the janitor returns the job to the
  queue for somebody else.
- A runner whose lease is gone stops rather than reporting a result nobody will
  record, and a job that kills five runners in a row is failed instead of cycling.

Three bugs found while building it, each worth the test that now covers it:

- Background loops lived in Run(), so the tests — which serve the SSH listener
  themselves — never had ingestion or lease expiry. They start in New() now.
- post-receive derived the repository name from GIT_DIR, which git sets to "." for
  hooks, so every event was queued for a repository called ".". The same trap had
  already appeared in a pre-receive message; both now resolve the absolute path.
- Leases ordered only by creation time, so two pushes in the same second could be
  handed out in any order; the id breaks the tie, and ULIDs sort by time.

Verified end to end on a live instance: a push queues a job, a runner leases it,
clones the commit, prints "hi", and the run closes success with the log attached.
The integration tests cover the two exit criteria that matter — a push runs a
pipeline, and killing a runner mid-job requeues the work onto a second runner
(attempts=2, success) — plus the property that editing a workflow after a push does
not change what the already-queued run executes.

Not built yet: statuses on change requests and required-check gating (slice 3), and
signed run records (Phase 7). Runs live in data/runs.db and are visible there, not
on the web.

cmd/vole/main.go +17−2

@@ -75,6 +75,7 @@ Usage:
75 75 vole request update <id> --head <rev>
76 76 vole request approve <id> [--as <user>]
77 77 vole request close|reopen <id>
78 vole runner --remote ssh://git@host:port
78 79 vole workflow check [--dir path]
79 80 vole workflow list [--dir path]
80 81
@@ -110,6 +111,8 @@ func run(args []string) error {
110 111 return cmdRequest(args[1:])
111 112 case "workflow":
112 113 return cmdWorkflow(args[1:])
114 case "runner":
115 return cmdRunner(args[1:])
113 116 case "hook":
114 117 return cmdHook(args[1:])
115 118 case "version", "-v", "--version":
@@ -611,13 +614,25 @@ func cmdIndex(args []string) error {
611 614 // cmdHook runs inside a git push. It must never be optimistic: anything it
612 615 // cannot verify is a rejection.
613 616 func cmdHook(args []string) error {
614 if len(args) == 0 || args[0] != "pre-receive" {
615 return errors.New("usage: vole hook pre-receive")
617 if len(args) == 0 {
618 return errors.New("usage: vole hook pre-receive|post-receive")
616 619 }
617 620 dir := os.Getenv("GIT_DIR")
618 621 if dir == "" {
619 622 dir = "."
620 623 }
624 if args[0] == "post-receive" {
625 hook.PostReceive(hook.Config{
626 RepoDir: dir,
627 SpoolDir: os.Getenv("VOLE_SPOOL"),
628 In: os.Stdin,
629 Out: os.Stderr,
630 })
631 return nil
632 }
633 if args[0] != "pre-receive" {
634 return usagef("unknown hook %q", args[0])
635 }
621 636 maxBlob := int64(0)
622 637 if v := os.Getenv("VOLE_MAX_BLOB"); v != "" {
623 638 if n, err := strconv.ParseInt(v, 10, 64); err == nil {