Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 4 slice 2: the queue and the runner

b44cbfaf873fda01a458b527e5684b2386266117 · Keenan Nemetz · 18d ago · parent 8fe238d4

Phase 4 slice 2: the queue and the runner

A push now runs a pipeline. post-receive spools one file per accepted ref update —
the hook must never write to a database or fail a commit — and the server ingests
those into data/runs.db, pinning each run to the commit and to the workflow blob it
was queued from. The unique index on (repo, ref, sha, workflow, job) is what makes
ingestion idempotent, so a spool file read twice after a crash does not run twice.

vole runner connects over SSH with the key it would push with: no token, no stored
secret, and the same identity the rest of the system uses. It may only run jobs for
repositories where its account is an admin, because running a job executes that
repository's code — a bigger power than writing to it.

Delivery is at-least-once by construction:

- A lease hands out a job for a fixed window; the runner heartbeats while it works.
- An expired lease means the runner died, and the janitor returns the job to the
  queue for somebody else.
- A runner whose lease is gone stops rather than reporting a result nobody will
  record, and a job that kills five runners in a row is failed instead of cycling.

Three bugs found while building it, each worth the test that now covers it:

- Background loops lived in Run(), so the tests — which serve the SSH listener
  themselves — never had ingestion or lease expiry. They start in New() now.
- post-receive derived the repository name from GIT_DIR, which git sets to "." for
  hooks, so every event was queued for a repository called ".". The same trap had
  already appeared in a pre-receive message; both now resolve the absolute path.
- Leases ordered only by creation time, so two pushes in the same second could be
  handed out in any order; the id breaks the tie, and ULIDs sort by time.

Verified end to end on a live instance: a push queues a job, a runner leases it,
clones the commit, prints "hi", and the run closes success with the log attached.
The integration tests cover the two exit criteria that matter — a push runs a
pipeline, and killing a runner mid-job requeues the work onto a second runner
(attempts=2, success) — plus the property that editing a workflow after a push does
not change what the already-queued run executes.

Not built yet: statuses on change requests and required-check gating (slice 3), and
signed run records (Phase 7). Runs live in data/runs.db and are visible there, not
on the web.

internal/config/config.go +21−2

@@ -5,6 +5,7 @@ import (
5 5 "fmt"
6 6 "os"
7 7 "path/filepath"
8 "time"
8 9
9 10 "github.com/BurntSushi/toml"
10 11 )
@@ -20,6 +21,9 @@ type Config struct {
20 21 // ReceiveMaxBytes caps a whole push, enforced by git's own
21 22 // receive.maxInputSize in each repo rather than by buffering in the proxy.
22 23 ReceiveMaxBytes int64 `toml:"receive_max_bytes"`
24 // LeaseSeconds is how long a runner may hold a job without a heartbeat before
25 // the job is handed to somebody else.
26 LeaseSeconds int `toml:"lease_seconds"`
23 27
24 28 Path string `toml:"-"`
25 29 }
@@ -58,8 +62,23 @@ func (c *Config) ReposDir() string { return filepath.Join(c.DataDir, "repos") }
58 62 func (c *Config) SystemRepo() string { return filepath.Join(c.DataDir, "system.git") }
59 63
60 64 // AuditLog records privileged operations. Roster history also lives in git.
61 func (c *Config) AuditLog() string { return filepath.Join(c.DataDir, "audit.log") }
62 func (c *Config) IndexDB() string { return filepath.Join(c.DataDir, "index.db") }
65 func (c *Config) AuditLog() string { return filepath.Join(c.DataDir, "audit.log") }
66 func (c *Config) IndexDB() string { return filepath.Join(c.DataDir, "index.db") }
67
68 // RunsDB holds the CI queue and finished runs. Unlike index.db it is not derived:
69 // a run happened or it did not.
70 func (c *Config) RunsDB() string { return filepath.Join(c.DataDir, "runs.db") }
71
72 // SpoolDir receives accepted push events from post-receive hooks.
73 func (c *Config) SpoolDir() string { return filepath.Join(c.DataDir, "queue", "spool") }
74
75 // Lease returns how long a runner may hold a job without a heartbeat.
76 func (c *Config) Lease() time.Duration {
77 if c.LeaseSeconds <= 0 {
78 return 60 * time.Second
79 }
80 return time.Duration(c.LeaseSeconds) * time.Second
81 }
63 82 func (c *Config) InstanceKey() string { return filepath.Join(c.DataDir, "instance.key") }
64 83 func (c *Config) SSHHostKey() string { return filepath.Join(c.DataDir, "ssh_host_ed25519_key") }
65 84