Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 4 slice 2: the queue and the runner

b44cbfaf873fda01a458b527e5684b2386266117 · Keenan Nemetz · 17d ago · parent 8fe238d4

Phase 4 slice 2: the queue and the runner

A push now runs a pipeline. post-receive spools one file per accepted ref update —
the hook must never write to a database or fail a commit — and the server ingests
those into data/runs.db, pinning each run to the commit and to the workflow blob it
was queued from. The unique index on (repo, ref, sha, workflow, job) is what makes
ingestion idempotent, so a spool file read twice after a crash does not run twice.

vole runner connects over SSH with the key it would push with: no token, no stored
secret, and the same identity the rest of the system uses. It may only run jobs for
repositories where its account is an admin, because running a job executes that
repository's code — a bigger power than writing to it.

Delivery is at-least-once by construction:

- A lease hands out a job for a fixed window; the runner heartbeats while it works.
- An expired lease means the runner died, and the janitor returns the job to the
  queue for somebody else.
- A runner whose lease is gone stops rather than reporting a result nobody will
  record, and a job that kills five runners in a row is failed instead of cycling.

Three bugs found while building it, each worth the test that now covers it:

- Background loops lived in Run(), so the tests — which serve the SSH listener
  themselves — never had ingestion or lease expiry. They start in New() now.
- post-receive derived the repository name from GIT_DIR, which git sets to "." for
  hooks, so every event was queued for a repository called ".". The same trap had
  already appeared in a pre-receive message; both now resolve the absolute path.
- Leases ordered only by creation time, so two pushes in the same second could be
  handed out in any order; the id breaks the tie, and ULIDs sort by time.

Verified end to end on a live instance: a push queues a job, a runner leases it,
clones the commit, prints "hi", and the run closes success with the log attached.
The integration tests cover the two exit criteria that matter — a push runs a
pipeline, and killing a runner mid-job requeues the work onto a second runner
(attempts=2, success) — plus the property that editing a workflow after a push does
not change what the already-queued run executes.

Not built yet: statuses on change requests and required-check gating (slice 3), and
signed run records (Phase 7). Runs live in data/runs.db and are visible there, not
on the web.

internal/workflow/workflow.go +34−1

@@ -55,7 +55,10 @@ type Job struct {
55 55 }
56 56
57 57 type Workflow struct {
58 Path string
58 Path string
59 // BlobSHA identifies the exact file contents queued for a run, so a workflow
60 // edited after enqueue cannot change what an already-queued job executes.
61 BlobSHA string
59 62 Version int
60 63 Branches []string
61 64 Jobs []Job
@@ -257,6 +260,36 @@ func firstLine(s string) string {
257 260 return s
258 261 }
259 262
263 // At loads and validates the workflows in a git tree, with the blob id of each
264 // file so a run can be pinned to the version it was queued from.
265 func At(g *gitx.Repo, commit string) ([]*Workflow, []string) {
266 if commit == "" || commit == gitx.ZeroSHA {
267 return nil, nil
268 }
269 blobs, err := g.LsTreeR(commit)
270 if err != nil {
271 return nil, []string{fmt.Sprintf("cannot read %s: %v", Dir, err)}
272 }
273 files := map[string]string{}
274 shas := map[string]string{}
275 for _, e := range blobs {
276 if !strings.HasPrefix(e.Name, Dir+"/") {
277 continue
278 }
279 data, err := g.BlobBySHA(e.SHA)
280 if err != nil {
281 return nil, []string{fmt.Sprintf("cannot read %s: %v", e.Name, err)}
282 }
283 files[e.Name] = string(data)
284 shas[e.Name] = e.SHA
285 }
286 wfs, problems := Check(files)
287 for _, wf := range wfs {
288 wf.BlobSHA = shas[wf.Path]
289 }
290 return wfs, problems
291 }
292
260 293 // ValidateTree checks the workflows in a git tree. Called from pre-receive, so a
261 294 // bad workflow fails the push that introduced it.
262 295 func ValidateTree(g *gitx.Repo, commit string) []string {