vole
Clone
Read only, no credentials:
https://rodent.software/git/vole.git
Write over SSH with a registered key:
git clone ssh://git@0.0.0.0:2222/vole.git
A free and open-source forge that lives in your Git repo.
Phase 4: record the queue's new shape, and let the pins be tested while running
d798eaacf9b210fcdc2b51372d64a0e21e239bab · Keenan Nemetz · 16d ago · parent ac433531
Phase 4: record the queue's new shape, and let the pins be tested while running Docs, plus the two tests whose setup the new cancellation invalidates. docs/roadmap.md gains slices 7 and 8, rewrites the Phase 4 remaining list (the push-event table is gone by decision; what is left is the toolchain in the record, a narrowing/targeting filter for a mixed fleet, and a per-repository cap somebody would have to tune), adds an exit criterion for "a push whose spool file was never written still runs", and appends four decision-log rows: reconcile from the refs rather than persist events, park an idle lease on a wake-up channel, cancel queued superseded runs and order the lease by in-flight count, and keep runner presence in the queue filtered by the clone rule. README describes the new queue behaviour and /runners, and its stale "instance ed25519 key generated at first boot" line is replaced: the instance holds no signing key. Two tests built their scenario on a superseded run being started — queue v1, push v2, then run both. That is exactly what the queue no longer does, so both now hold the first build inside its step with a sentinel file, land the second push while it is running, and release it. The properties are unchanged and now tested where they actually apply: a build that is running when the branch moves must not publish a site (TestWorkflowRefusesAStaleSiteDeploy), and a run executes the steps it was queued with, not the workflow as the branch has it now (TestQueuedRunUsesTheWorkflowItWasQueuedFrom). A run that is still waiting when the new tip arrives is canceled, which TestANewerTipCancelsTheQueuedBuildOfTheOlderOne covers. scripts/ci.sh green before these comment-and-doc edits (internal/server 132.9s, everything else included); gofmt, vet, build and the markdown link check green after.
docs/roadmap.md +65−2
| @@ -443,18 +443,77 @@ commit while an unsigned one was refused; a writer who is not a runner was refus | |||
| 443 | 443 | `pre-receive` naming the allowlist; `runs.db` deleted, `vole index rebuild` restored | |
| 444 | 444 | the row, the attestation and the log object id.) | |
| 445 | 445 | ||
| 446 | ✅ **Slice 7 — the queue follows the refs, and a lease waits instead of polls.** The | ||
| 447 | push path used to be one statement: a spool file the hook writes after the refs move, | ||
| 448 | which it is allowed to fail without failing the push. A crash or a full disk in that | ||
| 449 | window left a live commit with no build and nothing that would ever notice, and | ||
| 450 | `rm data/runs.db` lost every queued job while this package's own header claimed only | ||
| 451 | in-flight work was lost. Both are now the same question with the same answer: a ref tip | ||
| 452 | is the durable statement that a commit is live. `ref_cursors` records the tip each ref | ||
| 453 | had when its work was queued, and `reconcileRefs` compares the refs against it once at | ||
| 454 | startup and every 30 seconds. The spool stays what it should be — the sub-second path, | ||
| 455 | not the promise. The trade is stated rather than discovered: a tip no cursor has | ||
| 456 | recorded is work owed, so a queue that was deleted (or an instance upgrading onto this) | ||
| 457 | queues its current tips once, and the unique index stops a tip that already ran from | ||
| 458 | running twice once `vole index rebuild` has restored its row from the record on the | ||
| 459 | branch. | ||
| 460 | |||
| 461 | An idle runner also stopped costing anything. The lease takes a `wait` and the session | ||
| 462 | parks on a wake-up channel that a committed enqueue closes — taken *before* the lease | ||
| 463 | attempt, so a run queued in between cannot be missed — and the account's eligible | ||
| 464 | repositories are reused for the roster's five-second window instead of being re-derived | ||
| 465 | on every request, which was O(repos) per idle runner every two seconds. | ||
| 466 | |||
| 467 | A newer tip cancels the queued builds of the older one (`StatusCanceled` finally has a | ||
| 468 | writer, and only queued rows: a running job is never interrupted), and the lease orders | ||
| 469 | by in-flight count per repository before age, so one repository's burst cannot occupy | ||
| 470 | every runner while a quiet repository waits. | ||
| 471 | |||
| 472 | (Measured: `TestReconcileQueuesATipTheQueueHasNeverSeen`, | ||
| 473 | `TestReconcileRebuildsAQueueThatWasDeleted` — `runs.db` and its WAL deleted, the live | ||
| 474 | tip queued again — `TestReconcileIgnoresTheMetadataBranch`, | ||
| 475 | `TestReconcileDropsTheCursorOfADeletedBranch`, | ||
| 476 | `TestALeaseWaitsForWorkInsteadOfPolling` (a lease parked for 30 s returns the job | ||
| 477 | within 5 s of it being queued), `TestALeaseWithoutAWaitAnswersAtOnce`, | ||
| 478 | `TestQueuingWorkWakesTheParked`, | ||
| 479 | `TestCancelSupersededOnlyTouchesWaitingRunsOfTheSameRef`, | ||
| 480 | `TestLeaseServesTheLeastBusyRepoFirst`, `TestANewerTipCancelsTheQueuedBuildOfTheOlderOne`.) | ||
| 481 | |||
| 482 | ✅ **Slice 8 — runner status on the web.** `runner_sessions` is presence rather than | ||
| 483 | history: one row per connected channel, written by events a runner already sends | ||
| 484 | (connect, lease, heartbeat, finish), dropped when the channel closes and forgotten | ||
| 485 | after ten silent minutes — the only trace a killed process leaves. `GET /runners` shows | ||
| 486 | who is connected, what each is building (linked to the run) or "waiting for work", and | ||
| 487 | the queue depth per repository, which is the page somebody needs when a build does not | ||
| 488 | start. It follows the clone rule like every other page: a session building a private | ||
| 489 | repository is not shown at all and that repository's queue is not counted, because | ||
| 490 | naming either would confirm the repository exists. | ||
| 491 | (Measured: `TestARunnerSessionIsVisibleWhileItIsConnected`, | ||
| 492 | `TestRunnersPageShowsTheRunnerHoldingAJobAndTheQueueDepth`, | ||
| 493 | `TestRunnersPageKeepsAPrivateRepositoryOut`.) | ||
| 494 | |||
| 446 | 495 | Remaining: | |
| 447 | 496 | ||
| 448 | - Push-event queue persisted in SQLite, with at-least-once delivery | ||
| 449 | 497 | - Toolchain environment in the record (the runner's Go version, OS, architecture) — the | |
| 450 | 498 | record names the job, not the machine | |
| 499 | - A narrowing filter on the runner (`--repos`, labels) and job-level targeting | ||
| 500 | (`runs-on`): one instance-wide pool is right for identical machines and wrong for one | ||
| 501 | macOS runner in a Linux pool. The filter can only narrow what the allowlist grants, | ||
| 502 | never widen it | ||
| 503 | - No per-repository in-flight cap. Fairness is ordering — least busy repository first — | ||
| 504 | which stops a burst taking the whole fleet without a knob to tune; a cap is what an | ||
| 505 | operator wants when they need to *bound* one repository's footprint, and that is a | ||
| 506 | config key to add when somebody asks for the bound rather than the ordering | ||
| 451 | 507 | ||
| 452 | 508 | **Exit criteria:** a push runs a pipeline end-to-end ✅; killing a runner mid-job results in | |
| 453 | 509 | the job being requeued and completing on another runner ✅; a workflow that mutates its | |
| 454 | 510 | own definition mid-run still executes the version it was queued with ✅ (steps are pinned | |
| 455 | 511 | at enqueue); a run record can be verified against the SHA it claims ✅ (the runner signs | |
| 456 | 512 | the commit, `git verify-commit` accepts it under the roster afterwards, and the record | |
| 457 | is on the branch a plain clone carries, so the check needs no access to the instance). | ||
| 513 | is on the branch a plain clone carries, so the check needs no access to the instance); | ||
| 514 | a push whose spool file was never written still runs ✅ (the refs are reconciled, measured | ||
| 515 | by `TestReconcileRebuildsAQueueThatWasDeleted`, which deletes the queue and its WAL and | ||
| 516 | gets the live tip queued again). | ||
| 458 | 517 | ||
| 459 | 518 | ### Phase 5 — LFS — ✅ complete (artifact uploads and locking deferred) | |
| 460 | 519 | ||
| @@ -940,6 +999,10 @@ Filled in as phases land. Append; don't rewrite. | |||
| 940 | 999 | | 2026-10-23 | Access rules live in `data/users.db`, not `refs/vole/acl` | The rules are configuration the instance owns: only the instance writes them, every request and every push reads them, and they are keyed by account id, so a copy in a clone cannot be read without this instance's roster. In git they cost a commit, a compare-and-swap retry and a rev-parse per read to store a handful of rows — the same argument that moved the roster out of `data/system.git`. What the ref did buy, surviving a deleted instance, a backup of one file does better. Existing instances import the ref once at startup (`repo.Store.ImportACLs`), and a change keeps its author in the row and a line in `data/audit.log`, which is what the commit message used to say | | |
| 941 | 1000 | | 2026-10-23 | `runs.db` holds the queue; finished runs are rebuilt from the branch | A queued or leased run has no record because it has not happened, and that is exactly what a database is for. Once an attempt finishes, the record is the copy that matters, so `vole index rebuild` reconstructs the row, the attestation and the artifact list from git — the queue stops being the only place a result exists, which is what made run records non-portable | | |
| 942 | 1001 | | 2026-10-23 | "Would this merge?" is answered in memory and surfaced, never gated | `merge-tree --write-tree` merges without a worktree or an index, so the cost is one git process on a page view; gating on it would be wrong because the base can move after an approval, and the roadmap asked for detection before the push rather than a second refusal. The exit status alone is ambiguous — 1 is both "conflicts" and "bad revision" — so the answer is believed only when the output starts with a tree id, and silence otherwise | | |
| 1002 | | 2026-10-23 | The queue is reconciled from the refs, not persisted as a push-event log | The roadmap item asked for push events in SQLite, and a row there would not have been atomic with the ref update either: the hook is another process writing another file, so it shrinks the crash window instead of closing it, puts a database write into the path that must never fail a push, and dies with the same `rm data/runs.db` it was meant to survive. A ref tip is already the durable statement that a commit is live, so `ref_cursors` plus a 30-second pass answers the same question without a second copy of the fact — and the spool becomes a latency hint. The cost, accepted and written down: an emptied queue or a first start after an upgrade queues each current tip once | | ||
| 1003 | | 2026-10-23 | An idle lease parks on an in-process wake-up channel, and eligibility is cached for one poll window | The "long poll" was the client's own timer — the queue answered at once and the runner slept — and every ask re-derived the account's repositories by opening each one's rules, which is O(repos) per idle runner every two seconds. The wait is a hint and never a durability mechanism (the lease is still the database's decision), so a missed wake-up costs one poll interval; in-process is enough because one `vole serve` owns the queue, and a second process would have to ask the database anyway | | ||
| 1004 | | 2026-10-23 | A new tip cancels the queued runs of the same ref, and the lease serves the least busy repository first | Pushing twice in a minute built the first commit twice while `canceled` existed with no writer. The cancellation covers queued rows only: interrupting a running build is a different promise from not starting one. Fairness is ordering rather than a per-repository cap, because a cap is a number somebody has to tune and ordering needs none — a burst still runs, it just cannot take every runner | | ||
| 1005 | | 2026-10-23 | Runner presence is a queue row per channel, filtered by the clone rule | A runner was an SSH channel invisible to both the web and the queue, so "is my build waiting because nobody is listening" had no answer anywhere. Presence is derived rather than stored: written by connect, lease, heartbeat and finish, dropped when the channel closes, aged out after ten silent minutes. The web is anonymous, so a session appears only while the repository it is building is public — naming a private repository would confirm it exists | | ||
| 943 | 1006 | ||
| 944 | 1007 | ## Open questions | |
| 945 | 1008 | ||