vole

Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Code

392 commit(s) on main

Compare History

Merge change request 01M70WES5ZZDJ6RP226WB11BBZ: Let people rename and remove their own passkeys keenan · 16m ago 4b8c9e1d

source preview history blame raw 28.8 KiB

1	# Runners
2
3	A runner is a machine that leases jobs over SSH, executes their steps in
4	containers, and reports the result. Workflows themselves are described in
5	[Workflows](workflows.md); this page is about the machines that run them.
6
7	## Start a runner
8
9	To have a machine set up for you — the unit, the account rootless Podman needs,
10	and the pinned host key — see [CI runners](deploy/runner.md).
11
12	A runner needs an OCI runtime: Docker, rootless Podman, or nerdctl. Rootless
13	Podman is preferred, because membership in the Docker group is effectively root on
14	the host. Start a runner anywhere it can reach the Vole server and that runtime:
15
16	```sh
17	vole runner --remote ssh://git@host:2222 --container-runtime podman
18	vole runner --remote ssh://git@host:2222 --container-runtime podman --once
19	```
20
21	You can describe the machine and limit which repositories it serves:
22
23	```sh
24	vole runner --remote ssh://git@host:2222 --label linux --label arm64
25	vole runner --remote ssh://git@host:2222 --repos demo --repos docs
26	```
27
28	- `--label` says what the machine is. A job is sent only to a runner that has
29	every label in its `runs-on`. A job with no `runs-on` can run anywhere.
30	- `--repos` narrows what the account is already allowed to run. It cannot grant
31	access to a repository.
32	- `--dir` chooses where checkouts and caches are kept.
33	- `--jobs` is how many jobs this machine runs at once. Each gets its own container;
34	see [Running several jobs at once](#running-several-jobs-at-once).
35	- `--once` leases one job and exits, which is useful for a runner that is
36	scheduled per job.
37
38	## Before it takes a job
39
40	The runtime is probed once at startup: the runner runs `<runtime> info` and refuses
41	to serve if it fails, before it connects and before anything is leased. A runtime
42	whose binary is installed but which cannot create a container — a missing OCI
43	runtime, a run root it cannot write, a storage driver the kernel will not load — is
44	the machine's problem, and a runner that took a job in that state would spend an
45	attempt and put the machine's failure on a commit that never ran a step.
46
47	The message repeats the runtime's own last line:
48
49	```
50	vole runner: cannot run a container with podman: Error: default OCI runtime "crun" not found: invalid argument
51	```
52
53	The probe is one query and pulls no image. It runs once: a runtime that breaks while
54	the runner is serving fails the job it was building, and restarting the runner probes
55	again. As a system service the unit exits on this error, so `Restart=on-failure`
56	retries it and `journalctl -u vole-runner` carries the message. `vole workflow run`
57	makes the same check before a local run.
58
59	## Running several jobs at once
60
61	A build is usually one step waiting on one thing at a time, so most of a big machine
62	sits idle behind it. `--jobs` puts more jobs on the one machine:
63
64	```sh
65	vole runner --remote ssh://git@host:2222 --container-runtime podman --jobs 4
66	```
67
68	Each job slot is a session of its own, so the instance hands work to each one
69	independently and the runners page shows four rows rather than one. The slots share
70	the machine: one cache, one set of saved checkouts, one set of images. What they do
71	not share is a container.
72
73	Two things need sizing for it:
74
75	- The container limits are per job. `--container-memory`, `--container-cpus` and
76	`--container-pids` apply to each container, so a machine running four jobs needs
77	four times what they allow. On the instance, `max_jobs_per_repo` is what stops one
78	repository's burst from claiming every slot (see
79	[Configuration](configuration.md)).
80	- A job's cache is shared with the same repository's other jobs. Two jobs of the
81	same repository, trust and image get the same `/cache`, which is what lets them
82	reuse each other's downloads. The tools in the image are what make that safe: `go`,
83	`cargo` and `npm` all lock their own caches. A job that empties `/cache` or keeps
84	state there between runs is competing with its own siblings, so give it a path of
85	its own or its own runner.
86
87	An idle slot holds one open connection while it waits for work, so a machine with
88	`--jobs 8` uses eight of the instance's SSH connections (`ssh_max_connections`).
89	Losing a connection costs that slot its job — the job is requeued when its lease
90	expires — and the slot reconnects rather than stopping the rest.
91
92	With `--once`, `--jobs 4` starts up to four jobs, finishes them, and exits, which is
93	the shape a scheduler that starts a container per job wants.
94
95	## Stopping a runner
96
97	A runner that is asked to stop — `systemctl stop`, Ctrl-C, a `SIGTERM` — stops
98	taking work and lets the jobs it already has finish and report. That is the
99	difference between a build that is nearly done being published and being thrown
100	away, and it means the run is not left unclaimed until its lease expires. A second
101	signal ends the process at once.
102
103	Under systemd the packaged unit allows 60 seconds for that
104	(`TimeoutStopSec`), so a job that runs longer than that is killed anyway. If builds
105	on a machine take longer and you would rather wait for them, raise it:
106
107	```sh
108	sudo systemctl edit vole-runner # [Service] / TimeoutStopSec=30min
109	```
110
111	## The runner's account
112
113	A runner connects as an account, and that account is the runner's identity: it is
114	what the repository's runner allowlist names, what the audit log records, and what
115	the server checks before it hands over a job. For a machine nobody signs in to,
116	make the account a bot:
117
118	```sh
119	vole admin account create ci-runner-1 --bot --key ci-runner-1.pub
120	vole admin acl runners add demo ci-runner-1
121	vole admin acl set-role demo ci-runner-1 read # private repositories only
122	```
123
124	From another machine, name the instance; the key file is read where the command
125	runs:
126
127	```sh
128	vole admin --remote ssh://git@host:2222 account create ci-runner-1 --bot --key ci-runner-1.pub
129	```
130
131	A bot has no browser sign-in, so its key cannot be turned into a web session, and
132	it cannot be an instance admin. That second part matters for a runner: admins
133	count as trusted reporters even when a repository's allowlist does not name them,
134	so an admin account would make every result count and quietly bypass the list. A
135	named runner needs no repository role for its result to count — the allowlist is
136	that decision — but the build still has to clone the code, and a private
137	repository grants no read role by default. The `set-role ... read` line covers the
138	clone without giving the machine write access.
139
140	The `--key` is long-lived: it authenticates the runner process for as long as the
141	account is active. It is not the credential a job uses. A workflow job that names
142	a bot with `acts-as:` still gets a key minted for that run, scoped to the repository
143	and the run, and gone when the run ends; that key cannot lease work. Keep one key
144	per runner machine, readable only by the runner user, so a host you stop trusting
145	is revoked by removing that key or retiring the account.
146
147	## Host key verification
148
149	A runner executes repository code, so the instance it trusts is a security
150	boundary: a host that can answer in its place can send it any job. The SSH host
151	key is verified against your known_hosts, and an unknown or changed host is
152	refused. Populate the file once, out of band:
153
154	```sh
155	ssh-keyscan -p 2222 git@host >> ~/.ssh/known_hosts
156	vole runner --remote ssh://git@host:2222 --known-hosts ~/.ssh/vole_known_hosts
157	```
158
159	- `--known-hosts <path>` checks against that file instead of the default
160	`~/.ssh/known_hosts`.
161	- `--insecure-host-key` turns verification off and accepts any host that answers.
162	It exists for tests and throwaway instances; a real runner should not use it.
163	- `VOLE_KNOWN_HOSTS` and `VOLE_INSECURE_HOST_KEY` do the same for every command,
164	including `vole attach`, `vole admin --remote`, and the CLI verbs.
165
166	The policy covers git's own ssh calls as well as the runner's control channel, so
167	a clone, a fetch, a release push and the job stream all verify the same host. An
168	environment variable cannot loosen it: an explicit
169	`GIT_SSH_COMMAND=... StrictHostKeyChecking=no` is overridden by the runner's own
170	options, which come first.
171
172	## Images and image builds
173
174	An `image-file` definition is built by the configured runtime, so a rootless
175	Podman runner builds rootless. The build runs with the runtime's own network
176	(bridge by default), because the base image and any `run:` commands need to fetch
177	packages; if builds should have no network, preload the base image and set the
178	runtime's default network to none on that host. The definition is pinned by its
179	digest — the runner receives the definition text and digest with the job, tags the
180	built image by that digest, and reuses it — so a definition that changes after a
181	job is queued is not the one that was reviewed. Pin the base image in `from:` by
182	digest (`image@sha256:...`) when you need the build itself to be reproducible.
183
184	## What a job may name
185
186	The server validates a workflow when it is pushed, and the runner validates the
187	job again before it acts on it, because the runner is the party that turns the
188	payload into paths, container arguments and scripts. A job whose repository,
189	commit, ref, branch, tag, pages path, artifact pattern, step or environment does
190	not have the shape the instance documents is refused, and the runner stops rather
191	than running it. In particular a path that climbs out of the checkout (`..`), an
192	absolute path, a `:` or NUL in a value used to build a container argument, and a
193	commit id that is not a commit id are all refused at the runner, not only at the
194	server.
195
196	## What a job may consume
197
198	A job runs untrusted build code, and everything it produces is stored by the
199	instance or kept on the runner, so each of those channels has a bound:
200
201	- Logs. One attempt's log is capped at 32 MiB. Past that the runner keeps the
202	first 32 MiB and appends a line saying it was truncated, so a build that loops
203	printing cannot fill the runner's disk or the instance's log directory. The log
204	is written straight to a file on the runner and streamed to the server in chunks
205	while the job runs; it is not held in memory.
206	- Artifacts. A workflow declares at most 32 patterns, and a run may publish at
207	most 256 files totalling 4 GiB. The expanded set is checked before anything is
208	uploaded; a pattern that matches a directory tree is the usual way to hit this.
209	- Cache. `--cache-size` (5 GiB by default) caps the cache root, enforced by
210	evicting whole namespaces, and a namespace a job has mounted is left for a later
211	pass. Each repository gets its own namespace, and each
212	contributor account gets its own inside a repository, so one contributor's
213	branch cannot seed the cache another contributor's branch builds from. A
214	protected branch never shares a cache with a contributor branch. The saved
215	checkouts jobs clone from are bounded separately; see
216	[Saved checkouts](#saved-checkouts).
217	- Container. `--container-memory`, `--container-cpus`, `--container-pids` and
218	`--container-tmpfs` are the runtime's own limits, per container, so `--jobs`
219	multiplies them. `--container-pids` (2048) and `--container-tmpfs` (1G) have
220	defaults and cannot be removed: `/tmp` is a tmpfs, so its size is memory, and a job
221	that can fork without a bound is a fork bomb away from taking the machine. Set
222	`--container-memory` — the runtime's default is the whole machine — and see the
223	unit's `MemoryMax` backstop below. `--container-network` is what this machine can
224	offer — `none` (the default) or `bridge` — and a job's own `network:` decides what
225	it gets, so a bridge runner can still run a job that asks for none. See
226	[Network](#network). The runtime's storage driver, not Vole, is what bounds the
227	container's writable layer, so size it on the host.
228	- Disk. A job's tree is a host directory the runtime cannot quota, so the runner
229	keeps `--disk-reserve` (2G) free under `--dir` and stops a job that takes the
230	filesystem below it while the job is running. It is one `statfs` per second, and the
231	log says what was left and what the reserve is. That is the promise — the runner
232	notices — rather than a hard bound; for one of those, put `--dir` on its own
233	filesystem and size that. The unit's `MemoryMax` is the same idea for RAM.
234
235	## Images and image builds
236
237	Every job names its image in the workflow. A registry reference is used as
238	written; an `image-file` definition is built locally first. The definition fields
239	are described in the [Image reference](reference/image-reference.md).
240
241	For a custom definition, the runner receives the pinned definition text and its
242	digest from the queue, so it cannot build a definition that changed after the job
243	was queued. It renders the definition to a Dockerfile, builds it with the
244	configured OCI runtime, and tags the result by the definition digest. A later job
245	with the same definition reuses that tag.
246
247	With `--jobs`, jobs that name the same definition build it once: a job whose image
248	another job is already building waits for that build, saying so at the top of its
249	log, and keeps its lease alive while it waits. If the build fails, the jobs waiting
250	on it fail with the same reason.
251
252	The image build uses the runtime's own default network, so a base image and its
253	`run:` commands can fetch packages. The job's own network request applies to the job
254	container, not to the build.
255
256	An image definition is code. Its build runs on the runner through the OCI
257	builder, so approving a contributor tip that changes a definition also approves
258	that build. Prefer rootless Podman, and pin the base image by digest.
259
260	## How a job runs
261
262	One container is created for the job, plus one for each service the job declares;
263	all are removed afterward. The workflow's steps run inside the job container, but
264	the runner process stays on the host: the SSH key, the trusted Git cache, and the
265	finish channel are never exposed to the job. Each job gets a fresh
266	clone of the runner's saved copy of the repository (see
267	[Saved checkouts](#saved-checkouts)), so a job cannot leave a Git hook or a modified
268	checkout behind for the next one.
269
270	A service is a second container on a private user-defined network created for the
271	attempt. It gets the job's user, capabilities, and resource limits, but no job tree,
272	no cache, and no secrets. Its declared name is the network alias a step connects to.
273	The runner starts services after the image is ready and before the first step,
274	fails the job if one is not running, and adds the end of their output to a failed
275	attempt's log. Services, the job, and the network are removed together; a killed
276	runner finds its leftovers by label on the next start.
277
278	Container defaults are intentionally closed:
279
280	- no network unless the job asks for one with `network: bridge`, or the runner was
281	started with it and the job says nothing;
282	- no capabilities, `no-new-privileges`, and no host mounts except the job tree;
283	- no Docker socket, SSH agent, or host environment;
284	- `/tmp` is a writable, executable tmpfs (compilers write their output there and
285	run it);
286	- the steps run as the runner user's `uid:gid` by default.
287
288	A run started by a non-push trigger also gets a directory mounted at `/run/vole`
289	with the pinned event as `event.json`, plus `VOLE_EVENT`, `VOLE_EVENT_ACTION`, and
290	`VOLE_EVENT_PATH`. The file is data for a step to read; a workflow cannot replace
291	those variables through `env:`. See [Workflow reference § Event data available to steps](reference/workflow-reference.md#event-data-available-to-steps).
292
293	A job that declares `secrets:` gets the same directory, with a `secrets.env` beside
294	the event. The runner writes one `export NAME='value'` line per secret, quotes it so
295	a value keeps its line breaks, and sources the file before each step, so a secret is
296	a variable in the step's shell and never an argument to the runtime. Values are at
297	most 128 KiB each, are delivered only for the job that declared them, and a name the
298	repository does not have leaves its variable unset with one line in the log. The
299	value arrives over the SSH channel with the lease, is written `0600` in the job's
300	directory, and is removed with it.
301
302	Every delivered value is replaced with `***` in the log as the log is written, so
303	the file the server stores never holds one, and a value split across two writes is
304	still caught. The runner's own error text goes through the same mask. It is best
305	effort: a re-encoded value (base64, hex, a URL escape) no longer matches, and a
306	value shorter than 4 bytes is left alone because masking every occurrence of it
307	would rewrite the log. A step that writes a secret into an artifact publishes it —
308	the mask covers the log, not the files a job chooses to upload.
309
310	A runner can read every secret a repository's jobs declare. That is what running
311	the job means: the values arrive on the machine. The runner allowlist is the control
312	that decides whose machine it is, not the container. See
313	[Accounts and access § Roles](accounts-and-access.md#roles).
314
315	The container's own filesystem is writable, and thrown away with the container:
316	nothing a job writes there survives the run, and an unprivileged uid cannot write
317	the image's root-owned paths anyway. What protects the machine is the mount set —
318	the job tree is the only host path a job can see — not the writability of the
319	container's own layer.
320
321	Because the uid is the runner's own, an image whose tools call `getpwuid` — OpenSSH
322	is the common case — needs a passwd entry for that uid, or `ssh` and `ssh-keygen`
323	exit with `No user exists for uid N`. Define one in the image definition, or pass
324	`--container-user` naming a user the image already has. This is the same container
325	`vole workflow run` starts, so an image that fails here fails there too.
326
327	The uid decides how immutable the image is. An ordinary user cannot write the
328	image's root-owned files, so a job's view of its own toolchain is the one that was
329	built. If the runner runs as root (or `--container-user` says so), the job can
330	rewrite its own image, and the runner prints a warning at startup: run the runner
331	as an unprivileged user.
332
333	Use `--container-runtime` to choose `docker`, `podman`, or `nerdctl`. The runner
334	user must be able to talk to that runtime; rootless Podman is preferred. Do not
335	mount the host's Docker socket into a job; that would undo the isolation.
336
337	`--container-pull` controls whether an image is pulled (`missing`, `always`, or
338	`never`). `--container-user`, `--container-userns`, `--container-memory`,
339	`--container-cpus`, and `--container-pids` override the user, namespace mode, and
340	resource limits. Rootless Podman uses `keep-id` by default so the numeric user can
341	write the bind-mounted job tree; for rootless Docker, choose a user that matches
342	the runtime's mapping (container root is the runner user) or use a runtime with
343	`keep-id`.
344
345	The run record names the image, the definition digest when there is one, and the
346	runtime, so the run page says what actually executed.
347
348	## Network
349
350	`--container-network` says what this machine can give a job: `none` (the default) or
351	`bridge`. It is a capability, not a setting for every job. The job's `network:`
352	decides what the container gets, and the runner applies it:
353
354	\| Job says \| This runner says `bridge` \| This runner says `none` \|
355	\|---\|---\|---\|
356	\| nothing \| bridge \| none \|
357	\| `none` \| none \| none \|
358	\| `bridge` \| bridge \| the runner never receives the job \|
359
360	So a machine with a network is not a machine that hands it to every build. A job
361	that asks for none is isolated on either kind of runner, and a job that asks for
362	bridge waits in the queue for a runner that advertises one instead of running
363	without the network it needs. A bridge job that declares services gets a
364	user-defined network for that attempt rather than the runtime's default bridge; the
365	job and its services are its only members, so the service names resolve there and
366	nowhere else. The runner advertises this with every lease, the same
367	way it advertises `--label`, and the job's request is pinned with the run when it is
368	queued.
369
370	A runner can be started with `--container-network bridge` and nothing breaks; that
371	is a policy choice for a machine whose owner trusts everything it builds. What no
372	runner can do is take the choice away from the job, and this is why. A job runs code
373	from the repository, including a contributor branch a maintainer approved for
374	reasons that had nothing to do with network access. With a network, that code
375	reaches whatever the machine can reach: the instance handing out the work, since a
376	deployed instance listens beyond loopback, the other services on that host, and
377	anything on the internet to send out what the job can read. With no network the only
378	things it can touch are the job tree and the container itself, both of which are
379	discarded, so a job that declares a secret and does not need to reach anything
380	cannot send it anywhere. No network also keeps a build honest: a job that fetches
381	its dependencies at run time builds whatever the remote serves that day, while a
382	closed job builds what its image carries, which is the image the run pinned by
383	digest.
384
385	## Caching
386
387	A job starts with an empty container, so without help every run downloads its
388	dependencies and compiles them again: in this repository that is about 23 seconds
389	of `go build`/`go vet` that a warm cache turns into one. The runner keeps a directory
390	per job and mounts it at `/cache`.
391
392	It mounts the directory and sets no variable that points at it. Which tool uses
393	`/cache`, and therefore whether a job shares state with the job before it, is the
394	repository's decision. A runner that guessed would be making a claim about a
395	toolchain it does not test, and a job that wants no shared state would have to
396	defend against a default it never asked for.
397
398	Opt in where the environment already lives — the image definition:
399
400	```yaml
401	# .vole/images/verify.yaml
402	version: 1
403	from: golang:1.27-bookworm
404	env:
405	GOMODCACHE: "/cache/go-mod"
406	GOCACHE: "/cache/go-build"
407	```
408
409	or per job, which works for a registry image too:
410
411	```yaml
412	jobs:
413	test:
414	image: golang:1.27-bookworm
415	env:
416	GOCACHE: /cache/go-build
417	GOMODCACHE: /cache/go-mod
418	steps:
419	- run: go test ./...
420	```
421
422	The variables for the common toolchains are the tools' own; there is nothing
423	Vole-specific to learn. `GOMODCACHE`/`GOCACHE` for Go, `CARGO_HOME`,
424	`npm_config_cache`, `PIP_CACHE_DIR`, `MAVEN_ARGS`/`-Dmaven.repo.local`,
425	`GRADLE_USER_HOME`, `NUGET_PACKAGES`, `PUB_CACHE`, `COMPOSER_CACHE_DIR`,
426	`CCACHE_DIR`, and `XDG_CACHE_HOME` for anything that follows the XDG convention.
427	An image that sets none of them gets an empty directory and no behaviour change.
428
429	The cache lives under `--dir/cache`, keyed by repository, trust, and image:
430	`<dir>/cache/<repo>/<direct\|contrib>/<image>`. The parts matter. A cache is shared
431	mutable state: a repository never sees another's, and a contributor branch gets its
432	own namespace, so a branch nobody has vouched for cannot seed the cache a protected
433	branch builds into. A combined result derived from a contributor branch belongs to
434	that contributor's namespace for the same reason. The image is part of the key so a
435	cache built with one toolchain is not consumed by another.
436
437	Namespaces are evicted whole and least-recently-written first, so a cache is never
438	left half-deleted — a Go module cache missing one file is a checksum error deep in
439	a build rather than a download. A job holds its namespace for as long as it runs,
440	and eviction leaves a held namespace for a later pass, so two runners sharing one
441	`--dir` never delete each other's tree in the middle of a build. `--cache-size` caps
442	the total (5G by default), and the runner enforces it at startup and then on its own
443	timer: a cache that is in use when the cap is reached comes down when the job
444	finishes, not under it. `--no-cache` keeps nothing between jobs: the directory is
445	still mounted — an image that points at `/cache` has to work either way — but it is
446	emptied after every job, while that job still holds it, so a shared `--dir` is not
447	emptied under somebody else's build.
448
449	## Saved checkouts
450
451	The first job for a repository downloads it once, and the runner keeps that copy
452	under `--dir`. A later job clones from the copy and fetches only the commits it does
453	not have — usually none at all — so history crosses the network once instead of once
454	per run. The copy is never mounted into a job: the workflow gets a clone with its own
455	`.git`, which is why a hook it writes still dies with the job.
456
457	Two things keep the copies from filling the disk:
458
459	- Repacking. A copy that is most of the cap is repacked. That drops history no
460	branch or tag can reach any more: commits from a force-push or a deleted branch,
461	and a commit a job once pinned by id. Nothing a job can still ask for is touched,
462	so the next job for the same revision is still a local copy with no download.
463	- Deleting. When the cap is still reached, the copy nobody has built from in the
464	longest time is deleted, whole. The next job for that repository downloads it
465	again, which is why it is the last resort rather than the first.
466
467	`--git-cache-size` caps all saved checkouts together (10G by default; `0` keeps
468	every one). A copy another runner process is using is passed over, and the
469	repository this runner is about to build is never the one deleted.
470
471	## Who can run jobs and report results
472
473	A job runs repository code, so Vole uses a runner allowlist for both taking a job
474	and reporting its result:
475
476	```sh
477	vole admin acl runners list demo # show the list
478	vole admin acl runners set demo ci-runner-1 # replace the list
479	vole admin acl runners add demo ci-runner-2 # add one account
480	vole admin acl runners remove demo ci-runner-1 # remove one account
481	vole admin acl runners clear demo # back to admins only
482	```
483
484	Admins always count. For everyone else, a maintainer running CI on their laptop does
485	not turn a required check green. The result is still recorded, and the run page
486	says whether it counts.
487
488	While a job runs, the runner streams its log to the server in chunks over the same
489	SSH channel that leased it. The server writes one file under `data/logs/` and the
490	run page can open it before the job finishes. When the job ends, the runner sends
491	the attempt's record with its finish message; Vole checks that the record names the
492	account holding the lease, that it describes the job the queue handed out, and that
493	every artifact pointer exists in the store. It then stores the attempt and the run's
494	result in `data/runs.db` in one transaction. A tagged build that publishes release
495	artifacts also pushes those pointers to the `vole` branch after the record is
496	stored.
497
498	The runner did the work, so the runner reports the result. The SSH connection
499	identifies the runner, and `data/audit.log` records the decision.
500
501	## Logs
502
503	The runner streams the log while the steps run. The server appends each chunk to
504	the attempt's file, and the run page serves that file directly with range requests,
505	so it does not read a large log into memory. The file is complete when the attempt's
506	record is stored.
507
508	The stored attempt also names each step's byte range in that one log, so the run
509	page can show a step's output without downloading the image build and other steps.
510	Step timing is recorded beside the range.
511
512	```text
513	/<repo>/runs/<id>/log # latest attempt
514	/<repo>/runs/<id>/log?attempt=N # a particular retry
515	/<repo>/runs/<id>/log?step=N # one step inside that attempt
516	```
517
518	If a runner is killed, its attempt has no log. A record written before step
519	timing existed still opens; it simply has no step ranges to show.
520
521	## Scheduling and the queue
522
523	All runners share one pool. An idle runner waits for work instead of asking
524	repeatedly, and Vole gives the next job to the repository with the least work in
525	progress. Set `max_jobs_per_repo` when you need a hard limit per repository.
526
527	The instance-admin Admin page at `/-/admin?tab=runners` shows who is connected,
528	what each runner is building, and how much work is queued. The old `/-/runners` URL
529	redirects there.
530
531	How the queue behaves:
532
533	- Finished runs are stored in `data/runs.db`, not on the `vole` branch. A clone or
534	backup of the repository does not carry CI history, so back up `runs.db`.
535	- A push normally reaches the queue in under a second. Every 30 seconds Vole also
536	compares each branch and tag with the last tip it queued, so a missed
537	notification still produces a run.
538	- Pushing again cancels queued runs for the commit it replaced. It never stops a run
539	that is already in progress. A maintainer can stop one from its page or with
540	`vole run cancel <id>`: the queue records the request, the runner sees it on its next
541	heartbeat, stops the container, and reports the partial attempt. The run stays
542	`canceled` rather than becoming a failure, and its page says who asked.
543	- If a runner disappears, its job is requeued. After five attempts the job fails
544	instead of cycling forever.
545	- A job whose attempt stops before its first step — the image build, the container,
546	the network — is recorded as `error` (the page says could not run) with the
547	runner's reason, not as a failure of the commit. A required check waits on it
548	rather than going red; rerunning the job clears it.
549	- Deleting `data/runs.db` loses CI history and attempts that were in flight. Current
550	branch and tag tips are queued again from Git, but only a backup can restore the
551	finished runs.
552