vole

Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Blame · docs/runner.md

Line attribution on main · 566 line(s)

View file History

70b70264 Keenan Nemetz 15d ago	1	# Runners
	2
	3	A runner is a machine that leases jobs over SSH, executes their steps in
	4	containers, and reports the result. Workflows themselves are described in
	5	[Workflows](workflows.md); this page is about the machines that run them.
	6
	7	## Start a runner
	8
e21f6c5e Keenan Nemetz 3d ago	9	To have a machine set up for you — the unit, the account rootless Podman needs,
	10	and the pinned host key — see [CI runners](deploy/runner.md).
	11
70b70264 Keenan Nemetz 15d ago	12	A runner needs an OCI runtime: Docker, rootless Podman, or nerdctl. Rootless
	13	Podman is preferred, because membership in the Docker group is effectively root on
	14	the host. Start a runner anywhere it can reach the Vole server and that runtime:
	15
	16	```sh
	17	vole runner --remote ssh://git@host:2222 --container-runtime podman
	18	vole runner --remote ssh://git@host:2222 --container-runtime podman --once
	19	```
	20
	21	You can describe the machine and limit which repositories it serves:
	22
	23	```sh
	24	vole runner --remote ssh://git@host:2222 --label linux --label arm64
	25	vole runner --remote ssh://git@host:2222 --repos demo --repos docs
	26	```
	27
	28	- `--label` says what the machine is. A job is sent only to a runner that has
	29	every label in its `runs-on`. A job with no `runs-on` can run anywhere.
	30	- `--repos` narrows what the account is already allowed to run. It cannot grant
	31	access to a repository.
	32	- `--dir` chooses where checkouts and caches are kept.
18c559ea Keenan Nemetz 3d ago	33	- `--jobs` is how many jobs this machine runs at once. Each gets its own container;
18c559ea Keenan Nemetz 3d ago	34	see [Running several jobs at once](#running-several-jobs-at-once).
70b70264 Keenan Nemetz 15d ago	35	- `--once` leases one job and exits, which is useful for a runner that is
	36	scheduled per job.
	37
f28097af Keenan Nemetz 4h ago	38	## Before it takes a job
	39
	40	The runtime is probed once at startup: the runner runs `<runtime> info` and refuses
	41	to serve if it fails, before it connects and before anything is leased. A runtime
	42	whose binary is installed but which cannot create a container — a missing OCI
	43	runtime, a run root it cannot write, a storage driver the kernel will not load — is
	44	the machine's problem, and a runner that took a job in that state would spend an
	45	attempt and put the machine's failure on a commit that never ran a step.
	46
	47	The message repeats the runtime's own last line:
	48
	49	```
	50	vole runner: cannot run a container with podman: Error: default OCI runtime "crun" not found: invalid argument
	51	```
	52
	53	The probe is one query and pulls no image. It runs once: a runtime that breaks while
	54	the runner is serving fails the job it was building, and restarting the runner probes
	55	again. As a system service the unit exits on this error, so `Restart=on-failure`
	56	retries it and `journalctl -u vole-runner` carries the message. `vole workflow run`
	57	makes the same check before a local run.
	58
18c559ea Keenan Nemetz 3d ago	59	## Running several jobs at once
	60
	61	A build is usually one step waiting on one thing at a time, so most of a big machine
	62	sits idle behind it. `--jobs` puts more jobs on the one machine:
	63
	64	```sh
	65	vole runner --remote ssh://git@host:2222 --container-runtime podman --jobs 4
	66	```
	67
	68	Each job slot is a session of its own, so the instance hands work to each one
	69	independently and the runners page shows four rows rather than one. The slots share
	70	the machine: one cache, one set of saved checkouts, one set of images. What they do
	71	not share is a container.
	72
	73	Two things need sizing for it:
	74
	75	- The container limits are per job. `--container-memory`, `--container-cpus` and
	76	`--container-pids` apply to each container, so a machine running four jobs needs
	77	four times what they allow. On the instance, `max_jobs_per_repo` is what stops one
	78	repository's burst from claiming every slot (see
	79	[Configuration](configuration.md)).
	80	- A job's cache is shared with the same repository's other jobs. Two jobs of the
	81	same repository, trust and image get the same `/cache`, which is what lets them
	82	reuse each other's downloads. The tools in the image are what make that safe: `go`,
	83	`cargo` and `npm` all lock their own caches. A job that empties `/cache` or keeps
	84	state there between runs is competing with its own siblings, so give it a path of
	85	its own or its own runner.
	86
	87	An idle slot holds one open connection while it waits for work, so a machine with
	88	`--jobs 8` uses eight of the instance's SSH connections (`ssh_max_connections`).
	89	Losing a connection costs that slot its job — the job is requeued when its lease
	90	expires — and the slot reconnects rather than stopping the rest.
	91
	92	With `--once`, `--jobs 4` starts up to four jobs, finishes them, and exits, which is
	93	the shape a scheduler that starts a container per job wants.
	94
f642e0aa Keenan Nemetz 2d ago	95	## Stopping a runner
	96
	97	A runner that is asked to stop — `systemctl stop`, Ctrl-C, a `SIGTERM` — stops
	98	taking work and lets the jobs it already has finish and report. That is the
	99	difference between a build that is nearly done being published and being thrown
	100	away, and it means the run is not left unclaimed until its lease expires. A second
	101	signal ends the process at once.
	102
	103	Under systemd the packaged unit allows 60 seconds for that
	104	(`TimeoutStopSec`), so a job that runs longer than that is killed anyway. If builds
	105	on a machine take longer and you would rather wait for them, raise it:
	106
	107	```sh
	108	sudo systemctl edit vole-runner # [Service] / TimeoutStopSec=30min
	109	```
	110
338a19dd Keenan Nemetz 6d ago	111	## The runner's account
	112
	113	A runner connects as an account, and that account is the runner's identity: it is
	114	what the repository's runner allowlist names, what the audit log records, and what
	115	the server checks before it hands over a job. For a machine nobody signs in to,
	116	make the account a bot:
	117
	118	```sh
	119	vole admin account create ci-runner-1 --bot --key ci-runner-1.pub
	120	vole admin acl runners add demo ci-runner-1
	121	vole admin acl set-role demo ci-runner-1 read # private repositories only
	122	```
	123
	124	From another machine, name the instance; the key file is read where the command
	125	runs:
	126
	127	```sh
	128	vole admin --remote ssh://git@host:2222 account create ci-runner-1 --bot --key ci-runner-1.pub
	129	```
	130
	131	A bot has no browser sign-in, so its key cannot be turned into a web session, and
	132	it cannot be an instance admin. That second part matters for a runner: admins
	133	count as trusted reporters even when a repository's allowlist does not name them,
	134	so an admin account would make every result count and quietly bypass the list. A
	135	named runner needs no repository role for its result to count — the allowlist is
	136	that decision — but the build still has to clone the code, and a private
	137	repository grants no read role by default. The `set-role ... read` line covers the
	138	clone without giving the machine write access.
	139
	140	The `--key` is long-lived: it authenticates the runner process for as long as the
	141	account is active. It is not the credential a job uses. A workflow job that names
ac541b26 Keenan Nemetz 6d ago	142	a bot with `acts-as:` still gets a key minted for that run, scoped to the repository
338a19dd Keenan Nemetz 6d ago	143	and the run, and gone when the run ends; that key cannot lease work. Keep one key
	144	per runner machine, readable only by the runner user, so a host you stop trusting
	145	is revoked by removing that key or retiring the account.
	146
f5456470 Keenan Nemetz 12d ago	147	## Host key verification
	148
	149	A runner executes repository code, so the instance it trusts is a security
	150	boundary: a host that can answer in its place can send it any job. The SSH host
	151	key is verified against your known_hosts, and an unknown or changed host is
	152	refused. Populate the file once, out of band:
	153
	154	```sh
	155	ssh-keyscan -p 2222 git@host >> ~/.ssh/known_hosts
	156	vole runner --remote ssh://git@host:2222 --known-hosts ~/.ssh/vole_known_hosts
	157	```
	158
	159	- `--known-hosts <path>` checks against that file instead of the default
	160	`~/.ssh/known_hosts`.
	161	- `--insecure-host-key` turns verification off and accepts any host that answers.
	162	It exists for tests and throwaway instances; a real runner should not use it.
	163	- `VOLE_KNOWN_HOSTS` and `VOLE_INSECURE_HOST_KEY` do the same for every command,
	164	including `vole attach`, `vole admin --remote`, and the CLI verbs.
	165
	166	The policy covers git's own ssh calls as well as the runner's control channel, so
b312e166 Keenan Nemetz 11d ago	167	a clone, a fetch, a release push and the job stream all verify the same host. An
f5456470 Keenan Nemetz 12d ago	168	environment variable cannot loosen it: an explicit
	169	`GIT_SSH_COMMAND=... StrictHostKeyChecking=no` is overridden by the runner's own
	170	options, which come first.
	171
6bfc4294 Keenan Nemetz 12d ago	172	## Images and image builds
	173
	174	An `image-file` definition is built by the configured runtime, so a rootless
	175	Podman runner builds rootless. The build runs with the runtime's own network
	176	(bridge by default), because the base image and any `run:` commands need to fetch
	177	packages; if builds should have no network, preload the base image and set the
	178	runtime's default network to none on that host. The definition is pinned by its
	179	digest — the runner receives the definition text and digest with the job, tags the
	180	built image by that digest, and reuses it — so a definition that changes after a
	181	job is queued is not the one that was reviewed. Pin the base image in `from:` by
	182	digest (`image@sha256:...`) when you need the build itself to be reproducible.
	183
f5456470 Keenan Nemetz 12d ago	184	## What a job may name
	185
	186	The server validates a workflow when it is pushed, and the runner validates the
	187	job again before it acts on it, because the runner is the party that turns the
	188	payload into paths, container arguments and scripts. A job whose repository,
	189	commit, ref, branch, tag, pages path, artifact pattern, step or environment does
	190	not have the shape the instance documents is refused, and the runner stops rather
	191	than running it. In particular a path that climbs out of the checkout (`..`), an
	192	absolute path, a `:` or NUL in a value used to build a container argument, and a
	193	commit id that is not a commit id are all refused at the runner, not only at the
	194	server.
	195
6bfc4294 Keenan Nemetz 12d ago	196	## What a job may consume
	197
	198	A job runs untrusted build code, and everything it produces is stored by the
	199	instance or kept on the runner, so each of those channels has a bound:
	200
	201	- Logs. One attempt's log is capped at 32 MiB. Past that the runner keeps the
	202	first 32 MiB and appends a line saying it was truncated, so a build that loops
5ece70a5 Keenan Nemetz 10d ago	203	printing cannot fill the runner's disk or the instance's log directory. The log
	204	is written straight to a file on the runner and streamed to the server in chunks
	205	while the job runs; it is not held in memory.
6bfc4294 Keenan Nemetz 12d ago	206	- Artifacts. A workflow declares at most 32 patterns, and a run may publish at
	207	most 256 files totalling 4 GiB. The expanded set is checked before anything is
	208	uploaded; a pattern that matches a directory tree is the usual way to hit this.
	209	- Cache. `--cache-size` (5 GiB by default) caps the cache root, enforced by
a29ce1f7 Keenan Nemetz 3d ago	210	evicting whole namespaces, and a namespace a job has mounted is left for a later
a29ce1f7 Keenan Nemetz 3d ago	211	pass. Each repository gets its own namespace, and each
6bfc4294 Keenan Nemetz 12d ago	212	contributor account gets its own inside a repository, so one contributor's
6bfc4294 Keenan Nemetz 12d ago	213	branch cannot seed the cache another contributor's branch builds from. A
f594f1a1 Keenan Nemetz 5d ago	214	protected branch never shares a cache with a contributor branch. The saved
	215	checkouts jobs clone from are bounded separately; see
	216	[Saved checkouts](#saved-checkouts).
4e3759da Keenan Nemetz 4h ago	217	- Container. `--container-memory`, `--container-cpus`, `--container-pids` and
	218	`--container-tmpfs` are the runtime's own limits, per container, so `--jobs`
	219	multiplies them. `--container-pids` (2048) and `--container-tmpfs` (1G) have
	220	defaults and cannot be removed: `/tmp` is a tmpfs, so its size is memory, and a job
	221	that can fork without a bound is a fork bomb away from taking the machine. Set
	222	`--container-memory` — the runtime's default is the whole machine — and see the
	223	unit's `MemoryMax` backstop below. `--container-network` is what this machine can
	224	offer — `none` (the default) or `bridge` — and a job's own `network:` decides what
d7958024 Keenan Nemetz 7d ago	225	it gets, so a bridge runner can still run a job that asks for none. See
d7958024 Keenan Nemetz 7d ago	226	[Network](#network). The runtime's storage driver, not Vole, is what bounds the
6bfc4294 Keenan Nemetz 12d ago	227	container's writable layer, so size it on the host.
4e3759da Keenan Nemetz 4h ago	228	- Disk. A job's tree is a host directory the runtime cannot quota, so the runner
	229	keeps `--disk-reserve` (2G) free under `--dir` and stops a job that takes the
	230	filesystem below it while the job is running. It is one `statfs` per second, and the
	231	log says what was left and what the reserve is. That is the promise — the runner
	232	notices — rather than a hard bound; for one of those, put `--dir` on its own
	233	filesystem and size that. The unit's `MemoryMax` is the same idea for RAM.
6bfc4294 Keenan Nemetz 12d ago	234
70b70264 Keenan Nemetz 15d ago	235	## Images and image builds
	236
	237	Every job names its image in the workflow. A registry reference is used as
	238	written; an `image-file` definition is built locally first. The definition fields
	239	are described in the [Image reference](reference/image-reference.md).
	240
	241	For a custom definition, the runner receives the pinned definition text and its
	242	digest from the queue, so it cannot build a definition that changed after the job
	243	was queued. It renders the definition to a Dockerfile, builds it with the
	244	configured OCI runtime, and tags the result by the definition digest. A later job
	245	with the same definition reuses that tag.
	246
77056ba8 Keenan Nemetz 2d ago	247	With `--jobs`, jobs that name the same definition build it once: a job whose image
	248	another job is already building waits for that build, saying so at the top of its
	249	log, and keeps its lease alive while it waits. If the build fails, the jobs waiting
	250	on it fail with the same reason.
	251
d7958024 Keenan Nemetz 7d ago	252	The image build uses the runtime's own default network, so a base image and its
	253	`run:` commands can fetch packages. The job's own network request applies to the job
	254	container, not to the build.
70b70264 Keenan Nemetz 15d ago	255
	256	An image definition is code. Its build runs on the runner through the OCI
	257	builder, so approving a contributor tip that changes a definition also approves
	258	that build. Prefer rootless Podman, and pin the base image by digest.
	259
	260	## How a job runs
	261
c1929d2f Keenan Nemetz 5d ago	262	One container is created for the job, plus one for each service the job declares;
	263	all are removed afterward. The workflow's steps run inside the job container, but
	264	the runner process stays on the host: the SSH key, the trusted Git cache, and the
	265	finish channel are never exposed to the job. Each job gets a fresh
f594f1a1 Keenan Nemetz 5d ago	266	clone of the runner's saved copy of the repository (see
	267	[Saved checkouts](#saved-checkouts)), so a job cannot leave a Git hook or a modified
	268	checkout behind for the next one.
70b70264 Keenan Nemetz 15d ago	269
c1929d2f Keenan Nemetz 5d ago	270	A service is a second container on a private user-defined network created for the
	271	attempt. It gets the job's user, capabilities, and resource limits, but no job tree,
	272	no cache, and no secrets. Its declared name is the network alias a step connects to.
	273	The runner starts services after the image is ready and before the first step,
	274	fails the job if one is not running, and adds the end of their output to a failed
	275	attempt's log. Services, the job, and the network are removed together; a killed
	276	runner finds its leftovers by label on the next start.
	277
70b70264 Keenan Nemetz 15d ago	278	Container defaults are intentionally closed:
70b70264 Keenan Nemetz 15d ago	279
d7958024 Keenan Nemetz 7d ago	280	- no network unless the job asks for one with `network: bridge`, or the runner was
d7958024 Keenan Nemetz 7d ago	281	started with it and the job says nothing;
70b70264 Keenan Nemetz 15d ago	282	- no capabilities, `no-new-privileges`, and no host mounts except the job tree;
70b70264 Keenan Nemetz 15d ago	283	- no Docker socket, SSH agent, or host environment;
3b39a84c Keenan Nemetz 14d ago	284	- `/tmp` is a writable, executable tmpfs (compilers write their output there and
3b39a84c Keenan Nemetz 14d ago	285	run it);
70b70264 Keenan Nemetz 15d ago	286	- the steps run as the runner user's `uid:gid` by default.
70b70264 Keenan Nemetz 15d ago	287
eb07ff56 Keenan Nemetz 8d ago	288	A run started by a non-push trigger also gets a directory mounted at `/run/vole`
	289	with the pinned event as `event.json`, plus `VOLE_EVENT`, `VOLE_EVENT_ACTION`, and
	290	`VOLE_EVENT_PATH`. The file is data for a step to read; a workflow cannot replace
	291	those variables through `env:`. See [Workflow reference § Event data available to steps](reference/workflow-reference.md#event-data-available-to-steps).
	292
4ac42680 Keenan Nemetz 7d ago	293	A job that declares `secrets:` gets the same directory, with a `secrets.env` beside
	294	the event. The runner writes one `export NAME='value'` line per secret, quotes it so
	295	a value keeps its line breaks, and sources the file before each step, so a secret is
	296	a variable in the step's shell and never an argument to the runtime. Values are at
	297	most 128 KiB each, are delivered only for the job that declared them, and a name the
	298	repository does not have leaves its variable unset with one line in the log. The
	299	value arrives over the SSH channel with the lease, is written `0600` in the job's
	300	directory, and is removed with it.
	301
e82bbe7d Keenan Nemetz 7d ago	302	Every delivered value is replaced with `***` in the log as the log is written, so
	303	the file the server stores never holds one, and a value split across two writes is
	304	still caught. The runner's own error text goes through the same mask. It is best
	305	effort: a re-encoded value (base64, hex, a URL escape) no longer matches, and a
	306	value shorter than 4 bytes is left alone because masking every occurrence of it
	307	would rewrite the log. A step that writes a secret into an artifact publishes it —
	308	the mask covers the log, not the files a job chooses to upload.
	309
4ac42680 Keenan Nemetz 7d ago	310	A runner can read every secret a repository's jobs declare. That is what running
	311	the job means: the values arrive on the machine. The runner allowlist is the control
	312	that decides whose machine it is, not the container. See
	313	[Accounts and access § Roles](accounts-and-access.md#roles).
	314
3b39a84c Keenan Nemetz 14d ago	315	The container's own filesystem is writable, and thrown away with the container:
	316	nothing a job writes there survives the run, and an unprivileged uid cannot write
	317	the image's root-owned paths anyway. What protects the machine is the mount set —
	318	the job tree is the only host path a job can see — not the writability of the
	319	container's own layer.
	320
	321	Because the uid is the runner's own, an image whose tools call `getpwuid` — OpenSSH
	322	is the common case — needs a passwd entry for that uid, or `ssh` and `ssh-keygen`
	323	exit with `No user exists for uid N`. Define one in the image definition, or pass
	324	`--container-user` naming a user the image already has. This is the same container
	325	`vole workflow run` starts, so an image that fails here fails there too.
	326
	327	The uid decides how immutable the image is. An ordinary user cannot write the
	328	image's root-owned files, so a job's view of its own toolchain is the one that was
	329	built. If the runner runs as root (or `--container-user` says so), the job can
	330	rewrite its own image, and the runner prints a warning at startup: run the runner
	331	as an unprivileged user.
	332
70b70264 Keenan Nemetz 15d ago	333	Use `--container-runtime` to choose `docker`, `podman`, or `nerdctl`. The runner
	334	user must be able to talk to that runtime; rootless Podman is preferred. Do not
	335	mount the host's Docker socket into a job; that would undo the isolation.
	336
	337	`--container-pull` controls whether an image is pulled (`missing`, `always`, or
	338	`never`). `--container-user`, `--container-userns`, `--container-memory`,
	339	`--container-cpus`, and `--container-pids` override the user, namespace mode, and
	340	resource limits. Rootless Podman uses `keep-id` by default so the numeric user can
	341	write the bind-mounted job tree; for rootless Docker, choose a user that matches
	342	the runtime's mapping (container root is the runner user) or use a runtime with
	343	`keep-id`.
	344
	345	The run record names the image, the definition digest when there is one, and the
	346	runtime, so the run page says what actually executed.
	347
d7958024 Keenan Nemetz 7d ago	348	## Network
	349
	350	`--container-network` says what this machine can give a job: `none` (the default) or
	351	`bridge`. It is a capability, not a setting for every job. The job's `network:`
	352	decides what the container gets, and the runner applies it:
	353
	354	\| Job says \| This runner says `bridge` \| This runner says `none` \|
	355	\|---\|---\|---\|
	356	\| nothing \| bridge \| none \|
	357	\| `none` \| none \| none \|
	358	\| `bridge` \| bridge \| the runner never receives the job \|
	359
	360	So a machine with a network is not a machine that hands it to every build. A job
	361	that asks for none is isolated on either kind of runner, and a job that asks for
	362	bridge waits in the queue for a runner that advertises one instead of running
c1929d2f Keenan Nemetz 5d ago	363	without the network it needs. A bridge job that declares services gets a
	364	user-defined network for that attempt rather than the runtime's default bridge; the
	365	job and its services are its only members, so the service names resolve there and
	366	nowhere else. The runner advertises this with every lease, the same
d7958024 Keenan Nemetz 7d ago	367	way it advertises `--label`, and the job's request is pinned with the run when it is
	368	queued.
	369
0b9b10f6 Keenan Nemetz 6d ago	370	A runner can be started with `--container-network bridge` and nothing breaks; that
	371	is a policy choice for a machine whose owner trusts everything it builds. What no
	372	runner can do is take the choice away from the job, and this is why. A job runs code
	373	from the repository, including a contributor branch a maintainer approved for
	374	reasons that had nothing to do with network access. With a network, that code
	375	reaches whatever the machine can reach: the instance handing out the work, since a
	376	deployed instance listens beyond loopback, the other services on that host, and
	377	anything on the internet to send out what the job can read. With no network the only
	378	things it can touch are the job tree and the container itself, both of which are
	379	discarded, so a job that declares a secret and does not need to reach anything
	380	cannot send it anywhere. No network also keeps a build honest: a job that fetches
	381	its dependencies at run time builds whatever the remote serves that day, while a
	382	closed job builds what its image carries, which is the image the run pinned by
	383	digest.
	384
3b39a84c Keenan Nemetz 14d ago	385	## Caching
	386
	387	A job starts with an empty container, so without help every run downloads its
ab326d3a Keenan Nemetz 14d ago	388	dependencies and compiles them again: in this repository that is about 23 seconds
ab326d3a Keenan Nemetz 14d ago	389	of `go build`/`go vet` that a warm cache turns into one. The runner keeps a directory
3b39a84c Keenan Nemetz 14d ago	390	per job and mounts it at `/cache`.
	391
	392	It mounts the directory and sets no variable that points at it. Which tool uses
	393	`/cache`, and therefore whether a job shares state with the job before it, is the
	394	repository's decision. A runner that guessed would be making a claim about a
	395	toolchain it does not test, and a job that wants no shared state would have to
	396	defend against a default it never asked for.
	397
	398	Opt in where the environment already lives — the image definition:
	399
	400	```yaml
91e6a33a Keenan Nemetz 14d ago	401	# .vole/images/verify.yaml
3b39a84c Keenan Nemetz 14d ago	402	version: 1
	403	from: golang:1.27-bookworm
	404	env:
	405	GOMODCACHE: "/cache/go-mod"
	406	GOCACHE: "/cache/go-build"
	407	```
	408
	409	or per job, which works for a registry image too:
	410
	411	```yaml
	412	jobs:
	413	test:
	414	image: golang:1.27-bookworm
	415	env:
	416	GOCACHE: /cache/go-build
	417	GOMODCACHE: /cache/go-mod
	418	steps:
	419	- run: go test ./...
	420	```
	421
	422	The variables for the common toolchains are the tools' own; there is nothing
	423	Vole-specific to learn. `GOMODCACHE`/`GOCACHE` for Go, `CARGO_HOME`,
	424	`npm_config_cache`, `PIP_CACHE_DIR`, `MAVEN_ARGS`/`-Dmaven.repo.local`,
	425	`GRADLE_USER_HOME`, `NUGET_PACKAGES`, `PUB_CACHE`, `COMPOSER_CACHE_DIR`,
	426	`CCACHE_DIR`, and `XDG_CACHE_HOME` for anything that follows the XDG convention.
	427	An image that sets none of them gets an empty directory and no behaviour change.
	428
	429	The cache lives under `--dir/cache`, keyed by repository, trust, and image:
	430	`<dir>/cache/<repo>/<direct\|contrib>/<image>`. The parts matter. A cache is shared
	431	mutable state: a repository never sees another's, and a contributor branch gets its
	432	own namespace, so a branch nobody has vouched for cannot seed the cache a protected
e5478085 Keenan Nemetz 11d ago	433	branch builds into. A combined result derived from a contributor branch belongs to
	434	that contributor's namespace for the same reason. The image is part of the key so a
	435	cache built with one toolchain is not consumed by another.
3b39a84c Keenan Nemetz 14d ago	436
	437	Namespaces are evicted whole and least-recently-written first, so a cache is never
	438	left half-deleted — a Go module cache missing one file is a checksum error deep in
a29ce1f7 Keenan Nemetz 3d ago	439	a build rather than a download. A job holds its namespace for as long as it runs,
	440	and eviction leaves a held namespace for a later pass, so two runners sharing one
	441	`--dir` never delete each other's tree in the middle of a build. `--cache-size` caps
	442	the total (5G by default), and the runner enforces it at startup and then on its own
	443	timer: a cache that is in use when the cap is reached comes down when the job
	444	finishes, not under it. `--no-cache` keeps nothing between jobs: the directory is
	445	still mounted — an image that points at `/cache` has to work either way — but it is
	446	emptied after every job, while that job still holds it, so a shared `--dir` is not
	447	emptied under somebody else's build.
3b39a84c Keenan Nemetz 14d ago	448
f594f1a1 Keenan Nemetz 5d ago	449	## Saved checkouts
	450
	451	The first job for a repository downloads it once, and the runner keeps that copy
	452	under `--dir`. A later job clones from the copy and fetches only the commits it does
	453	not have — usually none at all — so history crosses the network once instead of once
	454	per run. The copy is never mounted into a job: the workflow gets a clone with its own
	455	`.git`, which is why a hook it writes still dies with the job.
	456
	457	Two things keep the copies from filling the disk:
	458
	459	- Repacking. A copy that is most of the cap is repacked. That drops history no
	460	branch or tag can reach any more: commits from a force-push or a deleted branch,
	461	and a commit a job once pinned by id. Nothing a job can still ask for is touched,
	462	so the next job for the same revision is still a local copy with no download.
	463	- Deleting. When the cap is still reached, the copy nobody has built from in the
	464	longest time is deleted, whole. The next job for that repository downloads it
	465	again, which is why it is the last resort rather than the first.
	466
	467	`--git-cache-size` caps all saved checkouts together (10G by default; `0` keeps
	468	every one). A copy another runner process is using is passed over, and the
	469	repository this runner is about to build is never the one deleted.
	470
70b70264 Keenan Nemetz 15d ago	471	## Who can run jobs and report results
	472
	473	A job runs repository code, so Vole uses a runner allowlist for both taking a job
	474	and reporting its result:
	475
	476	```sh
2e024cc9 Keenan Nemetz 14d ago	477	vole admin acl runners list demo # show the list
	478	vole admin acl runners set demo ci-runner-1 # replace the list
	479	vole admin acl runners add demo ci-runner-2 # add one account
	480	vole admin acl runners remove demo ci-runner-1 # remove one account
	481	vole admin acl runners clear demo # back to admins only
70b70264 Keenan Nemetz 15d ago	482	```
	483
	484	Admins always count. For everyone else, a maintainer running CI on their laptop does
	485	not turn a required check green. The result is still recorded, and the run page
	486	says whether it counts.
	487
5ece70a5 Keenan Nemetz 10d ago	488	While a job runs, the runner streams its log to the server in chunks over the same
	489	SSH channel that leased it. The server writes one file under `data/logs/` and the
	490	run page can open it before the job finishes. When the job ends, the runner sends
	491	the attempt's record with its finish message; Vole checks that the record names the
	492	account holding the lease, that it describes the job the queue handed out, and that
	493	every artifact pointer exists in the store. It then stores the attempt and the run's
	494	result in `data/runs.db` in one transaction. A tagged build that publishes release
	495	artifacts also pushes those pointers to the `vole` branch after the record is
	496	stored.
70b70264 Keenan Nemetz 15d ago	497
b312e166 Keenan Nemetz 11d ago	498	The runner did the work, so the runner reports the result. The SSH connection
b312e166 Keenan Nemetz 11d ago	499	identifies the runner, and `data/audit.log` records the decision.
70b70264 Keenan Nemetz 15d ago	500
	501	## Logs
	502
5ece70a5 Keenan Nemetz 10d ago	503	The runner streams the log while the steps run. The server appends each chunk to
	504	the attempt's file, and the run page serves that file directly with range requests,
	505	so it does not read a large log into memory. The file is complete when the attempt's
	506	record is stored.
c765ae2e Keenan Nemetz 13d ago	507
b312e166 Keenan Nemetz 11d ago	508	The stored attempt also names each step's byte range in that one log, so the run
	509	page can show a step's output without downloading the image build and other steps.
	510	Step timing is recorded beside the range.
70b70264 Keenan Nemetz 15d ago	511
70b70264 Keenan Nemetz 15d ago	512	```text
b7396a11 Keenan Nemetz 6h ago	513	/<repo>/runs/<id>/log # latest attempt
	514	/<repo>/runs/<id>/log?attempt=N # a particular retry
	515	/<repo>/runs/<id>/log?step=N # one step inside that attempt
70b70264 Keenan Nemetz 15d ago	516	```
70b70264 Keenan Nemetz 15d ago	517
92d8afc8 Keenan Nemetz 1h ago	518	The same bytes are one command away from a terminal:
	519
	520	```sh
	521	vole run logs <id> # the default attempt, whole
	522	vole run logs <id> --attempt 2 --step 1 # step 1 of the second attempt
	523	vole run show <id> # status, steps, attempts, artifacts
	524	vole run list --state failed --limit 5 # find the run to open
	525	```
	526
	527	`vole run show` and `vole run list` read the queue; `vole run logs` writes the log to
	528	stdout and sends a refusal to stderr, so a redirect gets the log or an empty file.
	529	Reading is the repository's read capability. `vole change show` lists the runs for a
	530	change's head and for its combined result, which is where the id usually comes from
	531	when a check is red.
	532
c765ae2e Keenan Nemetz 13d ago	533	If a runner is killed, its attempt has no log. A record written before step
c765ae2e Keenan Nemetz 13d ago	534	timing existed still opens; it simply has no step ranges to show.
70b70264 Keenan Nemetz 15d ago	535
	536	## Scheduling and the queue
	537
	538	All runners share one pool. An idle runner waits for work instead of asking
	539	repeatedly, and Vole gives the next job to the repository with the least work in
	540	progress. Set `max_jobs_per_repo` when you need a hard limit per repository.
	541
b7396a11 Keenan Nemetz 6h ago	542	The instance-admin Admin page at `/-/admin?tab=runners` shows who is connected,
b7396a11 Keenan Nemetz 6h ago	543	what each runner is building, and how much work is queued. The old `/-/runners` URL
43e681d8 Keenan Nemetz 4d ago	544	redirects there.
70b70264 Keenan Nemetz 15d ago	545
	546	How the queue behaves:
	547
b312e166 Keenan Nemetz 11d ago	548	- Finished runs are stored in `data/runs.db`, not on the `vole` branch. A clone or
b312e166 Keenan Nemetz 11d ago	549	backup of the repository does not carry CI history, so back up `runs.db`.
70b70264 Keenan Nemetz 15d ago	550	- A push normally reaches the queue in under a second. Every 30 seconds Vole also
	551	compares each branch and tag with the last tip it queued, so a missed
	552	notification still produces a run.
0c14f078 Keenan Nemetz 6d ago	553	- Pushing again cancels queued runs for the commit it replaced. It never stops a run
deb024d4 Keenan Nemetz 6d ago	554	that is already in progress. A maintainer can stop one from its page or with
	555	`vole run cancel <id>`: the queue records the request, the runner sees it on its next
	556	heartbeat, stops the container, and reports the partial attempt. The run stays
	557	`canceled` rather than becoming a failure, and its page says who asked.
70b70264 Keenan Nemetz 15d ago	558	- If a runner disappears, its job is requeued. After five attempts the job fails
70b70264 Keenan Nemetz 15d ago	559	instead of cycling forever.
70c9ad4e Keenan Nemetz 4h ago	560	- A job whose attempt stops before its first step — the image build, the container,
	561	the network — is recorded as `error` (the page says could not run) with the
	562	runner's reason, not as a failure of the commit. A required check waits on it
	563	rather than going red; rerunning the job clears it.
b312e166 Keenan Nemetz 11d ago	564	- Deleting `data/runs.db` loses CI history and attempts that were in flight. Current
b312e166 Keenan Nemetz 11d ago	565	branch and tag tips are queued again from Git, but only a backup can restore the
59badc63 Keenan Nemetz 11d ago	566	finished runs.