Runners

A runner is a machine that leases jobs over SSH, executes their steps in containers, and reports the result. Workflows themselves are described in Workflows; this page is about the machines that run them.

Start a runner

To have a machine set up for you — the unit, the account rootless Podman needs, and the pinned host key — see CI runners.

A runner needs an OCI runtime: Docker, rootless Podman, or nerdctl. Rootless Podman is preferred, because membership in the Docker group is effectively root on the host. Start a runner anywhere it can reach the Vole server and that runtime:

vole runner --remote ssh://git@host:2222 --container-runtime podman
vole runner --remote ssh://git@host:2222 --container-runtime podman --once

You can describe the machine and limit which repositories it serves:

vole runner --remote ssh://git@host:2222 --label linux --label arm64
vole runner --remote ssh://git@host:2222 --repos demo --repos docs

• --label says what the machine is. A job is sent only to a runner that has every label in its runs-on. A job with no runs-on can run anywhere.
• --repos narrows what the account is already allowed to run. It cannot grant access to a repository.
• --dir chooses where checkouts and caches are kept.
• --jobs is how many jobs this machine runs at once. Each gets its own container; see Running several jobs at once.
• --once leases one job and exits, which is useful for a runner that is scheduled per job.

Before it takes a job

The runtime is probed once at startup: the runner runs <runtime> info and refuses to serve if it fails, before it connects and before anything is leased. A runtime whose binary is installed but which cannot create a container — a missing OCI runtime, a run root it cannot write, a storage driver the kernel will not load — is the machine’s problem, and a runner that took a job in that state would spend an attempt and put the machine’s failure on a commit that never ran a step.

The message repeats the runtime’s own last line:

 vole runner: cannot run a container with podman: Error: default OCI runtime "crun" not found: invalid argument

The probe is one query and pulls no image. It runs once: a runtime that breaks while the runner is serving fails the job it was building, and restarting the runner probes again. As a system service the unit exits on this error, so Restart=on-failure retries it and journalctl -u vole-runner carries the message. vole workflow run makes the same check before a local run.

Running several jobs at once

A build is usually one step waiting on one thing at a time, so most of a big machine sits idle behind it. --jobs puts more jobs on the one machine:

vole runner --remote ssh://git@host:2222 --container-runtime podman --jobs 4

Each job slot is a session of its own, so the instance hands work to each one independently and the runners page shows four rows rather than one. The slots share the machine: one cache, one set of saved checkouts, one set of images. What they do not share is a container.

Two things need sizing for it:

• The container limits are per job. --container-memory, --container-cpus and --container-pids apply to each container, so a machine running four jobs needs four times what they allow. On the instance, max_jobs_per_repo is what stops one repository’s burst from claiming every slot (see Configuration).
• A job’s cache is shared with the same repository’s other jobs. Two jobs of the same repository, trust and image get the same /cache, which is what lets them reuse each other’s downloads. The tools in the image are what make that safe: go, cargo and npm all lock their own caches. A job that empties /cache or keeps state there between runs is competing with its own siblings, so give it a path of its own or its own runner.

An idle slot holds one open connection while it waits for work, so a machine with --jobs 8 uses eight of the instance’s SSH connections (ssh_max_connections). Losing a connection costs that slot its job — the job is requeued when its lease expires — and the slot reconnects rather than stopping the rest.

With --once, --jobs 4 starts up to four jobs, finishes them, and exits, which is the shape a scheduler that starts a container per job wants.

Stopping a runner

A runner that is asked to stop — systemctl stop, Ctrl-C, a SIGTERM — stops taking work and lets the jobs it already has finish and report. That is the difference between a build that is nearly done being published and being thrown away, and it means the run is not left unclaimed until its lease expires. A second signal ends the process at once.

Under systemd the packaged unit allows 60 seconds for that (TimeoutStopSec), so a job that runs longer than that is killed anyway. If builds on a machine take longer and you would rather wait for them, raise it:

sudo systemctl edit vole-runner    # [Service] / TimeoutStopSec=30min

The runner’s account

A runner connects as an account, and that account is the runner’s identity: it is what the repository’s runner allowlist names, what the audit log records, and what the server checks before it hands over a job. For a machine nobody signs in to, make the account a bot:

vole admin account create ci-runner-1 --bot --key ci-runner-1.pub
vole admin acl runners add demo ci-runner-1
vole admin acl set-role demo ci-runner-1 read   # private repositories only

From another machine, name the instance; the key file is read where the command runs:

vole admin --remote ssh://git@host:2222 account create ci-runner-1 --bot --key ci-runner-1.pub

A bot has no browser sign-in, so its key cannot be turned into a web session, and it cannot be an instance admin. That second part matters for a runner: admins count as trusted reporters even when a repository’s allowlist does not name them, so an admin account would make every result count and quietly bypass the list. A named runner needs no repository role for its result to count — the allowlist is that decision — but the build still has to clone the code, and a private repository grants no read role by default. The set-role ... read line covers the clone without giving the machine write access.

The --key is long-lived: it authenticates the runner process for as long as the account is active. It is not the credential a job uses. A workflow job that names a bot with acts-as: still gets a key minted for that run, scoped to the repository and the run, and gone when the run ends; that key cannot lease work. Keep one key per runner machine, readable only by the runner user, so a host you stop trusting is revoked by removing that key or retiring the account.

Host key verification

A runner executes repository code, so the instance it trusts is a security boundary: a host that can answer in its place can send it any job. The SSH host key is verified against your known_hosts, and an unknown or changed host is refused. Populate the file once, out of band:

ssh-keyscan -p 2222 git@host >> ~/.ssh/known_hosts
vole runner --remote ssh://git@host:2222 --known-hosts ~/.ssh/vole_known_hosts

• --known-hosts <path> checks against that file instead of the default ~/.ssh/known_hosts.
• --insecure-host-key turns verification off and accepts any host that answers. It exists for tests and throwaway instances; a real runner should not use it.
• VOLE_KNOWN_HOSTS and VOLE_INSECURE_HOST_KEY do the same for every command, including vole attach, vole admin --remote, and the CLI verbs.

The policy covers git’s own ssh calls as well as the runner’s control channel, so a clone, a fetch, a release push and the job stream all verify the same host. An environment variable cannot loosen it: an explicit GIT_SSH_COMMAND=... StrictHostKeyChecking=no is overridden by the runner’s own options, which come first.

Images and image builds

An image-file definition is built by the configured runtime, so a rootless Podman runner builds rootless. The build runs with the runtime’s own network (bridge by default), because the base image and any run: commands need to fetch packages; if builds should have no network, preload the base image and set the runtime’s default network to none on that host. The definition is pinned by its digest — the runner receives the definition text and digest with the job, tags the built image by that digest, and reuses it — so a definition that changes after a job is queued is not the one that was reviewed. Pin the base image in from: by digest (image@sha256:...) when you need the build itself to be reproducible.

What a job may name

The server validates a workflow when it is pushed, and the runner validates the job again before it acts on it, because the runner is the party that turns the payload into paths, container arguments and scripts. A job whose repository, commit, ref, branch, tag, pages path, artifact pattern, step or environment does not have the shape the instance documents is refused, and the runner stops rather than running it. In particular a path that climbs out of the checkout (..), an absolute path, a : or NUL in a value used to build a container argument, and a commit id that is not a commit id are all refused at the runner, not only at the server.

What a job may consume

A job runs untrusted build code, and everything it produces is stored by the instance or kept on the runner, so each of those channels has a bound:

• Logs. One attempt’s log is capped at 32 MiB. Past that the runner keeps the first 32 MiB and appends a line saying it was truncated, so a build that loops printing cannot fill the runner’s disk or the instance’s log directory. The log is written straight to a file on the runner and streamed to the server in chunks while the job runs; it is not held in memory.
• Artifacts. A workflow declares at most 32 patterns, and a run may publish at most 256 files totalling 4 GiB. The expanded set is checked before anything is uploaded; a pattern that matches a directory tree is the usual way to hit this.
• Cache. --cache-size (5 GiB by default) caps the cache root, enforced by evicting whole namespaces, and a namespace a job has mounted is left for a later pass. Each repository gets its own namespace, and each contributor account gets its own inside a repository, so one contributor’s branch cannot seed the cache another contributor’s branch builds from. A protected branch never shares a cache with a contributor branch. The saved checkouts jobs clone from are bounded separately; see Saved checkouts.
• Container. --container-memory, --container-cpus, --container-pids and --container-tmpfs are the runtime’s own limits, per container, so --jobs multiplies them. --container-pids (2048) and --container-tmpfs (1G) have defaults and cannot be removed: /tmp is a tmpfs, so its size is memory, and a job that can fork without a bound is a fork bomb away from taking the machine. Set --container-memory — the runtime’s default is the whole machine — and see the unit’s MemoryMax backstop below. --container-network is what this machine can offer — none (the default) or bridge — and a job’s own network: decides what it gets, so a bridge runner can still run a job that asks for none. See Network. The runtime’s storage driver, not Vole, is what bounds the container’s writable layer, so size it on the host.
• Disk. A job’s tree is a host directory the runtime cannot quota, so the runner keeps --disk-reserve (2G) free under --dir and stops a job that takes the filesystem below it while the job is running. It is one statfs per second, and the log says what was left and what the reserve is. That is the promise — the runner notices — rather than a hard bound; for one of those, put --dir on its own filesystem and size that. The unit’s MemoryMax is the same idea for RAM.

Images and image builds

Every job names its image in the workflow. A registry reference is used as written; an image-file definition is built locally first. The definition fields are described in the Image reference.

For a custom definition, the runner receives the pinned definition text and its digest from the queue, so it cannot build a definition that changed after the job was queued. It renders the definition to a Dockerfile, builds it with the configured OCI runtime, and tags the result by the definition digest. A later job with the same definition reuses that tag.

With --jobs, jobs that name the same definition build it once: a job whose image another job is already building waits for that build, saying so at the top of its log, and keeps its lease alive while it waits. If the build fails, the jobs waiting on it fail with the same reason.

The image build uses the runtime’s own default network, so a base image and its run: commands can fetch packages. The job’s own network request applies to the job container, not to the build.

An image definition is code. Its build runs on the runner through the OCI builder, so approving a contributor tip that changes a definition also approves that build. Prefer rootless Podman, and pin the base image by digest.

How a job runs

One container is created for the job, plus one for each service the job declares; all are removed afterward. The workflow’s steps run inside the job container, but the runner process stays on the host: the SSH key, the trusted Git cache, and the finish channel are never exposed to the job. Each job gets a fresh clone of the runner’s saved copy of the repository (see Saved checkouts), so a job cannot leave a Git hook or a modified checkout behind for the next one.

A service is a second container on a private user-defined network created for the attempt. It gets the job’s user, capabilities, and resource limits, but no job tree, no cache, and no secrets. Its declared name is the network alias a step connects to. The runner starts services after the image is ready and before the first step, fails the job if one is not running, and adds the end of their output to a failed attempt’s log. Services, the job, and the network are removed together; a killed runner finds its leftovers by label on the next start.

Container defaults are intentionally closed:

• no network unless the job asks for one with network: bridge, or the runner was started with it and the job says nothing;
• no capabilities, no-new-privileges, and no host mounts except the job tree;
• no Docker socket, SSH agent, or host environment;
• /tmp is a writable, executable tmpfs (compilers write their output there and run it);
• the steps run as the runner user’s uid:gid by default.

A run started by a non-push trigger also gets a directory mounted at /run/vole with the pinned event as event.json, plus VOLE_EVENT, VOLE_EVENT_ACTION, and VOLE_EVENT_PATH. The file is data for a step to read; a workflow cannot replace those variables through env:. See Workflow reference § Event data available to steps.

A job that declares secrets: gets the same directory, with a secrets.env beside the event. The runner writes one export NAME='value' line per secret, quotes it so a value keeps its line breaks, and sources the file before each step, so a secret is a variable in the step’s shell and never an argument to the runtime. Values are at most 128 KiB each, are delivered only for the job that declared them, and a name the repository does not have leaves its variable unset with one line in the log. The value arrives over the SSH channel with the lease, is written 0600 in the job’s directory, and is removed with it.

Every delivered value is replaced with *** in the log as the log is written, so the file the server stores never holds one, and a value split across two writes is still caught. The runner’s own error text goes through the same mask. It is best effort: a re-encoded value (base64, hex, a URL escape) no longer matches, and a value shorter than 4 bytes is left alone because masking every occurrence of it would rewrite the log. A step that writes a secret into an artifact publishes it — the mask covers the log, not the files a job chooses to upload.

A runner can read every secret a repository’s jobs declare. That is what running the job means: the values arrive on the machine. The runner allowlist is the control that decides whose machine it is, not the container. See Accounts and access § Roles.

The container’s own filesystem is writable, and thrown away with the container: nothing a job writes there survives the run, and an unprivileged uid cannot write the image’s root-owned paths anyway. What protects the machine is the mount set — the job tree is the only host path a job can see — not the writability of the container’s own layer.

Because the uid is the runner’s own, an image whose tools call getpwuid — OpenSSH is the common case — needs a passwd entry for that uid, or ssh and ssh-keygen exit with No user exists for uid N. Define one in the image definition, or pass --container-user naming a user the image already has. This is the same container vole workflow run starts, so an image that fails here fails there too.

The uid decides how immutable the image is. An ordinary user cannot write the image’s root-owned files, so a job’s view of its own toolchain is the one that was built. If the runner runs as root (or --container-user says so), the job can rewrite its own image, and the runner prints a warning at startup: run the runner as an unprivileged user.

Use --container-runtime to choose docker, podman, or nerdctl. The runner user must be able to talk to that runtime; rootless Podman is preferred. Do not mount the host’s Docker socket into a job; that would undo the isolation.

--container-pull controls whether an image is pulled (missing, always, or never). --container-user, --container-userns, --container-memory, --container-cpus, and --container-pids override the user, namespace mode, and resource limits. Rootless Podman uses keep-id by default so the numeric user can write the bind-mounted job tree; for rootless Docker, choose a user that matches the runtime’s mapping (container root is the runner user) or use a runtime with keep-id.

The run record names the image, the definition digest when there is one, and the runtime, so the run page says what actually executed.

Network

--container-network says what this machine can give a job: none (the default) or bridge. It is a capability, not a setting for every job. The job’s network: decides what the container gets, and the runner applies it:

Job says	This runner says bridge	This runner says none
nothing	bridge	none
none	none	none
bridge	bridge	the runner never receives the job

So a machine with a network is not a machine that hands it to every build. A job that asks for none is isolated on either kind of runner, and a job that asks for bridge waits in the queue for a runner that advertises one instead of running without the network it needs. A bridge job that declares services gets a user-defined network for that attempt rather than the runtime’s default bridge; the job and its services are its only members, so the service names resolve there and nowhere else. The runner advertises this with every lease, the same way it advertises --label, and the job’s request is pinned with the run when it is queued.

A runner can be started with --container-network bridge and nothing breaks; that is a policy choice for a machine whose owner trusts everything it builds. What no runner can do is take the choice away from the job, and this is why. A job runs code from the repository, including a contributor branch a maintainer approved for reasons that had nothing to do with network access. With a network, that code reaches whatever the machine can reach: the instance handing out the work, since a deployed instance listens beyond loopback, the other services on that host, and anything on the internet to send out what the job can read. With no network the only things it can touch are the job tree and the container itself, both of which are discarded, so a job that declares a secret and does not need to reach anything cannot send it anywhere. No network also keeps a build honest: a job that fetches its dependencies at run time builds whatever the remote serves that day, while a closed job builds what its image carries, which is the image the run pinned by digest.

Caching

A job starts with an empty container, so without help every run downloads its dependencies and compiles them again: in this repository that is about 23 seconds of go build/go vet that a warm cache turns into one. The runner keeps a directory per job and mounts it at /cache.

It mounts the directory and sets no variable that points at it. Which tool uses /cache, and therefore whether a job shares state with the job before it, is the repository’s decision. A runner that guessed would be making a claim about a toolchain it does not test, and a job that wants no shared state would have to defend against a default it never asked for.

Opt in where the environment already lives — the image definition:

# .vole/images/verify.yaml
version: 1
from: golang:1.27-bookworm
env:
  GOMODCACHE: "/cache/go-mod"
  GOCACHE: "/cache/go-build"

or per job, which works for a registry image too:

jobs:
  test:
    image: golang:1.27-bookworm
    env:
      GOCACHE: /cache/go-build
      GOMODCACHE: /cache/go-mod
    steps:
      - run: go test ./...

The variables for the common toolchains are the tools’ own; there is nothing Vole-specific to learn. GOMODCACHE/GOCACHE for Go, CARGO_HOME, npm_config_cache, PIP_CACHE_DIR, MAVEN_ARGS/-Dmaven.repo.local, GRADLE_USER_HOME, NUGET_PACKAGES, PUB_CACHE, COMPOSER_CACHE_DIR, CCACHE_DIR, and XDG_CACHE_HOME for anything that follows the XDG convention. An image that sets none of them gets an empty directory and no behaviour change.

The cache lives under --dir/cache, keyed by repository, trust, and image: <dir>/cache/<repo>/<direct|contrib>/<image>. The parts matter. A cache is shared mutable state: a repository never sees another’s, and a contributor branch gets its own namespace, so a branch nobody has vouched for cannot seed the cache a protected branch builds into. A combined result derived from a contributor branch belongs to that contributor’s namespace for the same reason. The image is part of the key so a cache built with one toolchain is not consumed by another.

Namespaces are evicted whole and least-recently-written first, so a cache is never left half-deleted — a Go module cache missing one file is a checksum error deep in a build rather than a download. A job holds its namespace for as long as it runs, and eviction leaves a held namespace for a later pass, so two runners sharing one --dir never delete each other’s tree in the middle of a build. --cache-size caps the total (5G by default), and the runner enforces it at startup and then on its own timer: a cache that is in use when the cap is reached comes down when the job finishes, not under it. --no-cache keeps nothing between jobs: the directory is still mounted — an image that points at /cache has to work either way — but it is emptied after every job, while that job still holds it, so a shared --dir is not emptied under somebody else’s build.

Saved checkouts

The first job for a repository downloads it once, and the runner keeps that copy under --dir. A later job clones from the copy and fetches only the commits it does not have — usually none at all — so history crosses the network once instead of once per run. The copy is never mounted into a job: the workflow gets a clone with its own .git, which is why a hook it writes still dies with the job.

Two things keep the copies from filling the disk:

• Repacking. A copy that is most of the cap is repacked. That drops history no branch or tag can reach any more: commits from a force-push or a deleted branch, and a commit a job once pinned by id. Nothing a job can still ask for is touched, so the next job for the same revision is still a local copy with no download.
• Deleting. When the cap is still reached, the copy nobody has built from in the longest time is deleted, whole. The next job for that repository downloads it again, which is why it is the last resort rather than the first.

--git-cache-size caps all saved checkouts together (10G by default; 0 keeps every one). A copy another runner process is using is passed over, and the repository this runner is about to build is never the one deleted.

Who can run jobs and report results

A job runs repository code, so Vole uses a runner allowlist for both taking a job and reporting its result:

vole admin acl runners list demo                           # show the list
vole admin acl runners set demo ci-runner-1                # replace the list
vole admin acl runners add demo ci-runner-2                # add one account
vole admin acl runners remove demo ci-runner-1             # remove one account
vole admin acl runners clear demo                          # back to admins only

Admins always count. For everyone else, a maintainer running CI on their laptop does not turn a required check green. The result is still recorded, and the run page says whether it counts.

While a job runs, the runner streams its log to the server in chunks over the same SSH channel that leased it. The server writes one file under data/logs/ and the run page can open it before the job finishes. When the job ends, the runner sends the attempt’s record with its finish message; Vole checks that the record names the account holding the lease, that it describes the job the queue handed out, and that every artifact pointer exists in the store. It then stores the attempt and the run’s result in data/runs.db in one transaction. A tagged build that publishes release artifacts also pushes those pointers to the vole branch after the record is stored.

The runner did the work, so the runner reports the result. The SSH connection identifies the runner, and data/audit.log records the decision.

Logs

The runner streams the log while the steps run. The server appends each chunk to the attempt’s file, and the run page serves that file directly with range requests, so it does not read a large log into memory. The file is complete when the attempt’s record is stored.

The stored attempt also names each step’s byte range in that one log, so the run page can show a step’s output without downloading the image build and other steps. Step timing is recorded beside the range.

/<repo>/runs/<id>/log             # latest attempt
/<repo>/runs/<id>/log?attempt=N   # a particular retry
/<repo>/runs/<id>/log?step=N      # one step inside that attempt

If a runner is killed, its attempt has no log. A record written before step timing existed still opens; it simply has no step ranges to show.

Scheduling and the queue

All runners share one pool. An idle runner waits for work instead of asking repeatedly, and Vole gives the next job to the repository with the least work in progress. Set max_jobs_per_repo when you need a hard limit per repository.

The instance-admin Admin page at /-/admin?tab=runners shows who is connected, what each runner is building, and how much work is queued. The old /-/runners URL redirects there.

How the queue behaves:

• Finished runs are stored in data/runs.db, not on the vole branch. A clone or backup of the repository does not carry CI history, so back up runs.db.
• A push normally reaches the queue in under a second. Every 30 seconds Vole also compares each branch and tag with the last tip it queued, so a missed notification still produces a run.
• Pushing again cancels queued runs for the commit it replaced. It never stops a run that is already in progress. A maintainer can stop one from its page or with vole run cancel <id>: the queue records the request, the runner sees it on its next heartbeat, stops the container, and reports the partial attempt. The run stays canceled rather than becoming a failure, and its page says who asked.
• If a runner disappears, its job is requeued. After five attempts the job fails instead of cycling forever.
• A job whose attempt stops before its first step — the image build, the container, the network — is recorded as error (the page says could not run) with the runner’s reason, not as a failure of the commit. A required check waits on it rather than going red; rerunning the job clears it.
• Deleting data/runs.db loses CI history and attempts that were in flight. Current branch and tag tips are queued again from Git, but only a backup can restore the finished runs.