CI runners

A runner is a machine that leases jobs from your instance over SSH, runs each job in a container, and reports the result. It is a second machine, never the forge’s own: it executes repository code and can read every secret its jobs declare, so it should not share a disk with the account database and the SSH host key.

Give it its own machine the same way you created the forge — a rendered first-boot file pasted into the console — or set one up on a machine you already have. Either way there is one step only an admin can do, and it is per repository.

Create a runner machine

Create a server the way the Hetzner, DigitalOcean, or AWS guide describes, with three differences: the runner’s file instead of the forge’s, no inbound ports except 22, and a disk sized for container images.

cd deploy/cloud-init
FORGE_HOST=forge.example.com \
VOLE_PACKAGE_URL=https://rodent.software/vole/releases/v0.2.0/artifacts/vole_0.2.0-1_amd64.deb \
  ./render.sh runner > runner-user-data.yaml

render.sh runner also reads RUNNER_LABEL_NAMES (default linux), RUNNER_NETWORK (default bridge), RUNNER_RUNTIME (default podman), RUNNER_JOBS (default 1, how many jobs run at once), RUNNER_HOSTNAME, RUNNER_MEMORY (default 4G, what one job may take from RAM), RUNNER_PIDS (default 2048), RUNNER_TMPFS (default 1G, the job’s /tmp, which is a tmpfs and therefore memory), and RUNNER_DISK_RESERVE (default 2G, free space kept under RUNNER_DIR).

Size the machine for RUNNER_JOBS times RUNNER_MEMORY, and keep that below the unit’s MemoryMax=90% backstop, which bounds the whole runner if a job gets past its own cgroup. RUNNER_DISK_RESERVE is what the machine keeps for itself: a job that takes the filesystem under RUNNER_DIR below it is stopped while it runs, and the job’s log says so. See Runners § What a job may consume for what each limit means, and Running several jobs at once for RUNNER_JOBS. The file installs the package, creates an unprivileged ci-runner account with the subordinate ids rootless Podman needs, generates the runner’s key, pins the instance’s host key once, and enables vole-runner.service.

If you would rather do it by hand on a machine that already exists, the steps are below.

On a machine you already have

A runner needs an OCI runtime — rootless Podman is preferred, because membership in the Docker group is effectively root on the host — and it needs to reach the instance’s SSH port.

On a machine you configure yourself, the package carries the unit, so it is three steps: install it, set up the account the runtime needs, and fill in the arguments.

sudo apt install ./vole_0.2.0-1_amd64.deb          # dnf on Fedora and RHEL
sudo useradd --create-home --home-dir /var/lib/vole-runner --shell /usr/sbin/nologin ci-runner
sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 ci-runner
sudo apt install podman slirp4netns fuse-overlayfs uidmap
sudo install -d -o ci-runner -g ci-runner -m 0700 /var/lib/vole-runner/.ssh
sudo -u ci-runner ssh-keygen -q -t ed25519 -N '' -f /var/lib/vole-runner/.ssh/id_ed25519

# Pin the instance's host key. A runner executes repository code, so a host that
# can answer in the instance's place could send it any job.
ssh-keyscan -p 2222 git@forge.example.com | sudo tee /etc/vole/runner_known_hosts

Then put the arguments in /etc/vole/runner.env. The package ships an example with each option explained:

sudo tee /etc/vole/runner.env >/dev/null <<'EOF'
VOLE_RUNNER_ARGS=--remote ssh://git@forge.example.com:2222 --container-runtime podman --container-network bridge --container-memory 4G --container-tmpfs 1G --disk-reserve 2G --dir /var/lib/vole-runner --known-hosts /etc/vole/runner_known_hosts --label linux
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now vole-runner

Add --jobs N to that line to run several jobs at once on a machine with room for them, and raise --container-memory with it: the memory limit is per job, so a machine running two jobs at 4G needs room for 8G below the unit’s MemoryMax backstop.

systemctl stop vole-runner stops taking work and lets the jobs in flight finish, up to the unit’s TimeoutStopSec (60 seconds). Raise it in a drop-in if this machine’s builds run longer and you would rather wait for them:

sudo systemctl edit vole-runner    # [Service] / TimeoutStopSec=30min

systemctl status vole-runner and journalctl -u vole-runner are where a runner that is not taking work says why.

What the package configures in the container runtime

A rootless runtime needs two settings the distributions do not agree on, so the package installs both as drop-ins. Each merges with the distribution’s own file instead of replacing it, and both reach the runner’s account: system configuration is read for rootless users too, so nothing has to be written into a home directory.

A site that wants a different registry list writes its own drop-in with a higher number, such as 99-site.conf, which is read after these.

The unit also runs with NoNewPrivileges=false, alone among the units this package installs. Rootless podman, Docker, and nerdctl all map the calling user into the container through setuid newuidmap and newgidmap; under NoNewPrivileges=true a setuid helper cannot gain privilege, so the mapping fails before a container exists (newuidmap: write to uid_map failed: Operation not permitted). The isolation a job gets is its container, not the unit’s sandbox, so the rest of the hardening stays.

If you would rather give podman the systemd user manager it prefers, run the runner as a user unit instead: loginctl enable-linger ci-runner, a unit in ci-runner’s own systemctl --user session, and no cgroup_manager setting. That session lives at /run/user/<uid>, so the unit’s RuntimeDirectory=vole-runner and XDG_RUNTIME_DIR=/run/vole-runner have to go with it — the two arrangements cannot both be assumed, and this one is not what the shipped unit does.

Register the runner, then allow it on a repository

The runner’s key is not an admin key. It becomes a bot account, and the allowlist decides what it may build:

# The public key is on the runner; this copies it to the instance as a new bot.
ssh root@<runner-address> cat /var/lib/vole-runner/.ssh/id_ed25519.pub \
  | vole admin --remote ssh://git@forge.example.com:2222 \
      account create ci-runner-1 --bot --key -

# Per repository: let it take jobs, and give it read so it can clone.
vole admin --remote ssh://git@forge.example.com:2222 acl runners add demo ci-runner-1
vole admin --remote ssh://git@forge.example.com:2222 acl set-role demo ci-runner-1 read

This is the part that cannot happen at first boot: repositories do not exist yet, and the allowlist is per repository. Add each new repository to it as you create it, or the runner will sit idle while its jobs wait in the queue.

A bot cannot sign in to the browser and cannot be an instance admin. That second part matters: admins count as trusted reporters, so a bot that was an admin would make every result count whatever the allowlist said.

Check that it works

• The Admin page’s Runners tab (/-/admin?tab=runners) shows what is connected, what each runner is building, and how much is queued.
• Push a commit on a branch with a workflow. The run page shows which runner took it and whether the result counts.
• A run that executed on a runner outside the allowlist still exists and still says so; it just does not count for the branch.

What to know before you give it a repository

A runner can read every secret its jobs declare. That is what running the job means: the values arrive on the machine. If the machine is not yours, the secrets are not either. This is also why an instance admin’s machine should not be a runner for a repository whose secrets it should not see.

The allowlist is the control, not the container. A job that escapes its container finds a machine that can reach the instance and the internet. Keep the runner on its own machine, run it as an unprivileged account, and give it no credentials except the one key.

Network is a per-job decision this machine permits. A bridge runner still gives a job that asks for none no network. What it cannot do is take the choice away from a job, so a workflow you approve for a contributor branch is a workflow that may reach the internet with whatever it can read.

Disk fills where builds happen. Container images, the /cache mount, and the saved checkouts live under --dir. --cache-size (5 GiB) and --git-cache-size (10 GiB) bound the last two; the images are the runtime’s own storage, so size the machine for them.

Upgrading

The runner is the same package as the instance, and the two talk over a versioned protocol. Upgrade the instance first, then the runner machines:

sudo apt install ./vole-<new-version>.deb
sudo systemctl restart vole-runner

Next

• Runners reference — every option, the container defaults, caching, and how a lease is reported
• Workflows — what a job asks a runner for, including runs-on