Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Blame · docs/deploy/runner.md

Line attribution on main · 196 line(s)

View file History
e21f6c5e
Keenan Nemetz
3d ago
 1 # CI runners
2
3 A runner is a machine that leases jobs from your instance over SSH, runs each job
942c93cd
Keenan Nemetz
3d ago
 4 in a container, and reports the result. It is a second machine, never the forge's
5 own: it executes repository code and can read every secret its jobs declare, so it
6 should not share a disk with the account database and the SSH host key.
e21f6c5e
Keenan Nemetz
3d ago
 7
942c93cd
Keenan Nemetz
3d ago
 8 Give it its own machine the same way you created the forge — a rendered first-boot
9 file pasted into the console — or set one up on a machine you already have. Either
10 way there is one step only an admin can do, and it is per repository.
e21f6c5e
Keenan Nemetz
3d ago
 11
942c93cd
Keenan Nemetz
3d ago
 12 ## Create a runner machine
e21f6c5e
Keenan Nemetz
3d ago
 13
942c93cd
Keenan Nemetz
3d ago
 14 Create a server the way the [Hetzner](hetzner.md), [DigitalOcean](digitalocean.md),
15 or [AWS](aws.md) guide describes, with three differences: the runner's file instead
16 of the forge's, no inbound ports except 22, and a disk sized for container images.
e21f6c5e
Keenan Nemetz
3d ago
 17
18 ```sh
942c93cd
Keenan Nemetz
3d ago
 19 cd deploy/cloud-init
20 FORGE_HOST=forge.example.com \
b7396a11
Keenan Nemetz
2h ago
 21 VOLE_PACKAGE_URL=https://rodent.software/vole/releases/v0.2.0/artifacts/vole_0.2.0-1_amd64.deb \
942c93cd
Keenan Nemetz
3d ago
 22 ./render.sh runner > runner-user-data.yaml
e21f6c5e
Keenan Nemetz
3d ago
 23 ```
24
942c93cd
Keenan Nemetz
3d ago
 25 `render.sh runner` also reads `RUNNER_LABEL_NAMES` (default `linux`),
18c559ea
Keenan Nemetz
2d ago
 26 `RUNNER_NETWORK` (default `bridge`), `RUNNER_RUNTIME` (default `podman`),
4e3759da
Keenan Nemetz
42m ago
 27 `RUNNER_JOBS` (default `1`, how many jobs run at once), `RUNNER_HOSTNAME`,
28 `RUNNER_MEMORY` (default `4G`, what one job may take from RAM), `RUNNER_PIDS`
29 (default `2048`), `RUNNER_TMPFS` (default `1G`, the job's `/tmp`, which is a tmpfs
30 and therefore memory), and `RUNNER_DISK_RESERVE` (default `2G`, free space kept under
31 `RUNNER_DIR`).
32
33 Size the machine for `RUNNER_JOBS` times `RUNNER_MEMORY`, and keep that below the
34 unit's `MemoryMax=90%` backstop, which bounds the whole runner if a job gets past its
35 own cgroup. `RUNNER_DISK_RESERVE` is what the machine keeps for itself: a job that
36 takes the filesystem under `RUNNER_DIR` below it is stopped while it runs, and the
37 job's log says so. See
38 [Runners § What a job may consume](../runner.md#what-a-job-may-consume) for what each
39 limit means, and [Running several jobs at once](../runner.md#running-several-jobs-at-once)
40 for `RUNNER_JOBS`.
18c559ea
Keenan Nemetz
2d ago
 41 The file installs the package, creates an unprivileged
942c93cd
Keenan Nemetz
3d ago
 42 `ci-runner` account with the subordinate ids rootless Podman needs, generates the
43 runner's key, pins the instance's host key once, and enables `vole-runner.service`.
e21f6c5e
Keenan Nemetz
3d ago
 44
942c93cd
Keenan Nemetz
3d ago
 45 If you would rather do it by hand on a machine that already exists, the steps are
46 below.
e21f6c5e
Keenan Nemetz
3d ago
 47
48 ## On a machine you already have
49
50 A runner needs an OCI runtime — rootless Podman is preferred, because membership
51 in the Docker group is effectively root on the host — and it needs to reach the
52 instance's SSH port.
53
54 On a machine you configure yourself, the package carries the unit, so it is three
55 steps: install it, set up the account the runtime needs, and fill in the
56 arguments.
57
58 ```sh
59 sudo apt install ./vole_0.2.0-1_amd64.deb # dnf on Fedora and RHEL
60 sudo useradd --create-home --home-dir /var/lib/vole-runner --shell /usr/sbin/nologin ci-runner
61 sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 ci-runner
62 sudo apt install podman slirp4netns fuse-overlayfs uidmap
63 sudo install -d -o ci-runner -g ci-runner -m 0700 /var/lib/vole-runner/.ssh
64 sudo -u ci-runner ssh-keygen -q -t ed25519 -N '' -f /var/lib/vole-runner/.ssh/id_ed25519
65
66 # Pin the instance's host key. A runner executes repository code, so a host that
67 # can answer in the instance's place could send it any job.
68 ssh-keyscan -p 2222 git@forge.example.com | sudo tee /etc/vole/runner_known_hosts
69 ```
70
71 Then put the arguments in `/etc/vole/runner.env`. The package ships an example
72 with each option explained:
73
74 ```sh
75 sudo tee /etc/vole/runner.env >/dev/null <<'EOF'
4e3759da
Keenan Nemetz
42m ago
 76 VOLE_RUNNER_ARGS=--remote ssh://git@forge.example.com:2222 --container-runtime podman --container-network bridge --container-memory 4G --container-tmpfs 1G --disk-reserve 2G --dir /var/lib/vole-runner --known-hosts /etc/vole/runner_known_hosts --label linux
e21f6c5e
Keenan Nemetz
3d ago
 77 EOF
78 sudo systemctl daemon-reload
79 sudo systemctl enable --now vole-runner
80 ```
81
18c559ea
Keenan Nemetz
2d ago
 82 Add `--jobs N` to that line to run several jobs at once on a machine with room for
4e3759da
Keenan Nemetz
42m ago
 83 them, and raise `--container-memory` with it: the memory limit is per job, so a machine
84 running two jobs at `4G` needs room for `8G` below the unit's `MemoryMax` backstop.
18c559ea
Keenan Nemetz
2d ago
 85
f642e0aa
Keenan Nemetz
2d ago
 86 `systemctl stop vole-runner` stops taking work and lets the jobs in flight finish,
87 up to the unit's `TimeoutStopSec` (60 seconds). Raise it in a drop-in if this
88 machine's builds run longer and you would rather wait for them:
89
90 ```sh
91 sudo systemctl edit vole-runner # [Service] / TimeoutStopSec=30min
92 ```
93
e21f6c5e
Keenan Nemetz
3d ago
 94 `systemctl status vole-runner` and `journalctl -u vole-runner` are where a runner
95 that is not taking work says why.
96
9704d735
Keenan Nemetz
1h ago
 97 ## What the package configures in the container runtime
98
99 A rootless runtime needs two settings the distributions do not agree on, so the
100 package installs both as drop-ins. Each merges with the distribution's own file
101 instead of replacing it, and both reach the runner's account: system configuration
102 is read for rootless users too, so nothing has to be written into a home directory.
103
104 | File | What it sets, and why |
105 |---|---|
106 | `/etc/containers/registries.conf.d/50-vole-runner.conf` | `unqualified-search-registries = ["docker.io"]`. Debian and Ubuntu ship `registries.conf` entirely commented out, so a base image named as a short name has no registry to resolve against and the build fails with `short-name … did not resolve to an alias and no unqualified-search registries are defined`. The digest in `.vole/images` is the pin; this list only says where to ask. |
107 | `/etc/containers/containers.conf.d/50-vole-runner.conf` | `[engine] cgroup_manager = "cgroupfs"`. `vole-runner.service` is a system service, so the runner's account has no systemd user manager for podman to ask for a cgroup scope. It falls back to sd-bus, which answers `Interactive authentication required`, and the first `RUN` step of an image build fails. `cgroupfs` writes into the cgroup subtree `Delegate=yes` already gives the service. |
108
109 A site that wants a different registry list writes its own drop-in with a higher
110 number, such as `99-site.conf`, which is read after these.
111
112 The unit also runs with `NoNewPrivileges=false`, alone among the units this package
113 installs. Rootless podman, Docker, and nerdctl all map the calling user into the
114 container through setuid `newuidmap` and `newgidmap`; under `NoNewPrivileges=true`
115 a setuid helper cannot gain privilege, so the mapping fails before a container
116 exists (`newuidmap: write to uid_map failed: Operation not permitted`). The
117 isolation a job gets is its container, not the unit's sandbox, so the rest of the
118 hardening stays.
119
120 If you would rather give podman the systemd user manager it prefers, run the
121 runner as a user unit instead: `loginctl enable-linger ci-runner`, a unit in
122 `ci-runner`'s own `systemctl --user` session, and no `cgroup_manager` setting. That
123 session lives at `/run/user/<uid>`, so the unit's `RuntimeDirectory=vole-runner` and
124 `XDG_RUNTIME_DIR=/run/vole-runner` have to go with it — the two arrangements cannot
125 both be assumed, and this one is not what the shipped unit does.
126
e21f6c5e
Keenan Nemetz
3d ago
 127 ## Register the runner, then allow it on a repository
128
129 The runner's key is not an admin key. It becomes a bot account, and the
130 allowlist decides what it may build:
131
132 ```sh
133 # The public key is on the runner; this copies it to the instance as a new bot.
134 ssh root@<runner-address> cat /var/lib/vole-runner/.ssh/id_ed25519.pub \
135 | vole admin --remote ssh://git@forge.example.com:2222 \
136 account create ci-runner-1 --bot --key -
137
138 # Per repository: let it take jobs, and give it read so it can clone.
139 vole admin --remote ssh://git@forge.example.com:2222 acl runners add demo ci-runner-1
140 vole admin --remote ssh://git@forge.example.com:2222 acl set-role demo ci-runner-1 read
141 ```
142
143 This is the part that cannot happen at first boot: repositories do not exist yet,
144 and the allowlist is per repository. Add each new repository to it as you create
145 it, or the runner will sit idle while its jobs wait in the queue.
146
147 A bot cannot sign in to the browser and cannot be an instance admin. That second
148 part matters: admins count as trusted reporters, so a bot that was an admin would
149 make every result count whatever the allowlist said.
150
151 ## Check that it works
152
b7396a11
Keenan Nemetz
2h ago
 153 - The Admin page's **Runners** tab (`/-/admin?tab=runners`) shows what is connected,
e21f6c5e
Keenan Nemetz
3d ago
 154 what each runner is building, and how much is queued.
155 - Push a commit on a branch with a workflow. The run page shows which runner took
156 it and whether the result counts.
157 - A run that executed on a runner outside the allowlist still exists and still
158 says so; it just does not count for the branch.
159
160 ## What to know before you give it a repository
161
162 **A runner can read every secret its jobs declare.** That is what running the job
163 means: the values arrive on the machine. If the machine is not yours, the secrets
164 are not either. This is also why an instance admin's machine should not be a
165 runner for a repository whose secrets it should not see.
166
167 **The allowlist is the control, not the container.** A job that escapes its
168 container finds a machine that can reach the instance and the internet. Keep the
169 runner on its own machine, run it as an unprivileged account, and give it no
170 credentials except the one key.
171
172 **Network is a per-job decision this machine permits.** A `bridge` runner still
173 gives a job that asks for none no network. What it cannot do is take the choice
174 away from a job, so a workflow you approve for a contributor branch is a workflow
175 that may reach the internet with whatever it can read.
176
177 **Disk fills where builds happen.** Container images, the `/cache` mount, and the
178 saved checkouts live under `--dir`. `--cache-size` (5 GiB) and `--git-cache-size`
179 (10 GiB) bound the last two; the images are the runtime's own storage, so size the
180 machine for them.
181
182 ## Upgrading
183
184 The runner is the same package as the instance, and the two talk over a versioned
185 protocol. Upgrade the instance first, then the runner machines:
186
187 ```sh
188 sudo apt install ./vole-<new-version>.deb
189 sudo systemctl restart vole-runner
190 ```
191
192 ## Next
193
194 - [Runners reference](../runner.md) — every option, the container defaults, caching,
195 and how a lease is reported
196 - [Workflows](../workflows.md) — what a job asks a runner for, including `runs-on`