vole
Clone
Read only, no credentials:
https://rodent.software/git/vole.git
Write over SSH with a registered key:
git clone ssh://git@0.0.0.0:2222/vole.git
A free and open-source forge that lives in your Git repo.
Bound the runner's containers and keep its disk free
4e3759da08a603ede50107e89a23ff6d5432b7ac · Keenan Nemetz · 1h ago · parent c4ad0d5a
Bound the runner's containers and keep its disk free The container boundary was good and the resource the job consumes was not. A job could fork without a bound, write /tmp until the machine's RAM was gone, and fill the disk its workspace sits on — and the runner would only notice when a build died. The isolation protects the forge's data; this protects the machine that runs the code. - `--container-pids` now defaults to 2048 and zero means the default, not "no limit": a bound nobody set is the easiest one to be missing, and a job that can fork without one is a fork bomb away from taking the machine. - `--container-tmpfs` is new and defaults to 1G. /tmp is a tmpfs, so its size is memory, and an unsized one is a build that fills RAM without writing a byte to the workspace. It is passed to service containers too. - `--container-memory` stays the operator's setting rather than a default, because the right number is the machine's size and a wrong default would show up as an OOM kill in someone's build. All three places a runner is configured now set it: the packaged runner.env example, the manual steps in the deploy guide, and the cloud-init template (RUNNER_MEMORY, default 4G, alongside RUNNER_PIDS, RUNNER_TMPFS and RUNNER_DISK_RESERVE). - `--disk-reserve` (default 2G) is how the job tree is bounded. It is a host bind mount, so the runtime cannot quota it and a filesystem quota needs root; a walker over the tree would cost more than it protects. What the runner can promise is that it notices: one statfs per second while a command runs, and the job is stopped below the reserve with the numbers in its log. `--dir` on its own sized filesystem is the hard bound, and the docs say so. - The runner unit gets `MemoryMax=90%` as the backstop under all of it, for a job that gets past its own cgroup. It is a percentage so the unit does not have to know the machine. TasksMax is deliberately not set: the container's own --container-pids already applies, and a unit value would have to track --jobs and --container-pids and would fail builds, looking like the workflow's fault, when they drifted apart. A workflow still cannot raise or remove any of this: the limits are flags on the runner and the unit, never fields a job can set. scripts/check-deploy.py now fails when the rendered runner arguments stop carrying the memory, pids, tmpfs or disk limits, so the first-boot file cannot quietly lose them. Closes 01M6S798BH0BK00R2B5N1KDJYM
docs/deploy/runner.md +17−6
| @@ -24,10 +24,20 @@ VOLE_PACKAGE_URL=https://rodent.software/vole/releases/v0.2.0/artifacts/vole_0.2 | |||
| 24 | 24 | ||
| 25 | 25 | `render.sh runner` also reads `RUNNER_LABEL_NAMES` (default `linux`), | |
| 26 | 26 | `RUNNER_NETWORK` (default `bridge`), `RUNNER_RUNTIME` (default `podman`), | |
| 27 | `RUNNER_JOBS` (default `1`, how many jobs run at once), and `RUNNER_HOSTNAME`. Set | ||
| 28 | `RUNNER_JOBS` to the number of containers the machine can carry at once; the | ||
| 29 | container limits apply to each, and see | ||
| 30 | [Runners § Running several jobs at once](../runner.md#running-several-jobs-at-once). | ||
| 27 | `RUNNER_JOBS` (default `1`, how many jobs run at once), `RUNNER_HOSTNAME`, | ||
| 28 | `RUNNER_MEMORY` (default `4G`, what one job may take from RAM), `RUNNER_PIDS` | ||
| 29 | (default `2048`), `RUNNER_TMPFS` (default `1G`, the job's `/tmp`, which is a tmpfs | ||
| 30 | and therefore memory), and `RUNNER_DISK_RESERVE` (default `2G`, free space kept under | ||
| 31 | `RUNNER_DIR`). | ||
| 32 | |||
| 33 | Size the machine for `RUNNER_JOBS` times `RUNNER_MEMORY`, and keep that below the | ||
| 34 | unit's `MemoryMax=90%` backstop, which bounds the whole runner if a job gets past its | ||
| 35 | own cgroup. `RUNNER_DISK_RESERVE` is what the machine keeps for itself: a job that | ||
| 36 | takes the filesystem under `RUNNER_DIR` below it is stopped while it runs, and the | ||
| 37 | job's log says so. See | ||
| 38 | [Runners § What a job may consume](../runner.md#what-a-job-may-consume) for what each | ||
| 39 | limit means, and [Running several jobs at once](../runner.md#running-several-jobs-at-once) | ||
| 40 | for `RUNNER_JOBS`. | ||
| 31 | 41 | The file installs the package, creates an unprivileged | |
| 32 | 42 | `ci-runner` account with the subordinate ids rootless Podman needs, generates the | |
| 33 | 43 | runner's key, pins the instance's host key once, and enables `vole-runner.service`. | |
| @@ -63,14 +73,15 @@ with each option explained: | |||
| 63 | 73 | ||
| 64 | 74 | ```sh | |
| 65 | 75 | sudo tee /etc/vole/runner.env >/dev/null <<'EOF' | |
| 66 | VOLE_RUNNER_ARGS=--remote ssh://git@forge.example.com:2222 --container-runtime podman --container-network bridge --dir /var/lib/vole-runner --known-hosts /etc/vole/runner_known_hosts --label linux | ||
| 76 | VOLE_RUNNER_ARGS=--remote ssh://git@forge.example.com:2222 --container-runtime podman --container-network bridge --container-memory 4G --container-tmpfs 1G --disk-reserve 2G --dir /var/lib/vole-runner --known-hosts /etc/vole/runner_known_hosts --label linux | ||
| 67 | 77 | EOF | |
| 68 | 78 | sudo systemctl daemon-reload | |
| 69 | 79 | sudo systemctl enable --now vole-runner | |
| 70 | 80 | ``` | |
| 71 | 81 | ||
| 72 | 82 | Add `--jobs N` to that line to run several jobs at once on a machine with room for | |
| 73 | them. | ||
| 83 | them, and raise `--container-memory` with it: the memory limit is per job, so a machine | ||
| 84 | running two jobs at `4G` needs room for `8G` below the unit's `MemoryMax` backstop. | ||
| 74 | 85 | ||
| 75 | 86 | `systemctl stop vole-runner` stops taking work and lets the jobs in flight finish, | |
| 76 | 87 | up to the unit's `TimeoutStopSec` (60 seconds). Raise it in a drop-in if this | |