Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Bound the runner's containers and keep its disk free

4e3759da08a603ede50107e89a23ff6d5432b7ac · Keenan Nemetz · 1h ago · parent c4ad0d5a

Bound the runner's containers and keep its disk free

The container boundary was good and the resource the job consumes was not. A job
could fork without a bound, write /tmp until the machine's RAM was gone, and fill
the disk its workspace sits on — and the runner would only notice when a build
died. The isolation protects the forge's data; this protects the machine that runs
the code.

- `--container-pids` now defaults to 2048 and zero means the default, not "no
  limit": a bound nobody set is the easiest one to be missing, and a job that can
  fork without one is a fork bomb away from taking the machine.
- `--container-tmpfs` is new and defaults to 1G. /tmp is a tmpfs, so its size is
  memory, and an unsized one is a build that fills RAM without writing a byte to
  the workspace. It is passed to service containers too.
- `--container-memory` stays the operator's setting rather than a default, because
  the right number is the machine's size and a wrong default would show up as an
  OOM kill in someone's build. All three places a runner is configured now set it:
  the packaged runner.env example, the manual steps in the deploy guide, and the
  cloud-init template (RUNNER_MEMORY, default 4G, alongside RUNNER_PIDS,
  RUNNER_TMPFS and RUNNER_DISK_RESERVE).
- `--disk-reserve` (default 2G) is how the job tree is bounded. It is a host bind
  mount, so the runtime cannot quota it and a filesystem quota needs root; a walker
  over the tree would cost more than it protects. What the runner can promise is
  that it notices: one statfs per second while a command runs, and the job is
  stopped below the reserve with the numbers in its log. `--dir` on its own sized
  filesystem is the hard bound, and the docs say so.
- The runner unit gets `MemoryMax=90%` as the backstop under all of it, for a job
  that gets past its own cgroup. It is a percentage so the unit does not have to
  know the machine. TasksMax is deliberately not set: the container's own
  --container-pids already applies, and a unit value would have to track --jobs and
  --container-pids and would fail builds, looking like the workflow's fault, when
  they drifted apart.

A workflow still cannot raise or remove any of this: the limits are flags on the
runner and the unit, never fields a job can set.

scripts/check-deploy.py now fails when the rendered runner arguments stop carrying
the memory, pids, tmpfs or disk limits, so the first-boot file cannot quietly lose
them.

Closes 01M6S798BH0BK00R2B5N1KDJYM

docs/deploy/runner.md +17−6

@@ -24,10 +24,20 @@ VOLE_PACKAGE_URL=https://rodent.software/vole/releases/v0.2.0/artifacts/vole_0.2
24 24
25 25 `render.sh runner` also reads `RUNNER_LABEL_NAMES` (default `linux`),
26 26 `RUNNER_NETWORK` (default `bridge`), `RUNNER_RUNTIME` (default `podman`),
27 `RUNNER_JOBS` (default `1`, how many jobs run at once), and `RUNNER_HOSTNAME`. Set
28 `RUNNER_JOBS` to the number of containers the machine can carry at once; the
29 container limits apply to each, and see
30 [Runners § Running several jobs at once](../runner.md#running-several-jobs-at-once).
27 `RUNNER_JOBS` (default `1`, how many jobs run at once), `RUNNER_HOSTNAME`,
28 `RUNNER_MEMORY` (default `4G`, what one job may take from RAM), `RUNNER_PIDS`
29 (default `2048`), `RUNNER_TMPFS` (default `1G`, the job's `/tmp`, which is a tmpfs
30 and therefore memory), and `RUNNER_DISK_RESERVE` (default `2G`, free space kept under
31 `RUNNER_DIR`).
32
33 Size the machine for `RUNNER_JOBS` times `RUNNER_MEMORY`, and keep that below the
34 unit's `MemoryMax=90%` backstop, which bounds the whole runner if a job gets past its
35 own cgroup. `RUNNER_DISK_RESERVE` is what the machine keeps for itself: a job that
36 takes the filesystem under `RUNNER_DIR` below it is stopped while it runs, and the
37 job's log says so. See
38 [Runners § What a job may consume](../runner.md#what-a-job-may-consume) for what each
39 limit means, and [Running several jobs at once](../runner.md#running-several-jobs-at-once)
40 for `RUNNER_JOBS`.
31 41 The file installs the package, creates an unprivileged
32 42 `ci-runner` account with the subordinate ids rootless Podman needs, generates the
33 43 runner's key, pins the instance's host key once, and enables `vole-runner.service`.
@@ -63,14 +73,15 @@ with each option explained:
63 73
64 74 ```sh
65 75 sudo tee /etc/vole/runner.env >/dev/null <<'EOF'
66 VOLE_RUNNER_ARGS=--remote ssh://git@forge.example.com:2222 --container-runtime podman --container-network bridge --dir /var/lib/vole-runner --known-hosts /etc/vole/runner_known_hosts --label linux
76 VOLE_RUNNER_ARGS=--remote ssh://git@forge.example.com:2222 --container-runtime podman --container-network bridge --container-memory 4G --container-tmpfs 1G --disk-reserve 2G --dir /var/lib/vole-runner --known-hosts /etc/vole/runner_known_hosts --label linux
67 77 EOF
68 78 sudo systemctl daemon-reload
69 79 sudo systemctl enable --now vole-runner
70 80 ```
71 81
72 82 Add `--jobs N` to that line to run several jobs at once on a machine with room for
73 them.
83 them, and raise `--container-memory` with it: the memory limit is per job, so a machine
84 running two jobs at `4G` needs room for `8G` below the unit's `MemoryMax` backstop.
74 85
75 86 `systemctl stop vole-runner` stops taking work and lets the jobs in flight finish,
76 87 up to the unit's `TimeoutStopSec` (60 seconds). Raise it in a drop-in if this