Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Bound the runner's containers and keep its disk free

4e3759da08a603ede50107e89a23ff6d5432b7ac · Keenan Nemetz · 1h ago · parent c4ad0d5a

Bound the runner's containers and keep its disk free

The container boundary was good and the resource the job consumes was not. A job
could fork without a bound, write /tmp until the machine's RAM was gone, and fill
the disk its workspace sits on — and the runner would only notice when a build
died. The isolation protects the forge's data; this protects the machine that runs
the code.

- `--container-pids` now defaults to 2048 and zero means the default, not "no
  limit": a bound nobody set is the easiest one to be missing, and a job that can
  fork without one is a fork bomb away from taking the machine.
- `--container-tmpfs` is new and defaults to 1G. /tmp is a tmpfs, so its size is
  memory, and an unsized one is a build that fills RAM without writing a byte to
  the workspace. It is passed to service containers too.
- `--container-memory` stays the operator's setting rather than a default, because
  the right number is the machine's size and a wrong default would show up as an
  OOM kill in someone's build. All three places a runner is configured now set it:
  the packaged runner.env example, the manual steps in the deploy guide, and the
  cloud-init template (RUNNER_MEMORY, default 4G, alongside RUNNER_PIDS,
  RUNNER_TMPFS and RUNNER_DISK_RESERVE).
- `--disk-reserve` (default 2G) is how the job tree is bounded. It is a host bind
  mount, so the runtime cannot quota it and a filesystem quota needs root; a walker
  over the tree would cost more than it protects. What the runner can promise is
  that it notices: one statfs per second while a command runs, and the job is
  stopped below the reserve with the numbers in its log. `--dir` on its own sized
  filesystem is the hard bound, and the docs say so.
- The runner unit gets `MemoryMax=90%` as the backstop under all of it, for a job
  that gets past its own cgroup. It is a percentage so the unit does not have to
  know the machine. TasksMax is deliberately not set: the container's own
  --container-pids already applies, and a unit value would have to track --jobs and
  --container-pids and would fail builds, looking like the workflow's fault, when
  they drifted apart.

A workflow still cannot raise or remove any of this: the limits are flags on the
runner and the unit, never fields a job can set.

scripts/check-deploy.py now fails when the rendered runner arguments stop carrying
the memory, pids, tmpfs or disk limits, so the first-boot file cannot quietly lose
them.

Closes 01M6S798BH0BK00R2B5N1KDJYM

docs/runner.md +14−3

@@ -214,12 +214,23 @@ instance or kept on the runner, so each of those channels has a bound:
214 214 protected branch never shares a cache with a contributor branch. The saved
215 215 checkouts jobs clone from are bounded separately; see
216 216 [Saved checkouts](#saved-checkouts).
217 - **Container.** `--container-memory`, `--container-cpus` and `--container-pids`
218 pass the runtime's own limits, per container, so `--jobs` multiplies them.
219 `--container-network` is what this machine can offer — `none` (the default) or `bridge` — and a job's own `network:` decides what
217 - **Container.** `--container-memory`, `--container-cpus`, `--container-pids` and
218 `--container-tmpfs` are the runtime's own limits, per container, so `--jobs`
219 multiplies them. `--container-pids` (2048) and `--container-tmpfs` (1G) have
220 defaults and cannot be removed: `/tmp` is a tmpfs, so its size is memory, and a job
221 that can fork without a bound is a fork bomb away from taking the machine. Set
222 `--container-memory` — the runtime's default is the whole machine — and see the
223 unit's `MemoryMax` backstop below. `--container-network` is what this machine can
224 offer — `none` (the default) or `bridge` — and a job's own `network:` decides what
220 225 it gets, so a bridge runner can still run a job that asks for none. See
221 226 [Network](#network). The runtime's storage driver, not Vole, is what bounds the
222 227 container's writable layer, so size it on the host.
228 - **Disk.** A job's tree is a host directory the runtime cannot quota, so the runner
229 keeps `--disk-reserve` (2G) free under `--dir` and stops a job that takes the
230 filesystem below it while the job is running. It is one `statfs` per second, and the
231 log says what was left and what the reserve is. That is the promise — the runner
232 notices — rather than a hard bound; for one of those, put `--dir` on its own
233 filesystem and size that. The unit's `MemoryMax` is the same idea for RAM.
223 234
224 235 ## Images and image builds
225 236