vole

Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Bound the runner's containers and keep its disk free

4e3759da08a603ede50107e89a23ff6d5432b7ac · Keenan Nemetz · 1h ago · parent c4ad0d5a

Bound the runner's containers and keep its disk free

The container boundary was good and the resource the job consumes was not. A job
could fork without a bound, write /tmp until the machine's RAM was gone, and fill
the disk its workspace sits on — and the runner would only notice when a build
died. The isolation protects the forge's data; this protects the machine that runs
the code.

- `--container-pids` now defaults to 2048 and zero means the default, not "no
  limit": a bound nobody set is the easiest one to be missing, and a job that can
  fork without one is a fork bomb away from taking the machine.
- `--container-tmpfs` is new and defaults to 1G. /tmp is a tmpfs, so its size is
  memory, and an unsized one is a build that fills RAM without writing a byte to
  the workspace. It is passed to service containers too.
- `--container-memory` stays the operator's setting rather than a default, because
  the right number is the machine's size and a wrong default would show up as an
  OOM kill in someone's build. All three places a runner is configured now set it:
  the packaged runner.env example, the manual steps in the deploy guide, and the
  cloud-init template (RUNNER_MEMORY, default 4G, alongside RUNNER_PIDS,
  RUNNER_TMPFS and RUNNER_DISK_RESERVE).
- `--disk-reserve` (default 2G) is how the job tree is bounded. It is a host bind
  mount, so the runtime cannot quota it and a filesystem quota needs root; a walker
  over the tree would cost more than it protects. What the runner can promise is
  that it notices: one statfs per second while a command runs, and the job is
  stopped below the reserve with the numbers in its log. `--dir` on its own sized
  filesystem is the hard bound, and the docs say so.
- The runner unit gets `MemoryMax=90%` as the backstop under all of it, for a job
  that gets past its own cgroup. It is a percentage so the unit does not have to
  know the machine. TasksMax is deliberately not set: the container's own
  --container-pids already applies, and a unit value would have to track --jobs and
  --container-pids and would fail builds, looking like the workflow's fault, when
  they drifted apart.

A workflow still cannot raise or remove any of this: the limits are flags on the
runner and the unit, never fields a job can set.

scripts/check-deploy.py now fails when the rendered runner arguments stop carrying
the memory, pids, tmpfs or disk limits, so the first-boot file cannot quietly lose
them.

Closes 01M6S798BH0BK00R2B5N1KDJYM

deploy/cloud-init/render.sh +14−1

@@ -79,6 +79,15 @@ runner)
	79	79	# How many jobs this machine runs at once. Each is a container, so the machine
	80	80	# has to carry this many times the container limits.
	81	81	: "${RUNNER_JOBS:=1}"
		82	# What one job may take, and what the machine keeps for itself. The memory limit
		83	# is the one to size for: RUNNER_JOBS times RUNNER_MEMORY is the runner at full
		84	# stretch, and the unit's MemoryMax backstop is a percentage of the machine above
		85	# it. RUNNER_DISK_RESERVE is free space kept under RUNNER_DIR: a job that takes
		86	# the filesystem below it is stopped while it runs.
		87	: "${RUNNER_MEMORY:=4G}"
		88	: "${RUNNER_PIDS:=2048}"
		89	: "${RUNNER_TMPFS:=1G}"
		90	: "${RUNNER_DISK_RESERVE:=2G}"
	82	91
	83	92	labels=""
	84	93	for name in $RUNNER_LABEL_NAMES; do
@@ -91,10 +100,14 @@ runner)
	91	100	export runner_network="$RUNNER_NETWORK"
	92	101	export runner_dir="$RUNNER_DIR"
	93	102	export runner_jobs="--jobs $RUNNER_JOBS"
		103	export runner_memory="$RUNNER_MEMORY"
		104	export runner_pids="$RUNNER_PIDS"
		105	export runner_tmpfs="$RUNNER_TMPFS"
		106	export runner_disk_reserve="$RUNNER_DISK_RESERVE"
	94	107	export runner_labels="$labels"
	95	108	export vole_package_url="$VOLE_PACKAGE_URL"
	96	109
	97		render runner.yaml runner_hostname forge_host runner_runtime runner_network runner_dir runner_jobs runner_labels vole_package_url
		110	render runner.yaml runner_hostname forge_host runner_runtime runner_network runner_dir runner_jobs runner_labels runner_memory runner_pids runner_tmpfs runner_disk_reserve vole_package_url
	98	111	;;
	99	112	*)
	100	113	echo "unknown template $what; use vole or runner" >&2