Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#259

Bound the runner's containers and keep its disk free

merged · by keenan 3h ago · merged into main at 11ee410e by keenan 3h ago

keenan opened this 3h ago

Closes #211. The container boundary was good and the resource the job consumes was not: a job could fork without a bound, write /tmp until the machine’s RAM was gone, and fill the disk its workspace sits on, and the runner would only notice when a build died. The isolation protects the forge’s data; this protects the machine that runs the code.

What bounds what

Bound Default Why it is that
--container-pids 2048 A container’s processes. Zero now means the default, not “no limit”: a bound nobody set is the easiest one to be missing.
--container-tmpfs 1G The job’s /tmp, which is a tmpfs — its size is memory. Passed to service containers too.
--container-memory none (operator) Per job, and the right number is the machine’s size. A wrong default would appear as an OOM kill in someone’s build.
--disk-reserve 2G Free space the runner keeps under --dir; a job that takes the filesystem below it is stopped while it runs.
unit MemoryMax 90% Backstop under all of it, for a job that gets past its own cgroup.

Decisions

  • The job tree is bounded by noticing, not by a quota. It is a host bind mount, so the runtime cannot quota it and a filesystem quota needs root and a filesystem that supports one; a walker over the tree on a timer would cost more than it protects. One statfs per second while a command runs, and the log gets what was left and what the reserve is. --dir on its own sized filesystem is the hard bound, and the docs say so.
  • Memory has no default. Every other limit is safe to set blind; this one is a property of the machine, and the failure mode of getting it wrong is an OOM kill that reads like a bug in the workflow.
  • No TasksMax/LimitNPROC on the unit. The container’s own pids limit already applies to what the runner starts, and a unit-level value would have to track --jobs and --container-pids; when it drifted the builds would fail in a way that looks like the workflow’s fault. MemoryMax is the backstop because a cgroup memory escape is the one that takes the whole machine down.
  • A workflow still cannot raise any of it. These are runner flags and unit settings, never fields a job can name.

Where it is configured

All three places a runner is documented or created now carry the limits: packaging/runner.env (the shipped example), the manual steps in docs/deploy/runner.md, and deploy/cloud-init/runner.yaml via new render variables RUNNER_MEMORY (4G), RUNNER_PIDS, RUNNER_TMPFS and RUNNER_DISK_RESERVE, documented with the sizing rule (--jobs × memory below the MemoryMax backstop).

Testing

  • ci/container: an unconfigured runtime still gets finite pids and a sized tmpfs; an operator’s values reach both a job and a service container; a tmpfs value with whitespace is refused.
  • ci/agent: diskLow against a real filesystem — below the reserve, above it, and the zero that disables it; --container-tmpfs reaches container.Config.
  • cmd/vole: a --disk-reserve that is not a size is refused where the flag is read.
  • scripts/check-deploy.py fails when the rendered runner arguments stop carrying --container-memory, --container-pids, --container-tmpfs or --disk-reserve (verified by removing one and watching it fail).

vole workflow run verify passes (format, vet, build, licenses, govulncheck, packages, deployment files, links, race suite).

Discussion (0)

No discussion yet.