Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Bound the runner's containers and keep its disk free

4e3759da08a603ede50107e89a23ff6d5432b7ac · Keenan Nemetz · 2h ago · parent c4ad0d5a 

Bound the runner's containers and keep its disk free

The container boundary was good and the resource the job consumes was not. A job
could fork without a bound, write /tmp until the machine's RAM was gone, and fill
the disk its workspace sits on — and the runner would only notice when a build
died. The isolation protects the forge's data; this protects the machine that runs
the code.

- `--container-pids` now defaults to 2048 and zero means the default, not "no
  limit": a bound nobody set is the easiest one to be missing, and a job that can
  fork without one is a fork bomb away from taking the machine.
- `--container-tmpfs` is new and defaults to 1G. /tmp is a tmpfs, so its size is
  memory, and an unsized one is a build that fills RAM without writing a byte to
  the workspace. It is passed to service containers too.
- `--container-memory` stays the operator's setting rather than a default, because
  the right number is the machine's size and a wrong default would show up as an
  OOM kill in someone's build. All three places a runner is configured now set it:
  the packaged runner.env example, the manual steps in the deploy guide, and the
  cloud-init template (RUNNER_MEMORY, default 4G, alongside RUNNER_PIDS,
  RUNNER_TMPFS and RUNNER_DISK_RESERVE).
- `--disk-reserve` (default 2G) is how the job tree is bounded. It is a host bind
  mount, so the runtime cannot quota it and a filesystem quota needs root; a walker
  over the tree would cost more than it protects. What the runner can promise is
  that it notices: one statfs per second while a command runs, and the job is
  stopped below the reserve with the numbers in its log. `--dir` on its own sized
  filesystem is the hard bound, and the docs say so.
- The runner unit gets `MemoryMax=90%` as the backstop under all of it, for a job
  that gets past its own cgroup. It is a percentage so the unit does not have to
  know the machine. TasksMax is deliberately not set: the container's own
  --container-pids already applies, and a unit value would have to track --jobs and
  --container-pids and would fail builds, looking like the workflow's fault, when
  they drifted apart.

A workflow still cannot raise or remove any of this: the limits are flags on the
runner and the unit, never fields a job can set.

scripts/check-deploy.py now fails when the rendered runner arguments stop carrying
the memory, pids, tmpfs or disk limits, so the first-boot file cannot quietly lose
them.

Closes 01M6S798BH0BK00R2B5N1KDJYM

cmd/vole/runner.go +26−2

@@ -85,7 +85,12 @@ Container execution:
85 85 --container-userns user namespace mode passed to the runtime. Defaults to
86 86 keep-id for rootless Podman so the job tree is writable by this user.
87 87 --container-memory, --container-cpus, --container-pids
88 optional resource limits passed to the runtime.
88 resource limits passed to the runtime. --container-pids defaults to 2048
89 and cannot be removed: a job that can fork without a bound is a fork bomb
90 away from taking the machine.
91 --container-tmpfs size of the job's /tmp (default 1G). It is a tmpfs, so it is
92 memory: an unsized one lets a build fill the machine's RAM without writing
93 to the workspace. Raise it for a build that writes large temporary files.
89 94
90 95 Job cache:
91 96 Each job gets a writable directory at /cache, kept under --dir/cache and reused
@@ -109,7 +114,11 @@ Saved checkouts:
109 114 most of the cap is repacked first, which drops history no branch or tag
110 115 can reach any more and costs nothing to re-download; when the cap is
111 116 still reached, the least recently used repository's checkout is deleted
112 and its next job downloads it again. 0 keeps every saved checkout.`
117 and its next job downloads it again. 0 keeps every saved checkout.
118 --disk-reserve free space to keep under --dir (default 2G, 0 disables). A job
119 that takes the filesystem below it is stopped while it runs. A job tree is
120 a host directory the runtime cannot quota, and a build that fills the disk
121 takes the runner out of service for every repository it serves.`
113 122 }
114 123
115 124 func cmdRunner(args []string) error {
@@ -120,6 +129,7 @@ func cmdRunner(args []string) error {
120 129 flagSpec{long: "container-user"}, flagSpec{long: "container-userns"},
121 130 flagSpec{long: "container-memory"},
122 131 flagSpec{long: "container-cpus"}, flagSpec{long: "container-pids"},
132 flagSpec{long: "container-tmpfs"}, flagSpec{long: "disk-reserve"},
123 133 flagSpec{long: "cache-size"}, flagSpec{long: "git-cache-size"},
124 134 flagSpec{long: "no-cache", boolean: true},
125 135 flagSpec{long: "once", boolean: true},
@@ -169,6 +179,18 @@ func cmdRunner(args []string) error {
169 179 return usagef("--git-cache-size %v", err)
170 180 }
171 181 }
182 // The disk reserve is on by default: a runner that fills its own filesystem is out
183 // of service for every repository it serves, so the bound is the runner's problem
184 // rather than the operator's memory. 0 turns it off for a --dir on its own sized
185 // filesystem.
186 diskReserve := int64(2 << 30)
187 if raw := f.one("disk-reserve"); raw != "" {
188 if strings.TrimSpace(raw) == "0" {
189 diskReserve = 0
190 } else if diskReserve, err = jobcache.ParseSize(raw); err != nil {
191 return usagef("--disk-reserve %v", err)
192 }
193 }
172 194 // Host-key policy for this runner and every git child it starts. The explicit
173 195 // flag replaces what the environment asked for, so a test can be explicit and an
174 196 // operator can point at a file.
@@ -220,6 +242,8 @@ func cmdRunner(args []string) error {
220 242 ContainerMemory: f.one("container-memory"),
221 243 ContainerCPUs: f.one("container-cpus"),
222 244 ContainerPIDs: pids,
245 ContainerTmpfs: f.one("container-tmpfs"),
246 DiskReserve: diskReserve,
223 247 NoCache: f.has("no-cache"),
224 248 CacheMaxBytes: cacheMax,
225 249 GitCacheMaxBytes: gitCacheMax,