Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Bound the runner's containers and keep its disk free

4e3759da08a603ede50107e89a23ff6d5432b7ac · Keenan Nemetz · 2h ago · parent c4ad0d5a 

Bound the runner's containers and keep its disk free

The container boundary was good and the resource the job consumes was not. A job
could fork without a bound, write /tmp until the machine's RAM was gone, and fill
the disk its workspace sits on — and the runner would only notice when a build
died. The isolation protects the forge's data; this protects the machine that runs
the code.

- `--container-pids` now defaults to 2048 and zero means the default, not "no
  limit": a bound nobody set is the easiest one to be missing, and a job that can
  fork without one is a fork bomb away from taking the machine.
- `--container-tmpfs` is new and defaults to 1G. /tmp is a tmpfs, so its size is
  memory, and an unsized one is a build that fills RAM without writing a byte to
  the workspace. It is passed to service containers too.
- `--container-memory` stays the operator's setting rather than a default, because
  the right number is the machine's size and a wrong default would show up as an
  OOM kill in someone's build. All three places a runner is configured now set it:
  the packaged runner.env example, the manual steps in the deploy guide, and the
  cloud-init template (RUNNER_MEMORY, default 4G, alongside RUNNER_PIDS,
  RUNNER_TMPFS and RUNNER_DISK_RESERVE).
- `--disk-reserve` (default 2G) is how the job tree is bounded. It is a host bind
  mount, so the runtime cannot quota it and a filesystem quota needs root; a walker
  over the tree would cost more than it protects. What the runner can promise is
  that it notices: one statfs per second while a command runs, and the job is
  stopped below the reserve with the numbers in its log. `--dir` on its own sized
  filesystem is the hard bound, and the docs say so.
- The runner unit gets `MemoryMax=90%` as the backstop under all of it, for a job
  that gets past its own cgroup. It is a percentage so the unit does not have to
  know the machine. TasksMax is deliberately not set: the container's own
  --container-pids already applies, and a unit value would have to track --jobs and
  --container-pids and would fail builds, looking like the workflow's fault, when
  they drifted apart.

A workflow still cannot raise or remove any of this: the limits are flags on the
runner and the unit, never fields a job can set.

scripts/check-deploy.py now fails when the rendered runner arguments stop carrying
the memory, pids, tmpfs or disk limits, so the first-boot file cannot quietly lose
them.

Closes 01M6S798BH0BK00R2B5N1KDJYM

ci/container/container_test.go +54−2

@@ -53,7 +53,7 @@ func TestRunArgsAreClosed(t *testing.T) {
53 53 "--pids-limit 512",
54 54 "--memory 2g",
55 55 "--cpus 2",
56 "--tmpfs /tmp:rw,nosuid,nodev,exec,mode=1777",
56 "--tmpfs /tmp:rw,nosuid,nodev,exec,mode=1777,size=1g",
57 57 "--entrypoint /bin/sh",
58 58 "golang:1.27@sha256:abc",
59 59 } {
@@ -256,7 +256,7 @@ func TestServiceRunArgsAreClosed(t *testing.T) {
256 256 "--userns keep-id",
257 257 "--network-alias db",
258 258 "--env POSTGRES_PASSWORD=test",
259 "--tmpfs /tmp:rw,nosuid,nodev,exec,mode=1777",
259 "--tmpfs /tmp:rw,nosuid,nodev,exec,mode=1777,size=1g",
260 260 "--cap-drop ALL",
261 261 "--security-opt no-new-privileges",
262 262 "--network vole-runner1-run1-1-net",
@@ -361,3 +361,55 @@ func TestLastLineSkipsWarningsAndBlankLines(t *testing.T) {
361 361 }
362 362 }
363 363 }
364
365 // The resource limits belong to the operator, and the defaults are limits too: a job
366 // cannot arrive with no pids bound, and its /tmp is a tmpfs in RAM, so an unsized one
367 // is a build that fills the machine without writing to the workspace.
368 func TestContainerLimitsAreFiniteWithoutBeingSet(t *testing.T) {
369 fakeRuntime(t, "docker", "#!/bin/sh\nexit 0\n")
370 cfg, err := Validate(Config{Runtime: "docker"})
371 if err != nil {
372 t.Fatal(err)
373 }
374 if cfg.PIDs != DefaultPIDs {
375 t.Fatalf("pids = %d, want the default %d", cfg.PIDs, DefaultPIDs)
376 }
377 if cfg.Tmpfs != DefaultTmpfs {
378 t.Fatalf("tmpfs = %q, want the default %q", cfg.Tmpfs, DefaultTmpfs)
379 }
380 args := strings.Join(RunArgs(cfg, Spec{Image: "alpine:3.22", Name: "job"}), " ")
381 for _, want := range []string{
382 "--pids-limit 2048",
383 "--tmpfs /tmp:rw,nosuid,nodev,exec,mode=1777,size=1g",
384 } {
385 if !strings.Contains(args, want) {
386 t.Fatalf("container arguments do not contain %q:\n%s", want, args)
387 }
388 }
389
390 // What the operator sets is what both a job and its services get.
391 cfg, err = Validate(Config{Runtime: "docker", PIDs: 256, Tmpfs: "2g", Memory: "4g"})
392 if err != nil {
393 t.Fatal(err)
394 }
395 for _, got := range []string{
396 strings.Join(RunArgs(cfg, Spec{Image: "alpine:3.22", Name: "job"}), " "),
397 strings.Join(ServiceRunArgs(cfg, ServiceSpec{Name: "db", Image: "postgres:17"}), " "),
398 } {
399 for _, want := range []string{"--pids-limit 256", "size=2g", "--memory 4g"} {
400 if !strings.Contains(got, want) {
401 t.Fatalf("arguments do not contain %q:\n%s", want, got)
402 }
403 }
404 }
405 }
406
407 // A size with a space or a newline in it would split into two container arguments.
408 func TestValidateRefusesATmpfsThatIsNotASize(t *testing.T) {
409 fakeRuntime(t, "docker", "#!/bin/sh\nexit 0\n")
410 for _, raw := range []string{"1g 2g", "1g\nextra", "1g\textra"} {
411 if _, err := Validate(Config{Runtime: "docker", Tmpfs: raw}); err == nil {
412 t.Fatalf("--container-tmpfs %q was accepted", raw)
413 }
414 }
415 }