Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#262

P1: Test-spawned runners abort jobs on the job's 1 GiB /tmp

closed · opened by keenan 1h ago

keenan opened this 1h ago

Why

main’s verify run 01M7189M5A5P0PJ170QA8F3CK2 (3229011, “update readme”) failed its test step. That commit changes only README.md; the failure is in the test harness, not the code under test.

Two runner defaults interact:

  • --container-tmpfs defaults to 1g (ci/container/container.go), so the job’s /tmp is a 1 GiB tmpfs.
  • --disk-reserve defaults to 2G (cmd/vole/runner.go), checked once a second (ci/agent/agent.go) against the filesystem that holds --dir.

The server integration tests spawn nested runners with --dir t.TempDir() (server/integration_test.go). t.TempDir() is under /tmp, so those runners measure a 1 GiB tmpfs against a 2 GiB reserve, stop the job at the first one-second check, and the test fails:

--- FAIL: TestKilledRunnerIsRequeued (6.55s)
    integration_test.go:3437: the retried run did not succeed: failed (exit 1)
        the runner has 945.8 MiB free, below its 2.0 GiB reserve;
        stopping the job so the machine does not fill (--disk-reserve)
--- FAIL: TestRunLogStreamsWhileTheJobRuns (2.64s)

Only tests whose steps run longer than a second are hit. TestKilledRunnerIsRequeued holds the lease with sleep 3, and TestRunLogStreamsWhileTheJobRuns streams output slowly on purpose; the rest of the package finishes each step before the first check, which is why they pass.

Reproduced locally in the same file:verify image with --tmpfs /tmp:size=1g:

the runner has 769.8 MiB free, below its 2.0 GiB reserve; stopping the job
--- FAIL: TestKilledRunnerIsRequeued (6.55s)
--- FAIL: TestRunLogStreamsWhileTheJobRuns (2.43s)

The reserve protects a host filesystem. A test’s t.TempDir() is a throwaway directory on a tmpfs the runtime already bounds, so enforcing the host reserve there proves nothing and makes the harness depend on the runner’s flag defaults.

Scope

  • Turn the disk reserve off for the runners the integration tests spawn, at the one place their command line is built.
  • Say why in a comment, so a new test does not reintroduce it.
  • Confirm the two tests pass with the suite running in a 1 GiB /tmp.

Acceptance criteria

  • TestKilledRunnerIsRequeued and TestRunLogStreamsWhileTheJobRuns pass when go test ./server runs in a job container whose /tmp is a 1 GiB tmpfs.
  • go test -race -count=1 ./server passes locally and vole workflow run verify passes in the verify image.
  • No production behavior changes: the runner keeps the 2 GiB --disk-reserve default, and ci/agent/disk_test.go still covers the check itself.

Comments (1)

keenan 58m ago

Fixed in #263 (01M71GENHTB7EK2GS84YQMDEQQ): fixture.runnerCmd builds every runner a test spawns with --disk-reserve 0, and the reason is in a comment there.

CI on the branch passed (01M71GEDW1AWY3QYHG29BW65J2, test step 2m39s, all steps ok), and the suite now passes locally in the file:verify image with --tmpfs /tmp:size=1g.