Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

#236

P1: ci/cache Trim evicts the wrong namespace, so verify's test step fails on a runner

closed · opened by keenan 19h ago

keenan opened this 19h ago

The test step of the verify workflow fails in the job image, on a test that passes on every machine I have tried it on:

--- FAIL: TestTrimDropsTheOldestNamespaceWhole (0.00s)
    cache_test.go:127: the oldest namespace survived: <nil>
FAIL	github.com/rodent-software/vole/ci/cache	0.039s

Seen on run 01M6YSX585GD5BDT45H0FE65AE (verify · main @ 973fb80906d6a1ec79ecd6b449e2e51e9cfb77d5) on a DigitalOcean runner: Ubuntu 24.04, rootless podman, overlay storage. go test ./ci/cache -run TestTrim -count=5 passes on ext4 locally, so something about the job filesystem changes the answer.

Why it matters

checks are required on main, so a red verify blocks every change request that would land there. A test that depends on which namespace a filesystem clock happens to order first is a gate that fails for reasons unrelated to the change.

What the failure says

The assertion before it passed — trim = 1000, <nil>; want 1000 bytes — so Trim reported removing one whole namespace and no error, yet the oldest one was still on disk. It evicted the newer one.

Two mechanisms in ci/cache/cache.go can produce exactly that:

  • Trim (line 151) sorts on newest, which measure (line 272) computes as the newest mtime in the namespace tree, directories included. In the test the file mtimes differ by two hours, but the namespace directories are both created “now”, so the sort key is nearly identical and sort.Slice — unstable — breaks the tie arbitrarily.
  • Trim silently continues when TryLockExclusive cannot be held (lines 181-190) and then evicts the next namespace to get under the cap. A lock that cannot be opened for the oldest namespace yields the same signature: something was removed, no error surfaced, the oldest survived.

Scope

  • Decide what Trim should order on: file mtimes only, or a single mtime recorded per namespace when it is written.
  • Decide whether a namespace that cannot be locked is reported rather than skipped silently, given it is still counted in total.
  • Make the test’s two namespaces distinguishable by the key Trim actually uses, so the outcome does not depend on clock granularity.

Acceptance criteria

  • go test ./ci/cache -count=20 passes inside the verify image on a runner, not just on a developer’s ext4 checkout.
  • The oldest namespace is evicted when both are created inside the same clock tick.
  • A namespace that cannot be locked is either reported or excluded from the byte accounting, so removed cannot overstate what was freed.

Comments (1)

keenan 19h ago

Seen while setting up a runner on a fresh Ubuntu 24.04 machine. The three deployment failures that came out of that session are #237 (the packaged unit cannot run rootless podman), #239 (no unqualified-search-registries) and #241 (no systemd user manager for cgroups). The container that produced this failure was working by the time of the last attempt, so the report stands on its own.