The test step of the verify workflow fails in the job image, on a test that passes
on every machine I have tried it on:
--- FAIL: TestTrimDropsTheOldestNamespaceWhole (0.00s)
cache_test.go:127: the oldest namespace survived: <nil>
FAIL github.com/rodent-software/vole/ci/cache 0.039s
Seen on run 01M6YSX585GD5BDT45H0FE65AE (verify · main @ 973fb80906d6a1ec79ecd6b449e2e51e9cfb77d5) on a
DigitalOcean runner: Ubuntu 24.04, rootless podman, overlay storage. go test ./ci/cache -run TestTrim -count=5 passes on ext4 locally, so something about the job filesystem
changes the answer.
Why it matters
checks are required on main, so a red verify blocks every change request that
would land there. A test that depends on which namespace a filesystem clock happens to
order first is a gate that fails for reasons unrelated to the change.
What the failure says
The assertion before it passed — trim = 1000, <nil>; want 1000 bytes — so Trim
reported removing one whole namespace and no error, yet the oldest one was still on
disk. It evicted the newer one.
Two mechanisms in ci/cache/cache.go can produce exactly that:
Trim(line 151) sorts onnewest, whichmeasure(line 272) computes as the newest mtime in the namespace tree, directories included. In the test the file mtimes differ by two hours, but the namespace directories are both created “now”, so the sort key is nearly identical andsort.Slice— unstable — breaks the tie arbitrarily.Trimsilentlycontinues whenTryLockExclusivecannot be held (lines 181-190) and then evicts the next namespace to get under the cap. A lock that cannot be opened for the oldest namespace yields the same signature: something was removed, no error surfaced, the oldest survived.
Scope
- Decide what
Trimshould order on: file mtimes only, or a single mtime recorded per namespace when it is written. - Decide whether a namespace that cannot be locked is reported rather than skipped
silently, given it is still counted in
total. - Make the test’s two namespaces distinguishable by the key
Trimactually uses, so the outcome does not depend on clock granularity.
Acceptance criteria
go test ./ci/cache -count=20passes inside theverifyimage on a runner, not just on a developer’s ext4 checkout.- The oldest namespace is evicted when both are created inside the same clock tick.
- A namespace that cannot be locked is either reported or excluded from the byte
accounting, so
removedcannot overstate what was freed.