Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#83

Store CI run records in the database, not the repository

merged · by keenan 11d ago · merged into main at fdf2340f by keenan 10d ago

keenan opened this 11d ago

Why

#80: CI history is instance-local and should not travel with a clone or backup. At the time of the move, runs/ was 849 of the 959 files on the vole branch, and every human metadata write carried them.

What changed

  • data/runs.db now stores every finished attempt: run_attempts, run_steps, run_attempt_artifacts, plus the existing log, attestation, and artifact rows. RecordAttempt writes the attempt and the run’s result in one transaction.
  • The runner sends the record with finish over the lease’s SSH channel. The server checks the account, run, attempt, commit, workflow, job, steps digest, and image, and verifies every log and artifact pointer is in the object store.
  • The runner no longer pushes a record commit. A tagged build pushes its release pointers after the record is stored, so a release cannot exist without the build record.
  • Pre-receive rejects additions and edits under runs/; deletions are allowed for cleanup.
  • vole admin index rebuild imports records still on an old vole branch (ImportLegacyRecords), idempotently and without deleting database records. It is the upgrade path.
  • The run page reads the stored attempt; the “commit that records it” link is gone.
  • data/runs.db is documented as a source of truth: backup and restore include it, and the upgrade notes say server and runners move together.

What this deliberately does not do

Existing runs/ files remain in past commits, and in the current tip until an operator deletes them. Removing them from history would require rewriting the protected vole branch and every clone, which the workflow forbids. New records never go there.

Tests

  • ci/runs: RecordAttempt stores once, is idempotent on retry, refuses different bytes for the same attempt, and refuses a record for another commit.
  • server/ssh: finish with no record, wrong account, wrong attempt, wrong commit, the real record, an idempotent retry, and a changed record.
  • server/hook: a push adding runs/ is refused and names the database.
  • server integration: an end-to-end run stores the attempt in the database and leaves no record on the branch; index rebuild imports a legacy record; a release build pushes pointers and no runs/.
  • Existing record tests still cover the legacy import format.

Checks

  • go test -count=1 ./...
  • go test -race ./ci/runs ./ci/agent ./server/ssh ./server/hook ./server/web
  • go vet ./...
  • vole workflow check
  • vole workflow run verify

Rollout

Server and runners must be upgraded together; an older runner’s finish carries no record and is refused. Run vole admin index rebuild once to import old records, then back up runs.db.

Discussion (2)

keenan 10d ago

Updated: the compatibility path is gone (59badc6). No runs/ parser, importer, pre-receive allowance, or RunsDir; metadata.ValidateIssues rejects runs/ like any unknown directory, and vole admin index rebuild rebuilds only repositories and issues.

The migration is now a one-time local step, not code we ship:

  1. Build the previous commit (b312e16) and run vole admin index rebuild once to copy runs/ into runs.db.
  2. Rewrite the vole branch to drop runs/ (filter only that path), then prune the old objects.
  3. Run this build. Nothing reads the old branch.

I dry-ran the rewrite on a copy of the server repository: 329 commits became 204, no runs/ remained, and the issue/request addition order was identical (119 paths), so [#42](/vole/n/42) numbering survives.

keenan 10d ago

The live local instance is migrated and its history is rewritten.

  • Backups: pre-migration data and bare repo at /home/keenan/vole-backups/vole-data-20261029-161801.tar.gz and vole.git-20261029-161801; post-migration runs-post-20261029-162031.db and vole.git-post-20261029-162031.tar.gz.
  • Imported 126 attempts (651 step results, 125 logs and attestations) into runs.db.
  • Rewrote the forge repository vole branch: 331 commits to 205, removing only runs/. Issue/request addition order is identical (120 paths), the tree has no runs/, old objects were pruned, and git fsck is clean. The bare repo went from 23 MB to 2.9 MB.
  • Force-fetched the local clone and origin/vole to 49bc113, rebuilt ./vole, and restarted the server and runner on 59badc6.
  • Verified issue numbering (#79/#80/#81 unchanged), CR #83 head 59badc6, old run pages and step logs from runs.db, and a clean index rebuild that leaves runs.db untouched.

The repo-root ./vole was still the old build; it has been rebuilt from this branch, because its index rebuild would have emptied the finished-run rows. Existing clones or mirrors of the forge need a re-clone or a forced vole ref update.