Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#80

P1: CI run records make every human metadata write slower

closed · opened by keenan 11d ago

keenan opened this 11d ago

Why

runs/ holds machine-written CI records, but it lives on the same vole branch as issues and change requests. The package comment in metadata/issues.go says machine-written data must not go there; the code and the layout in docs/architecture.md currently put it there anyway.

The branch today has 45 files under issues/, 65 under requests/, and 849 under runs/ — 118 runs, about 7 files per run. Every issue, comment, edit, approval, and release write reads and rewrites all of them, because of the whole-tree commit path in #79.

So filing an issue gets permanently slower even when nobody files issues. At about 13 ms per file per write, every 100 CI runs adds roughly 9 seconds to every metadata write, on top of the current 12 seconds.

It also means the read limits do not bound the branch. MaxEntries and MaxTreeBytes are checked per prefix (issues/ and requests/), and neither runs/ nor CommitFiles is bounded. Nothing stops machine-written history from making the branch unusable.

Scope

  • Decide where run records live: a separate ref (refs/heads/vole-runs or a sidecar refs/vole/runs), or keep them on vole only if #79 makes the cost independent of tree size.
  • Move existing records with a migration that preserves their history and still passes ValidateIssues.
  • Update ci/runs.WriteRecord, the runner in ci/agent/agent.go, pre-receive, index rebuild, and the docs that name the layout.
  • Preserve the one-push-per-run property: a result and the artifacts it produced stay one commit.
  • Add a test that a run-record write does not touch the issue tree, and an issue write does not read run records.

Acceptance

  • Filing an issue and publishing a run record touch disjoint trees.
  • The number of files on the human metadata branch depends only on issues, requests, and releases.
  • A repository with 10,000 run records still files an issue in the same time as one with none.
  • vole admin index rebuild and ValidateIssues still cover everything they are responsible for.

Related

  • #79 for the write path itself.

Comments (2)

keenan 11d ago

Decision: CI history is instance-local. Move run records out of the vole branch and into data/runs.db; a clone or backup of the repository no longer needs to carry them.

Plan:

  • runs.db becomes the durable store for finished attempts (run_attempts, run_steps, run_attempt_artifacts), not only a rebuildable read model.
  • The runner sends the record with finish over the existing authenticated SSH channel. The server validates it against the lease and the job, checks the log/artifact pointers exist, and writes the record and the run status in one transaction.
  • The runner no longer pushes a record commit. A tagged build that publishes release artifacts still pushes the release pointers, after the record is stored (releases/ stays in Git; it is product metadata, not CI history).
  • Pre-receive rejects new runs/ additions. Existing records are imported into the database by vole admin index rebuild, which becomes the upgrade step; it no longer deletes and rebuilds the finished history from Git.
  • The run page reads steps and artifacts from the database, and the link to “the commit that records it” goes away.
  • Backup/restore and operations docs now treat runs.db as the source of truth for CI history.

Existing runs/ commits cannot be erased without rewriting the protected vole branch and every clone. This change stops new writes and imports the old records; it does not rewrite history.

keenan 11d ago

Implemented on ci-records-in-db, opened as #83.

data/runs.db is now the durable store for finished attempts (run_attempts, run_steps, run_attempt_artifacts). The runner sends the record with finish; the server validates it against the lease and the job, checks the log and artifact pointers, and stores the attempt and the run status in one transaction. The runner no longer pushes a record commit. A tagged build still pushes its release pointers after the record is stored.

Pre-receive refuses new runs/ files. vole admin index rebuild imports records still on an old branch, idempotently, and keeps runs.db as the source going forward. Server and runners must be upgraded together; an old runner finish carries no record and is refused.

Existing runs/ commits remain in history. Erasing them would require rewriting the protected vole branch and every clone, which this change deliberately does not do; it stops new records and imports the old ones.