Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#117

Index the metadata branch: lists, counts, detail pages, and search

merged · by keenan 10d ago · merged into main at 73ec800b by keenan 10d ago

keenan opened this 10d ago

Why

The read path is the part of a forge that anonymous traffic reaches. Before this stack, one page render spawned between 38 and 114 git processes, and the busy part grew with the repository: the metadata branch was read one cat-file per issue, comment and change request. Indexing the branch makes lists, counts, detail pages and search database reads, and leaves git for the work that is genuinely git’s: content, diffs, merge checks and history.

Closes #113. The three stages are in three reviewable commits groups on the branch; this request carries them together because the later stages build on the index tables the first one adds.

What changed

Stage 1 — index the whole metadata branch. New requests, request_closes, request_approvals, comments and artifacts tables, refreshed from the same path delta as issues; header counts, the change list, its chips and approval counts, and the release artifact list read the index. The metadata loaders take an already-resolved commit, and GetRequest reads one request instead of every request.

Stage 2 — search. An FTS5 table with one document per issue, change and comment, fed by the same refresh. GET /search (linked from the header) ranks across all three, answers [#42](/vole/n/42)/id prefixes first, marks the matched sentence, and passes the query only the repository ids the viewer may read. A search refreshes the repositories it may see first, so freshness does not depend on a page having been visited.

Stage 3 — detail pages. Issues, changes and the activity feed render from the index. The assembled records are tested field-for-field against metadata’s git readers, including labels, assignees, replies, an anchored comment, closes in file order and approvals. A write request still reads git, because it may have just changed the record.

Git calls per page

Fixture: 40 files, 30 issues, 30 change requests. Original → final.

page before after
overview 114 18 / 6 warm
code 45 9 / 8 warm
tree, blob 45 7
commits 39 4
compare 44 8
issues list 38 2
issue view — 3
changes list 72 3
change view — 9
activity whole-tree reads 6
releases 38 3
search — 2 + one rev-parse per visible repo

The counts that remain are content work: the metadata commit check, branch and tree reads the page shows, the diff and merge check on a change, the backlink search, the state-change log, and tags.

Build and operations

  • Search needs SQLite’s FTS5. The verify and release images, both workflows, README.md, getting-started.md and CONTRIBUTING.md pass sqlite_fts5 beside libsqlite3; the system library must include FTS5 (Debian, Ubuntu, Fedora, RHEL and Rocky all do), and packaging.md says so.
  • index.db now stores bodies and the FTS index, so it grows with the metadata branch. It stays derived: vole admin index rebuild, and the schema bump rebuilds each repository on its first read after upgrade (schema 3 → 6 across this stack).
  • docs/architecture.md records the new index contents.

Checks

./vole workflow run verify passed in the verify image: format, vet, build, license and reachability scans, the .deb/.rpm packaging step, markdown links, and the full -race suite against the system SQLite with FTS5. That is the check that proves the build-tag change, not just the code.

New tests: batched metadata reads, request/comment/approval/close/artifact indexing and incremental updates, rebuild-versus-delta equality, ref listing, FTS5 search (kinds, scoping, numbers, prefixes, deletion, rebuild), private-repo search isolation, and index-assembled detail records versus git.

Discussion (1)

keenan 10d ago

This request recorded head 73ec800, which was already main’s tip: the local perf/fewer-git-processes branch it resolved was stale, and the remote branch was at 14b5c64. Merging it therefore landed nothing, and it closed #113 while the work was still only on the feature branch.

The corrected request is #120 (base main, head 14b5c648, closes #113), and the trap that caused this is filed as #121. #113 has been reopened.