Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#113

Index the metadata branch for lists, counts, and search

closed · opened by keenan 10d ago

keenan opened this 10d ago

Why

index.db only holds issues: issues, issue_labels, issue_assignees, entity_numbers. Change requests, approvals, closes, comments and release artifacts are read from the vole branch on every page that needs them, and the repository header reads the whole requests subtree just to print a count.

#112 cut the page cost from 38–114 git calls to 5–19 on the fixture. What is left is mostly the metadata read itself: rev-parse, ls-tree -r, cat-file --batch-check, cat-file --batch on every page whose header shows the open-changes badge. That read grows with the number of issues and comments, so it is the part that gets worse with use.

Indexing the rest of the branch makes the header, the lists and search SQLite reads, and it is the prerequisite for a search surface across issues, changes and comments. The index stays derived: git remains authoritative, Rebuild and the schema-version reset remain the recovery path.

Related: #112, #99, #100.

Scope

Stage 1 — index the whole branch

  • Tables: requests, request_closes, request_approvals, comments (issues and requests), artifacts (release pointers). Schema version bumped so the existing drop-and-rebuild path applies.
  • Ensure refreshes issues and requests from the same path delta; a changed issue or request replaces its own rows, its comments, and its closes/approvals. Deletions remove the rows; numbers still come from history.
  • Rebuild (vole admin index rebuild) and the first-read path index every table, not just issues.
  • Web: the repository header’s open-changes count, the change list and its state filters, the number lookup and the release artifact list read the index. Detail pages keep reading git, but a change page reads its one request instead of every request.
  • The index is still disposable: Rebuild reproduces every row from the branch alone.

Stage 2 — universal search

  • FTS5 documents table over issue titles/bodies, request titles/bodies and comment bodies, fed by the same refresh. Requires sqlite_fts5 in the build (verify workflow, packaging, CONTRIBUTING.md).
  • One search page across issues, changes and comments, plus the per-repo surfaces that exist today. [#42](/vole/n/42) numbers and ULID prefixes keep working.
  • Results are filtered by the same visibility rule the pages use: a private repository’s titles must not appear in a global search.

Stage 3 — drop the redundant reads

  • Once stage 1 has run for a while, render detail pages from the index (Ensure already runs before the read) and remove the remaining git reads from page rendering.
  • Re-measure with the harness from #99 before deciding whether the persistent cat-file pool is still worth it.

Acceptance

  • A page that only shows the header count performs no metadata cat-file/ls-tree (verified with a git shim, as in #112).
  • vole admin index rebuild on a fixture with issues, changes, comments, approvals and artifacts reproduces identical rows.
  • The change list, its filters and its approval counts match what the git-read page showed, including stale approvals that name an older head.
  • Search (stage 2) finds an issue title, a change title and a comment body from one query, and never returns a repository the viewer cannot read.

Costs

The DB grows with the branch (bodies are allowed up to 2 MiB per comment); Rebuild becomes a full metadata parse per repository; every push re-indexes the entities it touched.

Comments (3)

keenan 10d ago

Stage 1 is implemented as #114, stacked on #112 (base perf/fewer-git-processes), so it can be reviewed while that one waits for an approval. It adds the requests, comments, approvals, closes and artifacts tables and switches the header badge, the change list and the release artifact list to the index. Conversation pages still read their record from git, as planned.

Git calls per page on the fixture: issues list 5 → 2, change list 6 → 3, releases 6 → 3, commits 7 → 4, overview 19 → 6 on a repeat view.

Next is stage 2 (FTS5 documents table and the search surfaces), which needs the sqlite_fts5 build tag in the verify workflow, packaging, and CONTRIBUTING.md.

keenan 10d ago

Stage 2 is #115 (“Search issues, changes and comments from one page”), based on perf/fewer-git-processes now that #112 and #114 have landed there.

It adds the FTS5 search table (fed by the same refresh as the list rows, including comment bodies and deletions), GET /search linked from the header, exact [#42](/vole/n/42)/id-prefix results ahead of ranked text matches, and ACL filtering by passing the query only the repository ids the viewer may read. Search refreshes the repositories it may see before querying, so freshness does not depend on a page having been visited.

The build now asks for FTS5: the verify/release images and workflows, README, getting-started and CONTRIBUTING pass sqlite_fts5 beside libsqlite3, and packaging notes that the system SQLite must include it. A build without the tag fails at index.Open with that reason rather than a bare “no such module”.

Not in stage 2: a per-repository unified search page (the existing issue/change filters and code search are unchanged), and a push-time index refresh from the pre-receive hook — a search currently resolves each visible repository once. Stage 3 (detail pages and the activity feed off git) is next, after #99’s harness gives us numbers.

keenan 10d ago

Stage 3 is #116 (“Render issues, changes and the activity feed from the index”), based on perf/fewer-git-processes now that #115 has landed there.

The issue and change detail pages and the activity feed now read the rows the request already refreshed. A write request still reads git, because it has no memo and may have just changed the record. Git calls per page on the fixture: issue view 7 → 3, change view 12 → 9, activity from whole-tree reads to one metadata log, one commit and one tag listing. What remains is content work (backlink search, diff, merge check, state-change log, tags).

Two follow-ups are now the obvious next candidates, both with their own measurement: a push-time index refresh from the pre-receive hook (removes the rev-parse every read and every search pays) and indexing state changes so the activity feed stops reading the metadata log. The persistent cat-file pool still waits for the #99 harness, since we have call counts rather than throughput.