Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#112

Read the metadata tree once per page

merged · by keenan 10d ago · merged into main at 73ec800b by keenan 10d ago

keenan opened this 10d ago

Why

The read path is anonymous and every page render shells out to git. On a repository with 40 files and 30 change requests, one page load cost 38 to 114 git process invocations:

page before after
overview 114 calls, 98 ms 21 calls, 21 ms
code 45 calls, 57 ms 13 calls, 19 ms
issues 38 calls, 48 ms 7 calls, 12 ms
changes 72 calls, 90 ms 8 calls, 13 ms
releases 38 calls, 45 ms 7 calls, 12 ms

The worst of it scaled with the repository’s contents: the metadata branch was read one git cat-file per file, so every page paid for every issue and comment. The rest was repeated work inside one request — the repository opened two or three times, and the change-request list read once for the header and again for the page.

What changed

  • metadata: the loaders and the push validators read every listed blob in one cat-file --batch instead of one process per file.
  • core/git: Refs lists branches and tags in one for-each-ref, with creation times so the tag picker stays newest-first.
  • server/web: one memo per read request holds the opened repository, the open counts, and the change-request list. Write requests get no memo, so the cache dies with the request and a push is visible on the next page load without a TTL or an invalidation protocol.
  • The repository header takes the page it renders (nav .Page "issues") instead of six arguments the view already carries. Output is unchanged.

Checks

  • ./vole workflow check
  • go vet ./..., gofmt
  • CGO_ENABLED=1 go test -race ./... (the libsqlite3-tagged build could not run here: no sqlite3.h on this machine)
  • New tests: TestLoadRequestsReadsEveryFile (batched read, including two paths that share one object) and TestRefsListsBranchesAndTagsTogether.

Further read-path work (memoising the index checks, ETags on immutable views) follows on this branch, so the measurements above are a floor, not the final numbers.

Discussion (1)

keenan 10d ago

Pushed a second round on the branch, so this tip needs its own approval and checks again.

What changed since the first push

  • The index refresh (issue rows, filters, [#42](/vole/n/42) numbers) ran on every surface that shows one, and each run resolved the metadata ref first. It now runs once per request, and the resolved commit is shared with the change-request read.
  • The overview walked the whole history on every load: shortlog for contributors, rev-list --count for the code page, and the tree grouping for languages. A commit id is content-addressed, so those answers cannot go stale: they are remembered by commit id in a bounded per-process cache that evicts oldest-first (256 entries). Nothing account-specific is cached.

Git calls per page (fixture: 40 files, 30 issues, 30 change requests)

First load / repeat load, after all four commits:

page before after (first / repeat)
overview 114 19 / 9
code 45 12 / 11
tree 45 10 / 10
blob 45 10 / 10
commits 39 7 / 7
file history 40 8 / 8
compare 44 11 / 11
issues 38 5 / 5
changes 72 6 / 6
releases 38 6 / 6

Wall-clock per request is roughly calls × 1.2 ms plus rendering, but this machine’s load swung 14–66 ms for the same code between runs, so the call counts are the number to trust here. #99/#100 should replace them with measured percentiles on a quiet instance.

What is left, and why I did not do it here

  • The metadata read is still four calls (shared rev-parse, ls-tree -r, cat-file --batch-check, cat-file --batch). Collapsing the last two into one needs cat-file --batch-command, which is git 2.36+; packaging targets distributions that still ship 2.34, so that would need a documented minimum version first.
  • A cross-request cache of the parsed request list, keyed by the metadata commit, would remove those four calls from every page. It needs a byte budget as well as an entry budget (the tree is capped at 64 MiB), which is its own change request.
  • ETags on immutable views save bandwidth but not server work: the header’s open counts change with every issue, so a whole-page validator is not stable the way /static/ is.