Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

#129

Paginate the whole activity feed

merged · by keenan 9d ago · merged into main at 20442b59 by keenan 9d ago

keenan opened this 9d ago

Why

Closes #128. Both activity pages stopped at a fixed number of events: the repository feed was truncated at 100, and the instance feed merged 100 per repository and then took 200 before its pager. The instance pager ran out after four pages, and the repository feed had no pager at all. Now that issues, changes, comments and state changes are index rows (#113, #126), the whole feed can be read and paginated.

What changed

  • The repository feed and the instance feed keep the whole sorted, filtered event list; both show 50 events per page.
  • The repository feed gains a pager. Its links swap the #activity-feed fragment over HTMX (like the chips and the filter form), and stay plain hrefs when JavaScript is not there.
  • The kind, actor and assignee filters travel through the page links, so paging never widens what the reader asked for; changing a filter starts at page one.
  • The feed reads comment summaries (parent, id, author, created) instead of every comment body, which it never rendered.
  • The home rail still asks for a bounded feed (homeEventLimit), so the landing page does not become a feed.

Behaviour

A repository with 55 events:

  • page one shows 50, the newest, with Showing 1–50 of 55;
  • page two shows the oldest 5, with Showing 51–55 of 55;
  • the instance feed shows the same;
  • ?kind=issues&actor=keenan keeps both filters in the next/previous links.

The test creates 55 issues directly on the metadata branch and reads both pages of both feeds, plus the filtered link.

Checks

gofmt, go vet ./..., python3 scripts/check-links.py, ./vole workflow check, and the full -race suite with GOFLAGS=-tags=sqlite_fts5.

Trade-off

The feed is assembled in memory per request, so a page costs O(history) rather than O(page). That is fine at this size; if a large history makes it visible, the follow-up is a SQL event stream (a UNION over the source tables) that paginates without materialising every event.

Discussion (3)

keenan 9d ago

Second commit: the feed is now read as a SQL stream.

The first commit paginated the in-memory feed and left an O(history) page cost; this one removes that materialisation. forge/index.Activity and ActivityCount are one UNION over the source rows (issues, comments, approvals, state changes), filtered/ordered/limited in SQL; only the page’s rows cross back into the process. Release events are still read from git (a tag is a ref) and merged with the page in time order. The handlers refresh each repository first, because browser writes do not pass through the push hook, and the repository feed keeps the in-memory path as a fallback for a Web without an index.

What remains: the count and the page each scan the union, so SQLite still sorts the events rather than reading an index in order. A materialised events table with a (repo_id, at DESC) index would make a page O(page) rather than O(history log history); it is the next step if a large history shows the sort.

Numbers on the load harness (concurrency 4, three repositories, a small seeded history of a few dozen events):

page before after
activity 496.4 req/s, p50 7 ms 577.0 req/s, p50 6 ms
instance activity 230.4 req/s, p50 17 ms 222.0 req/s, p50 18 ms
home 202.2 req/s, p50 20 ms 165.2 req/s, p50 23 ms

The fixture is too small for the SQL stream to show its scale benefit; the repository page improves, and the instance/home figures are within run-to-run variance on this machine. The change is about not materialising every event per request, which the fixture cannot exercise.

Checks: gofmt, go vet, the link check, ./vole workflow check, and the full -race suite. Existing activity, filter and pagination tests pass against the SQL stream.

keenan 9d ago

Third commit: the feed is materialised in an events table, which is the design the first two stopped short of.

The UNION version removed the Go-side materialisation but SQLite still scanned and sorted the union to find a page, so a page still cost the history. events has one row per feed event, indexed by (repository, at DESC, id DESC) and, for the instance feed, by (at DESC, id DESC):

  • a repository page is an index range read (no sort, no scan of history);
  • the instance page walks the time index and filters by repository, stopping after offset+limit matches;
  • an unfiltered page total is a stored per-repository count, so the pager does not count the history;
  • a filtered total is the one query that scans, and it is an index scan without a sort.

The row carries the title and assignees the feed shows, and ids rather than URLs, so a repository rename cannot leave a dead link.

Maintenance is part of the existing refresh: an entity’s events are replaced when it is reloaded (a title or assignee edit reaches old events); state changes are rebuilt into events for the entities their commits name, which also catches a close-and-reopen that leaves the tree where it started; releases are refreshed from tags after every push (post-receive) and after a release written in the browser, since a tag is a ref; the stored count moves by what a refresh adds or removes.

Scale check with 100,000 events across 50 repositories:

query time
repository page 1 0.45 ms
repository page 20 (offset 950) 0.28 ms
instance page 1 (50 repos visible) 0.51 ms
instance page 20 0.94 ms
unfiltered total 0.11 ms
filtered total (kind=issue) 12 ms

Load harness (concurrency 4, three repositories, small seeded history):

page UNION events table
activity 577 req/s, p50 6 ms 762 req/s, p50 5 ms
instance activity 222 req/s, p50 18 ms 482 req/s, p50 8 ms
home 165 req/s, p50 23 ms 178 req/s, p50 21 ms

home still uses the in-memory card path; only the activity pages were the target here.

What is left, if it ever matters: deep pages scan offset index entries (inherent to numbered pages; keyset pagination would remove it), and a filtered total scans the filtered index range. Neither sorts the history.

New tests cover the feed’s rows (all event kinds), edits and deletions updating and removing events, the close/reopen round trip an endpoint diff cannot see, tag refresh, multi-repository ordering, pagination, and a query-plan assertion that the page queries use an index and do not create a temp B-tree.

keenan 9d ago

Fourth commit: the home rail uses the events table too.

The landing page still built every visible repository’s whole feed in memory (the old repoActivity path) to show the newest ten events across the instance. It now asks events for one bounded page, with repository names attached, so the rail costs a page — the same query the activity page uses at offset 0.

The cards were split from the feed:

  • open issue/change counts come from the index;
  • a card’s newest event time is one grouped query (LastEventTimes), not a by-product of building its feed;
  • the release chip comes from the newest release event (LastReleases), not a Tags() call per card;
  • the head commit and language still come from git. That is the remaining per-repository work: symbolic-ref, rev-parse, log -1, ls-tree -r per card. A push-time head summary in the index would make the cards SQL-only; it is a separate change.

visibleRepos now uses Repos.Meta (no HEAD lookup), which also removes a symbolic-ref per repository from the instance activity page.

Load harness, concurrency 4, three repositories: home 178 → 243 req/s, p50 21 → 16 ms. New tests cover the rail (the newest ten across repositories, not the whole feed) and LastEventTimes per repository; existing home, activity and pagination tests pass. Full -race suite, vet, gofmt and link check pass.