Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 0: measure metadata writes, git transport, and tree serving

c5626f1b87084870524ea54987ef47a65ea4740b · Keenan Nemetz · 18d ago · parent 17311880 

Phase 0: measure metadata writes, git transport, and tree serving

Three throwaway spikes under docs/spikes, each with its own harness,
README, and reproduce command. Findings are folded into the roadmap and
its decision log.

- ref-cas: sharded + batched refs sustain 25,637 entity writes/s at 20k
  entities with zero lost writes; a single unsharded ref manages 30/s
  with 22 CAS retries per write and decays quadratically as it fills.
  Metadata refs must be sharded. Unsharded is now off the table.
- smart-http: driving git http-backend as a subprocess is correct (hook
  stderr reaches the client, aborts leave no ref and no orphaned
  children) and pushes 150 MiB/s on compressible data, but a 1 GiB push
  peaks at 1.13 GB of git-child RSS. Phase 1 needs a push size cap and
  a concurrency semaphore, plus chunked-upload spooling.
- tree-serving: sites can be served straight from git trees with no
  export step - 1.2 ms p50 cold, 0.075 ms warm, 170k req/s behind an
  18.5 MiB cache.

Harness bugs caught during the work are recorded in the spike READMEs,
including one that silently dropped 83% of writes while reporting
plausible throughput.

docs/roadmap.md +60−37

@@ -46,26 +46,19 @@ Explicitly **not** built. Each is additive later; none is load-bearing for the c
46 46 Each phase ends with something runnable and a measurement that proves it, not a checklist
47 47 of merged PRs.
48 48
49 ### Phase 0 — Spikes and decisions (~1 week)
49 ### Phase 0 — Spikes and decisions — ✅ complete
50 50
51 Throwaway code. Nothing merges. The point is to kill bad assumptions while they're cheap.
51 Throwaway code in [`docs/spikes/`](spikes/README.md). Nothing merges. The point
52 is to kill bad assumptions while they're cheap.
52 53
53 **Spike A — metadata write path.** Drive `git update-ref --stdin` from N concurrent Go
54 writers committing to a metadata tree. Measure commits/sec and CAS-retry rate with (a) a
55 single unsharded ref and (b) 256 hash-sharded refs. Also measure batched commits (coalescing
56 N entity writes into one commit) against per-entity commits.
57
58 **Spike B — smart HTTP through Go.** Proxy `git http-backend` as a subprocess, set the CGI
59 env correctly, run a 1 GB push, a `pre-receive` rejection, and a client abort mid-transfer.
60 Confirm hook stderr reaches the client and no orphaned `receive-pack` processes survive.
61
62 **Spike C — serving files from git trees.** Render a 10k-file tree as static pages with a
63 read-through cache. Record p50/p99 and memory for cold and warm cache.
54 | Spike | Result | Decision |
55 |---|---|---|
56 | [`ref-cas`](spikes/ref-cas/README.md) | Sharded + batched metadata writes: 6,966/s at 2k entities, 25,637/s at 20k, zero lost writes. One shared ref: **30/s** with 22 CAS retries per write, decaying to 511/s at 20k. | Metadata refs are **sharded (256)** and writes are **batched**. Non-negotiable. |
57 | [`smart-http`](spikes/smart-http/README.md) | Push 1 GiB at 36 MiB/s (incompressible) or 150 MiB/s (compressible); clone 118 MiB/s. Hook stderr propagates; abort leaves no ref, no orphans. Peak git RSS **1.13 GB** per 1 GiB push. | `git http-backend` subprocess confirmed. Push **size cap + concurrency semaphore** required. |
58 | [`tree-serving`](spikes/tree-serving/README.md) | Cold git read 1.19 ms p50 (12k req/s); warm cache 0.075 ms p50 (170k req/s) holding an 18.5 MiB site. | Serve **straight from trees**, no export step. Cache is required. |
64 59
65 **Exit criteria:** commits/sec at the 95th percentile under 32 writers with retry rate
66 recorded; a 1 GB push succeeds and a rejected push reports a real reason to the client;
67 p99 for a warm static file served from a 10k-file tree. Decisions written into the
68 [decision log](#decision-log).
60 Everything the roadmap assumed survived contact except one thing: an unsharded
61 metadata ref, which is now off the table permanently.
69 62
70 63 ### Phase 1 — It's a git server
71 64
@@ -73,6 +66,11 @@ The unglamorous foundation. If this phase isn't solid, nothing above it is trust
73 66
74 67 - Bare-repo store on disk with a stable repo identity; create/rename/delete via CLI and web
75 68 - Smart HTTP read and write, auth via HTTP basic + personal access tokens
69 - Chunked request handling: spool to temp, set `CONTENT_LENGTH`, clean up on
70 error *and* abort, plus a startup janitor for orphans (spike B: git's default
71 client config produces a chunked request on any large push)
72 - Push size cap (default 1 GiB) and a push concurrency semaphore — a 1 GiB push
73 peaks at ~1.13 GB of git-child RSS, so this is a memory bound, not politeness
76 74 - Users, sessions, PATs (hashed at rest, scoped, revocable)
77 75 - ACL v1: owner / write / read, stored on `refs/vole/acl`
78 76 - `pre-receive` + `update` hooks enforcing ACLs, protected branches, and object size limits
@@ -83,15 +81,19 @@ The unglamorous foundation. If this phase isn't solid, nothing above it is trust
83 81
84 82 **Exit criteria:** authenticated clone/push/fetch works against a fresh instance; an
85 83 unauthorized push is rejected by the hook with an actionable message; `vole index rebuild`
86 reproduces state after deleting the index file; a rejected push leaves no partial ref update.
84 reproduces state after deleting the index file; a rejected push leaves no partial ref update;
85 an aborted 1 GiB push leaves no ref and no spooled temp file; a second concurrent 1 GiB push
86 is queued at the semaphore instead of doubling memory.
87 87
88 88 ### Phase 2 — Metadata layer and issues
89 89
90 90 The phase where the core bet is either proven or abandoned.
91 91
92 - Metadata tree schema: file-per-entity, sharded directories, stable IDs generated as
93 monotonic counters committed alongside the entities
94 - CAS commit helper shared by all metadata writers, with retry and a bounded write queue
92 - Metadata tree schema: file-per-entity, **256 hash-sharded refs** (spike A), stable IDs
93 generated as monotonic counters committed alongside the entities
94 - One CAS commit helper that every metadata writer goes through: shard-local batching on a
95 short timer, retry with jitter on a lost race, and a bounded write queue. No other code
96 path may move a metadata ref
95 97 - Derived index with incremental update on ref change and full rebuild; every query path
96 98 works from a cold index
97 99 - Issues: create, comment, edit, close/reopen, labels, assignees, cross-references
@@ -99,10 +101,11 @@ The phase where the core bet is either proven or abandoned.
99 101 - Markdown rendering with HTML sanitization
100 102 - Notifications: in-app first, email via SMTP config second
101 103
102 **Exit criteria:** deleting the SQLite file and running `vole index rebuild` reproduces
103 every issue byte-for-byte (diffed, not eyeballed); 32 concurrent commenters lose zero
104 comments; `git fetch 'refs/vole/*:refs/vole/*'` on a fresh clone yields a browsable issue
105 history with `git log`.
104 **Exit criteria:** deleting the SQLite file and running `vole index rebuild` reproduces every
105 issue byte-for-byte (diffed, not eyeballed); 32 concurrent commenters lose zero comments;
106 `git fetch 'refs/vole/*:refs/vole/*'` on a fresh clone yields a browsable issue history with
107 `git log`; sustained ≥500 writes/s at 100k entities **including the index update** — spike A
108 measured the ref path alone, so the index is the unmeasured half and must be proven.
106 109
107 110 ### Phase 3 — Change requests
108 111
@@ -152,11 +155,15 @@ from a live ref.
152 155
153 156 ### Phase 6 — Site hosting
154 157
155 - A designated pages ref per repo, served as static files directly from the git tree
158 - A designated pages ref per repo, served as static files directly from the git tree — no
159 export step (spike C). A pool of `git cat-file --batch` processes (default 4; 1→4 doubled
160 throughput) with a read-through in-memory blob cache keyed by `(commit, path)`
161 - Cache is **required, not an optimisation**: cold git reads are ~1.2 ms p50, warm is
162 0.075 ms. LRU eviction keyed by commit, since a busy pages ref would otherwise grow it
156 163 - Per-repo hostname (`<repo>.<instance-domain>`) plus optional custom domain
157 164 - Automatic TLS via ACME when a domain is configured, with DNS-01 for wildcards
158 165 - ETag/`Last-Modified` from blob SHAs; correct `Content-Type` from extension and content sniff
159 - Cache invalidation on push to the pages ref, without a full rebuild
166 - Cache invalidation keyed by commit, so there is no rebuild step to drift
160 167
161 168 **Exit criteria:** a pushed doc tree is live in under 5 seconds; conditional requests return
162 169 `304`; a custom domain gets a certificate with no manual steps; serving a repo with a
@@ -194,11 +201,22 @@ instance with one command on a clean box.
194 201
195 202 ## Risks
196 203
197 **Metadata writes are ref CAS on a shared ref.** The whole state model rides on
198 `git update-ref` throughput, which is roughly hundreds of writes/sec per ref before retry
199 storms. Mitigations, in order of preference: shard refs (Spike A measures the gain),
200 coalesce writes into batched commits, and only then consider a write-ahead log in front of
201 the ref. This is settled in Phase 0, *before* the schema is built around per-entity commits.
204 **Metadata writes: measured, resolved.** Spike A closed this. Sharded + batched refs sustain
205 6,966/s at 2k entities and 25,637/s at 20k with zero lost writes; the unsharded alternative
206 is 30/s and decays quadratically. The decision is made and Phase 2 just has to implement it
207 correctly. The remaining unknown is the **derived-index cost**, which spike A deliberately
208 excluded — hence the ≥500 writes/s-at-100k criterion in Phase 2.
209
210 **Push memory is bounded by client behaviour, not by us.** A single 1 GiB push peaked at
211 1.13 GB of git-child RSS. Three concurrent 1 GiB pushes would OOM a 4 GB box. Mitigation:
212 size cap plus concurrency semaphore in Phase 1, both verified by test, not by configuration
213 review.
214
215 **Chunked uploads consume temp disk per concurrent push.** git's default client config sends
216 chunked requests, which must be spooled to learn `CONTENT_LENGTH`. Observed 48.9 MiB for a
217 1 GiB push, but the quantity scales with what the client sends and with concurrency.
218 Mitigation: spool size cap, cleanup on error and abort, and a startup janitor for files
219 orphaned by a hard crash.
202 220
203 221 **Derived index can silently drift from refs.** Any path that writes refs without updating
204 222 the index is a bug that shows up as missing data. Mitigation: the index is never written
@@ -210,9 +228,9 @@ server-initiated ref updates must route through the same validation as `pre-rece
210 228 the ACL model has two implementations that will disagree. Mitigation: one policy function,
211 229 called from both, with a shared test fixture table.
212 230
213 **Diffs and file serving get expensive at scale before anything else does.** Static serving
214 straight from a tree is elegant and fast until it isn't. Spike C exists to find the cliff
215 before Phase 6 discovers it.
231 **Large diffs remain unmeasured.** Spike C covered static file serving, not three-dot diffs
232 on a CR with thousands of changed files. Phase 3 should measure that before the diff viewer
233 is designed around an assumption.
216 234
217 235 ## Decision log
218 236
@@ -220,8 +238,13 @@ Filled in as phases land. Append; don't rewrite.
220 238
221 239 | Date | Decision | Rationale |
222 240 |---|---|---|
223 | — | Single-tenant, no federation | Keeps scope finite; removes global IDs, peer protocol, and multi-writer storage |
224 | — | Git authoritative, SQLite derived | Rebuildability makes the storage choice reversible |
241 | 2026-10-22 | Single-tenant, no federation | Keeps scope finite; removes global IDs, peer protocol, and multi-writer storage |
242 | 2026-10-22 | Git authoritative, SQLite derived | Rebuildability makes the storage choice reversible |
243 | 2026-10-22 | Metadata refs sharded 256 ways, writes batched | Spike A: 30/s unsharded (and decaying) vs 6,966/s at 2k and 25,637/s at 20k sharded+batched |
244 | 2026-10-22 | CAS ref updates as the only write primitive | Spike A: zero lost writes in every configuration, including 32 writers on one ref |
245 | 2026-10-22 | Smart HTTP via `git http-backend` subprocess | Spike B: hook stderr propagates, aborts leave no ref and no orphans, 150 MiB/s on compressible data |
246 | 2026-10-22 | Push size cap + concurrency semaphore in Phase 1 | Spike B: 1 GiB push peaked at 1.13 GB git-child RSS |
247 | 2026-10-22 | Static sites served from trees, no export step | Spike C: cold 1.2 ms p50; 170k req/s warm with an 18.5 MiB cache |
225 248
226 249 ## Open questions
227 250