vole

Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 0: measure metadata writes, git transport, and tree serving

c5626f1b87084870524ea54987ef47a65ea4740b · Keenan Nemetz · 18d ago · parent 17311880

Phase 0: measure metadata writes, git transport, and tree serving

Three throwaway spikes under docs/spikes, each with its own harness,
README, and reproduce command. Findings are folded into the roadmap and
its decision log.

- ref-cas: sharded + batched refs sustain 25,637 entity writes/s at 20k
  entities with zero lost writes; a single unsharded ref manages 30/s
  with 22 CAS retries per write and decays quadratically as it fills.
  Metadata refs must be sharded. Unsharded is now off the table.
- smart-http: driving git http-backend as a subprocess is correct (hook
  stderr reaches the client, aborts leave no ref and no orphaned
  children) and pushes 150 MiB/s on compressible data, but a 1 GiB push
  peaks at 1.13 GB of git-child RSS. Phase 1 needs a push size cap and
  a concurrency semaphore, plus chunked-upload spooling.
- tree-serving: sites can be served straight from git trees with no
  export step - 1.2 ms p50 cold, 0.075 ms warm, 170k req/s behind an
  18.5 MiB cache.

Harness bugs caught during the work are recorded in the spike READMEs,
including one that silently dropped 83% of writes while reporting
plausible throughput.

docs/spikes/ref-cas/README.md +96

@@ -0,0 +1,96 @@
	1	# Spike A — metadata writes as CAS ref updates
	2
	3	Question: can issue/CR/ACL writes be commits on sidecar refs, and at what
	4	throughput? The roadmap's core risk is that a comment becomes a `git update-ref`,
	5	so this decides the Phase 2 schema shape.
	6
	7	## Method
	8
	9	Each write produces a blob for one entity, folds it into the shard's tree, makes
	10	a commit, and moves the ref with `git update-ref <ref> <new> <old>` — the old
	11	value makes a lost race fail loudly instead of clobbering. A loser retries with
	12	jittered backoff (capped at 200 attempts).
	13
	14	Four configurations, each run on a fresh repo:
	15
	16	\| Mode \| Refs \| Writers \| Batching \|
	17	\|---\|---\|---\|---\|
	18	\| `shared-naive` \| 1 \| 32 \| one commit per entity \|
	19	\| `sharded-naive` \| 256 \| 32 \| one commit per entity \|
	20	\| `shared-batched` \| 1 \| 1 batcher \| 64 entities per commit \|
	21	\| `sharded-batched` \| 256 \| 8 batchers \| 64 entities per commit \|
	22
	23	Entities are ~450-byte JSON docs. Every run ends with a **verification pass that
	24	counts tree entries in every ref**, so a silently dropped write cannot hide in a
	25	good-looking throughput number. That check earned its place: the first version of
	26	this harness lost 249 of 300 writes in the batched modes because a batcher
	27	discarded work owned by a peer that had already exited. Lossless-looking
	28	throughput was worthless; the write-verification pass caught it.
	29
	30	Hardware: 16-core Linux 7.0, 14 GB RAM, ext4 on local NVMe, git 2.53.
	31
	32	## Results — 2,000 entities
	33
	34	\| Mode \| Entities/s \| CAS retries/entity \| p50 \| p95 \| p99 \| Lost writes \|
	35	\|---\|---\|---\|---\|---\|---\|---\|
	36	\| `shared-naive` \| 30 \| 22.0 \| 623 ms \| 3,641 ms \| 6,368 ms \| 0 \|
	37	\| `sharded-naive` \| 1,240 \| 0.06 \| 23 ms \| 45 ms \| 62 ms \| 0 \|
	38	\| `shared-batched` \| 1,425 \| 0 \| 41 ms/commit \| 53 ms \| 58 ms \| 0 \|
	39	\| `sharded-batched` \| 6,966 \| 0 \| 6.7 ms/commit \| 24 ms \| 36 ms \| 0 \|
	40
	41	A single `git` invocation costs 1.10 ms on this box, and a per-entity commit
	42	costs 5–6 of them. That floor is what all of this is fighting.
	43
	44	## Results — 20,000 entities
	45
	46	\| Mode \| Entities/s \| p50 \| Lost writes \|
	47	\|---\|---\|---\|---\|
	48	\| `shared-batched` \| 511 \| 83 ms/commit \| 0 \|
	49	\| `sharded-batched` \| 25,637 \| 9.8 ms/commit \| 0 \|
	50
	51	This is the important pair. The single-ref design gets worse as it fills:
	52	1,425/s at 2k entities collapses to 511/s at 20k, because every commit re-lists
	53	and rewrites a tree that now holds 20,000 entries. That is O(n²) total work and
	54	it has no floor. The sharded design instead gets faster (6,966 → 25,637/s),
	55	because more entities per shard means batches fill completely.
	56
	57	## Findings
	58
	59	1. An unsharded metadata ref is not viable. 30 writes/s with 22 retries per
	60	write at 32 writers. Comment threads would visibly stall. Rejected outright.
	61	2. Sharding is the single biggest win — 41× on its own (30 → 1,240/s) by
	62	removing ref contention rather than adding hardware.
	63	3. Batching amortises the spawn cost — 5.6× on top of sharding
	64	(1,240 → 6,966/s), by turning 6 processes per entity into 6 per 64 entities.
	65	4. Composition matters more than scale-out. Sharded + batched is 232× the
	66	naive single-ref path on the same box, with zero retries.
	67	5. CAS is the right primitive. Zero lost writes in every configuration,
	68	including the pathological one. Nothing needed a lock server.
	69	6. Sharding bounds tree growth, which is what keeps throughput flat instead
	70	of degrading quadratically. This is the finding that constrains Phase 2.
	71
	72	## Decision
	73
	74	- Metadata refs are sharded (256 shards, FNV-1a of entity id, matching
	75	`shardOf` here). Non-negotiable: the 20k-entity run shows the alternative decays.
	76	- Writers batch entity writes per shard on a short timer.
	77	- Writes go through one CAS commit helper; no other code path may move a
	78	metadata ref.
	79	- Target of ≥500 writes/s sustained at 100k entities is comfortably met with
	80	headroom. Realistic issue traffic is orders of magnitude below this.
	81
	82	## Caveats
	83
	84	- Throughput here excludes the derived-index update, which will add per-write
	85	cost. Re-measure in Phase 2 once the index exists.
	86	- Payloads are small. Large issue bodies or long comment threads will shift
	87	numbers; the batching decision is what protects against that.
	88	- Single box, warm page cache, local disk. Object storage over a network would
	89	move the spawn cost line and should be re-measured if that's ever on the table.
	90
	91	## Reproduce
	92
	93	```sh
	94	go run ./ref-cas -n 2000
	95	go run ./ref-cas -n 20000 -modes shared-batched,sharded-batched
	96	```