vole

Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Docs: rewrite the README and organize the guides

520470e19cd906bd6fff987e2d132d793ce56486 · Keenan Nemetz · 16d ago · parent c5902451

Docs: rewrite the README and organize the guides

Replace the long README with a landing page, split feature details into focused guides, and remove the roadmap and spike notes. Update comments that referenced them.

docs/spikes/smart-http/README.md −93

@@ -1,93 +0,0 @@
	1	# Spike B — smart HTTP by driving `git http-backend` as a subprocess
	2
	3	Question: does proxying `git http-backend` from Go give correct and fast git
	4	transport, and what breaks at 1 GiB? Specifically: do hook rejections reach the
	5	client, is a client abort safe, and what does a large push cost in memory?
	6
	7	## Method
	8
	9	A Go `net/http` server runs `git http-backend` per request with CGI env
	10	(`GIT_PROJECT_ROOT`, `PATH_INFO`, `REQUEST_METHOD`, `CONTENT_LENGTH`, …), parses
	11	the CGI response headers and streams the body. Real `git` clients push and clone
	12	over loopback against it.
	13
	14	Fixture: 1,024 MiB in 64 files of 16 MiB, generated two ways, each file made
	15	distinct so nothing dedupes:
	16
	17	- incompressible — random bytes, xor-varied per file. Worst case for git.
	18	- compressible — repeating text. Closer to real source.
	19
	20	Scenarios: hook rejection, incompressible push, compressible push, chunked
	21	upload (`http.postBuffer=1MiB`), client abort mid-push, clone.
	22
	23	## Results
	24
	25	\| Scenario \| Payload \| Wall \| Rate \| Notes \|
	26	\|---\|---\|---\|---\|---\|
	27	\| `hook-rejection` \| 1,024 MiB \| — \| — \| `pre-receive` message reached client verbatim; ref did not move \|
	28	\| `large-push` (incompressible) \| 1,024 MiB \| 28.1 s \| 36.4 MiB/s \| peak git-child RSS 1,134 MB, 4 children \|
	29	\| `large-push` (compressible) \| 1,024 MiB → 7 MiB on disk \| 6.8 s \| 149.6 MiB/s \| same transport, 4.1× faster \|
	30	\| `chunked-upload` (1 MiB postBuffer) \| 1,024 MiB \| 28.1 s \| 36.4 MiB/s \| 1 chunked request, 48.9 MiB spooled to disk \|
	31	\| `client-abort-mid-push` \| — \| — \| — \| no ref created, 0 orphaned git children, 0 leaked spool files \|
	32	\| `clone` \| 1,024 MiB \| 8.7 s \| 117.5 MiB/s \| 3.2× faster than the same data pushed \|
	33
	34	## Findings
	35
	36	1. It works. Hook stderr propagates verbatim to the client
	37	(`remote: POLICY: push rejected by vole`), a rejected push leaves the ref
	38	untouched, and an aborted push creates nothing. No correctness compromise
	39	from using the CGI subprocess.
	40	2. The 36 MiB/s figure is zlib, not the proxy. The same 1 GiB goes at
	41	149.6 MiB/s when compressible and 36.4 MiB/s when random. So the transport
	42	isn't the bottleneck — pack compression is. Do not read 36 MiB/s as "Vole's
	43	push speed"; read it as "worst-case for already-compressed binaries".
	44	3. Push costs ~3.2× clone on identical data (28.1 s vs 8.7 s). `receive-pack`
	45	must index a pack it has never seen; `upload-pack` only streams. This is git's
	46	asymmetry and there is no reason to fight it.
	47	4. Memory is the real constraint. A single 1 GiB push peaked at 1,134 MB
	48	of git-child RSS. Two concurrent pushes of that size would put a 2.5 GB hole
	49	in a small box. Push size must be capped and concurrent pushes bounded.
	50	5. Chunked upload is not theoretical. With git's default `postBuffer` of
	51	1 MiB, a chunked request arrives and `Content-Length` is absent — CGI cannot
	52	proceed without a length, so the body has to be spooled. Observed cost was
	53	48.9 MiB of temp space for a 1 GiB push, not a full second copy, but it is
	54	unbounded per concurrent push and must be capped.
	55	6. Abort cleanup is the sharp edge. Killing only the `git push` parent left
	56	`git-remote-https` running and the push completed anyway. Only killing the
	57	process group produced a real abort. On the server side the equivalent hazard
	58	is a half-received pack that still moves a ref — `receive-pack` already
	59	refuses that, which is why the abort scenario shows no ref.
	60	7. Temp files are cleaned on abort (0 leaked), but that path only runs if the
	61	handler notices the disconnect. A crash or SIGKILL would orphan them, so a
	62	startup janitor is still needed.
	63
	64	## Decision
	65
	66	- Phase 1 transport is smart HTTP via `git http-backend` subprocess. SSH can
	67	wait; this is correct and fast enough.
	68	- Chunked request handling is required, not optional. Spool to a temp file,
	69	set `CONTENT_LENGTH`, stream from disk, delete on completion and on error.
	70	- Add a maximum push size (default 1 GiB) and a concurrency semaphore for
	71	push operations. Without both, memory is unbounded by client behaviour.
	72	- Add a spool janitor on startup for temp files orphaned by a hard crash.
	73	- Document that operators should raise `http.postBuffer` client-side for large
	74	pushes; the server must keep working if they don't.
	75
	76	## Caveats
	77
	78	- Loopback measurement, warm page cache, local NVMe. Network numbers will be
	79	lower; the push/clone ratio should hold.
	80	- The split between the chunked request (48.9 MiB) and the rest of the 1 GiB
	81	transfer was not investigated further — the counters are reliable, the
	82	reason git splits it this way is inferred, not measured.
	83	- Peak RSS is the sum across matching git children (`receive-pack`,
	84	`index-pack`, `pack-objects`, `http-backend`) sampled every 100 ms, so a
	85	shorter spike could be missed.
	86	- Single push at a time. Concurrent-push behaviour is untested and is exactly
	87	what the size cap and semaphore are there to make predictable.
	88
	89	## Reproduce
	90
	91	```sh
	92	go run ./smart-http -sizemb 1024
	93	```