vole
Clone
Read only, no credentials:
https://rodent.software/vole.git
Write over SSH with a registered key:
git clone ssh://git@rodent.software:2222/vole.git
A free and open-source forge that lives in your Git repo.
Phase 0: measure metadata writes, git transport, and tree serving
c5626f1b87084870524ea54987ef47a65ea4740b · Keenan Nemetz · 18d ago · parent 17311880
Phase 0: measure metadata writes, git transport, and tree serving Three throwaway spikes under docs/spikes, each with its own harness, README, and reproduce command. Findings are folded into the roadmap and its decision log. - ref-cas: sharded + batched refs sustain 25,637 entity writes/s at 20k entities with zero lost writes; a single unsharded ref manages 30/s with 22 CAS retries per write and decays quadratically as it fills. Metadata refs must be sharded. Unsharded is now off the table. - smart-http: driving git http-backend as a subprocess is correct (hook stderr reaches the client, aborts leave no ref and no orphaned children) and pushes 150 MiB/s on compressible data, but a 1 GiB push peaks at 1.13 GB of git-child RSS. Phase 1 needs a push size cap and a concurrency semaphore, plus chunked-upload spooling. - tree-serving: sites can be served straight from git trees with no export step - 1.2 ms p50 cold, 0.075 ms warm, 170k req/s behind an 18.5 MiB cache. Harness bugs caught during the work are recorded in the spike READMEs, including one that silently dropped 83% of writes while reporting plausible throughput.
docs/spikes/smart-http/README.md +93
| @@ -0,0 +1,93 @@ | |||
| 1 | # Spike B — smart HTTP by driving `git http-backend` as a subprocess | ||
| 2 | |||
| 3 | **Question:** does proxying `git http-backend` from Go give correct and fast git | ||
| 4 | transport, and what breaks at 1 GiB? Specifically: do hook rejections reach the | ||
| 5 | client, is a client abort safe, and what does a large push cost in memory? | ||
| 6 | |||
| 7 | ## Method | ||
| 8 | |||
| 9 | A Go `net/http` server runs `git http-backend` per request with CGI env | ||
| 10 | (`GIT_PROJECT_ROOT`, `PATH_INFO`, `REQUEST_METHOD`, `CONTENT_LENGTH`, …), parses | ||
| 11 | the CGI response headers and streams the body. Real `git` clients push and clone | ||
| 12 | over loopback against it. | ||
| 13 | |||
| 14 | Fixture: 1,024 MiB in 64 files of 16 MiB, generated two ways, each file made | ||
| 15 | distinct so nothing dedupes: | ||
| 16 | |||
| 17 | - **incompressible** — random bytes, xor-varied per file. Worst case for git. | ||
| 18 | - **compressible** — repeating text. Closer to real source. | ||
| 19 | |||
| 20 | Scenarios: hook rejection, incompressible push, compressible push, chunked | ||
| 21 | upload (`http.postBuffer=1MiB`), client abort mid-push, clone. | ||
| 22 | |||
| 23 | ## Results | ||
| 24 | |||
| 25 | | Scenario | Payload | Wall | Rate | Notes | | ||
| 26 | |---|---|---|---|---| | ||
| 27 | | `hook-rejection` | 1,024 MiB | — | — | `pre-receive` message reached client verbatim; ref did not move | | ||
| 28 | | `large-push` (incompressible) | 1,024 MiB | 28.1 s | **36.4 MiB/s** | peak git-child RSS **1,134 MB**, 4 children | | ||
| 29 | | `large-push` (compressible) | 1,024 MiB → 7 MiB on disk | 6.8 s | **149.6 MiB/s** | same transport, 4.1× faster | | ||
| 30 | | `chunked-upload` (1 MiB postBuffer) | 1,024 MiB | 28.1 s | 36.4 MiB/s | 1 chunked request, **48.9 MiB spooled to disk** | | ||
| 31 | | `client-abort-mid-push` | — | — | — | **no ref created**, 0 orphaned git children, 0 leaked spool files | | ||
| 32 | | `clone` | 1,024 MiB | 8.7 s | **117.5 MiB/s** | 3.2× faster than the same data pushed | | ||
| 33 | |||
| 34 | ## Findings | ||
| 35 | |||
| 36 | 1. **It works.** Hook stderr propagates verbatim to the client | ||
| 37 | (`remote: POLICY: push rejected by vole`), a rejected push leaves the ref | ||
| 38 | untouched, and an aborted push creates nothing. No correctness compromise | ||
| 39 | from using the CGI subprocess. | ||
| 40 | 2. **The 36 MiB/s figure is zlib, not the proxy.** The same 1 GiB goes at | ||
| 41 | 149.6 MiB/s when compressible and 36.4 MiB/s when random. So the transport | ||
| 42 | isn't the bottleneck — pack compression is. Do not read 36 MiB/s as "Vole's | ||
| 43 | push speed"; read it as "worst-case for already-compressed binaries". | ||
| 44 | 3. **Push costs ~3.2× clone** on identical data (28.1 s vs 8.7 s). `receive-pack` | ||
| 45 | must index a pack it has never seen; `upload-pack` only streams. This is git's | ||
| 46 | asymmetry and there is no reason to fight it. | ||
| 47 | 4. **Memory is the real constraint.** A single 1 GiB push peaked at **1,134 MB** | ||
| 48 | of git-child RSS. Two concurrent pushes of that size would put a 2.5 GB hole | ||
| 49 | in a small box. Push size must be capped and concurrent pushes bounded. | ||
| 50 | 5. **Chunked upload is not theoretical.** With git's default `postBuffer` of | ||
| 51 | 1 MiB, a chunked request arrives and `Content-Length` is absent — CGI cannot | ||
| 52 | proceed without a length, so the body has to be spooled. Observed cost was | ||
| 53 | 48.9 MiB of temp space for a 1 GiB push, not a full second copy, but it is | ||
| 54 | unbounded per concurrent push and must be capped. | ||
| 55 | 6. **Abort cleanup is the sharp edge.** Killing only the `git push` parent left | ||
| 56 | `git-remote-https` running and *the push completed anyway*. Only killing the | ||
| 57 | process group produced a real abort. On the server side the equivalent hazard | ||
| 58 | is a half-received pack that still moves a ref — `receive-pack` already | ||
| 59 | refuses that, which is why the abort scenario shows no ref. | ||
| 60 | 7. Temp files are cleaned on abort (0 leaked), but that path only runs if the | ||
| 61 | handler notices the disconnect. A crash or SIGKILL would orphan them, so a | ||
| 62 | startup janitor is still needed. | ||
| 63 | |||
| 64 | ## Decision | ||
| 65 | |||
| 66 | - Phase 1 transport is **smart HTTP via `git http-backend` subprocess**. SSH can | ||
| 67 | wait; this is correct and fast enough. | ||
| 68 | - **Chunked request handling is required, not optional.** Spool to a temp file, | ||
| 69 | set `CONTENT_LENGTH`, stream from disk, delete on completion *and* on error. | ||
| 70 | - Add a **maximum push size** (default 1 GiB) and a **concurrency semaphore** for | ||
| 71 | push operations. Without both, memory is unbounded by client behaviour. | ||
| 72 | - Add a **spool janitor** on startup for temp files orphaned by a hard crash. | ||
| 73 | - Document that operators should raise `http.postBuffer` client-side for large | ||
| 74 | pushes; the server must keep working if they don't. | ||
| 75 | |||
| 76 | ## Caveats | ||
| 77 | |||
| 78 | - Loopback measurement, warm page cache, local NVMe. Network numbers will be | ||
| 79 | lower; the push/clone ratio should hold. | ||
| 80 | - The split between the chunked request (48.9 MiB) and the rest of the 1 GiB | ||
| 81 | transfer was **not** investigated further — the counters are reliable, the | ||
| 82 | reason git splits it this way is inferred, not measured. | ||
| 83 | - Peak RSS is the sum across matching git children (`receive-pack`, | ||
| 84 | `index-pack`, `pack-objects`, `http-backend`) sampled every 100 ms, so a | ||
| 85 | shorter spike could be missed. | ||
| 86 | - Single push at a time. Concurrent-push behaviour is untested and is exactly | ||
| 87 | what the size cap and semaphore are there to make predictable. | ||
| 88 | |||
| 89 | ## Reproduce | ||
| 90 | |||
| 91 | ```sh | ||
| 92 | go run ./smart-http -sizemb 1024 | ||
| 93 | ``` | ||