vole
Clone
Read only, no credentials:
https://rodent.software/vole.git
Write over SSH with a registered key:
git clone ssh://git@rodent.software:2222/vole.git
A free and open-source forge that lives in your Git repo.
Docs: rewrite the README and organize the guides
520470e19cd906bd6fff987e2d132d793ce56486 · Keenan Nemetz · 16d ago · parent c5902451
Docs: rewrite the README and organize the guides Replace the long README with a landing page, split feature details into focused guides, and remove the roadmap and spike notes. Update comments that referenced them.
docs/spikes/tree-serving/README.md −83
| @@ -1,83 +0,0 @@ | |||
| 1 | # Spike C — serving static sites straight from git trees | ||
| 2 | |||
| 3 | **Question:** the README promises "every repo is a hosted website". Can we serve | ||
| 4 | pages by reading blobs out of the git object store on demand, or do we need an | ||
| 5 | export step before Phase 6? | ||
| 6 | |||
| 7 | ## Method | ||
| 8 | |||
| 9 | A bare repo with **10,000 files** (~18.5 MiB) across 100 directories, built with | ||
| 10 | `git fast-import` in 373 ms. Four servers, 4,000 requests each at 16 concurrent | ||
| 11 | clients, requesting files in a deliberately strided order (step 7919) so the | ||
| 12 | load walks the whole tree rather than hammering one directory: | ||
| 13 | |||
| 14 | | Configuration | Backend | | ||
| 15 | |---|---| | ||
| 16 | | `tree-nocache-1proc` | one long-lived `git cat-file --batch`, no cache | | ||
| 17 | | `tree-nocache-4proc` | four `cat-file` processes round-robin, no cache | | ||
| 18 | | `tree-warmcache-4proc` | four processes + in-memory blob cache | | ||
| 19 | | `filesystem-baseline` | the same tree exported to disk, `http.FileServer` | | ||
| 20 | |||
| 21 | Correctness is asserted, not assumed: a verification pass compares the bytes | ||
| 22 | returned for `dir-00/file-000.html` against the bytes committed. It reports | ||
| 23 | `bytes_match: true`. | ||
| 24 | |||
| 25 | ## Results | ||
| 26 | |||
| 27 | | Server | req/s | p50 | p95 | p99 | Errors | | ||
| 28 | |---|---|---|---|---|---| | ||
| 29 | | `tree-nocache-1proc` | 12,122 | 1.19 ms | 2.65 ms | 4.13 ms | 0 | | ||
| 30 | | `tree-nocache-4proc` | 24,782 | 0.45 ms | 1.77 ms | 2.76 ms | 0 | | ||
| 31 | | `tree-warmcache-4proc` | **169,845** | **0.075 ms** | 0.15 ms | 0.52 ms | 0 | | ||
| 32 | | `filesystem-baseline` | 41,847 | 0.20 ms | 0.78 ms | 3.01 ms | 0 | | ||
| 33 | |||
| 34 | Cache footprint for the entire 10,000-file site: **18.45 MiB**. | ||
| 35 | |||
| 36 | ## Findings | ||
| 37 | |||
| 38 | 1. **Serving straight from trees is viable — no export step needed.** A cold | ||
| 39 | read from git is 1.19 ms at p50, and p99 stays under 5 ms. That is a fine | ||
| 40 | cold-start experience for a docs site. | ||
| 41 | 2. **Cold git reads cost ~6× the filesystem** in throughput (12.1k vs 41.8k | ||
| 42 | req/s single-process), which is the honest price of not keeping an exported | ||
| 43 | copy on disk. Note the p99 of the 4-process git server (2.76 ms) is actually | ||
| 44 | *better* than the filesystem baseline (3.01 ms) — the mean difference is the | ||
| 45 | interesting number, not the tail. | ||
| 46 | 3. **A cache erases the gap entirely** and then some: 4× faster than the | ||
| 47 | filesystem baseline, 170k req/s, sub-millisecond p50. 18.5 MiB to hold a | ||
| 48 | whole 10k-file site in memory is trivially affordable. | ||
| 49 | 4. **One `cat-file` process is not the ceiling.** Going 1→4 processes doubled | ||
| 50 | throughput (12.1k → 24.8k), so reads parallelise across processes. This also | ||
| 51 | means a single serialized pipe is a real bottleneck once traffic is real. | ||
| 52 | 5. Because ETags fall out of the blob SHA for free, conditional requests and | ||
| 53 | cache invalidation are exact — no mtime heuristics, no rebuilding. | ||
| 54 | |||
| 55 | ## Decision | ||
| 56 | |||
| 57 | - Phase 6 serves static sites **directly from the pages ref** via a pool of | ||
| 58 | `git cat-file --batch` processes, with a read-through in-memory blob cache | ||
| 59 | keyed by `(commit, path)`. | ||
| 60 | - **Cache is required, not an optimisation** — it is the difference between | ||
| 61 | "comparable to a filesystem" and "faster than one". | ||
| 62 | - ETag/`Last-Modified` derive from the blob SHA. Cache invalidation on push is | ||
| 63 | keyed by commit, so no rebuild step exists to get out of sync. | ||
| 64 | - Pool size is configurable; default 4, since 1→4 doubled throughput. | ||
| 65 | - No export-to-disk step. That removes an entire class of "the served site | ||
| 66 | doesn't match the repo" bugs. | ||
| 67 | |||
| 68 | ## Caveats | ||
| 69 | |||
| 70 | - The load generator runs on the same box over loopback, so these numbers are | ||
| 71 | dominated by HTTP client overhead at the top end. Relative comparisons are | ||
| 72 | sound; absolute req/s is not a capacity plan. | ||
| 73 | - One commit, one flat-ish tree, all small files (~1.9 KiB). Large binaries in a | ||
| 74 | pages ref would move the cache sizing calculation well past "trivially | ||
| 75 | affordable", and should be pushed to LFS. | ||
| 76 | - Cache keyed by commit means a busy pages ref grows the cache. Needs an eviction | ||
| 77 | policy (LRU by commit) before Phase 6 ships, not a full redesign. | ||
| 78 | |||
| 79 | ## Reproduce | ||
| 80 | |||
| 81 | ```sh | ||
| 82 | go run ./tree-serving -files 10000 -conc 16 -reqs 4000 | ||
| 83 | ``` | ||