vole
Clone
Read only, no credentials:
https://rodent.software/vole.git
Write over SSH with a registered key:
git clone ssh://git@rodent.software:2222/vole.git
A free and open-source forge that lives in your Git repo.
Docs: rewrite the README and organize the guides
520470e19cd906bd6fff987e2d132d793ce56486 · Keenan Nemetz · 16d ago · parent c5902451
Docs: rewrite the README and organize the guides Replace the long README with a landing page, split feature details into focused guides, and remove the roadmap and spike notes. Update comments that referenced them.
docs/spikes/archive-streaming/README.md −156
| @@ -1,156 +0,0 @@ | |||
| 1 | # Spike D — streaming release archives out of git | ||
| 2 | |||
| 3 | **Question:** a release page wants a source tarball per tag. Do we store it, or | ||
| 4 | generate it per request from the tag? And what does generating cost in time, | ||
| 5 | memory and CPU when it happens on every download? | ||
| 6 | |||
| 7 | ## Method | ||
| 8 | |||
| 9 | Three fixtures built with `-build-fixture` (the builder is in `main.go`, so the | ||
| 10 | numbers below are re-runnable on any box) plus one real repository: | ||
| 11 | |||
| 12 | | Fixture | Files | Tree | What it exercises | | ||
| 13 | |---|---|---|---| | ||
| 14 | | `fix20k` | 20,040 | 99.7 MiB (80 MiB of it incompressible binaries) | A binary release: the byte-bound case | | ||
| 15 | | `fixtext` | 40,000 | 156.7 MiB of text | A compressible tree | | ||
| 16 | | `fix100k` | 100,000 | 98.8 MiB of text | Per-file metadata cost | | ||
| 17 | | `defradb` | 2,566 | 11.6 MiB | A real repository, real pack layout | | ||
| 18 | |||
| 19 | For each, git 2.53.0 streams `git archive --format=<fmt> HEAD` to `io.Discard` | ||
| 20 | while the harness measures wall time, produced bytes, peak child RSS | ||
| 21 | (`rusage`), and time to first byte. Five runs per configuration; the table shows | ||
| 22 | the fastest, because the question is "what can this cost" rather than "what did | ||
| 23 | one noisy run cost". | ||
| 24 | |||
| 25 | Configurations: plain `tar`, `tar.gz`, `tar.gz` with `core.compression=1` and | ||
| 26 | `=9`, and `tar.zst` declared as a custom format | ||
| 27 | (`-c tar.zst.command='zstd -3 -c'`), which git pipes to without a temp file. | ||
| 28 | Concurrency levels 1/4/8 measure aggregate throughput on the same revision. | ||
| 29 | |||
| 30 | **Verification is part of the harness.** For every fixture and the real repo, | ||
| 31 | the tar stream is parsed and each regular file's content is hashed as a git | ||
| 32 | object (`blob <size>\x00` + bytes) and compared with the blob id `ls-tree -r` | ||
| 33 | reports for that path. All four report every file matched, 0 mismatched — so | ||
| 34 | the fast numbers are the *correct* bytes, not a truncated or reordered stream. | ||
| 35 | Determinism is measured by hashing the same archive repeatedly (3 runs) and | ||
| 36 | again from a fresh `git clone` of the fixture. | ||
| 37 | |||
| 38 | ## Results | ||
| 39 | |||
| 40 | The binary release — this is the case that decides the design, because | ||
| 41 | compression has nothing to win: | ||
| 42 | |||
| 43 | | Configuration | MiB/s | wall s | out MiB | peak RSS MiB | TTFB ms | | ||
| 44 | |---|---|---|---|---|---| | ||
| 45 | | `tar` | **977.0** | **0.122** | 119.2 | 14.7 | 12.5 | | ||
| 46 | | `tar.gz` | 49.0 | 1.643 | 80.4 | 14.7 | 12.5 | | ||
| 47 | | `tar.gz` level 1 | 49.2 | 1.636 | 80.4 | 14.7 | 12.4 | | ||
| 48 | | `tar.gz` level 9 | 49.1 | 1.638 | 80.4 | 14.7 | 13.4 | | ||
| 49 | | `tar.zst` (zstd -3) | 685.1 | 0.117 | 80.1 | 87.9 | 23.3 | | ||
| 50 | |||
| 51 | Text trees (`fixtext` 40k files / 156.7 MiB, `fix100k` 100k files / 98.8 MiB), | ||
| 52 | where compression actually pays and MiB/s of *input* is a misleading figure — | ||
| 53 | read the wall time and the output size: | ||
| 54 | |||
| 55 | | Fixture | `tar` | `tar.gz` | `tar.zst` | | ||
| 56 | |---|---|---|---| | ||
| 57 | | `fixtext` | 0.136 s → 195.4 MiB | 0.532 s → 1.2 MiB | 0.129 s → 0.3 MiB | | ||
| 58 | | `fix100k` | 0.235 s → 195.8 MiB | 0.767 s → 2.0 MiB | 0.250 s → 0.5 MiB | | ||
| 59 | | `defradb` (real) | 0.054 s → 13.7 MiB | 0.221 s → 1.8 MiB | 0.065 s → 1.8 MiB | | ||
| 60 | |||
| 61 | Peak RSS by file count: 14.7 MiB at 20k files, 21.5 MiB at 40k, **55.0 MiB at | ||
| 62 | 100k**. Time to first byte: 4.1 ms on the real repo, 12–32 ms on the fixtures, | ||
| 63 | 53.7 ms at 100k files. | ||
| 64 | |||
| 65 | Aggregate throughput at concurrency 1 / 4 / 8 (MiB/s): | ||
| 66 | |||
| 67 | | Format | conc 1 | conc 4 | conc 8 | | ||
| 68 | |---|---|---|---| | ||
| 69 | | `tar` (`fix20k`) | 988.7 | 2,760.4 | 5,005.1 | | ||
| 70 | | `tar.gz` (`fix20k`) | 49.0 | 192.4 | 273.4 | | ||
| 71 | | `tar` (`fix100k`) | 845.8 | 2,848.7 | 4,240.5 | | ||
| 72 | | `tar.gz` (`fix100k`) | 2.6 | 10.3 | 14.1 | | ||
| 73 | |||
| 74 | Determinism: `tar` and `tar.gz` both hash **identically across three runs and | ||
| 75 | from a fresh clone** on every fixture — `git archive` sets entry mtimes from the | ||
| 76 | commit, so output is a pure function of the revision. | ||
| 77 | |||
| 78 | ## Findings | ||
| 79 | |||
| 80 | 1. **Generating a source tarball is a tens-of-milliseconds operation, not a | ||
| 81 | build.** A real 11.6 MiB repository archives in 54 ms plain and 221 ms | ||
| 82 | gzipped. The 100 MiB binary fixture gzips in 1.6 s; the same tree as plain | ||
| 83 | tar takes 0.12 s. Storing a tarball would buy nothing that the tag does not | ||
| 84 | already pin. | ||
| 85 | 2. **`core.compression` is a dead knob for `git archive`.** Levels 1, 6 and 9 | ||
| 86 | produce byte-identical output — same sha256, same size, same time | ||
| 87 | (measured on a 156.7 MiB text tree and on the 100 MiB binary tree). There is | ||
| 88 | no cheap way to trade ratio for speed with the built-in gzip; the only lever | ||
| 89 | is a different compressor. | ||
| 90 | 3. **zstd is 14× faster than gzip at the same output size** on the binary | ||
| 91 | fixture (0.117 s / 80.1 MiB vs 1.643 s / 80.4 MiB), for 6× the RSS (87.9 vs | ||
| 92 | 14.7 MiB). It needs no Vole code: `tar.<fmt>.command` makes git pipe to any | ||
| 93 | binary. That also makes it an operator opt-in — git errors on an unknown | ||
| 94 | format when the command is unset, so the default must not depend on it. | ||
| 95 | 4. **Compression barely pays for binaries.** 80.4 MiB gzipped out of 119.2 MiB | ||
| 96 | of tar is a 32% saving for 13× the CPU. For a release archive of build | ||
| 97 | outputs, plain `tar` is the right default and gzip is a request option. | ||
| 98 | 5. **Memory scales with file count, not bytes** — 55 MiB at 100k files against | ||
| 99 | 98.8 MiB of content, versus 14.7 MiB at 20k files. Streaming one archive per | ||
| 100 | download means peak memory is *concurrent downloads × that figure*, so the | ||
| 101 | serving path needs the same concurrency bound the push path already has. | ||
| 102 | (The real repo breaks the per-file trend — 27.8 MiB for 2,566 files — so pack | ||
| 103 | layout contributes too; treat any figure here as "this tree, this pack".) | ||
| 104 | 6. **First byte arrives in single-digit milliseconds on a real repo**, so a | ||
| 105 | download can start streaming immediately: no temp file, no buffering the | ||
| 106 | whole archive, no `Content-Length` requirement. | ||
| 107 | 7. **Plain tar scales to 8 concurrent streams (5,005 MiB/s aggregate); gzip | ||
| 108 | does not** (273 MiB/s, ~34 MiB/s per stream). gzip is the only configuration | ||
| 109 | that needs a semaphore. | ||
| 110 | 8. **`git archive` emits LFS pointers verbatim.** A pointer file committed to | ||
| 111 | the fixture comes out of the tarball as the three-line pointer text, so a | ||
| 112 | naive source archive of an LFS repository ships shopping lists instead of | ||
| 113 | bytes. | ||
| 114 | |||
| 115 | ## Decision | ||
| 116 | |||
| 117 | - **Release source archives are generated per request from the tag, never | ||
| 118 | stored.** Determinism (finding: identical across runs and clones) means the | ||
| 119 | `ETag` can be derived from the tag object id plus the format, so a cache is a | ||
| 120 | pure optimisation that cannot serve stale bytes — and it is not needed at | ||
| 121 | 54–221 ms per request for a real repository. | ||
| 122 | - **Default format is plain `tar`**; `tar.gz` is an explicit request option, and | ||
| 123 | `tar.<fmt>.command` (zstd) is documented as the operator lever. The gzip level | ||
| 124 | knob is not exposed anywhere, because it does nothing. | ||
| 125 | - **Bound concurrent archive requests** with a semaphore, at least for | ||
| 126 | compressed formats: 8 gzipped streams of a 100 MiB repo cost 13 s of CPU. | ||
| 127 | - **A source archive with LFS pointers in it must say so**, or resolve them from | ||
| 128 | the object store while streaming. Confirmed by measurement (finding 8); the | ||
| 129 | resolve-versus-document choice belongs to the slice, not the spike. | ||
| 130 | - **Binary artifacts are not affected by any of this.** They stay in the LFS | ||
| 131 | store, addressed by sha256, because they are uploads, not derivations. | ||
| 132 | |||
| 133 | ## Caveats | ||
| 134 | |||
| 135 | - One box, one client, `io.Discard` as the sink: no network, no TLS, no proxy. | ||
| 136 | Relative comparisons are sound; absolute throughput includes the disk cache. | ||
| 137 | - Fixtures are freshly built single-commit repos with tidy packs. The real repo's | ||
| 138 | RSS hint says pack layout matters, and a delta-heavy history may be slower than | ||
| 139 | any fixture here. | ||
| 140 | - `git archive` honours `export-ignore` and `export-subst` from `.gitattributes`. | ||
| 141 | None of the measured trees use them; a repository that does would fail the | ||
| 142 | verification pass, which is the intended behaviour — the criterion is "matches | ||
| 143 | `ls-tree` except where the repository says otherwise". | ||
| 144 | - zstd figures use the external `zstd` binary at level 3. No other levels were | ||
| 145 | measured, and git on this box has no built-in `tar.zst`. | ||
| 146 | |||
| 147 | ## Reproduce | ||
| 148 | |||
| 149 | ```sh | ||
| 150 | cd docs/spikes | ||
| 151 | go run ./archive-streaming -build-fixture /tmp/fix20k -files 20000 -dirs 200 -filekib 1 -binarymb 2 | ||
| 152 | go run ./archive-streaming -repo /tmp/fix20k -rev HEAD -runs 5 -conc 1,4,8 -clone-to /tmp/fix20k-clone | ||
| 153 | go run ./archive-streaming -build-fixture /tmp/fixtext -files 40000 -dirs 200 -filekib 4 -binarymb 0 | ||
| 154 | go run ./archive-streaming -repo /tmp/fixtext -rev HEAD -runs 5 -conc 1,4,8 | ||
| 155 | go run ./archive-streaming -repo /path/to/any/repo -rev HEAD | ||
| 156 | ``` | ||