Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Spike D: a source archive is a 54 ms stream, not a stored build

ada367f1067bf80ae094f2f0f158f69090b3a741 · Keenan Nemetz · 18d ago · parent 78912785 

Spike D: a source archive is a 54 ms stream, not a stored build

Answers whether release tarballs can be generated per request from a tag
instead of stored. Measured with git 2.53 on three generated fixtures
(20k files / 100 MiB with binaries, 40k and 100k file text trees) and one
real repository:

- a real 11.6 MiB repo archives in 54 ms plain, 221 ms gzipped, first byte
  at 4.1 ms; the 100 MiB binary fixture takes 0.12 s plain, 1.64 s gzipped
- output is byte-identical across runs and across a fresh clone, so an ETag
  can come from the tag object id and a cache could never serve stale bytes
- core.compression is inert for git archive: levels 1, 6 and 9 produce the
  same sha256 and the same time
- zstd via tar.<fmt>.command is 14x faster than gzip at the same output size
  (0.117 s vs 1.643 s for 80 MiB), for 6x the RSS
- gzip barely pays on binaries (119.2 -> 80.4 MiB) and does not parallelise:
  8 concurrent gzipped streams reach 273 MiB/s against 5,005 MiB/s plain
- peak RSS tracks file count (55 MiB at 100k files), so a per-request archive
  needs the same concurrency bound the push path has
- git archive emits LFS pointers verbatim, so a source tarball of an LFS repo
  ships pointers unless the server resolves them

Every fixture and the real repo are verified by hashing each tar entry as a
git object and comparing to ls-tree: 2,566 / 20,040 / 40,000 / 100,000 files
matched, none mismatched.

docs/spikes/README.md +3

@@ -9,6 +9,7 @@ decision it produced. Nothing here ships; the code exists to be re-run.
9 9 | [ref-cas](ref-cas/README.md) | Can metadata be commits on sidecar refs, and at what rate? | Yes. Sharded + batched: 6,966/s at 2k entities, 25,637/s at 20k. A single ref is 30/s and decays. |
10 10 | [smart-http](smart-http/README.md) | Is `git http-backend` as a subprocess correct and fast enough? | Yes. Hook messages propagate, aborts are clean. 1 GiB push peaks 1.13 GB RSS — needs a size cap. |
11 11 | [tree-serving](tree-serving/README.md) | Can sites be served straight from git trees? | Yes. Cold 1.2 ms p50 from git; 170k req/s warm with an 18 MiB cache. No export step needed. |
12 | [archive-streaming](archive-streaming/README.md) | Can release tarballs be generated per request instead of stored? | Yes. Real repo 54 ms plain / 221 ms gzipped, byte-identical across runs and clones. gzip is 14× slower than zstd and its level knob does nothing. |
12 13
13 14 Run any of them with:
14 15
@@ -16,6 +17,8 @@ Run any of them with:
16 17 go run ./ref-cas -n 2000
17 18 go run ./smart-http -sizemb 1024
18 19 go run ./tree-serving -files 10000 -conc 16 -reqs 4000
20 go run ./archive-streaming -build-fixture /tmp/fix -files 20000 -dirs 200 -binarymb 2
21 go run ./archive-streaming -repo /tmp/fix -rev HEAD
19 22 ```
20 23
21 24 Shared conventions worth keeping if these are ever rewritten: