Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#47

Bound and stream the anonymous read path

merged · by keenan 13d ago · merged into main at 4581690d by keenan 13d ago

keenan opened this 13d ago

Closes #3.

Why

Four anonymous read paths held a whole git object in memory before deciding what to do with it:

  • server/web/codebrowser.go read the entire blob, then kept the first MiB.
  • server/web/overview.go read the entire README to render it.
  • server/web/releases.go read the entire .gitattributes to look for one pattern.
  • server/pages/catfile.go allocated the full object for every site read.
  • metadata.LoadIssues/LoadRequests read every metadata blob in full.

A repository is untrusted content. One 100 MiB blob — which max_blob_bytes allows — turned every concurrent request for it into a 100 MiB allocation, with no account needed to make the request.

What

Read the header first. core/git grows BlobSize, BlobSizes (one cat-file --batch-check for a whole tree) and BlobStream: a one-object cat-file --batch whose header is parsed before any payload is touched. A stream that is only partly read stops the process instead of draining it, so a prefix read of a large blob costs nothing beyond the prefix.

Stream what is a download, cap what is a page. /raw copies the stream to the wire with a Content-Length taken from the object, so a 100 MiB asset costs one 64 KiB buffer. The inline viewer reads a one-MiB prefix and states the file’s real size rather than the size of what it shows. The README is capped at 1 MiB and is named and measured instead of rendered when it is larger. Site blobs are capped at 25 MiB with a 413 that says so; the commit and tree objects read to resolve a path are capped at 64 MiB.

Cap metadata on both sides. A metadata file is limited to 4 MiB, a body to 2 MiB (the web form limit, so a body the browser accepts cannot fail later), issues/ and requests/ to 20,000 files and 64 MiB together. The loader asks git for every size once and refuses the tree before reading it; LoadIssues also stops counting runs/ and releases/ against the issue file cap, which it was doing by accident. Pre-receive refuses the same tree, so a push that would break an issue list is rejected with a reason instead of accepted.

Tests

  • core/git: size, prefix, truncation, missing object, early close, and a 16 MiB stream copied through a 32 KiB buffer whose total allocation stays under 4 MiB.
  • server: an 8 MiB blob renders as a 1 MiB preview with its real size, /raw returns all of it with a matching Content-Length, an oversized README is reported rather than rendered, and a 5 MiB issue file is rejected at push.
  • server/pages: an oversized page is a 413 and the pool serves the next request normally after the refused process was killed.
  • metadata: the loader and the push validator both refuse an oversized file, and the writers refuse an oversized body.

gofmt -l, go vet ./..., go test ./..., and go test -race on the touched packages are clean.

Discussion (0)

No discussion yet.