vole
Clone
Read only, no credentials:
https://rodent.software/git/vole.git
Write over SSH with a registered key:
git clone ssh://git@0.0.0.0:2222/vole.git
A free and open-source forge that lives in your Git repo.
Phase 5: LFS objects, over SSH for writes and the proxy for reads
64ac5071a173e4e9ce8bc8d7170f9b0411bee52a · Keenan Nemetz · 17d ago · parent 35eaeccf
Phase 5: LFS objects, over SSH for writes and the proxy for reads Large files leave the git object database and land in one content-addressed store per instance. Reads are anonymous like clone; writes ride the SSH key that already authenticates the push, so no token exists anywhere. - internal/lfs: sha256-addressed store that verifies content and size before an object becomes visible; parser for the pointer format a client writes - HTTP: POST .../info/lfs/objects/batch for downloads, GET .../objects/<oid> with the object id as ETag; upload over HTTP is refused and names SSH - SSH: git-lfs-transfer speaking the pure SSH protocol (pkt-line), so `git lfs push` needs nothing but the key at the handshake - pre-receive: a pointer whose object is missing, or whose size disagrees with the stored object, fails the push that introduced it; a binary over lfs_warn_bytes is accepted with a `git lfs track` suggestion. Checks run before the protected-branch gate, because an admin's approval bypass is about review, not about whether the commit can be checked out - runs: a finished log becomes a content-addressed object in the same write that records the status, served at /r/<repo>/runs/<id>/log - vole gc [--dry-run] [--grace]: marks pointer targets from every ref in every repository plus archived logs, sweeps the rest. Mark-and-sweep rather than reference counts: a counter that is one too low deletes an artifact, a walk over the refs cannot be wrong about what is reachable
docs/roadmap.md +59−11
| @@ -269,7 +269,8 @@ Remaining: | |||
| 269 | 269 | timeout, streaming logs | |
| 270 | 270 | - Run records pin SHA + workflow file hash + toolchain env; signed with the instance key | |
| 271 | 271 | - Statuses surfaced on CRs and satisfying required-check gating from Phase 3 | |
| 272 | - Log and artifact storage onto the content-addressed store (shared with Phase 5) | ||
| 272 | - Log and artifact storage onto the content-addressed store (shared with Phase 5). | ||
| 273 | Logs moved in Phase 5 slice 4; workflow artifact uploads are still open | ||
| 273 | 274 | ||
| 274 | 275 | **Exit criteria:** a push runs a pipeline end-to-end ✅; killing a runner mid-job results in | |
| 275 | 276 | the job being requeued and completing on another runner ✅; a workflow that mutates its | |
| @@ -277,18 +278,65 @@ own definition mid-run still executes the version it was queued with ✅ (steps | |||
| 277 | 278 | at enqueue); a run record can be verified against the SHA it claims — the record carries | |
| 278 | 279 | it, but signing is Phase 7. | |
| 279 | 280 | ||
| 280 | ### Phase 5 — LFS | ||
| 281 | ### Phase 5 — LFS — ✅ complete (artifact uploads and locking deferred) | ||
| 282 | |||
| 283 | ✅ **Slice 1 — the store, and reads through the proxy.** Objects live at | ||
| 284 | `data/lfs/objects/<aa>/<bb>/<sha256>`, shared by every repository on the instance, and | ||
| 285 | are written through a temp file that is renamed only once the sha256 *and* the declared | ||
| 286 | size match, so nothing is ever filed under a key that lies about it. | ||
| 287 | `POST /git/<repo>.git/info/lfs/objects/batch` answers download requests with a plain | ||
| 288 | href, and `GET …/objects/<oid>` serves the bytes with the object id as its `ETag`. The | ||
| 289 | read rule is the clone rule: anonymous for a public repository, `404` for a private one. | ||
| 290 | `operation: upload` over HTTP is refused with a message naming SSH, rather than | ||
| 291 | half-working. | ||
| 292 | |||
| 293 | ✅ **Slice 2 — uploads over SSH.** When the remote is an SSH URL, git-lfs sends objects | ||
| 294 | over the same connection family as the push: the client runs | ||
| 295 | `git-lfs-transfer <repo> upload|download` and Vole answers the pure SSH protocol | ||
| 296 | (pkt-line: `version`, `batch`, `put-object`, `get-object`, `verify-object`, `quit`). No | ||
| 297 | token exists in the exchange — the key at the handshake is the identity, write access | ||
| 298 | comes from the repository's ACL, and bytes are hashed while they stream, so a client | ||
| 299 | that sends the wrong content gets an error instead of an object. | ||
| 300 | |||
| 301 | ✅ **Slice 3 — pointer hygiene at the gate.** `pre-receive` reads the blobs of the tree | ||
| 302 | the pushed ref will serve. A pointer whose object is missing fails the push, naming the | ||
| 303 | object id and the `git lfs push` command that would fix it; a pointer whose declared | ||
| 304 | size disagrees with the stored object fails too, because that checkout would be | ||
| 305 | truncated or stalled. A binary over `lfs_warn_bytes` (default 5 MiB) that is *not* a | ||
| 306 | pointer is accepted with a warning naming `git lfs track "*.ext"` and | ||
| 307 | `git lfs migrate`. The checks run before the protected-branch gate: an admin's approval | ||
| 308 | bypass is about review, not about whether the repository works. Deleting a broken | ||
| 309 | pointer remains possible, or there would be no way out of the mistake. | ||
| 310 | |||
| 311 | ✅ **Slice 4 — CI logs on the same store, and a sweep.** A finished run's log becomes a | ||
| 312 | content-addressed object in the same write that records the status, is served as | ||
| 313 | `text/plain` at `/r/<repo>/runs/<id>/log`, and is linked from the repository page. | ||
| 314 | `vole gc [--dry-run] [--grace]` marks the object ids named by pointers in every tree | ||
| 315 | reachable from every ref in every repository — plus archived logs, which no ref reaches | ||
| 316 | — and removes the rest. Anything younger than the grace period (default 1h) is kept, | ||
| 317 | because an upload that has not been pushed yet is not garbage. | ||
| 281 | 318 | ||
| 282 | - `POST /objects/batch` implementing `application/vnd.git-lfs+json`, upload and download | ||
| 283 | - Content-addressed object store with size limits and reference counting | ||
| 284 | - Server-side pointer hygiene: warn on committed binaries above a threshold, provide a | ||
| 285 | suggested `git lfs track` command in the rejection message | ||
| 286 | - `pre-receive` rejects LFS pointers whose objects are missing, before the ref moves | ||
| 287 | - Migrate CI artifacts onto the same store with the same GC story | ||
| 319 | Remaining: | ||
| 288 | 320 | ||
| 289 | **Exit criteria:** clone of an LFS repo pulls real objects through the proxy; a pointer | ||
| 290 | pushed without its object is rejected at hook time; GC never collects an object reachable | ||
| 291 | from a live ref. | ||
| 321 | - Artifact uploads from workflows. Logs are the only CI output today; there is no | ||
| 322 | `artifact:` step and no download endpoint for one. | ||
| 323 | - Per-repository object scoping. The store is instance-wide, so an object is fetchable | ||
| 324 | from any repository whose read rule you pass, if you know its id. Acceptable for | ||
| 325 | release artifacts, worth revisiting if repositories ever need to be isolated from | ||
| 326 | each other. | ||
| 327 | - Locking. `locking` is not advertised, so a client prints one harmless "does not | ||
| 328 | support the Git LFS locking API" line before an upload. | ||
| 329 | - Reference counting was replaced by mark-and-sweep. A counter that is one too low | ||
| 330 | deletes somebody's artifact after a crash; a walk over the refs cannot be wrong about | ||
| 331 | what is reachable, only slow — and `gc` is an operator command, not a hot path. | ||
| 332 | |||
| 333 | **Exit criteria:** a clone of an LFS repository pulls real objects through the proxy | ||
| 334 | (measured: a 256 KiB artifact pushed over SSH and pulled back byte-identical via | ||
| 335 | `http://…/git/<repo>.git`, with the sha256 checked on both ends); a pointer pushed | ||
| 336 | without its object is rejected at hook time (measured with `git push --no-verify`, so | ||
| 337 | the client's own pre-push check cannot stand in for the server's rule); GC never | ||
| 338 | collects an object reachable from a live ref (tested against an object reachable only | ||
| 339 | through history, and against an archived CI log). | ||
| 292 | 340 | ||
| 293 | 341 | ### Phase 6 — Site hosting | |
| 294 | 342 | ||