vole
Clone
Read only, no credentials:
https://rodent.software/vole.git
Write over SSH with a registered key:
git clone ssh://git@rodent.software:2222/vole.git
A free and open-source forge that lives in your Git repo.
Phase 5: an object is served only through a repository that holds it
65b88b3b1940623b6f7fbc686b19e986497c9ff2 · Keenan Nemetz · 17d ago · parent bf1e027c
Phase 5: an object is served only through a repository that holds it The store was instance-wide and content-addressed, so an object was fetchable through any repository whose read rule you passed, provided you knew its sha256. That is now impossible, and the fix is structural rather than a check: the pool still holds the bytes once, and each repository has a hard link into a namespace of its own (data/lfs/repos/<repo>/…). A fetch reads only that namespace, so "which repository owns these bytes" stops being a database question that can go stale and becomes a fact about the filesystem — and gc reconciles the links from the refs, which keeps the promise that derived state can always be thrown away and rebuilt. The alternatives were worse. A (repo, oid) table would have made the read path depend on the index, so deleting data/index.db — which the roadmap promises loses nothing — would have stopped LFS fetches until a rebuild. Namespacing the storage itself would have cost the deduplication the store exists for. Hard links give both: one copy of the bytes, one directory entry per repository, and removing one repository's link leaves the other's data alone. Three consequences that had to be handled rather than assumed: - pre-receive checks pointers against the pushing repository, not the pool. A pointer to an object that exists only in another repository would otherwise pass the push and then 404 on clone. The client re-uploads, the pool deduplicates and the link lands, which is the transfer cost isolation implies. - The sweep walks per repository, because reachability is per repository: an object reachable in one and not another keeps the first link and loses the second while the bytes stay. So `vole gc` is also the repair — it links what a repository reaches and unlinks what it no longer does. - An instance upgrading from the shared pool has no namespaces at all, so every fetch would 404 until somebody read an upgrade note. server.New notices a store with no namespace and runs the link pass in the background once, using gc.Link — the half of gc that creates links and deletes nothing, so an interrupted pass leaves the store more reachable than it found it. A repository rename moves its namespace with it. Reference counting stays rejected, and this makes the argument stronger: the thing that repairs the store is the same walk that cleans it, and it cannot invent a reference that does not exist. Measured: TestAnObjectIsInvisibleToARepositoryThatDoesNotHoldIt and the server level TestAnObjectIsNotFetchableThroughAnotherRepository (404 over the LFS endpoint for an object the shared store demonstrably has); TestTwoRepositoriesShareOneCopyOfTheBytes proves os.SameFile, so isolation costs no deduplication; TestOneRepositoryLosingAPointerKeepsTheOthersLink and TestLinkRepairsAMissingNamespace pin the per-repository sweep and the repair; TestAnUnscopedStoreIsLinkedOnStartup covers the upgrade path. Phase 5 is now complete: locking and object scoping both landed.
internal/server/releases_test.go +6−6
| @@ -318,7 +318,7 @@ func tarContents(t *testing.T, r io.Reader, gzipped bool) map[string]string { | |||
| 318 | 318 | // matters, and the hook enforces it — see the ordering test below. | |
| 319 | 319 | func attachArtifact(t *testing.T, f *fixture, tag, name string, data []byte) (string, string) { | |
| 320 | 320 | t.Helper() | |
| 321 | oid, err := f.srv.LFS.PutBytes(data) | ||
| 321 | oid, err := f.srv.LFS.PutBytes(f.repo, data) | ||
| 322 | 322 | if err != nil { | |
| 323 | 323 | t.Fatalf("store the artifact: %v", err) | |
| 324 | 324 | } | |
| @@ -435,13 +435,13 @@ func TestGCCountsReleaseArtifactsAsReachable(t *testing.T) { | |||
| 435 | 435 | keep := []byte("published artifact") | |
| 436 | 436 | drop := []byte("an upload nobody referenced") | |
| 437 | 437 | keepOID, _ := attachArtifact(t, f, "v1.0.0", "keep.bin", keep) | |
| 438 | dropOID, err := f.srv.LFS.PutBytes(drop) | ||
| 438 | dropOID, err := f.srv.LFS.PutBytes(f.repo, drop) | ||
| 439 | 439 | if err != nil { | |
| 440 | 440 | t.Fatal(err) | |
| 441 | 441 | } | |
| 442 | 442 | ||
| 443 | 443 | res, err := gc.Run(gc.Options{ | |
| 444 | Repos: []string{filepath.Join(f.cfg.ReposDir(), f.repo+".git")}, | ||
| 444 | Repos: []gc.Repo{{Name: f.repo, Dir: filepath.Join(f.cfg.ReposDir(), f.repo+".git")}}, | ||
| 445 | 445 | Store: f.srv.LFS, | |
| 446 | 446 | // Far enough ahead that the grace period protects nothing. | |
| 447 | 447 | Now: time.Now().Add(48 * time.Hour), | |
| @@ -449,10 +449,10 @@ func TestGCCountsReleaseArtifactsAsReachable(t *testing.T) { | |||
| 449 | 449 | if err != nil { | |
| 450 | 450 | t.Fatalf("gc: %v", err) | |
| 451 | 451 | } | |
| 452 | if !f.srv.LFS.Has(keepOID) { | ||
| 452 | if !f.srv.LFS.Has(f.repo, keepOID) { | ||
| 453 | 453 | t.Fatalf("gc swept the published artifact (%+v)", res) | |
| 454 | 454 | } | |
| 455 | if f.srv.LFS.Has(dropOID) { | ||
| 455 | if f.srv.LFS.Has(f.repo, dropOID) { | ||
| 456 | 456 | t.Fatalf("gc kept an object nothing reaches (%+v)", res) | |
| 457 | 457 | } | |
| 458 | 458 | } | |
| @@ -507,7 +507,7 @@ func pushMetaAs(t *testing.T, f *fixture, key string, files map[string]string, r | |||
| 507 | 507 | ||
| 508 | 508 | func pointerFor(t *testing.T, f *fixture, data []byte) string { | |
| 509 | 509 | t.Helper() | |
| 510 | oid, err := f.srv.LFS.PutBytes(data) | ||
| 510 | oid, err := f.srv.LFS.PutBytes(f.repo, data) | ||
| 511 | 511 | if err != nil { | |
| 512 | 512 | t.Fatal(err) | |
| 513 | 513 | } | |