Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 5: an object is served only through a repository that holds it

65b88b3b1940623b6f7fbc686b19e986497c9ff2 · Keenan Nemetz · 17d ago · parent bf1e027c

Phase 5: an object is served only through a repository that holds it

The store was instance-wide and content-addressed, so an object was fetchable
through any repository whose read rule you passed, provided you knew its
sha256. That is now impossible, and the fix is structural rather than a check:
the pool still holds the bytes once, and each repository has a hard link into
a namespace of its own (data/lfs/repos/<repo>/…). A fetch reads only that
namespace, so "which repository owns these bytes" stops being a database
question that can go stale and becomes a fact about the filesystem — and gc
reconciles the links from the refs, which keeps the promise that derived state
can always be thrown away and rebuilt.

The alternatives were worse. A (repo, oid) table would have made the read path
depend on the index, so deleting data/index.db — which the roadmap promises
loses nothing — would have stopped LFS fetches until a rebuild. Namespacing
the storage itself would have cost the deduplication the store exists for.
Hard links give both: one copy of the bytes, one directory entry per
repository, and removing one repository's link leaves the other's data alone.

Three consequences that had to be handled rather than assumed:

- pre-receive checks pointers against the pushing repository, not the pool. A
  pointer to an object that exists only in another repository would otherwise
  pass the push and then 404 on clone. The client re-uploads, the pool
  deduplicates and the link lands, which is the transfer cost isolation
  implies.
- The sweep walks per repository, because reachability is per repository: an
  object reachable in one and not another keeps the first link and loses the
  second while the bytes stay. So `vole gc` is also the repair — it links what
  a repository reaches and unlinks what it no longer does.
- An instance upgrading from the shared pool has no namespaces at all, so
  every fetch would 404 until somebody read an upgrade note. server.New
  notices a store with no namespace and runs the link pass in the background
  once, using gc.Link — the half of gc that creates links and deletes nothing,
  so an interrupted pass leaves the store more reachable than it found it.
  A repository rename moves its namespace with it.

Reference counting stays rejected, and this makes the argument stronger: the
thing that repairs the store is the same walk that cleans it, and it cannot
invent a reference that does not exist.

Measured: TestAnObjectIsInvisibleToARepositoryThatDoesNotHoldIt and the server
level TestAnObjectIsNotFetchableThroughAnotherRepository (404 over the LFS
endpoint for an object the shared store demonstrably has);
TestTwoRepositoriesShareOneCopyOfTheBytes proves os.SameFile, so isolation
costs no deduplication; TestOneRepositoryLosingAPointerKeepsTheOthersLink and
TestLinkRepairsAMissingNamespace pin the per-repository sweep and the repair;
TestAnUnscopedStoreIsLinkedOnStartup covers the upgrade path.

Phase 5 is now complete: locking and object scoping both landed.

internal/gc/gc_test.go +119−12

@@ -14,6 +14,32 @@ import (
14 14 // repoWithPointer builds a bare repository whose main branch commits a pointer
15 15 // naming oid, then a second commit that replaces it with plain text. The object
16 16 // is therefore reachable only through history — which is still reachable.
17 // repoWithoutPointer is a repository whose history never mentions an LFS pointer.
18 func repoWithoutPointer(t *testing.T) string {
19 t.Helper()
20 dir := t.TempDir() + "/repo.git"
21 if err := gitx.InitBare(dir, "main"); err != nil {
22 t.Fatal(err)
23 }
24 g := gitx.Open(dir)
25 text, err := g.HashObject([]byte("nothing but text here\n"))
26 if err != nil {
27 t.Fatal(err)
28 }
29 tree, err := g.MkTree([]gitx.Entry{{Mode: "100644", Type: "blob", SHA: text, Name: "README.md"}})
30 if err != nil {
31 t.Fatal(err)
32 }
33 c, err := g.CommitTree(tree, "", "text only")
34 if err != nil {
35 t.Fatal(err)
36 }
37 if err := g.UpdateRef("refs/heads/main", c); err != nil {
38 t.Fatal(err)
39 }
40 return dir
41 }
42
17 43 func repoWithPointer(t *testing.T, oid string) string {
18 44 t.Helper()
19 45 dir := t.TempDir() + "/repo.git"
@@ -63,7 +89,7 @@ func storeWith(t *testing.T, blobs ...[]byte) (*lfs.Store, []string) {
63 89 for _, b := range blobs {
64 90 sum := sha256.Sum256(b)
65 91 oid := hex.EncodeToString(sum[:])
66 if _, _, err := s.Put(oid, int64(len(b)), bytes.NewReader(b)); err != nil {
92 if _, _, err := s.Put("demo", oid, int64(len(b)), bytes.NewReader(b)); err != nil {
67 93 t.Fatal(err)
68 94 }
69 95 oids = append(oids, oid)
@@ -80,7 +106,7 @@ func TestReachableThroughHistorySurvivesAndGarbageDoesNot(t *testing.T) {
80 106 dir := repoWithPointer(t, reachable)
81 107
82 108 res, err := Run(Options{
83 Repos: []string{dir},
109 Repos: []Repo{{Name: "demo", Dir: dir}},
84 110 Store: store,
85 111 Grace: 0,
86 112 Now: time.Now().Add(time.Hour),
@@ -88,10 +114,10 @@ func TestReachableThroughHistorySurvivesAndGarbageDoesNot(t *testing.T) {
88 114 if err != nil {
89 115 t.Fatal(err)
90 116 }
91 if !store.Has(reachable) {
117 if !store.Has("demo", reachable) {
92 118 t.Fatal("swept an object a live ref reaches through history")
93 119 }
94 if store.Has(orphan) {
120 if store.Has("demo", orphan) {
95 121 t.Fatal("kept an object nothing references")
96 122 }
97 123 if res.Swept != 1 || res.Marked != 1 {
@@ -106,11 +132,11 @@ func TestGraceProtectsAFreshUpload(t *testing.T) {
106 132 store, oids := storeWith(t, kept, fresh)
107 133 dir := repoWithPointer(t, oids[0])
108 134
109 res, err := Run(Options{Repos: []string{dir}, Store: store, Grace: time.Hour})
135 res, err := Run(Options{Repos: []Repo{{Name: "demo", Dir: dir}}, Store: store, Grace: time.Hour})
110 136 if err != nil {
111 137 t.Fatal(err)
112 138 }
113 if !store.Has(oids[1]) {
139 if !store.Has("demo", oids[1]) {
114 140 t.Fatal("an upload inside the grace period was swept")
115 141 }
116 142 if res.Within != 1 || res.Swept != 0 {
@@ -118,16 +144,16 @@ func TestGraceProtectsAFreshUpload(t *testing.T) {
118 144 }
119 145 // The same store, an hour later: the grace has run out.
120 146 res, err = Run(Options{
121 Repos: []string{dir}, Store: store, Grace: time.Hour,
147 Repos: []Repo{{Name: "demo", Dir: dir}}, Store: store, Grace: time.Hour,
122 148 Now: time.Now().Add(2 * time.Hour),
123 149 })
124 150 if err != nil {
125 151 t.Fatal(err)
126 152 }
127 if store.Has(oids[1]) || res.Swept != 1 {
153 if store.Has("demo", oids[1]) || res.Swept != 1 {
128 154 t.Fatalf("result %+v; want the stale object swept", res)
129 155 }
130 if !store.Has(oids[0]) {
156 if !store.Has("demo", oids[0]) {
131 157 t.Fatal("swept the reachable object along with the stale one")
132 158 }
133 159 }
@@ -140,14 +166,14 @@ func TestArchivedLogIsARoot(t *testing.T) {
140 166 res, err := Run(Options{
141 167 Repos: nil,
142 168 Store: store,
143 RunOIDs: oids,
169 RunOIDs: map[string][]string{"demo": oids},
144 170 Grace: 0,
145 171 Now: time.Now().Add(time.Hour),
146 172 })
147 173 if err != nil {
148 174 t.Fatal(err)
149 175 }
150 if !store.Has(oids[0]) {
176 if !store.Has("demo", oids[0]) {
151 177 t.Fatal("swept an archived CI log; no git ref reaches one, so it must be a root")
152 178 }
153 179 if res.Swept != 0 || res.Marked != 1 {
@@ -162,10 +188,91 @@ func TestDryRunRemovesNothing(t *testing.T) {
162 188 if err != nil {
163 189 t.Fatal(err)
164 190 }
165 if !store.Has(oids[0]) {
191 if !store.Has("demo", oids[0]) {
166 192 t.Fatal("--dry-run deleted an object")
167 193 }
168 194 if res.Swept != 1 {
169 195 t.Fatalf("dry run reported %d removals; want 1", res.Swept)
170 196 }
171 197 }
198
199 // The sweep is also the repair. Nothing else creates a repository's links, so an object
200 // reachable from a ref but absent from its namespace is invisible to that repository until
201 // a pass links it — which is the path an instance upgrading onto scoped storage takes.
202 func TestLinkRepairsAMissingNamespace(t *testing.T) {
203 t.Parallel()
204 kept := []byte("the release artifact")
205 store, oids := storeWith(t, kept)
206 reachable := oids[0]
207 dir := repoWithPointer(t, reachable)
208
209 // Drop the link, which is what an unscoped store looks like from a repository's side.
210 if err := store.Unlink("demo", reachable); err != nil {
211 t.Fatal(err)
212 }
213 if store.Has("demo", reachable) {
214 t.Fatal("the object is still linked; the fixture proves nothing")
215 }
216
217 res, err := Link(Options{
218 Repos: []Repo{{Name: "demo", Dir: dir}},
219 Store: store,
220 Grace: 0,
221 Now: time.Now().Add(time.Hour),
222 })
223 if err != nil {
224 t.Fatal(err)
225 }
226 if res.Linked != 1 {
227 t.Fatalf("linked %d object(s), want 1 (%+v)", res.Linked, res)
228 }
229 if !store.Has("demo", reachable) {
230 t.Fatal("the repair did not make the object reachable")
231 }
232 // A repair deletes nothing, however stale the store looks.
233 if res.Swept != 0 || res.Unlinked != 0 {
234 t.Fatalf("the repair pass changed more than links: %+v", res)
235 }
236 // And it is idempotent.
237 again, err := Link(Options{Repos: []Repo{{Name: "demo", Dir: dir}}, Store: store, Now: time.Now()})
238 if err != nil {
239 t.Fatal(err)
240 }
241 if again.Linked != 0 {
242 t.Fatalf("a second repair linked %d more object(s)", again.Linked)
243 }
244 }
245
246 // One repository dropping its pointer must not unlink the other repository's view: the
247 // bytes are shared, the reachability is not.
248 func TestOneRepositoryLosingAPointerKeepsTheOthersLink(t *testing.T) {
249 t.Parallel()
250 shared := []byte("one artifact, two projects")
251 store, oids := storeWith(t, shared)
252 oid := oids[0]
253 // Give the second repository the same object, as a cross-repo upload would.
254 if _, _, err := store.Put("other", oid, int64(len(shared)), bytes.NewReader(shared)); err != nil {
255 t.Fatal(err)
256 }
257 dirA := repoWithPointer(t, oid) // demo reaches it
258 dirB := repoWithoutPointer(t) // other does not
259
260 res, err := Run(Options{
261 Repos: []Repo{{Name: "demo", Dir: dirA}, {Name: "other", Dir: dirB}},
262 Store: store,
263 Grace: 0,
264 Now: time.Now().Add(time.Hour),
265 })
266 if err != nil {
267 t.Fatal(err)
268 }
269 if !store.Has("demo", oid) {
270 t.Fatalf("the repository that still points at the object lost it (%+v)", res)
271 }
272 if store.Has("other", oid) {
273 t.Fatalf("a repository that no longer points at the object kept its link (%+v)", res)
274 }
275 if res.Swept != 0 {
276 t.Fatalf("swept the shared bytes while one repository still reaches them: %+v", res)
277 }
278 }