Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 5: an object is served only through a repository that holds it

65b88b3b1940623b6f7fbc686b19e986497c9ff2 · Keenan Nemetz · 17d ago · parent bf1e027c 

Phase 5: an object is served only through a repository that holds it

The store was instance-wide and content-addressed, so an object was fetchable
through any repository whose read rule you passed, provided you knew its
sha256. That is now impossible, and the fix is structural rather than a check:
the pool still holds the bytes once, and each repository has a hard link into
a namespace of its own (data/lfs/repos/<repo>/…). A fetch reads only that
namespace, so "which repository owns these bytes" stops being a database
question that can go stale and becomes a fact about the filesystem — and gc
reconciles the links from the refs, which keeps the promise that derived state
can always be thrown away and rebuilt.

The alternatives were worse. A (repo, oid) table would have made the read path
depend on the index, so deleting data/index.db — which the roadmap promises
loses nothing — would have stopped LFS fetches until a rebuild. Namespacing
the storage itself would have cost the deduplication the store exists for.
Hard links give both: one copy of the bytes, one directory entry per
repository, and removing one repository's link leaves the other's data alone.

Three consequences that had to be handled rather than assumed:

- pre-receive checks pointers against the pushing repository, not the pool. A
  pointer to an object that exists only in another repository would otherwise
  pass the push and then 404 on clone. The client re-uploads, the pool
  deduplicates and the link lands, which is the transfer cost isolation
  implies.
- The sweep walks per repository, because reachability is per repository: an
  object reachable in one and not another keeps the first link and loses the
  second while the bytes stay. So `vole gc` is also the repair — it links what
  a repository reaches and unlinks what it no longer does.
- An instance upgrading from the shared pool has no namespaces at all, so
  every fetch would 404 until somebody read an upgrade note. server.New
  notices a store with no namespace and runs the link pass in the background
  once, using gc.Link — the half of gc that creates links and deletes nothing,
  so an interrupted pass leaves the store more reachable than it found it.
  A repository rename moves its namespace with it.

Reference counting stays rejected, and this makes the argument stronger: the
thing that repairs the store is the same walk that cleans it, and it cannot
invent a reference that does not exist.

Measured: TestAnObjectIsInvisibleToARepositoryThatDoesNotHoldIt and the server
level TestAnObjectIsNotFetchableThroughAnotherRepository (404 over the LFS
endpoint for an object the shared store demonstrably has);
TestTwoRepositoriesShareOneCopyOfTheBytes proves os.SameFile, so isolation
costs no deduplication; TestOneRepositoryLosingAPointerKeepsTheOthersLink and
TestLinkRepairsAMissingNamespace pin the per-repository sweep and the repair;
TestAnUnscopedStoreIsLinkedOnStartup covers the upgrade path.

Phase 5 is now complete: locking and object scoping both landed.

internal/gc/gc.go +135−44

@@ -1,22 +1,28 @@
1 1 // Package gc sweeps the LFS store.
2 2 //
3 // The rule the sweep must never break: an object reachable from a live ref is
4 // kept. That is why nothing here trusts a counter — counters drift after a crash,
5 // a failed upload or an interrupted push, and a counter that is one too low
6 // deletes a release artifact. Instead the sweep marks what it can prove is
7 // reachable, and deletes only what remains.
3 // The rule the sweep must never break: an object reachable from a live ref is kept. That
4 // is why nothing here trusts a counter — counters drift after a crash, a failed upload or
5 // an interrupted push, and a counter that is one too low deletes a release artifact.
6 // Instead the sweep marks what it can prove is reachable, and deletes only what remains.
8 7 //
9 8 // Two kinds of root:
10 9 //
11 // - Pointer files in any tree reachable from any ref, in any repository. A
12 // pointer names one object by sha256; that is the whole reference.
13 // - Objects named by the run queue: archived CI logs and published run
14 // artifacts. No git ref reaches them, so a sweep that only marked git would
15 // delete every finished log and every build output a run published.
10 // - Pointer files in any tree reachable from any ref, in any repository. A pointer
11 // names one object by sha256; that is the whole reference.
12 // - Objects named by the run queue: archived CI logs and published run artifacts. No
13 // git ref reaches them, so a sweep that only marked git would delete every finished
14 // log and every build output a run published.
16 15 //
17 // Anything unmarked and younger than the grace period is kept: an upload that
18 // lands while the sweep is marking is not reachable yet, but is not garbage
19 // either.
16 // Reachability is per repository, because an object is served through the repository that
17 // owns it: the same bytes may be linked into two namespaces, and one repository dropping
18 // its pointer must not remove the other's link. So the sweep does three things in order —
19 // link what is reachable into the repository that reaches it, unlink what that repository
20 // no longer reaches, and finally drop pool objects that no namespace refers to any more.
21 // The middle step is what makes this pass the repair for a store that predates scoping,
22 // or one whose namespace was deleted: nothing else creates the links.
23 //
24 // Anything unmarked and younger than the grace period is kept: an upload that lands while
25 // the sweep is marking is not reachable yet, but is not garbage either.
20 26 package gc
21 27
22 28 import (
@@ -27,23 +33,34 @@ import (
27 33 "github.com/rodent-software/vole/internal/lfs"
28 34 )
29 35
36 // Repo is one repository to scan: the name its objects are served under, and the bare
37 // directory to read refs from.
38 type Repo struct {
39 Name string
40 Dir string
41 }
42
30 43 type Options struct {
31 // Repos are bare repository directories.
32 Repos []string
44 Repos []Repo
33 45 Store *lfs.Store
34 // RunOIDs are objects named by the run queue: archived logs and published run
35 // artifacts. Neither is reachable from a git ref, so a sweep that only marked
36 // git would delete every finished log and every build output a run published.
37 RunOIDs []string
38 // Grace protects recent objects that a concurrent upload has not made
39 // reachable yet.
46 // RunOIDs are objects named by the run queue, per repository: archived logs and
47 // published run artifacts. Neither is reachable from a git ref, so a sweep that only
48 // marked git would delete every finished log and every build output a run published.
49 RunOIDs map[string][]string
50 // Grace protects recent objects that a concurrent upload has not made reachable yet.
40 51 Grace time.Duration
41 52 DryRun bool
42 Now time.Time
53 // LinkOnly creates the missing links and deletes nothing. It is what an instance
54 // upgrading onto repository-scoped storage runs once, and it is the safe half of
55 // this package: it can only make more objects reachable.
56 LinkOnly bool
57 Now time.Time
43 58 }
44 59
45 60 type Result struct {
46 61 Marked int
62 Linked int // links created, which is repair work when it is not zero
63 Unlinked int
47 64 Swept int
48 65 SweptB int64
49 66 Within int // unmarked but inside the grace period
@@ -52,8 +69,8 @@ type Result struct {
52 69 Problems []string
53 70 }
54 71
55 // Run marks, then sweeps. A repository that cannot be scanned aborts the sweep:
56 // an unreadable ref means "I do not know what is reachable", and guessing costs
72 // Run marks, links, unlinks, then sweeps. A repository that cannot be scanned aborts the
73 // sweep: an unreadable ref means "I do not know what is reachable", and guessing costs
57 74 // somebody's data.
58 75 func Run(opts Options) (Result, error) {
59 76 var res Result
@@ -63,16 +80,20 @@ func Run(opts Options) (Result, error) {
63 80 if opts.Now.IsZero() {
64 81 opts.Now = time.Now()
65 82 }
66 marked := map[string]bool{}
67 for _, oid := range opts.RunOIDs {
68 marked[oid] = true
69 }
70 83
71 for _, dir := range opts.Repos {
72 g := gitx.Open(dir)
84 // Two sets: one per repository, for the links that make an object fetchable, and the
85 // union, which is what decides whether the pool may forget the bytes.
86 reachable := make(map[string]map[string]bool, len(opts.Repos))
87 all := map[string]bool{}
88 for _, repo := range opts.Repos {
89 g := gitx.Open(repo.Dir)
73 90 blobs, err := g.AllBlobs()
74 91 if err != nil {
75 return res, fmt.Errorf("gc: scan %s: %w", dir, err)
92 return res, fmt.Errorf("gc: scan %s: %w", repo.Dir, err)
93 }
94 set := map[string]bool{}
95 for _, oid := range opts.RunOIDs[repo.Name] {
96 set[oid] = true
76 97 }
77 98 for sha, size := range blobs {
78 99 if size > lfs.MaxPointerBytes {
@@ -80,24 +101,88 @@ func Run(opts Options) (Result, error) {
80 101 }
81 102 head, err := g.BlobPrefix(sha, int(lfs.MaxPointerBytes))
82 103 if err != nil {
83 return res, fmt.Errorf("gc: read %s in %s: %w", sha, dir, err)
104 return res, fmt.Errorf("gc: read %s in %s: %w", sha, repo.Dir, err)
84 105 }
85 106 p, ok := lfs.ParsePointer(head)
86 107 if !ok {
87 108 continue
88 109 }
89 marked[p.OID] = true
110 set[p.OID] = true
111 }
112 reachable[repo.Name] = set
113 for oid := range set {
114 all[oid] = true
115 }
116 }
117 // A run's objects belong to a repository that may not exist any more; those are
118 // still marked, so the sweep does not delete bytes a row still names.
119 for _, oids := range opts.RunOIDs {
120 for _, oid := range oids {
121 all[oid] = true
122 }
123 }
124 res.Marked = len(all)
125
126 // Link what each repository can reach, and (unless this is a repair-only pass)
127 // unlink what it cannot. Grace applies to the unlink exactly as it does to the sweep:
128 // a link created by an upload whose push has not landed yet is not garbage.
129 for _, repo := range opts.Repos {
130 set := reachable[repo.Name]
131 for oid := range set {
132 if opts.Store.Has(repo.Name, oid) {
133 continue
134 }
135 res.Linked++
136 if opts.DryRun {
137 continue
138 }
139 if err := opts.Store.Link(repo.Name, oid); err != nil {
140 res.Problems = append(res.Problems, fmt.Sprintf("%s: link %s: %v", repo.Name, oid, err))
141 }
142 }
143 if opts.LinkOnly {
144 continue
145 }
146 err := opts.Store.RepoWalk(repo.Name, func(oid string, size int64) error {
147 if set[oid] {
148 return nil
149 }
150 age, err := opts.Store.StoredAt(repo.Name, oid)
151 if err != nil {
152 res.Problems = append(res.Problems, fmt.Sprintf("%s: %s: %v", repo.Name, oid, err))
153 return nil
154 }
155 if opts.Now.Sub(age) < opts.Grace {
156 // Counted once, by the pool sweep: an object kept by grace here is kept
157 // there too unless another repository still reaches it.
158 return nil
159 }
160 res.Unlinked++
161 if opts.DryRun {
162 return nil
163 }
164 if err := opts.Store.Unlink(repo.Name, oid); err != nil {
165 res.Problems = append(res.Problems, fmt.Sprintf("%s: unlink %s: %v", repo.Name, oid, err))
166 }
167 return nil
168 })
169 if err != nil {
170 return res, fmt.Errorf("gc: reconcile links for %s: %w", repo.Name, err)
90 171 }
91 172 }
92 res.Marked = len(marked)
173 if opts.LinkOnly {
174 return res, nil
175 }
93 176
177 // Finally the pool: an object no repository reaches, and that nothing refers to any
178 // more, is the only thing that may be deleted.
94 179 err := opts.Store.Walk(func(oid string, size int64) error {
95 180 res.Scanned++
96 181 res.Left += size
97 if marked[oid] {
182 if all[oid] {
98 183 return nil
99 184 }
100 age, err := opts.Store.StoredAt(oid)
185 age, err := opts.Store.PoolStoredAt(oid)
101 186 if err != nil {
102 187 res.Problems = append(res.Problems, fmt.Sprintf("%s: %v", oid, err))
103 188 return nil
@@ -106,14 +191,11 @@ func Run(opts Options) (Result, error) {
106 191 res.Within++
107 192 return nil
108 193 }
109 if opts.DryRun {
110 res.Swept++
111 res.SweptB += size
112 return nil
113 }
114 if err := opts.Store.Remove(oid); err != nil {
115 res.Problems = append(res.Problems, fmt.Sprintf("%s: %v", oid, err))
116 return nil
194 if !opts.DryRun {
195 if err := opts.Store.Remove(oid); err != nil {
196 res.Problems = append(res.Problems, fmt.Sprintf("%s: %v", oid, err))
197 return nil
198 }
117 199 }
118 200 res.Swept++
119 201 res.SweptB += size
@@ -124,3 +206,12 @@ func Run(opts Options) (Result, error) {
124 206 }
125 207 return res, nil
126 208 }
209
210 // Link is the repair pass: it creates the links a repository is missing and deletes
211 // nothing. An instance that predates repository-scoped storage runs it once so its
212 // existing objects become fetchable again; so does a namespace somebody deleted.
213 func Link(opts Options) (Result, error) {
214 opts.LinkOnly = true
215 opts.DryRun = false
216 return Run(opts)
217 }