Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 5: an object is served only through a repository that holds it

65b88b3b1940623b6f7fbc686b19e986497c9ff2 · Keenan Nemetz · 17d ago · parent bf1e027c 

Phase 5: an object is served only through a repository that holds it

The store was instance-wide and content-addressed, so an object was fetchable
through any repository whose read rule you passed, provided you knew its
sha256. That is now impossible, and the fix is structural rather than a check:
the pool still holds the bytes once, and each repository has a hard link into
a namespace of its own (data/lfs/repos/<repo>/…). A fetch reads only that
namespace, so "which repository owns these bytes" stops being a database
question that can go stale and becomes a fact about the filesystem — and gc
reconciles the links from the refs, which keeps the promise that derived state
can always be thrown away and rebuilt.

The alternatives were worse. A (repo, oid) table would have made the read path
depend on the index, so deleting data/index.db — which the roadmap promises
loses nothing — would have stopped LFS fetches until a rebuild. Namespacing
the storage itself would have cost the deduplication the store exists for.
Hard links give both: one copy of the bytes, one directory entry per
repository, and removing one repository's link leaves the other's data alone.

Three consequences that had to be handled rather than assumed:

- pre-receive checks pointers against the pushing repository, not the pool. A
  pointer to an object that exists only in another repository would otherwise
  pass the push and then 404 on clone. The client re-uploads, the pool
  deduplicates and the link lands, which is the transfer cost isolation
  implies.
- The sweep walks per repository, because reachability is per repository: an
  object reachable in one and not another keeps the first link and loses the
  second while the bytes stay. So `vole gc` is also the repair — it links what
  a repository reaches and unlinks what it no longer does.
- An instance upgrading from the shared pool has no namespaces at all, so
  every fetch would 404 until somebody read an upgrade note. server.New
  notices a store with no namespace and runs the link pass in the background
  once, using gc.Link — the half of gc that creates links and deletes nothing,
  so an interrupted pass leaves the store more reachable than it found it.
  A repository rename moves its namespace with it.

Reference counting stays rejected, and this makes the argument stronger: the
thing that repairs the store is the same walk that cleans it, and it cannot
invent a reference that does not exist.

Measured: TestAnObjectIsInvisibleToARepositoryThatDoesNotHoldIt and the server
level TestAnObjectIsNotFetchableThroughAnotherRepository (404 over the LFS
endpoint for an object the shared store demonstrably has);
TestTwoRepositoriesShareOneCopyOfTheBytes proves os.SameFile, so isolation
costs no deduplication; TestOneRepositoryLosingAPointerKeepsTheOthersLink and
TestLinkRepairsAMissingNamespace pin the per-repository sweep and the repair;
TestAnUnscopedStoreIsLinkedOnStartup covers the upgrade path.

Phase 5 is now complete: locking and object scoping both landed.

internal/lfs/lfs.go +215−34

@@ -120,19 +120,94 @@ type Store struct {
120 120
121 121 func New(root string, maxBytes int64) *Store { return &Store{Root: root, MaxBytes: maxBytes} }
122 122
123 // ObjectPath is the on-disk location of an object. Two levels of sharding keeps
124 // any one directory readable at a million objects; the file keeps the whole id so
125 // a store can be read by eye and a sweep never has to reassemble a name.
123 // ObjectPath is the on-disk location of an object in the shared pool. Two levels of
124 // sharding keeps any one directory readable at a million objects; the file keeps the
125 // whole id so a store can be read by eye and a sweep never has to reassemble a name.
126 126 func (s *Store) ObjectPath(oid string) string {
127 127 return filepath.Join(s.Root, "objects", oid[:2], oid[2:4], oid)
128 128 }
129 129
130 // Stat returns the stored size, or ErrNotFound.
131 func (s *Store) Stat(oid string) (int64, error) {
130 // RepoPath is the same object as one repository sees it: a hard link into that
131 // repository's namespace.
132 //
133 // Why a link at all: the pool is content-addressed, so a plain lookup by id would let
134 // anyone who knows an id read an object through any repository whose read rule they pass.
135 // A repository's namespace is the set of objects it may serve, and a fetch reads only
136 // that, so "which repository owns these bytes" stops being a database question that can
137 // be stale and becomes a fact about the filesystem. The link costs no bytes: the same
138 // artifact uploaded to two repositories is still stored once, and removing one
139 // repository's link leaves the other's data alone.
140 func (s *Store) RepoPath(repo, oid string) string {
141 return filepath.Join(s.Root, "repos", repo, oid[:2], oid[2:4], oid)
142 }
143
144 // Link gives repo its view of an object already in the pool. It is idempotent, and a
145 // no-op when the object is not stored: the caller decides whether that is an error.
146 func (s *Store) Link(repo, oid string) error {
147 if !ValidOID(oid) {
148 return ErrBadOID
149 }
150 if err := validRepo(repo); err != nil {
151 return err
152 }
153 dst := s.RepoPath(repo, oid)
154 if _, err := os.Lstat(dst); err == nil {
155 return nil
156 }
157 src := s.ObjectPath(oid)
158 if _, err := os.Stat(src); err != nil {
159 if errors.Is(err, fs.ErrNotExist) {
160 return ErrNotFound
161 }
162 return err
163 }
164 if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
165 return err
166 }
167 if err := os.Link(src, dst); err != nil {
168 if errors.Is(err, fs.ErrExist) {
169 return nil
170 }
171 return err
172 }
173 return nil
174 }
175
176 // Unlink removes repo's view. The bytes survive while any other name refers to them,
177 // which is what lets two repositories share one artifact.
178 func (s *Store) Unlink(repo, oid string) error {
179 if !ValidOID(oid) {
180 return ErrBadOID
181 }
182 if err := validRepo(repo); err != nil {
183 return err
184 }
185 err := os.Remove(s.RepoPath(repo, oid))
186 if errors.Is(err, fs.ErrNotExist) {
187 return nil
188 }
189 return err
190 }
191
192 // validRepo refuses a repository name that could escape the namespace. Repository names
193 // are already restricted to [a-z0-9._-], so this is a belt for callers that are not the
194 // repository store.
195 func validRepo(repo string) error {
196 switch {
197 case repo == "":
198 return fmt.Errorf("lfs: a repository name is required")
199 case repo == "." || repo == ".." || strings.ContainsAny(repo, "/\\\x00"):
200 return fmt.Errorf("lfs: %q is not a repository name", repo)
201 }
202 return nil
203 }
204
205 // Stat returns the stored size as repo sees it, or ErrNotFound.
206 func (s *Store) Stat(repo, oid string) (int64, error) {
132 207 if !ValidOID(oid) {
133 208 return 0, ErrBadOID
134 209 }
135 st, err := os.Stat(s.ObjectPath(oid))
210 st, err := os.Stat(s.RepoPath(repo, oid))
136 211 if err != nil {
137 212 if errors.Is(err, fs.ErrNotExist) {
138 213 return 0, ErrNotFound
@@ -142,18 +217,19 @@ func (s *Store) Stat(oid string) (int64, error) {
142 217 return st.Size(), nil
143 218 }
144 219
145 // Has reports whether the object is present.
146 func (s *Store) Has(oid string) bool {
147 _, err := s.Stat(oid)
220 // Has reports whether repo may serve the object.
221 func (s *Store) Has(repo, oid string) bool {
222 _, err := s.Stat(repo, oid)
148 223 return err == nil
149 224 }
150 225
151 // StoredAt is when an object was written, used by the sweep's grace period.
152 func (s *Store) StoredAt(oid string) (time.Time, error) {
226 // StoredAt is when an object was written, used by the sweep's grace period. A link shares
227 // the pool object's inode, so this reads the same time either way.
228 func (s *Store) StoredAt(repo, oid string) (time.Time, error) {
153 229 if !ValidOID(oid) {
154 230 return time.Time{}, ErrBadOID
155 231 }
156 st, err := os.Stat(s.ObjectPath(oid))
232 st, err := os.Stat(s.RepoPath(repo, oid))
157 233 if err != nil {
158 234 if errors.Is(err, fs.ErrNotExist) {
159 235 return time.Time{}, ErrNotFound
@@ -163,12 +239,12 @@ func (s *Store) StoredAt(oid string) (time.Time, error) {
163 239 return st.ModTime(), nil
164 240 }
165 241
166 // Open returns the object for reading with its size. The caller closes it.
167 func (s *Store) Open(oid string) (*os.File, int64, error) {
242 // Open returns the object for reading with its size, as repo sees it. The caller closes it.
243 func (s *Store) Open(repo, oid string) (*os.File, int64, error) {
168 244 if !ValidOID(oid) {
169 245 return nil, 0, ErrBadOID
170 246 }
171 f, err := os.Open(s.ObjectPath(oid))
247 f, err := os.Open(s.RepoPath(repo, oid))
172 248 if err != nil {
173 249 if errors.Is(err, fs.ErrNotExist) {
174 250 return nil, 0, ErrNotFound
@@ -183,14 +259,20 @@ func (s *Store) Open(oid string) (*os.File, int64, error) {
183 259 return f, st.Size(), nil
184 260 }
185 261
186 // Put streams r into the store as oid. It verifies the sha256 and the size
187 // before the object becomes visible: a store whose key does not match its
188 // content is worse than an empty one, because every reader trusts it.
189 // It returns the number of bytes written and whether the object was new.
190 func (s *Store) Put(oid string, size int64, r io.Reader) (int64, bool, error) {
262 // Put streams r into the store as oid on behalf of repo. It verifies the sha256 and the
263 // size before the object becomes visible: a store whose key does not match its content is
264 // worse than an empty one, because every reader trusts it.
265 //
266 // It returns the number of bytes written and whether this is new to *this repository* —
267 // which is the answer that matters, since a second repository referring to an object that
268 // is already stored costs a link and no transfer.
269 func (s *Store) Put(repo, oid string, size int64, r io.Reader) (int64, bool, error) {
191 270 if !ValidOID(oid) {
192 271 return 0, false, ErrBadOID
193 272 }
273 if err := validRepo(repo); err != nil {
274 return 0, false, err
275 }
194 276 tmpDir := filepath.Join(s.Root, "tmp")
195 277 if err := os.MkdirAll(tmpDir, 0o755); err != nil {
196 278 return 0, false, err
@@ -238,37 +320,136 @@ func (s *Store) Put(oid string, size int64, r io.Reader) (int64, bool, error) {
238 320 }
239 321
240 322 dst := s.ObjectPath(oid)
323 createdPool := false
241 324 if st, err := os.Stat(dst); err == nil && st.Size() == n {
242 return n, false, nil // deduplicated
243 }
244 if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
245 return 0, false, err
325 // Already stored: the bytes are deduplicated and only the link is new.
326 } else {
327 if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
328 return 0, false, err
329 }
330 if err := os.Rename(tmp, dst); err != nil {
331 return 0, false, err
332 }
333 createdPool = true
246 334 }
247 if err := os.Rename(tmp, dst); err != nil {
335 // The repository's view is what a reader will look up, so an object that is stored
336 // but not linked is an object nobody can fetch. A failure here fails the upload: a
337 // successful put that leaves the object unreachable is the worse outcome.
338 hadLink := s.Has(repo, oid)
339 if err := s.Link(repo, oid); err != nil {
248 340 return 0, false, err
249 341 }
250 return n, true, nil
342 return n, !hadLink && (createdPool || !hadLink), nil
251 343 }
252 344
253 345 // PutBytes is Put for an in-memory payload (CI logs, small artifacts).
254 func (s *Store) PutBytes(data []byte) (string, error) {
346 func (s *Store) PutBytes(repo string, data []byte) (string, error) {
255 347 sum := sha256.Sum256(data)
256 348 oid := hex.EncodeToString(sum[:])
257 _, _, err := s.Put(oid, int64(len(data)), strings.NewReader(string(data)))
349 _, _, err := s.Put(repo, oid, int64(len(data)), strings.NewReader(string(data)))
258 350 return oid, err
259 351 }
260 352
261 // Remove deletes an object. Missing is not an error: a sweep races with nothing
262 // that recreates an object, but two sweeps can overlap.
353 // Remove deletes an object everywhere: the pool's bytes and every repository's link to
354 // them. Missing is not an error, because two sweeps can overlap.
355 //
356 // A sweep has usually unlinked each repository's name already — it must, to tell "still
357 // reachable in that repository" from "not" — so in practice this drops the pool name and
358 // finds nothing else. It is written the slow, complete way anyway, because `Remove` means
359 // the object is gone: a function that left a link behind would leave something fetchable
360 // that the caller believes it deleted.
263 361 func (s *Store) Remove(oid string) error {
264 362 if !ValidOID(oid) {
265 363 return ErrBadOID
266 364 }
267 err := os.Remove(s.ObjectPath(oid))
268 if errors.Is(err, fs.ErrNotExist) {
269 return nil
365 var firstErr error
366 names, err := os.ReadDir(filepath.Join(s.Root, "repos"))
367 if err != nil && !errors.Is(err, fs.ErrNotExist) {
368 return err
270 369 }
271 return err
370 for _, e := range names {
371 if !e.IsDir() {
372 continue
373 }
374 if err := os.Remove(s.RepoPath(e.Name(), oid)); err != nil &&
375 !errors.Is(err, fs.ErrNotExist) && firstErr == nil {
376 firstErr = err
377 }
378 }
379 if err := os.Remove(s.ObjectPath(oid)); err != nil && !errors.Is(err, fs.ErrNotExist) && firstErr == nil {
380 firstErr = err
381 }
382 return firstErr
383 }
384
385 // PoolStoredAt is when the pool object was written, whatever any repository thinks. The
386 // sweep's grace period is about the bytes, not about who has a link to them.
387 func (s *Store) PoolStoredAt(oid string) (time.Time, error) {
388 if !ValidOID(oid) {
389 return time.Time{}, ErrBadOID
390 }
391 st, err := os.Stat(s.ObjectPath(oid))
392 if err != nil {
393 if errors.Is(err, fs.ErrNotExist) {
394 return time.Time{}, ErrNotFound
395 }
396 return time.Time{}, err
397 }
398 return st.ModTime(), nil
399 }
400
401 // RepoWalk visits every object one repository may serve.
402 func (s *Store) RepoWalk(repo string, fn func(oid string, size int64) error) error {
403 if err := validRepo(repo); err != nil {
404 return err
405 }
406 root := filepath.Join(s.Root, "repos", repo)
407 return filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
408 if err != nil {
409 if errors.Is(err, fs.ErrNotExist) {
410 return nil
411 }
412 return err
413 }
414 if d.IsDir() {
415 return nil
416 }
417 if !ValidOID(d.Name()) {
418 return nil
419 }
420 info, err := d.Info()
421 if err != nil {
422 return err
423 }
424 return fn(d.Name(), info.Size())
425 })
426 }
427
428 // NamespaceExists reports whether this store has ever been scoped by repository. It is
429 // how a server knows to run the repair pass once on an instance that predates it.
430 func (s *Store) NamespaceExists() bool {
431 _, err := os.Stat(filepath.Join(s.Root, "repos"))
432 return err == nil
433 }
434
435 // RenameRepo moves a repository's namespace, so a rename does not strand every object it
436 // can serve. Missing is not an error: a repository with no objects has no namespace.
437 func (s *Store) RenameRepo(oldName, newName string) error {
438 if err := validRepo(oldName); err != nil {
439 return err
440 }
441 if err := validRepo(newName); err != nil {
442 return err
443 }
444 from := filepath.Join(s.Root, "repos", oldName)
445 to := filepath.Join(s.Root, "repos", newName)
446 if _, err := os.Stat(from); err != nil {
447 if errors.Is(err, fs.ErrNotExist) {
448 return nil
449 }
450 return err
451 }
452 return os.Rename(from, to)
272 453 }
273 454
274 455 // Walk visits every stored object.