Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.

vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 5: an object is served only through a repository that holds it

65b88b3b1940623b6f7fbc686b19e986497c9ff2 · Keenan Nemetz · 17d ago · parent bf1e027c

Phase 5: an object is served only through a repository that holds it

The store was instance-wide and content-addressed, so an object was fetchable
through any repository whose read rule you passed, provided you knew its
sha256. That is now impossible, and the fix is structural rather than a check:
the pool still holds the bytes once, and each repository has a hard link into
a namespace of its own (data/lfs/repos/<repo>/…). A fetch reads only that
namespace, so "which repository owns these bytes" stops being a database
question that can go stale and becomes a fact about the filesystem — and gc
reconciles the links from the refs, which keeps the promise that derived state
can always be thrown away and rebuilt.

The alternatives were worse. A (repo, oid) table would have made the read path
depend on the index, so deleting data/index.db — which the roadmap promises
loses nothing — would have stopped LFS fetches until a rebuild. Namespacing
the storage itself would have cost the deduplication the store exists for.
Hard links give both: one copy of the bytes, one directory entry per
repository, and removing one repository's link leaves the other's data alone.

Three consequences that had to be handled rather than assumed:

- pre-receive checks pointers against the pushing repository, not the pool. A
  pointer to an object that exists only in another repository would otherwise
  pass the push and then 404 on clone. The client re-uploads, the pool
  deduplicates and the link lands, which is the transfer cost isolation
  implies.
- The sweep walks per repository, because reachability is per repository: an
  object reachable in one and not another keeps the first link and loses the
  second while the bytes stay. So `vole gc` is also the repair — it links what
  a repository reaches and unlinks what it no longer does.
- An instance upgrading from the shared pool has no namespaces at all, so
  every fetch would 404 until somebody read an upgrade note. server.New
  notices a store with no namespace and runs the link pass in the background
  once, using gc.Link — the half of gc that creates links and deletes nothing,
  so an interrupted pass leaves the store more reachable than it found it.
  A repository rename moves its namespace with it.

Reference counting stays rejected, and this makes the argument stronger: the
thing that repairs the store is the same walk that cleans it, and it cannot
invent a reference that does not exist.

Measured: TestAnObjectIsInvisibleToARepositoryThatDoesNotHoldIt and the server
level TestAnObjectIsNotFetchableThroughAnotherRepository (404 over the LFS
endpoint for an object the shared store demonstrably has);
TestTwoRepositoriesShareOneCopyOfTheBytes proves os.SameFile, so isolation
costs no deduplication; TestOneRepositoryLosingAPointerKeepsTheOthersLink and
TestLinkRepairsAMissingNamespace pin the per-repository sweep and the repair;
TestAnUnscopedStoreIsLinkedOnStartup covers the upgrade path.

Phase 5 is now complete: locking and object scoping both landed.

internal/server/lfs_scope.go +70

@@ -0,0 +1,70 @@
1 package server
2
3 // Repository-scoped object storage needs a link per (repository, object). An instance
4 // that predates it has the pool and nothing else, so every LFS fetch would 404 until
5 // somebody knew to run `vole gc`. Rather than make that an upgrade note people skip, the
6 // server notices a store with no namespace and links it in the background on first boot.
7 //
8 // It is the safe half of gc: `Link` creates links and deletes nothing, so a pass that is
9 // interrupted leaves the store more reachable than it found it, and the next start simply
10 // finishes the job.
11
12 import (
13 "os"
14 "time"
15
16 "github.com/rodent-software/vole/internal/gc"
17 "github.com/rodent-software/vole/internal/runs"
18 )
19
20 // scopeLFSStore links an unscoped store into per-repository namespaces, in the
21 // background. It does nothing once a namespace exists, which is the marker.
22 func (s *Server) scopeLFSStore() {
23 if s.LFS == nil || s.LFS.NamespaceExists() {
24 return
25 }
26 go func() {
27 start := time.Now()
28 repos, err := s.Repos.List()
29 if err != nil {
30 s.Log.Warn("lfs: cannot list repositories for the scoping pass", "err", err)
31 return
32 }
33 if len(repos) == 0 {
34 return
35 }
36 scan := make([]gc.Repo, 0, len(repos))
37 for _, m := range repos {
38 scan = append(scan, gc.Repo{Name: m.Name, Dir: m.Dir})
39 }
40 // Run logs and artifacts are roots too, and they belong to the repository whose
41 // run produced them.
42 runOIDs := map[string][]string{}
43 if _, err := os.Stat(s.Cfg.RunsDB()); err == nil {
44 if rn, err := runs.Open(s.Cfg.RunsDB()); err == nil {
45 if logs, err := rn.LogOIDs(); err == nil {
46 for repo, oids := range logs {
47 runOIDs[repo] = append(runOIDs[repo], oids...)
48 }
49 }
50 if arts, err := rn.ArtifactOIDs(); err == nil {
51 for repo, oids := range arts {
52 runOIDs[repo] = append(runOIDs[repo], oids...)
53 }
54 }
55 rn.Close()
56 }
57 }
58 res, err := gc.Link(gc.Options{Repos: scan, Store: s.LFS, RunOIDs: runOIDs})
59 if err != nil {
60 s.Log.Error("lfs: the store could not be linked to repositories; "+
61 "objects stay unfetchable until `vole gc` succeeds", "err", err)
62 return
63 }
64 s.Log.Info("lfs: linked objects into per-repository storage",
65 "repos", len(scan), "objects", res.Linked, "took", time.Since(start).Round(time.Millisecond))
66 for _, p := range res.Problems {
67 s.Log.Warn("lfs: link", "problem", p)
68 }
69 }()
70 }