Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Phase 1: a working git remote with access control

653e0cfc227c992e5b26ea10057aa0b424b6ce4f · Keenan Nemetz · 18d ago · parent c5626f1b 

Phase 1: a working git remote with access control

Vole is now a functioning forge foundation: one CGO-free binary that serves
real git over smart HTTP and enforces per-repo policy in pre-receive.

What is in:

- internal/transport drives `git http-backend` as a CGI subprocess. Chunked
  uploads are spooled to learn CONTENT_LENGTH (git's default client config
  produces them), spool files are cleaned on error and abort, and a janitor
  sweeps anything a crash left behind. Pushes are capped by size and
  serialised behind a semaphore, because Phase 0 measured 1.13 GB of git RSS
  for a single 1 GiB push.
- internal/acl stores roles on refs/vole/acl inside each repository, so
  access rules travel with a clone. One evaluation function is shared by the
  hook and the web layer.
- internal/hook enforces policy in pre-receive: authentication, write role,
  protected refs, per-blob size limits, and a refusal to accept pushes to
  refs/vole/* - without that last check a writer could push their own
  acl.json and grant themselves admin.
- internal/auth holds users, personal access tokens and web sessions in
  SQLite. Passwords use PBKDF2-SHA256; tokens are 256-bit random values and
  are stored as SHA-256 hashes, since there is no low-entropy secret to grind.
- internal/index is the derived read model, rebuildable from disk and refs.
- internal/repo owns the bare-repo store and the metadata ref CAS helper.
- internal/web is the browse surface: repo list, tree, blob, raw, history,
  login, and per-repo settings for access rules, rename and delete.
- internal/signing creates the instance ed25519 key at first boot for the
  Phase 7 work.

Every Phase 1 exit criterion has a test that would fail if it regressed:
clone/push against a real listener, read-only and anonymous rejection with the
ref proven unmoved, rebuild-from-refs compared against the actual acl ref, a
spool file proven absent after an aborted chunked upload, and two pushes
completing through a single-slot semaphore. scripts/ci.sh runs gofmt, vet,
build, a markdown link check and the race-enabled suite.

Verified by hand as well: a clean `vole init` -> user -> token -> push ->
clone cycle, web login and browse, and a 256 MiB chunked push killed
mid-flight leaving no ref, no spool file and no orphaned git processes.

internal/transport/transport.go +532

@@ -0,0 +1,532 @@
1 // Package transport serves git over smart HTTP by driving `git http-backend`
2 // as a CGI subprocess, with authorization in front of it and policy enforced
3 // again in pre-receive behind it.
4 package transport
5
6 import (
7 "bufio"
8 "bytes"
9 "fmt"
10 "io"
11 "log/slog"
12 "net/http"
13 "os"
14 "os/exec"
15 "path/filepath"
16 "strconv"
17 "strings"
18 "sync"
19 "sync/atomic"
20 "time"
21
22 "github.com/rodent-software/vole/internal/acl"
23 "github.com/rodent-software/vole/internal/repo"
24 )
25
26 const (
27 spoolPrefix = "vole-spool-"
28 // A push may not exceed this much of the request body limit by more than the
29 // slack we use to detect overshoot.
30 overshootSlack = 1 << 20
31 )
32
33 type Op int
34
35 const (
36 OpRead Op = iota
37 OpPush
38 )
39
40 type Config struct {
41 // Root is the directory containing <name>.git bare repos.
42 Root string
43 // Open resolves a repo by name.
44 Open func(name string) (*repo.Meta, error)
45 // Actor resolves a Basic-auth pair to a username. An empty string with a nil
46 // error means "no credentials supplied".
47 Actor func(user, pass string) (string, error)
48 // MaxPushBytes caps a single push payload.
49 MaxPushBytes int64
50 // MaxBlobBytes is handed to the pre-receive hook for object size checks.
51 MaxBlobBytes int64
52 // SpoolDir is where chunked request bodies are spooled.
53 SpoolDir string
54 // PushConcurrency bounds simultaneous receive-pack processes. A 1 GiB push
55 // peaks at over 1 GiB of git RSS, so this is a memory guard.
56 PushConcurrency int
57 // QueueTimeout is how long a push will wait for a slot before 503.
58 QueueTimeout time.Duration
59 Log *slog.Logger
60 }
61
62 type Proxy struct {
63 cfg Config
64 sem chan struct{}
65 spool *Spooler
66 stats struct {
67 pushes atomic.Int64
68 reads atomic.Int64
69 denied atomic.Int64
70 queued atomic.Int64
71 spooled atomic.Int64
72 oversized atomic.Int64
73 }
74 }
75
76 func New(cfg Config) *Proxy {
77 if cfg.MaxPushBytes <= 0 {
78 cfg.MaxPushBytes = 1 << 30
79 }
80 if cfg.PushConcurrency <= 0 {
81 cfg.PushConcurrency = 1
82 }
83 if cfg.QueueTimeout <= 0 {
84 cfg.QueueTimeout = 5 * time.Minute
85 }
86 if cfg.Log == nil {
87 cfg.Log = slog.Default()
88 }
89 return &Proxy{
90 cfg: cfg,
91 sem: make(chan struct{}, cfg.PushConcurrency),
92 spool: NewSpooler(cfg.SpoolDir),
93 }
94 }
95
96 // Spooler tracks chunked-upload temp files so a periodic janitor can remove
97 // anything a crashed process left behind without touching live uploads.
98 type Spooler struct {
99 dir string
100 mu sync.Mutex
101 active map[string]struct{}
102 }
103
104 func NewSpooler(dir string) *Spooler {
105 return &Spooler{dir: dir, active: map[string]struct{}{}}
106 }
107
108 func (s *Spooler) Create() (*os.File, error) {
109 if err := os.MkdirAll(s.dir, 0o700); err != nil {
110 return nil, err
111 }
112 f, err := os.CreateTemp(s.dir, spoolPrefix+"*")
113 if err != nil {
114 return nil, err
115 }
116 s.mu.Lock()
117 s.active[f.Name()] = struct{}{}
118 s.mu.Unlock()
119 return f, nil
120 }
121
122 func (s *Spooler) Release(f *os.File) {
123 name := f.Name()
124 f.Close()
125 os.Remove(name)
126 s.mu.Lock()
127 delete(s.active, name)
128 s.mu.Unlock()
129 }
130
131 // Janitor removes spool files that no live upload owns. Safe to call at startup
132 // (nothing is active yet) and periodically thereafter.
133 func (s *Spooler) Janitor() (int, error) {
134 ents, err := os.ReadDir(s.dir)
135 if err != nil {
136 if os.IsNotExist(err) {
137 return 0, nil
138 }
139 return 0, err
140 }
141 n := 0
142 for _, e := range ents {
143 if e.IsDir() || !strings.HasPrefix(e.Name(), spoolPrefix) {
144 continue
145 }
146 p := filepath.Join(s.dir, e.Name())
147 s.mu.Lock()
148 _, live := s.active[p]
149 s.mu.Unlock()
150 if live {
151 continue
152 }
153 if err := os.Remove(p); err == nil {
154 n++
155 }
156 }
157 return n, nil
158 }
159
160 func (p *Proxy) Janitor() (int, error) { return p.spool.Janitor() }
161
162 func (p *Proxy) Stats() map[string]int64 {
163 return map[string]int64{
164 "pushes": p.stats.pushes.Load(),
165 "reads": p.stats.reads.Load(),
166 "denied": p.stats.denied.Load(),
167 "queued": p.stats.queued.Load(),
168 "spooled": p.stats.spooled.Load(),
169 "oversized": p.stats.oversized.Load(),
170 }
171 }
172
173 // Route parses /git/<name>.git/<rest> and returns the repo name and remainder.
174 func Route(path string) (name, rest string, ok bool) {
175 const prefix = "/git/"
176 if !strings.HasPrefix(path, prefix) {
177 return "", "", false
178 }
179 trimmed := strings.TrimPrefix(path, prefix)
180 i := strings.IndexByte(trimmed, '/')
181 if i <= 0 {
182 return "", "", false
183 }
184 dir := trimmed[:i]
185 rest = trimmed[i+1:]
186 if !strings.HasSuffix(dir, ".git") {
187 return "", "", false
188 }
189 return strings.TrimSuffix(dir, ".git"), rest, true
190 }
191
192 func classify(rest, rawQuery string) (Op, acl.Role, error) {
193 switch {
194 case rest == "info/refs":
195 q := parseQuery(rawQuery)
196 switch q["service"] {
197 case "git-receive-pack":
198 return OpPush, acl.Write, nil
199 case "git-upload-pack":
200 return OpRead, acl.Read, nil
201 }
202 return 0, acl.None, fmt.Errorf("client requested an unsupported service: dumb HTTP is not enabled (use smart HTTP)")
203 case strings.HasPrefix(rest, "git-receive-pack"):
204 return OpPush, acl.Write, nil
205 case strings.HasPrefix(rest, "git-upload-pack"):
206 return OpRead, acl.Read, nil
207 }
208 return 0, acl.None, fmt.Errorf("unsupported path %q", rest)
209 }
210
211 func parseQuery(q string) map[string]string {
212 out := map[string]string{}
213 for _, pair := range strings.Split(q, "&") {
214 if pair == "" {
215 continue
216 }
217 k, v, _ := strings.Cut(pair, "=")
218 out[k] = v
219 }
220 return out
221 }
222
223 func (p *Proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
224 name, rest, ok := Route(r.URL.Path)
225 if !ok {
226 http.NotFound(w, r)
227 return
228 }
229 m, err := p.cfg.Open(name)
230 if err != nil {
231 http.Error(w, "repository not found", http.StatusNotFound)
232 return
233 }
234 op, need, err := classify(rest, r.URL.RawQuery)
235 if err != nil {
236 http.Error(w, err.Error(), http.StatusForbidden)
237 return
238 }
239
240 actor, state := p.authenticate(r)
241 if state == authBad {
242 // Bad credentials are a failed authentication, not a permission
243 // problem: answer 401 so the client can retry, and say nothing about
244 // whether the repo exists.
245 p.stats.denied.Add(1)
246 w.Header().Set("WWW-Authenticate", `Basic realm="vole"`)
247 http.Error(w, "vole: invalid credentials", http.StatusUnauthorized)
248 return
249 }
250 a, err := m.ACL()
251 if err != nil {
252 p.cfg.Log.Error("read acl", "repo", name, "err", err)
253 http.Error(w, "access rules unavailable", http.StatusInternalServerError)
254 return
255 }
256 role := a.RoleFor(actor)
257 if !role.AtLeast(need) {
258 p.stats.denied.Add(1)
259 if state == authNone {
260 w.Header().Set("WWW-Authenticate", `Basic realm="vole"`)
261 http.Error(w, "authentication required", http.StatusUnauthorized)
262 return
263 }
264 http.Error(w, fmt.Sprintf("vole: %s does not have %s access to %s (your role: %s)",
265 displayActor(actor), need, displayRepo(actor, name), role), http.StatusForbidden)
266 return
267 }
268
269 if op == OpPush {
270 // Take a slot before reading a byte of the body: the memory cost of a
271 // push is git's, and it arrives with the pack.
272 start := time.Now()
273 select {
274 case p.sem <- struct{}{}:
275 defer func() { <-p.sem }()
276 case <-time.After(p.cfg.QueueTimeout):
277 p.stats.queued.Add(1)
278 http.Error(w, "vole: too many concurrent pushes, try again shortly", http.StatusServiceUnavailable)
279 return
280 }
281 if d := time.Since(start); d > 50*time.Millisecond {
282 p.cfg.Log.Info("push waited for a slot", "repo", name, "waited", d.String())
283 }
284 }
285
286 if r.ContentLength > p.cfg.MaxPushBytes {
287 p.stats.oversized.Add(1)
288 http.Error(w, fmt.Sprintf("vole: push is %d bytes, over the %d byte limit",
289 r.ContentLength, p.cfg.MaxPushBytes), http.StatusRequestEntityTooLarge)
290 return
291 }
292
293 body, length, cleanup, err := p.body(r)
294 if err != nil {
295 p.stats.oversized.Add(1)
296 http.Error(w, err.Error(), http.StatusRequestEntityTooLarge)
297 return
298 }
299 defer cleanup()
300
301 // The client can disappear while a chunked body is being spooled. There is
302 // no point running receive-pack for a push nobody will read the answer to.
303 if err := r.Context().Err(); err != nil {
304 p.cfg.Log.Info("client disconnected before backend start", "repo", name)
305 return
306 }
307
308 if err := p.backend(w, r, m, rest, actor, body, length); err != nil {
309 p.cfg.Log.Error("git backend", "repo", name, "err", err)
310 }
311 if op == OpPush {
312 p.stats.pushes.Add(1)
313 } else {
314 p.stats.reads.Add(1)
315 }
316 }
317
318 // body returns a reader for the request payload plus its true length.
319 // git http-backend is CGI and cannot proceed without CONTENT_LENGTH, so a
320 // chunked upload has to be spooled to disk first.
321 func (p *Proxy) body(r *http.Request) (io.Reader, int64, func(), error) {
322 noop := func() {}
323 if r.Method != http.MethodPost && r.Method != http.MethodPut {
324 return http.NoBody, 0, noop, nil
325 }
326 if r.ContentLength >= 0 {
327 // Stream directly, but never trust the declared length alone.
328 lim := &limitReader{r: r.Body, remaining: p.cfg.MaxPushBytes + overshootSlack, max: p.cfg.MaxPushBytes}
329 return lim, r.ContentLength, noop, nil
330 }
331
332 f, err := p.spool.Create()
333 if err != nil {
334 return nil, 0, noop, fmt.Errorf("vole: cannot spool upload: %w", err)
335 }
336 n, err := io.Copy(f, io.LimitReader(r.Body, p.cfg.MaxPushBytes+1))
337 if err != nil {
338 p.spool.Release(f)
339 return nil, 0, noop, fmt.Errorf("vole: upload interrupted: %w", err)
340 }
341 if n > p.cfg.MaxPushBytes {
342 p.spool.Release(f)
343 return nil, 0, noop, fmt.Errorf("vole: push exceeds the %d byte limit", p.cfg.MaxPushBytes)
344 }
345 if _, err := f.Seek(0, io.SeekStart); err != nil {
346 p.spool.Release(f)
347 return nil, 0, noop, fmt.Errorf("vole: spool seek: %w", err)
348 }
349 p.stats.spooled.Add(n)
350 return f, n, func() { p.spool.Release(f) }, nil
351 }
352
353 // limitReader fails loudly if a client sends more than it declared.
354 type limitReader struct {
355 r io.Reader
356 remaining int64
357 max int64
358 n int64
359 }
360
361 func (l *limitReader) Read(b []byte) (int, error) {
362 n, err := l.r.Read(b)
363 l.n += int64(n)
364 if l.n > l.max {
365 return n, fmt.Errorf("vole: push exceeds the %d byte limit", l.max)
366 }
367 return n, err
368 }
369
370 type authState int
371
372 const (
373 authNone authState = iota
374 authBad
375 authOK
376 )
377
378 func (p *Proxy) authenticate(r *http.Request) (string, authState) {
379 user, pass, ok := r.BasicAuth()
380 if !ok {
381 return "", authNone
382 }
383 actor, err := p.cfg.Actor(user, pass)
384 if err != nil {
385 return "", authBad
386 }
387 return actor, authOK
388 }
389
390 func (p *Proxy) backend(w http.ResponseWriter, r *http.Request, m *repo.Meta, rest, actor string, body io.Reader, length int64) error {
391 cmd := exec.Command("git", "http-backend")
392 cmd.Env = append(os.Environ(), p.cgiEnv(r, m, rest, actor, length)...)
393 stdin, err := cmd.StdinPipe()
394 if err != nil {
395 return err
396 }
397 stdout, err := cmd.StdoutPipe()
398 if err != nil {
399 return err
400 }
401 var stderr bytes.Buffer
402 cmd.Stderr = &stderr
403 if err := cmd.Start(); err != nil {
404 return err
405 }
406
407 copyErr := make(chan error, 1)
408 go func() {
409 defer stdin.Close()
410 _, err := io.Copy(stdin, body)
411 copyErr <- err
412 }()
413
414 br := bufio.NewReaderSize(stdout, 256<<10)
415 status := http.StatusOK
416 headers := http.Header{}
417 for {
418 line, err := br.ReadString('\n')
419 if err != nil {
420 cmd.Wait()
421 e := <-copyErr
422 if e != nil {
423 // Most likely the client vanished mid-upload.
424 return fmt.Errorf("client body: %w", e)
425 }
426 msg := strings.TrimSpace(stderr.String())
427 if msg == "" {
428 msg = err.Error()
429 }
430 http.Error(w, "git backend: "+msg, http.StatusInternalServerError)
431 return fmt.Errorf("cgi headers: %w", err)
432 }
433 line = strings.TrimRight(line, "\r\n")
434 if line == "" {
435 break
436 }
437 k, v, ok := strings.Cut(line, ":")
438 if !ok {
439 continue
440 }
441 k, v = strings.TrimSpace(k), strings.TrimSpace(v)
442 if strings.EqualFold(k, "Status") {
443 if f := strings.Fields(v); len(f) > 0 {
444 if code, err := strconv.Atoi(f[0]); err == nil {
445 status = code
446 }
447 }
448 continue
449 }
450 headers.Add(k, v)
451 }
452
453 h := w.Header()
454 for k, vs := range headers {
455 if strings.EqualFold(k, "Content-Length") {
456 continue // streamed; let net/http choose framing
457 }
458 for _, v := range vs {
459 h.Add(k, v)
460 }
461 }
462 w.WriteHeader(status)
463 flusher, _ := w.(http.Flusher)
464 buf := make([]byte, 128<<10)
465 for {
466 n, rerr := br.Read(buf)
467 if n > 0 {
468 if _, werr := w.Write(buf[:n]); werr != nil {
469 break
470 }
471 if flusher != nil {
472 flusher.Flush()
473 }
474 }
475 if rerr != nil {
476 break
477 }
478 }
479 <-copyErr
480 if err := cmd.Wait(); err != nil {
481 // A non-zero exit after the response was written is expected when git
482 // refuses the push; the client has already been told why.
483 if msg := strings.TrimSpace(stderr.String()); msg != "" && status >= 400 {
484 return fmt.Errorf("http-backend exit: %w: %s", err, msg)
485 }
486 }
487 return nil
488 }
489
490 func (p *Proxy) cgiEnv(r *http.Request, m *repo.Meta, rest, actor string, length int64) []string {
491 env := []string{
492 "GIT_PROJECT_ROOT=" + p.cfg.Root,
493 "GIT_HTTP_EXPORT_ALL=1",
494 "PATH_INFO=/" + m.Name + ".git/" + rest,
495 "REQUEST_METHOD=" + r.Method,
496 "QUERY_STRING=" + r.URL.RawQuery,
497 "REMOTE_USER=" + actor,
498 "REMOTE_ADDR=" + r.RemoteAddr,
499 "SERVER_PROTOCOL=" + r.Proto,
500 "GATEWAY_INTERFACE=CGI/1.1",
501 "CONTENT_LENGTH=" + strconv.FormatInt(length, 10),
502 // Hooks read these; http-backend passes its environment down.
503 "VOLE_ACTOR=" + actor,
504 "VOLE_REPO=" + m.Name,
505 "VOLE_MAX_BLOB=" + strconv.FormatInt(p.cfg.MaxBlobBytes, 10),
506 }
507 if ct := r.Header.Get("Content-Type"); ct != "" {
508 env = append(env, "CONTENT_TYPE="+ct)
509 }
510 // http-backend reads GIT_PROTOCOL, not HTTP_GIT_PROTOCOL, for protocol v2.
511 if gp := r.Header.Get("Git-Protocol"); gp != "" {
512 env = append(env, "GIT_PROTOCOL="+gp)
513 }
514 for name, vals := range r.Header {
515 key := "HTTP_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
516 switch key {
517 case "HTTP_CONTENT_TYPE", "HTTP_CONTENT_LENGTH", "HTTP_GIT_PROTOCOL", "HTTP_AUTHORIZATION":
518 continue
519 }
520 env = append(env, key+"="+strings.Join(vals, ","))
521 }
522 return env
523 }
524
525 func displayActor(a string) string {
526 if a == "" {
527 return "anonymous"
528 }
529 return a
530 }
531
532 func displayRepo(_ string, name string) string { return name }