Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Move writes to SSH, drop web authentication, keep identity in git

a5f19b3c19861e080953096f934df25dc38beaf7 · Keenan Nemetz · 18d ago · parent 653e0cfc 

Move writes to SSH, drop web authentication, keep identity in git

The authenticated-HTTP design is gone rather than extended. Writes now require
an SSH key, the web is read-only, and the user roster lives in git.

What replaced what:

- internal/sshserver is the only write transport. An embedded SSH server
  (golang.org/x/crypto/ssh) authenticates by public key and hands the principal
  to pre-receive through the environment, so the hook's policy code is unchanged.
  The exec surface is whitelisted to git-upload-pack and git-receive-pack, the
  repository argument is resolved through the store rather than concatenated, and
  everything else -- including `sh -c` -- is refused and asserted refused in a
  test.
- internal/identity stores users on refs/heads/main in data/system.git. The
  directory listing is the table: users/<name>/keys/<sha>.pub. Writes are CAS
  commits, reads come from an in-memory snapshot so the SSH auth callback is a
  map lookup. Nothing stored is a secret, so a clone of system.git is a complete
  backup and git log on it is the audit trail.
- internal/transport is now anonymous read-only: upload-pack behind a visibility
  check, 404 for private repositories, and no push path at all.
- internal/web has no login, no sessions and no mutations.

Deleted outright: internal/auth, the Basic-auth path, login/logout, web
settings/rename/delete, chunked body spooling, the spool janitor, the push
semaphore and the hand-rolled byte cap. The cap is now git's receive.maxInputSize
per repo, which enforces it on every transport without buffering.

Three things worth recording:

- The CAS helper read the tree outside the retry loop and compared against a
  freshly-read old value, so a concurrent writer's commit could be silently
  discarded while the CAS "succeeded". A test with two stores adding eight keys
  at once found it losing half of them; read, mutate and compare now all happen
  inside the loop.
- The roster moved from refs/vole/users to refs/heads/main because `git clone`
  only mirrors refs/heads/*: the documented backup command produced an instance
  with zero users. Storing on a branch also means git itself refuses to point the
  ref at a non-commit.
- Exec refusals are answered on the channel rather than by rejecting the request.
  Rejecting made the client print "exec request failed on channel 0", so the
  reason was unreadable.

Verified by hand as well as by test: init -> user -> add-key -> repo -> serve,
push over SSH, anonymous HTTP clone, web browse, a private repo answering 404 on
both anonymous surfaces while still cloning over SSH, a refused `sh -c` that
prints why and executes nothing, and a plain clone of system.git yielding the
whole roster with no side database present.

internal/transport/transport.go +56−316

@@ -1,6 +1,10 @@
1 // Package transport serves git over smart HTTP by driving `git http-backend`
2 // as a CGI subprocess, with authorization in front of it and policy enforced
3 // again in pre-receive behind it.
1 // Package transport serves anonymous, read-only git over smart HTTP.
2 //
3 // Writes live on SSH, so this package has no push path at all: no chunked
4 // spooling, no push semaphore, no byte cap. Those existed only for POST bodies
5 // that carried packs. What remains is upload-pack behind a visibility check.
6 // Private repositories answer 404 rather than 403, because a 403 confirms that
7 // the repository exists.
4 8 package transport
5 9
6 10 import (
@@ -12,165 +16,47 @@ import (
12 16 "net/http"
13 17 "os"
14 18 "os/exec"
15 "path/filepath"
16 19 "strconv"
17 20 "strings"
18 "sync"
19 21 "sync/atomic"
20 "time"
21 22
22 23 "github.com/rodent-software/vole/internal/acl"
23 24 "github.com/rodent-software/vole/internal/repo"
24 25 )
25 26
26 const (
27 spoolPrefix = "vole-spool-"
28 // A push may not exceed this much of the request body limit by more than the
29 // slack we use to detect overshoot.
30 overshootSlack = 1 << 20
31 )
32
33 type Op int
34
35 const (
36 OpRead Op = iota
37 OpPush
38 )
39
40 27 type Config struct {
41 28 // Root is the directory containing <name>.git bare repos.
42 29 Root string
43 30 // Open resolves a repo by name.
44 31 Open func(name string) (*repo.Meta, error)
45 // Actor resolves a Basic-auth pair to a username. An empty string with a nil
46 // error means "no credentials supplied".
47 Actor func(user, pass string) (string, error)
48 // MaxPushBytes caps a single push payload.
49 MaxPushBytes int64
50 // MaxBlobBytes is handed to the pre-receive hook for object size checks.
51 MaxBlobBytes int64
52 // SpoolDir is where chunked request bodies are spooled.
53 SpoolDir string
54 // PushConcurrency bounds simultaneous receive-pack processes. A 1 GiB push
55 // peaks at over 1 GiB of git RSS, so this is a memory guard.
56 PushConcurrency int
57 // QueueTimeout is how long a push will wait for a slot before 503.
58 QueueTimeout time.Duration
59 Log *slog.Logger
32 Log *slog.Logger
60 33 }
61 34
62 35 type Proxy struct {
63 36 cfg Config
64 sem chan struct{}
65 spool *Spooler
66 37 stats struct {
67 pushes atomic.Int64
68 reads atomic.Int64
69 denied atomic.Int64
70 queued atomic.Int64
71 spooled atomic.Int64
72 oversized atomic.Int64
38 reads atomic.Int64
39 hidden atomic.Int64
40 pushes atomic.Int64
73 41 }
74 42 }
75 43
76 44 func New(cfg Config) *Proxy {
77 if cfg.MaxPushBytes <= 0 {
78 cfg.MaxPushBytes = 1 << 30
79 }
80 if cfg.PushConcurrency <= 0 {
81 cfg.PushConcurrency = 1
82 }
83 if cfg.QueueTimeout <= 0 {
84 cfg.QueueTimeout = 5 * time.Minute
85 }
86 45 if cfg.Log == nil {
87 46 cfg.Log = slog.Default()
88 47 }
89 return &Proxy{
90 cfg: cfg,
91 sem: make(chan struct{}, cfg.PushConcurrency),
92 spool: NewSpooler(cfg.SpoolDir),
93 }
94 }
95
96 // Spooler tracks chunked-upload temp files so a periodic janitor can remove
97 // anything a crashed process left behind without touching live uploads.
98 type Spooler struct {
99 dir string
100 mu sync.Mutex
101 active map[string]struct{}
102 }
103
104 func NewSpooler(dir string) *Spooler {
105 return &Spooler{dir: dir, active: map[string]struct{}{}}
106 }
107
108 func (s *Spooler) Create() (*os.File, error) {
109 if err := os.MkdirAll(s.dir, 0o700); err != nil {
110 return nil, err
111 }
112 f, err := os.CreateTemp(s.dir, spoolPrefix+"*")
113 if err != nil {
114 return nil, err
115 }
116 s.mu.Lock()
117 s.active[f.Name()] = struct{}{}
118 s.mu.Unlock()
119 return f, nil
120 }
121
122 func (s *Spooler) Release(f *os.File) {
123 name := f.Name()
124 f.Close()
125 os.Remove(name)
126 s.mu.Lock()
127 delete(s.active, name)
128 s.mu.Unlock()
129 }
130
131 // Janitor removes spool files that no live upload owns. Safe to call at startup
132 // (nothing is active yet) and periodically thereafter.
133 func (s *Spooler) Janitor() (int, error) {
134 ents, err := os.ReadDir(s.dir)
135 if err != nil {
136 if os.IsNotExist(err) {
137 return 0, nil
138 }
139 return 0, err
140 }
141 n := 0
142 for _, e := range ents {
143 if e.IsDir() || !strings.HasPrefix(e.Name(), spoolPrefix) {
144 continue
145 }
146 p := filepath.Join(s.dir, e.Name())
147 s.mu.Lock()
148 _, live := s.active[p]
149 s.mu.Unlock()
150 if live {
151 continue
152 }
153 if err := os.Remove(p); err == nil {
154 n++
155 }
156 }
157 return n, nil
48 return &Proxy{cfg: cfg}
158 49 }
159 50
160 func (p *Proxy) Janitor() (int, error) { return p.spool.Janitor() }
161
162 51 func (p *Proxy) Stats() map[string]int64 {
163 52 return map[string]int64{
164 "pushes": p.stats.pushes.Load(),
165 "reads": p.stats.reads.Load(),
166 "denied": p.stats.denied.Load(),
167 "queued": p.stats.queued.Load(),
168 "spooled": p.stats.spooled.Load(),
169 "oversized": p.stats.oversized.Load(),
53 "reads": p.stats.reads.Load(),
54 "hidden": p.stats.hidden.Load(),
55 "push_attempts": p.stats.pushes.Load(),
170 56 }
171 57 }
172 58
173 // Route parses /git/<name>.git/<rest> and returns the repo name and remainder.
59 // Route parses /git/<name>.git/<rest>.
174 60 func Route(path string) (name, rest string, ok bool) {
175 61 const prefix = "/git/"
176 62 if !strings.HasPrefix(path, prefix) {
@@ -189,23 +75,12 @@ func Route(path string) (name, rest string, ok bool) {
189 75 return strings.TrimSuffix(dir, ".git"), rest, true
190 76 }
191 77
192 func classify(rest, rawQuery string) (Op, acl.Role, error) {
193 switch {
194 case rest == "info/refs":
195 q := parseQuery(rawQuery)
196 switch q["service"] {
197 case "git-receive-pack":
198 return OpPush, acl.Write, nil
199 case "git-upload-pack":
200 return OpRead, acl.Read, nil
201 }
202 return 0, acl.None, fmt.Errorf("client requested an unsupported service: dumb HTTP is not enabled (use smart HTTP)")
203 case strings.HasPrefix(rest, "git-receive-pack"):
204 return OpPush, acl.Write, nil
205 case strings.HasPrefix(rest, "git-upload-pack"):
206 return OpRead, acl.Read, nil
78 // IsPush reports whether a request is asking to write.
79 func IsPush(rest, rawQuery string) bool {
80 if rest == "info/refs" {
81 return parseQuery(rawQuery)["service"] == "git-receive-pack"
207 82 }
208 return 0, acl.None, fmt.Errorf("unsupported path %q", rest)
83 return strings.HasPrefix(rest, "git-receive-pack")
209 84 }
210 85
211 86 func parseQuery(q string) map[string]string {
@@ -228,168 +103,54 @@ func (p *Proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
228 103 }
229 104 m, err := p.cfg.Open(name)
230 105 if err != nil {
231 http.Error(w, "repository not found", http.StatusNotFound)
232 return
233 }
234 op, need, err := classify(rest, r.URL.RawQuery)
235 if err != nil {
236 http.Error(w, err.Error(), http.StatusForbidden)
237 return
238 }
239
240 actor, state := p.authenticate(r)
241 if state == authBad {
242 // Bad credentials are a failed authentication, not a permission
243 // problem: answer 401 so the client can retry, and say nothing about
244 // whether the repo exists.
245 p.stats.denied.Add(1)
246 w.Header().Set("WWW-Authenticate", `Basic realm="vole"`)
247 http.Error(w, "vole: invalid credentials", http.StatusUnauthorized)
106 http.NotFound(w, r)
248 107 return
249 108 }
250 109 a, err := m.ACL()
251 110 if err != nil {
252 111 p.cfg.Log.Error("read acl", "repo", name, "err", err)
253 http.Error(w, "access rules unavailable", http.StatusInternalServerError)
112 http.Error(w, "repository unavailable", http.StatusInternalServerError)
254 113 return
255 114 }
256 role := a.RoleFor(actor)
257 if !role.AtLeast(need) {
258 p.stats.denied.Add(1)
259 if state == authNone {
260 w.Header().Set("WWW-Authenticate", `Basic realm="vole"`)
261 http.Error(w, "authentication required", http.StatusUnauthorized)
262 return
263 }
264 http.Error(w, fmt.Sprintf("vole: %s does not have %s access to %s (your role: %s)",
265 displayActor(actor), need, displayRepo(actor, name), role), http.StatusForbidden)
115 // The reader has no identity here, so the default role is the only thing
116 // that can grant access. Anything else is indistinguishable from absent.
117 if !a.Can("", acl.Read) {
118 p.stats.hidden.Add(1)
119 http.NotFound(w, r)
266 120 return
267 121 }
268 122
269 if op == OpPush {
270 // Take a slot before reading a byte of the body: the memory cost of a
271 // push is git's, and it arrives with the pack.
272 start := time.Now()
273 select {
274 case p.sem <- struct{}{}:
275 defer func() { <-p.sem }()
276 case <-time.After(p.cfg.QueueTimeout):
277 p.stats.queued.Add(1)
278 http.Error(w, "vole: too many concurrent pushes, try again shortly", http.StatusServiceUnavailable)
279 return
280 }
281 if d := time.Since(start); d > 50*time.Millisecond {
282 p.cfg.Log.Info("push waited for a slot", "repo", name, "waited", d.String())
283 }
123 if IsPush(rest, r.URL.RawQuery) {
124 p.stats.pushes.Add(1)
125 http.Error(w, "vole: pushes are not accepted over HTTP; use an SSH remote", http.StatusForbidden)
126 return
284 127 }
285
286 if r.ContentLength > p.cfg.MaxPushBytes {
287 p.stats.oversized.Add(1)
288 http.Error(w, fmt.Sprintf("vole: push is %d bytes, over the %d byte limit",
289 r.ContentLength, p.cfg.MaxPushBytes), http.StatusRequestEntityTooLarge)
128 if rest != "info/refs" && !strings.HasPrefix(rest, "git-upload-pack") {
129 http.NotFound(w, r)
290 130 return
291 131 }
292
293 body, length, cleanup, err := p.body(r)
294 if err != nil {
295 p.stats.oversized.Add(1)
296 http.Error(w, err.Error(), http.StatusRequestEntityTooLarge)
132 if r.Method != http.MethodGet && r.Method != http.MethodPost {
133 http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
297 134 return
298 135 }
299 defer cleanup()
300
301 // The client can disappear while a chunked body is being spooled. There is
302 // no point running receive-pack for a push nobody will read the answer to.
303 if err := r.Context().Err(); err != nil {
304 p.cfg.Log.Info("client disconnected before backend start", "repo", name)
136 // upload-pack request bodies are small (wants/haves). CGI cannot express
137 // "length unknown", so refuse rather than buffer.
138 if r.Method == http.MethodPost && r.ContentLength < 0 {
139 http.Error(w, "vole: chunked request bodies are not supported on the read path",
140 http.StatusLengthRequired)
305 141 return
306 142 }
307 143
308 if err := p.backend(w, r, m, rest, actor, body, length); err != nil {
144 if err := p.backend(w, r, m, rest); err != nil {
309 145 p.cfg.Log.Error("git backend", "repo", name, "err", err)
146 return
310 147 }
311 if op == OpPush {
312 p.stats.pushes.Add(1)
313 } else {
314 p.stats.reads.Add(1)
315 }
316 }
317
318 // body returns a reader for the request payload plus its true length.
319 // git http-backend is CGI and cannot proceed without CONTENT_LENGTH, so a
320 // chunked upload has to be spooled to disk first.
321 func (p *Proxy) body(r *http.Request) (io.Reader, int64, func(), error) {
322 noop := func() {}
323 if r.Method != http.MethodPost && r.Method != http.MethodPut {
324 return http.NoBody, 0, noop, nil
325 }
326 if r.ContentLength >= 0 {
327 // Stream directly, but never trust the declared length alone.
328 lim := &limitReader{r: r.Body, remaining: p.cfg.MaxPushBytes + overshootSlack, max: p.cfg.MaxPushBytes}
329 return lim, r.ContentLength, noop, nil
330 }
331
332 f, err := p.spool.Create()
333 if err != nil {
334 return nil, 0, noop, fmt.Errorf("vole: cannot spool upload: %w", err)
335 }
336 n, err := io.Copy(f, io.LimitReader(r.Body, p.cfg.MaxPushBytes+1))
337 if err != nil {
338 p.spool.Release(f)
339 return nil, 0, noop, fmt.Errorf("vole: upload interrupted: %w", err)
340 }
341 if n > p.cfg.MaxPushBytes {
342 p.spool.Release(f)
343 return nil, 0, noop, fmt.Errorf("vole: push exceeds the %d byte limit", p.cfg.MaxPushBytes)
344 }
345 if _, err := f.Seek(0, io.SeekStart); err != nil {
346 p.spool.Release(f)
347 return nil, 0, noop, fmt.Errorf("vole: spool seek: %w", err)
348 }
349 p.stats.spooled.Add(n)
350 return f, n, func() { p.spool.Release(f) }, nil
351 }
352
353 // limitReader fails loudly if a client sends more than it declared.
354 type limitReader struct {
355 r io.Reader
356 remaining int64
357 max int64
358 n int64
359 }
360
361 func (l *limitReader) Read(b []byte) (int, error) {
362 n, err := l.r.Read(b)
363 l.n += int64(n)
364 if l.n > l.max {
365 return n, fmt.Errorf("vole: push exceeds the %d byte limit", l.max)
366 }
367 return n, err
368 }
369
370 type authState int
371
372 const (
373 authNone authState = iota
374 authBad
375 authOK
376 )
377
378 func (p *Proxy) authenticate(r *http.Request) (string, authState) {
379 user, pass, ok := r.BasicAuth()
380 if !ok {
381 return "", authNone
382 }
383 actor, err := p.cfg.Actor(user, pass)
384 if err != nil {
385 return "", authBad
386 }
387 return actor, authOK
148 p.stats.reads.Add(1)
388 149 }
389 150
390 func (p *Proxy) backend(w http.ResponseWriter, r *http.Request, m *repo.Meta, rest, actor string, body io.Reader, length int64) error {
151 func (p *Proxy) backend(w http.ResponseWriter, r *http.Request, m *repo.Meta, rest string) error {
391 152 cmd := exec.Command("git", "http-backend")
392 cmd.Env = append(os.Environ(), p.cgiEnv(r, m, rest, actor, length)...)
153 cmd.Env = append(os.Environ(), p.cgiEnv(r, m, rest)...)
393 154 stdin, err := cmd.StdinPipe()
394 155 if err != nil {
395 156 return err
@@ -404,11 +165,11 @@ func (p *Proxy) backend(w http.ResponseWriter, r *http.Request, m *repo.Meta, re
404 165 return err
405 166 }
406 167
407 copyErr := make(chan error, 1)
408 168 go func() {
409 169 defer stdin.Close()
410 _, err := io.Copy(stdin, body)
411 copyErr <- err
170 if r.Body != nil {
171 io.Copy(stdin, r.Body)
172 }
412 173 }()
413 174
414 175 br := bufio.NewReaderSize(stdout, 256<<10)
@@ -418,11 +179,6 @@ func (p *Proxy) backend(w http.ResponseWriter, r *http.Request, m *repo.Meta, re
418 179 line, err := br.ReadString('\n')
419 180 if err != nil {
420 181 cmd.Wait()
421 e := <-copyErr
422 if e != nil {
423 // Most likely the client vanished mid-upload.
424 return fmt.Errorf("client body: %w", e)
425 }
426 182 msg := strings.TrimSpace(stderr.String())
427 183 if msg == "" {
428 184 msg = err.Error()
@@ -476,33 +232,26 @@ func (p *Proxy) backend(w http.ResponseWriter, r *http.Request, m *repo.Meta, re
476 232 break
477 233 }
478 234 }
479 <-copyErr
480 if err := cmd.Wait(); err != nil {
481 // A non-zero exit after the response was written is expected when git
482 // refuses the push; the client has already been told why.
483 if msg := strings.TrimSpace(stderr.String()); msg != "" && status >= 400 {
484 return fmt.Errorf("http-backend exit: %w: %s", err, msg)
485 }
486 }
235 // A non-zero exit here is git refusing a read; the client has the reason.
236 cmd.Wait()
487 237 return nil
488 238 }
489 239
490 func (p *Proxy) cgiEnv(r *http.Request, m *repo.Meta, rest, actor string, length int64) []string {
240 func (p *Proxy) cgiEnv(r *http.Request, m *repo.Meta, rest string) []string {
241 length := r.ContentLength
242 if length < 0 {
243 length = 0
244 }
491 245 env := []string{
492 246 "GIT_PROJECT_ROOT=" + p.cfg.Root,
493 247 "GIT_HTTP_EXPORT_ALL=1",
494 248 "PATH_INFO=/" + m.Name + ".git/" + rest,
495 249 "REQUEST_METHOD=" + r.Method,
496 250 "QUERY_STRING=" + r.URL.RawQuery,
497 "REMOTE_USER=" + actor,
498 251 "REMOTE_ADDR=" + r.RemoteAddr,
499 252 "SERVER_PROTOCOL=" + r.Proto,
500 253 "GATEWAY_INTERFACE=CGI/1.1",
501 254 "CONTENT_LENGTH=" + strconv.FormatInt(length, 10),
502 // Hooks read these; http-backend passes its environment down.
503 "VOLE_ACTOR=" + actor,
504 "VOLE_REPO=" + m.Name,
505 "VOLE_MAX_BLOB=" + strconv.FormatInt(p.cfg.MaxBlobBytes, 10),
506 255 }
507 256 if ct := r.Header.Get("Content-Type"); ct != "" {
508 257 env = append(env, "CONTENT_TYPE="+ct)
@@ -521,12 +270,3 @@ func (p *Proxy) cgiEnv(r *http.Request, m *repo.Meta, rest, actor string, length
521 270 }
522 271 return env
523 272 }
524
525 func displayActor(a string) string {
526 if a == "" {
527 return "anonymous"
528 }
529 return a
530 }
531
532 func displayRepo(_ string, name string) string { return name }