Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Add phased implementation roadmap

17311880abe22e0b4e1ae7cca51fdb5ccfd8b73b · Keenan Nemetz · 18d ago · parent e4a9dc50 

Add phased implementation roadmap

Commits to single-tenant self-hosted scope with no federation, which
removes the peer replication protocol and multi-tenant storage design
from the plan. Covers Phase 0 spikes through Phase 8 operations, with
measurable exit criteria and the metadata write-path risk called out.

docs/roadmap.md +234

@@ -0,0 +1,234 @@
1 # Vole — Implementation Roadmap
2
3 Phased plan for building the forge described in [the README](../README.md).
4
5 ## Scope
6
7 **Single-tenant, self-hosted.** One Vole instance serves one team on one box. There is no
8 multi-tenancy, no hosted SaaS posture, and no organization/team hierarchy beyond a flat
9 list of users and repos.
10
11 **No federation.** Instances do not talk to each other. Read the README's "replicated
12 issues" bullet literally instead: because issues, change requests, and ACLs live in git
13 objects on sidecar refs, **every clone of a repo carries the full forge state**. Replication
14 is `git fetch`, not a protocol. State survives deleting the instance entirely.
15
16 These two constraints are what keep the project small enough to finish. They are assumed
17 throughout; see [Non-goals](#non-goals) for what they rule out.
18
19 ## Architectural decisions
20
21 | Decision | Choice | Why |
22 |---|---|---|
23 | State model | Git is authoritative; SQLite is a derived read model | Rebuildable index means storage is a swappable detail |
24 | Git integration | Shell out to the `git` binary | Hooks, CAS ref updates, and full `receive-pack` semantics for free; go-git lacks all three |
25 | Transport | Smart HTTP via `git http-backend` | Battle-tested; SSH transport deferred |
26 | Metadata storage | Sharded sidecar refs (`refs/vole/issues/<shard>`, `refs/vole/crs`, `refs/vole/acl`) | Parallel writers, no single-ref contention, still fetchable as one glob |
27 | Metadata writes | Compare-and-swap ref updates, retry on loss | No lock server, no lost updates |
28 | ACL enforcement | `pre-receive` hook, mirrored in web handlers | Web-only gating makes `git push` the bypass |
29 | Binary layout | One static CGO-free binary, three modes (`serve`, `runner`, `hook`) | Drop-anywhere deployment |
30 | Index store | SQLite via `modernc.org/sqlite` | CGO-free; rebuildable, so swapping engines later is cheap |
31 | Config | TOML on disk, no database-backed settings | One box, one file, greppable |
32
33 ## Non-goals
34
35 Explicitly **not** built. Each is additive later; none is load-bearing for the concept.
36
37 - Multi-tenant isolation, orgs, nested teams, per-tenant resource quotas
38 - Federation, peer replication, cross-instance search, global entity IDs
39 - SSH transport (HTTP + PATs first; SSH keys are Phase 7 work if demanded)
40 - Container-isolated CI steps (shell steps only in v1)
41 - Horizontal scaling, HA, multi-writer Postgres
42 - Kubernetes deployment story
43
44 ## Phases
45
46 Each phase ends with something runnable and a measurement that proves it, not a checklist
47 of merged PRs.
48
49 ### Phase 0 — Spikes and decisions (~1 week)
50
51 Throwaway code. Nothing merges. The point is to kill bad assumptions while they're cheap.
52
53 **Spike A — metadata write path.** Drive `git update-ref --stdin` from N concurrent Go
54 writers committing to a metadata tree. Measure commits/sec and CAS-retry rate with (a) a
55 single unsharded ref and (b) 256 hash-sharded refs. Also measure batched commits (coalescing
56 N entity writes into one commit) against per-entity commits.
57
58 **Spike B — smart HTTP through Go.** Proxy `git http-backend` as a subprocess, set the CGI
59 env correctly, run a 1 GB push, a `pre-receive` rejection, and a client abort mid-transfer.
60 Confirm hook stderr reaches the client and no orphaned `receive-pack` processes survive.
61
62 **Spike C — serving files from git trees.** Render a 10k-file tree as static pages with a
63 read-through cache. Record p50/p99 and memory for cold and warm cache.
64
65 **Exit criteria:** commits/sec at the 95th percentile under 32 writers with retry rate
66 recorded; a 1 GB push succeeds and a rejected push reports a real reason to the client;
67 p99 for a warm static file served from a 10k-file tree. Decisions written into the
68 [decision log](#decision-log).
69
70 ### Phase 1 — It's a git server
71
72 The unglamorous foundation. If this phase isn't solid, nothing above it is trustworthy.
73
74 - Bare-repo store on disk with a stable repo identity; create/rename/delete via CLI and web
75 - Smart HTTP read and write, auth via HTTP basic + personal access tokens
76 - Users, sessions, PATs (hashed at rest, scoped, revocable)
77 - ACL v1: owner / write / read, stored on `refs/vole/acl`
78 - `pre-receive` + `update` hooks enforcing ACLs, protected branches, and object size limits
79 - Web UI: repo list, tree/blob browser, raw view, commit log
80 - Instance signing key generated at first boot and persisted (used from Phase 7)
81 - `vole index rebuild` implemented early and exercised by tests, even though the index has
82 little to hold yet
83
84 **Exit criteria:** authenticated clone/push/fetch works against a fresh instance; an
85 unauthorized push is rejected by the hook with an actionable message; `vole index rebuild`
86 reproduces state after deleting the index file; a rejected push leaves no partial ref update.
87
88 ### Phase 2 — Metadata layer and issues
89
90 The phase where the core bet is either proven or abandoned.
91
92 - Metadata tree schema: file-per-entity, sharded directories, stable IDs generated as
93 monotonic counters committed alongside the entities
94 - CAS commit helper shared by all metadata writers, with retry and a bounded write queue
95 - Derived index with incremental update on ref change and full rebuild; every query path
96 works from a cold index
97 - Issues: create, comment, edit, close/reopen, labels, assignees, cross-references
98 (`#12`), full-text search over the index
99 - Markdown rendering with HTML sanitization
100 - Notifications: in-app first, email via SMTP config second
101
102 **Exit criteria:** deleting the SQLite file and running `vole index rebuild` reproduces
103 every issue byte-for-byte (diffed, not eyeballed); 32 concurrent commenters lose zero
104 comments; `git fetch 'refs/vole/*:refs/vole/*'` on a fresh clone yields a browsable issue
105 history with `git log`.
106
107 ### Phase 3 — Change requests
108
109 - CR entity on `refs/vole/crs` referencing base branch + head branch + head SHA
110 - Diff view (three-dot against merge base), file tree, per-file collapse for large diffs
111 - Line-anchored review comments that survive force-pushes via `(path, blob-sha, line)` anchoring
112 - Review states: comment / approve / request changes
113 - Merge as a CAS update of the base ref from the reviewed head SHA; reject stale merges
114 rather than clobbering
115 - Protected branches: required approvals, required status checks (wired in Phase 4)
116 - Conflict detection surfaced before merge is attempted
117
118 **Exit criteria:** a full CR flow on a real repo with review comments that stay attached
119 across a force-push; a merge raced by a concurrent push is rejected, not applied; a
120 protected branch cannot be pushed by a non-owner even with a valid token.
121
122 ### Phase 4 — CI/CD
123
124 - `.vole/workflows/*.yaml` parsed and validated at push time; invalid workflows fail loudly
125 on the CR rather than silently doing nothing
126 - Push-event queue persisted in SQLite, with at-least-once delivery
127 - Runner protocol: `vole runner` registers, long-polls for work, leases a job, heartbeats;
128 an expired lease requeues the job
129 - Job execution: checkout by exact SHA from the server, sequential shell steps, per-step
130 timeout, streaming logs
131 - Run records pin SHA + workflow file hash + toolchain env; signed with the instance key
132 - Statuses surfaced on CRs and satisfying required-check gating from Phase 3
133 - Log and artifact storage onto the content-addressed store (shared with Phase 5)
134
135 **Exit criteria:** a push runs a pipeline end-to-end; killing a runner mid-job results in
136 the job being requeued and completing on another runner; a run record can be verified
137 against the SHA it claims; a workflow that mutates its own definition mid-run still executes
138 the version it was queued with.
139
140 ### Phase 5 — LFS
141
142 - `POST /objects/batch` implementing `application/vnd.git-lfs+json`, upload and download
143 - Content-addressed object store with size limits and reference counting
144 - Server-side pointer hygiene: warn on committed binaries above a threshold, provide a
145 suggested `git lfs track` command in the rejection message
146 - `pre-receive` rejects LFS pointers whose objects are missing, before the ref moves
147 - Migrate CI artifacts onto the same store with the same GC story
148
149 **Exit criteria:** clone of an LFS repo pulls real objects through the proxy; a pointer
150 pushed without its object is rejected at hook time; GC never collects an object reachable
151 from a live ref.
152
153 ### Phase 6 — Site hosting
154
155 - A designated pages ref per repo, served as static files directly from the git tree
156 - Per-repo hostname (`<repo>.<instance-domain>`) plus optional custom domain
157 - Automatic TLS via ACME when a domain is configured, with DNS-01 for wildcards
158 - ETag/`Last-Modified` from blob SHAs; correct `Content-Type` from extension and content sniff
159 - Cache invalidation on push to the pages ref, without a full rebuild
160
161 **Exit criteria:** a pushed doc tree is live in under 5 seconds; conditional requests return
162 `304`; a custom domain gets a certificate with no manual steps; serving a repo with a
163 path-traversal-shaped filename is safe.
164
165 ### Phase 7 — Signing and audit
166
167 Scope is provenance and tamper-evidence for a single instance — no peer trust, no federation.
168
169 - User ed25519 key upload; commit signature verification with a visible verified/unverified badge
170 - All metadata commits signed with the instance key; verification of instance-signed
171 metadata on read so a hand-edited ref is flagged
172 - Append-only instance audit log (auth events, key changes, ACL changes, repo deletion)
173 as a hash chain in a dedicated ref, signed
174 - `git verify-commit` interop: signatures produced must verify with stock git tooling
175
176 **Exit criteria:** a commit signed by a registered key shows verified; a metadata commit
177 altered out-of-band is flagged on read; the audit chain detects a deleted middle entry;
178 signatures verify with `git verify-commit` outside Vole.
179
180 ### Phase 8 — Operate it
181
182 - Metrics and traces (job queue depth, CAS retry rate, push latency, index lag)
183 - Backup and restore: documented artifact set, plus a restore drill in CI
184 - Quotas: per-repo size, object count, LFS bytes
185 - Metadata GC that never breaks history: entities are tombstoned, not dropped
186 - Upgrade path with an explicit schema-version step and a downgrade refusal that explains itself
187 - Multi-arch container images and a `vole init` flow that produces a working instance in
188 one command
189 - Operator docs: TLS, SMTP, runner registration, backup cadence
190
191 **Exit criteria:** restore-from-backup drill passes in CI; upgrade N→N+1 succeeds on a repo
192 with live LFS objects and completes with no downtime; `vole init` produces a working
193 instance with one command on a clean box.
194
195 ## Risks
196
197 **Metadata writes are ref CAS on a shared ref.** The whole state model rides on
198 `git update-ref` throughput, which is roughly hundreds of writes/sec per ref before retry
199 storms. Mitigations, in order of preference: shard refs (Spike A measures the gain),
200 coalesce writes into batched commits, and only then consider a write-ahead log in front of
201 the ref. This is settled in Phase 0, *before* the schema is built around per-entity commits.
202
203 **Derived index can silently drift from refs.** Any path that writes refs without updating
204 the index is a bug that shows up as missing data. Mitigation: the index is never written
205 except through one function that is given the ref transaction it is updating; a scheduled
206 `rebuild` diff against incremental state runs in CI.
207
208 **Hook-enforced policy can be bypassed by server-side operations.** Merges and other
209 server-initiated ref updates must route through the same validation as `pre-receive`, or
210 the ACL model has two implementations that will disagree. Mitigation: one policy function,
211 called from both, with a shared test fixture table.
212
213 **Diffs and file serving get expensive at scale before anything else does.** Static serving
214 straight from a tree is elegant and fast until it isn't. Spike C exists to find the cliff
215 before Phase 6 discovers it.
216
217 ## Decision log
218
219 Filled in as phases land. Append; don't rewrite.
220
221 | Date | Decision | Rationale |
222 |---|---|---|
223 | — | Single-tenant, no federation | Keeps scope finite; removes global IDs, peer protocol, and multi-writer storage |
224 | — | Git authoritative, SQLite derived | Rebuildability makes the storage choice reversible |
225
226 ## Open questions
227
228 - Workflow YAML dialect: own minimal schema, or a subset of GitHub Actions for familiarity?
229 Leaning own schema — a subset that silently differs from the real thing is worse than a
230 clearly separate one.
231 - Whether Phase 1 ships SSH transport early because operators expect it, despite HTTP
232 being sufficient. Defer until someone actually complains.
233 - Email deliverability: rely on operator SMTP, or bundle a fallback? Leaning operator SMTP
234 only, with a loud warning when unconfigured.