vole

Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Blame · docs/operations.md

Line attribution on main · 288 line(s)

View file History

520470e1 Keenan Nemetz 16d ago	1	# Operations
	2
	3	A forge you own should be easy to back up and recover. Vole keeps the important
ab326d3a Keenan Nemetz 14d ago	4	state small: an account database, a handful of files, and ordinary Git repositories.
520470e1 Keenan Nemetz 16d ago	5	These are the tasks that keep an instance healthy.
520470e1 Keenan Nemetz 16d ago	6
b1499c72 Keenan Nemetz 13d ago	7	## Audit log
	8
	9	`data/audit.log` records every privileged action — roster and key changes, access
	10	rules, forced lock releases, forced pushes, release rewrites, admin commands,
	11	sessions, and ordinary pushes — as one tab-separated line:
	12
	13	```text
	14	timestamp action actor subject detail
	15	```
	16
	17	- One shape. Every writer goes through the same code, so a parser can split
	18	on tabs. `action` is a lower-case dotted name (`acl.update`, `push.force`,
	19	`session.create`, `admin`); `actor` is the account or OS user, `-` when there
	20	is none; `subject` is the repository, ref, or account the action was about.
	21	- Fields are escaped and truncated. A key comment, an LFS lock path, a
	22	repository name, or an admin argument cannot add a column or a line, and a
	23	field is capped at 4 KiB.
	24	- Mode 0600, owner only. The file is created that way and tightened on every
	25	write, so an instance upgraded from an older version stops being world-readable
	26	the first time it writes. Startup refuses a log that is a symlink, a directory,
	27	a non-regular file, or owned by another account.
	28	- Privileged changes fail closed. A roster, key, access-rule, session, forced
	29	lock release, or admin command that cannot be recorded is refused, and the
	30	command says so: `the audit log could not be written, so nothing was changed`.
	31	A push that overrides a lock, an approval, a release, or history is refused
	32	unless its override can be written. There is no setting that turns this off,
	33	because an override is exactly what an attacker with config access would set.
	34	- Ordinary pushes and web writes are reported, not refused. Those changes
	35	are already durable in Git; a failed write appears as `audit log write failed`
	36	in the server log (or on the pusher's terminal for `post-receive`) rather than
	37	making a working push look rejected.
	38	- A broken log stops the instance. `vole serve` and `vole admin` refuse to
	39	start when the log cannot be opened for append, which turns a silent, long-lived
	40	gap into a startup failure with a path in it.
	41
43e681d8 Keenan Nemetz 4d ago	42	An instance admin can read the log in the browser at Admin → Audit log. The
0db980ee Keenan Nemetz 4d ago	43	page shows one record per row, newest first, and pages back through the file. The
	44	Admin overview repeats the newest few, so a review can start there. Both are
	45	read-only views: they never edit, truncate, or rewrite a line, and a record the
	46	reader cannot parse is skipped rather than hidden from the rest of the page.
43e681d8 Keenan Nemetz 4d ago	47
b1499c72 Keenan Nemetz 13d ago	48	Ship the file off the host — a read-only mount, a log shipper, or a periodic
	49	`rsync` of an append-only copy — and alert on the absence of expected lines
	50	as well as on the presence of unexpected ones. Local append-only storage is not
	51	tamper-evident: anyone who can write `data/` can edit it, so keep an off-host copy
	52	that the service account cannot reach.
	53
520470e1 Keenan Nemetz 16d ago	54	## Back up an instance
520470e1 Keenan Nemetz 16d ago	55
642e5994 Keenan Nemetz 14d ago	56	Back up the instance state:
520470e1 Keenan Nemetz 16d ago	57
520470e1 Keenan Nemetz 16d ago	58	```sh
4506a4d3 Keenan Nemetz 14d ago	59	vole admin account backup backup/users.db # accounts, keys, sessions, rules, locks, views
b312e166 Keenan Nemetz 11d ago	60	sqlite3 data/runs.db ".backup 'backup/runs.db'" # CI history, queue, and approvals
5ece70a5 Keenan Nemetz 10d ago	61	cp -a data/logs backup/ # CI attempt logs
642e5994 Keenan Nemetz 14d ago	62	cp data/ssh_host_ed25519_key backup/ # the SSH host key
0c40a1d3 Keenan Nemetz 7d ago	63	sqlite3 data/secrets.db ".backup 'backup/secrets.db'" # sealed repository secrets
0c40a1d3 Keenan Nemetz 7d ago	64	cp data/secrets.key backup/ # the key those secrets need
642e5994 Keenan Nemetz 14d ago	65	cp -a data/acme backup/ 2>/dev/null # TLS account key and certificates, if ACME is on
520470e1 Keenan Nemetz 16d ago	66	cp data/audit.log backup/ 2>/dev/null # history of admin actions
	67	```
	68
2e024cc9 Keenan Nemetz 14d ago	69	`vole admin account backup` takes a consistent copy, even while Vole is running. Copying
520470e1 Keenan Nemetz 16d ago	70	`users.db` by hand can miss recent changes that have not been written to the main
b312e166 Keenan Nemetz 11d ago	71	database file yet. Do the same for `runs.db` with SQLite's backup command; copying it
b312e166 Keenan Nemetz 11d ago	72	by hand while the server is writing can miss WAL pages.
520470e1 Keenan Nemetz 16d ago	73
0c40a1d3 Keenan Nemetz 7d ago	74	Keep `data/secrets.key` apart from the copies of `data/`. A backup that holds both
	75	the database and its key is a backup of plaintext: the encryption protects a copy
	76	that loses the key, not one that keeps it. Restoring `data/secrets.db` beside a
	77	newly generated key reads nothing, and Vole says so rather than handing a job an
	78	empty value.
	79
ab326d3a Keenan Nemetz 14d ago	80	If you lose the account database, you need to recreate accounts and re-enroll
ab326d3a Keenan Nemetz 14d ago	81	their passkeys. You do not lose repositories. The backup contains account metadata,
6c84d387 Keenan Nemetz 16d ago	82	public keys, and hashes of session and enrollment tokens; it does not contain a
6c84d387 Keenan Nemetz 16d ago	83	usable password or a plaintext session token. Treat it as private anyway.
520470e1 Keenan Nemetz 16d ago	84
520470e1 Keenan Nemetz 16d ago	85	Repositories are ordinary Git repositories under `data/repos/`. You can back them
b312e166 Keenan Nemetz 11d ago	86	up by copying or cloning. Issues, change requests, and releases travel with them
	87	because they are stored in Git. CI history does not: it lives in `data/runs.db` and
	88	must be backed up with it.
520470e1 Keenan Nemetz 16d ago	89
	90	## Restore an instance
	91
5ece70a5 Keenan Nemetz 10d ago	92	1. Restore `data/users.db`, `data/runs.db`, `data/logs/`,
0c40a1d3 Keenan Nemetz 7d ago	93	`data/ssh_host_ed25519_key`, `data/audit.log`, `data/secrets.db` with the
	94	`data/secrets.key` that sealed it, and `data/acme/` if the instance obtains its
	95	own TLS certificates.
b312e166 Keenan Nemetz 11d ago	96	2. Restore the repositories under `data/repos/` and the object store under
5ece70a5 Keenan Nemetz 10d ago	97	`data/lfs/` if you keep CI artifacts.
3abbe749 Keenan Nemetz 16d ago	98	3. Run `vole admin index rebuild`.
2e024cc9 Keenan Nemetz 14d ago	99	4. Run `vole admin account revoke-sessions --all` so sessions from the restored copy
6c84d387 Keenan Nemetz 16d ago	100	cannot outlive their cookies.
6c84d387 Keenan Nemetz 16d ago	101	5. Start `vole serve`.
520470e1 Keenan Nemetz 16d ago	102
474a812d Keenan Nemetz 16d ago	103	Release notes for an upgrade: an older `users.db` is upgraded in place when it is
62576fa8 Keenan Nemetz 14d ago	104	opened. Accounts used to have an internal id and an optional display label; both
ab326d3a Keenan Nemetz 14d ago	105	are gone, and every key, passkey, enrollment link, session and admin grant is rebuilt
62576fa8 Keenan Nemetz 14d ago	106	around the registered name. Access-rule entries and file locks in that database
	107	are rewritten through the same mapping, and the retired `user_names` alias table
	108	is dropped. Take a backup first if you might roll the binary back. Approval files
	109	live in Git and are not rewritten; an approval written under an old account id
	110	stops matching, so ask for a fresh approval.
474a812d Keenan Nemetz 16d ago	111
	112	Contributor branches moved from `refs/heads/contrib/<account-id>/` to
	113	`refs/heads/contrib/<name>/`. Old refs stay in the repository and can still be
	114	fetched, but only a maintainer can push them; existing change requests still name
	115	the same head commit. A contributor who wants to keep working pushes the same
	116	commit under their new folder and opens a request from it.
	117
5ece70a5 Keenan Nemetz 10d ago	118	Large files and build artifacts live in `data/lfs/`; CI logs live in `data/logs/`.
	119	Back both directories up too if you need those files. `vole admin gc` treats files
	120	reachable from a repository or a published artifact as things to keep; logs are
	121	ordinary files and are not swept.
520470e1 Keenan Nemetz 16d ago	122
73755a56 Keenan Nemetz 9d ago	123	## Restart the server
	124
	125	A clean stop (`systemctl restart`, or Ctrl-C on a development instance) stops
	126	handing out new CI jobs and waits up to a minute for the jobs already leased to
	127	report their result. A job that finishes in that window is stored and is not run
	128	again. A job still running when the window ends loses its runner connection and
	129	is requeued as another attempt once its lease expires. The wait is best effort;
	130	start long builds well before a planned restart.
	131
520470e1 Keenan Nemetz 16d ago	132	## Clean up stored files
	133
	134	```sh
3abbe749 Keenan Nemetz 16d ago	135	vole admin gc --dry-run # show what would be removed
	136	vole admin gc # remove unused files
	137	vole admin gc --grace 24h # keep unused files for a day
520470e1 Keenan Nemetz 16d ago	138	```
520470e1 Keenan Nemetz 16d ago	139
5ece70a5 Keenan Nemetz 10d ago	140	`vole admin gc` cleans the shared storage used by LFS and build artifacts. CI logs are
5ece70a5 Keenan Nemetz 10d ago	141	ordinary files under `data/logs/` and are not part of the sweep. It
520470e1 Keenan Nemetz 16d ago	142	also repairs each repository's links into that storage, so it helps when files
	143	are missing or links are broken.
	144
	145	The default grace period is one hour. A recent upload that has not been pushed yet
	146	will not be removed.
	147
3abbe749 Keenan Nemetz 16d ago	148	You can run `vole admin gc` while people are pushing. There is no need to schedule it;
520470e1 Keenan Nemetz 16d ago	149	storage only grows when someone pushes files.
	150
	151	## Rebuild indexes
	152
	153	`data/index.db` is generated by Vole and can be deleted safely:
	154
	155	```sh
3abbe749 Keenan Nemetz 16d ago	156	vole admin index rebuild
520470e1 Keenan Nemetz 16d ago	157	```
520470e1 Keenan Nemetz 16d ago	158
b312e166 Keenan Nemetz 11d ago	159	`data/runs.db` holds the CI queue, finished-run records, and contributor-branch CI
	160	approvals. It is a source of truth, not a rebuildable index: deleting it loses CI
	161	history as well as attempts that were in progress. Back it up with the rest of the
59badc63 Keenan Nemetz 11d ago	162	instance. `vole admin index rebuild` rebuilds only the repository and issue index;
59badc63 Keenan Nemetz 11d ago	163	it cannot restore CI history.
520470e1 Keenan Nemetz 16d ago	164
	165	## Security notes
	166
0c40a1d3 Keenan Nemetz 7d ago	167	- Two long-term secrets are stored: the SSH host key, and the key that seals
	168	repository secrets. Keep both readable only by the Vole user (`0600`). Browser
	169	session and enrollment tokens are stored as hashes.
	170	- Repository secrets are encrypted at rest, which protects a copy of `data/` that
	171	leaves the host. It does not protect a host somebody already owns: file modes
	172	are the boundary, and the default key location is beside the database. Anything
	173	that can read the data directory can read the values.
ed181212 Keenan Nemetz 13d ago	174	- Vole creates its own secrets at `0600` and its data directory at `0700`: the
	175	roster (`users.db`), the audit log, the SSH host key and the ACME account
	176	directory. It also tightens an existing instance on startup, and logs the paths
	177	whose mode it changed, so an upgrade from an earlier version cannot leave a
	178	private key world-readable. If a mode cannot be changed, startup fails rather
	179	than serving keys it cannot protect.
	180	A site directory that is deliberately shared with a backup group is fine at
	181	`0750`: group access is left alone, world access is not.
520470e1 Keenan Nemetz 16d ago	182	- Nothing Vole writes is signed: not pushes, not approvals, not CI results. Vole
	183	relies on the SSH key at the connection, the checks before a push is accepted,
	184	and the audit log.
ab326d3a Keenan Nemetz 14d ago	185	- The web interface is read-only for anonymous visitors. Passkey sign-in enables
6c84d387 Keenan Nemetz 16d ago	186	issue, comment, and approval writes; HTTP Git clone remains read-only. There is
6c84d387 Keenan Nemetz 16d ago	187	no password store. Restoring an old `users.db` should be followed by
ff9870f1 Keenan Nemetz 1d ago	188	`vole admin account revoke-sessions --all`. The landing page introduction is
	189	not an exception: it is `profile/README.md` in the instance repository
	190	(`instance_repo`), edited by pushing like every other customization.
474a812d Keenan Nemetz 16d ago	191	- Account names are permanent. They name contributor branches, so an upgrade from
	192	a build where names could change keeps each account's current name and forgets
	193	the former ones.
f920a9f1 Keenan Nemetz 16d ago	194	- Open registration creates accounts without email verification. Leave
	195	`open_registration` off unless the instance is meant to accept public
	196	contributions.
3abbe749 Keenan Nemetz 16d ago	197	- Remote administration uses the same SSH listener and key. Only active instance
3abbe749 Keenan Nemetz 16d ago	198	admins can run it, and every admin command is audited.
520470e1 Keenan Nemetz 16d ago	199	- Private repositories return `404` on the web and over HTTP, so visitors cannot
520470e1 Keenan Nemetz 16d ago	200	confirm that they exist.
4506a4d3 Keenan Nemetz 14d ago	201	- Page-view metrics are daily totals keyed by account, plus one anonymous total.
	202	They store no path, address, referrer, or user agent; the admin-only status
	203	page uses them only to show how much the instance is read.
642e5994 Keenan Nemetz 14d ago	204	- Vole can obtain and renew TLS certificates itself when `https_addr` is set;
	205	otherwise put it behind a proxy that terminates TLS and keeps the `Host`
	206	header, so site routing still works. See
	207	[Configuration § Automatic HTTPS with ACME](configuration.md#automatic-https-with-acme).
ed181212 Keenan Nemetz 13d ago	208
	209	## Production baseline
	210
	211	A public instance is a web server, a Git server and a policy engine in one
	212	process. The settings below are the ones that matter on the day it is exposed;
	213	the linked sections explain each in full.
	214
	215	- Web behind TLS, on loopback. Leave `https_addr` empty, bind
	216	`http_addr = "127.0.0.1:8080"`, and put a proxy in front that terminates TLS
	217	and passes the `Host` header through unchanged. Alternatively set `https_addr`
	218	with `acme_email` and `acme_domains` and let Vole do it. Set `public_url` to
edf9db6e Keenan Nemetz 13d ago	219	the canonical `https://` address so links and passkeys agree with it. When the
	220	request arrives over TLS, or `public_url` is `https://`, Vole sends HSTS on
	221	every response, not only on the pages that use passkeys. See
ed181212 Keenan Nemetz 13d ago	222	[Configuration](configuration.md).
a76b2f38 Keenan Nemetz 12d ago	223	- Compression is already here. Forge pages, hosted sites, the stylesheet,
	224	scripts and source blobs are gzipped when the client asks for it, so a proxy in
	225	front should not compress them again: double compression costs CPU and hides
1b8675b1 Keenan Nemetz 9d ago	226	the `ETag` revalidation that keeps the shared scripts and each Prism grammar to
	227	one download per browser. Archives, LFS objects, Git responses and ranged
	228	downloads are passed through untouched.
25343521 Keenan Nemetz 3d ago	229	- Shared caches. If a caching proxy or CDN sits in front of the instance, it
	230	may keep public pages, static assets, and artifacts from public repositories.
	231	Private pages and downloads are marked `private` or `no-store`, so a cache that
	232	honours those directives will not serve them to an anonymous reader. Do not
	233	strip or rewrite those headers at the proxy.
ed181212 Keenan Nemetz 13d ago	234	- SSH firewalled. The SSH listener is the only write transport and the only
	235	place an account is authenticated. Expose it to the people who push, not to the
	236	internet at large, and leave it on a port other than 22 unless you also run
	237	`sshd` elsewhere.
	238	- No development shortcuts. `dev_only` and `dev_login` are refused together
	239	with a public address, and an instance that needs neither should set neither.
	240	The same goes for `open_registration`, which is off by default.
9e8cc155 Keenan Nemetz 12d ago	241	- Quotas. Every repository is bounded, so one contributor or maintainer cannot
	242	make every page slow or fill the disk: `repo_max_refs` (500), `repo_max_metadata_entries`
	243	(2000) and `repo_max_metadata_bytes` (32 MiB) of documents on the metadata branch
b312e166 Keenan Nemetz 11d ago	244	(issues, requests, comments, approvals, and release records),
9e8cc155 Keenan Nemetz 12d ago	245	`repo_max_bytes` (5 GiB) of Git objects, and `repo_max_lfs_bytes` (10 GiB) of
5ece70a5 Keenan Nemetz 10d ago	246	LFS objects and artifacts. The checks run in pre-receive, so a refused push
9e8cc155 Keenan Nemetz 12d ago	247	moves no refs, and the object bound is enforced on upload. When one bites, the
	248	message names it: raise it in `vole.toml`, delete the branches or metadata that
	249	are not wanted, or run `vole admin gc` to drop unreachable objects. Per-account
e5542ac6 Keenan Nemetz 12d ago	250	browser writes are budgeted for the untrusted (`contribute`) role too
	251	(`contributor_writes_per_hour`, 300); a person discussing never reaches it, and
	252	maintainers and admins are trusted roles that are not budgeted at all.
ed181212 Keenan Nemetz 13d ago	253	- Conservative limits. The defaults are chosen for a small instance:
ed181212 Keenan Nemetz 13d ago	254	`git_http_max_concurrent = 16`, `git_http_max_per_client = 8`,
78cd3437 Keenan Nemetz 13d ago	255	`ssh_max_connections = 64`, `ssh_max_git_processes = 16`,
	256	`ssh_handshake_timeout_seconds = 20`, `max_blob_bytes = 100 MiB`,
	257	`receive_max_bytes = 1 GiB`, and a 600 s budget for one clone. Raise them with
	258	measurements, not with optimism; the read path is anonymous, so a limit that is
	259	too low is a slow clone and a limit that is missing is an outage.
edf9db6e Keenan Nemetz 13d ago	260	- Bounded sign-in. The passkey ceremonies are the only anonymous endpoints
	261	that cost the instance work, so they are rate-limited per client address
	262	(`20` begins a minute), per account name (`10`), and for registrations (`5`),
	263	and the in-flight ceremony map is capped at `4096` entries with a five-minute
	264	expiry. A refused client gets `429` with `Retry-After`. Behind a proxy, set
	265	`git_http_client_ip_header` so the budgets follow the real address. Invite
	266	tokens live in the URL fragment, so they never reach an access log; links in
b7396a11 Keenan Nemetz 6h ago	267	the old `/-/invite/<token>` shape are redirected into it. See
edf9db6e Keenan Nemetz 13d ago	268	[Accounts § Browser sign-in](accounts-and-access.md#browser-sign-in).
78cd3437 Keenan Nemetz 13d ago	269	- Bounded before authentication. The SSH listener refuses connections past
	270	`ssh_max_connections` before the handshake and drops a connection that has not
	271	finished its handshake in `ssh_handshake_timeout_seconds`, so an anonymous
	272	client cannot hold a file descriptor by opening a socket and saying nothing.
	273	After the handshake one connection may hold eight session channels, and the
	274	git commands they run share `ssh_max_git_processes`; an exec line longer than
	275	4 KiB and a runner request line longer than 1 MiB are refused. Address-based
	276	backoff for repeated bad keys is a quota concern, not a transport bound, so it
	277	belongs with the abuse controls rather than here.
ed181212 Keenan Nemetz 13d ago	278	- Audited administration. Remote administration uses the same SSH listener
ed181212 Keenan Nemetz 13d ago	279	and is restricted to instance admins. Ship the audit log off the host: local
b1499c72 Keenan Nemetz 13d ago	280	append-only is not tamper-evident. See [Audit log](#audit-log) and
ed181212 Keenan Nemetz 13d ago	281	[Architecture § Trust model](architecture.md).
	282	- Backups you have restored once. `users.db` is not derived and not
	283	rebuildable; everything else either is, or can be re-pushed. Test the restore,
	284	not the backup. See [Back up an instance](#back-up-an-instance).
	285	- The service unit. The packaged unit drops all capabilities, sets
	286	`UMask=0027`, and hides other processes and devices. If you serve HTTPS or SSH
	287	on a port below 1024, grant `CAP_NET_BIND_SERVICE` with a drop-in rather than
	288	running the service as root. See [Packaging](packaging.md).