Operations

A forge you own should be easy to back up and recover. Vole keeps the important state small: an account database, a handful of files, and ordinary Git repositories. These are the tasks that keep an instance healthy.

Audit log

data/audit.log records every privileged action — roster and key changes, access rules, forced lock releases, forced pushes, release rewrites, admin commands, sessions, and ordinary pushes — as one tab-separated line:

timestamp	action	actor	subject	detail

• One shape. Every writer goes through the same code, so a parser can split on tabs. action is a lower-case dotted name (acl.update, push.force, session.create, admin); actor is the account or OS user, - when there is none; subject is the repository, ref, or account the action was about.
• Fields are escaped and truncated. A key comment, an LFS lock path, a repository name, or an admin argument cannot add a column or a line, and a field is capped at 4 KiB.
• Mode 0600, owner only. The file is created that way and tightened on every write, so an instance upgraded from an older version stops being world-readable the first time it writes. Startup refuses a log that is a symlink, a directory, a non-regular file, or owned by another account.
• Privileged changes fail closed. A roster, key, access-rule, session, forced lock release, or admin command that cannot be recorded is refused, and the command says so: the audit log could not be written, so nothing was changed. A push that overrides a lock, an approval, a release, or history is refused unless its override can be written. There is no setting that turns this off, because an override is exactly what an attacker with config access would set.
• Ordinary pushes and web writes are reported, not refused. Those changes are already durable in Git; a failed write appears as audit log write failed in the server log (or on the pusher’s terminal for post-receive) rather than making a working push look rejected.
• A broken log stops the instance. vole serve and vole admin refuse to start when the log cannot be opened for append, which turns a silent, long-lived gap into a startup failure with a path in it.

An instance admin can read the log in the browser at Admin → Audit log. The page shows one record per row, newest first, and pages back through the file. The Admin overview repeats the newest few, so a review can start there. Both are read-only views: they never edit, truncate, or rewrite a line, and a record the reader cannot parse is skipped rather than hidden from the rest of the page.

Ship the file off the host — a read-only mount, a log shipper, or a periodic rsync of an append-only copy — and alert on the absence of expected lines as well as on the presence of unexpected ones. Local append-only storage is not tamper-evident: anyone who can write data/ can edit it, so keep an off-host copy that the service account cannot reach.

Back up an instance

Back up the instance state:

vole admin account backup backup/users.db       # accounts, keys, sessions, rules, locks, views
sqlite3 data/runs.db ".backup 'backup/runs.db'" # CI history, queue, and approvals
cp -a data/logs backup/                        # CI attempt logs
cp data/ssh_host_ed25519_key backup/           # the SSH host key
sqlite3 data/secrets.db ".backup 'backup/secrets.db'"  # sealed repository secrets
cp data/secrets.key backup/                    # the key those secrets need
cp -a data/acme backup/ 2>/dev/null            # TLS account key and certificates, if ACME is on
cp data/audit.log backup/ 2>/dev/null          # history of admin actions

vole admin account backup takes a consistent copy, even while Vole is running. Copying users.db by hand can miss recent changes that have not been written to the main database file yet. Do the same for runs.db with SQLite’s backup command; copying it by hand while the server is writing can miss WAL pages.

Keep data/secrets.key apart from the copies of data/. A backup that holds both the database and its key is a backup of plaintext: the encryption protects a copy that loses the key, not one that keeps it. Restoring data/secrets.db beside a newly generated key reads nothing, and Vole says so rather than handing a job an empty value.

If you lose the account database, you need to recreate accounts and re-enroll their passkeys. You do not lose repositories. The backup contains account metadata, public keys, and hashes of session and enrollment tokens; it does not contain a usable password or a plaintext session token. Treat it as private anyway.

Repositories are ordinary Git repositories under data/repos/. You can back them up by copying or cloning. Issues, change requests, and releases travel with them because they are stored in Git. CI history does not: it lives in data/runs.db and must be backed up with it.

Restore an instance

1. Restore data/users.db, data/runs.db, data/logs/, data/ssh_host_ed25519_key, data/audit.log, data/secrets.db with the data/secrets.key that sealed it, and data/acme/ if the instance obtains its own TLS certificates.
2. Restore the repositories under data/repos/ and the object store under data/lfs/ if you keep CI artifacts.
3. Run vole admin index rebuild.
4. Run vole admin account revoke-sessions --all so sessions from the restored copy cannot outlive their cookies.
5. Start vole serve.

Release notes for an upgrade: an older users.db is upgraded in place when it is opened. Accounts used to have an internal id and an optional display label; both are gone, and every key, passkey, enrollment link, session and admin grant is rebuilt around the registered name. Access-rule entries and file locks in that database are rewritten through the same mapping, and the retired user_names alias table is dropped. Take a backup first if you might roll the binary back. Approval files live in Git and are not rewritten; an approval written under an old account id stops matching, so ask for a fresh approval.

Contributor branches moved from refs/heads/contrib/<account-id>/ to refs/heads/contrib/<name>/. Old refs stay in the repository and can still be fetched, but only a maintainer can push them; existing change requests still name the same head commit. A contributor who wants to keep working pushes the same commit under their new folder and opens a request from it.

Large files and build artifacts live in data/lfs/; CI logs live in data/logs/. Back both directories up too if you need those files. vole admin gc treats files reachable from a repository or a published artifact as things to keep; logs are ordinary files and are not swept.

Restart the server

A clean stop (systemctl restart, or Ctrl-C on a development instance) stops handing out new CI jobs and waits up to a minute for the jobs already leased to report their result. A job that finishes in that window is stored and is not run again. A job still running when the window ends loses its runner connection and is requeued as another attempt once its lease expires. The wait is best effort; start long builds well before a planned restart.

Clean up stored files

vole admin gc --dry-run          # show what would be removed
vole admin gc                    # remove unused files
vole admin gc --grace 24h        # keep unused files for a day

vole admin gc cleans the shared storage used by LFS and build artifacts. CI logs are ordinary files under data/logs/ and are not part of the sweep. It also repairs each repository’s links into that storage, so it helps when files are missing or links are broken.

The default grace period is one hour. A recent upload that has not been pushed yet will not be removed.

You can run vole admin gc while people are pushing. There is no need to schedule it; storage only grows when someone pushes files.

Rebuild indexes

data/index.db is generated by Vole and can be deleted safely:

vole admin index rebuild

data/runs.db holds the CI queue, finished-run records, and contributor-branch CI approvals. It is a source of truth, not a rebuildable index: deleting it loses CI history as well as attempts that were in progress. Back it up with the rest of the instance. vole admin index rebuild rebuilds only the repository and issue index; it cannot restore CI history.

Security notes

• Two long-term secrets are stored: the SSH host key, and the key that seals repository secrets. Keep both readable only by the Vole user (0600). Browser session and enrollment tokens are stored as hashes.
• Repository secrets are encrypted at rest, which protects a copy of data/ that leaves the host. It does not protect a host somebody already owns: file modes are the boundary, and the default key location is beside the database. Anything that can read the data directory can read the values.
• Vole creates its own secrets at 0600 and its data directory at 0700: the roster (users.db), the audit log, the SSH host key and the ACME account directory. It also tightens an existing instance on startup, and logs the paths whose mode it changed, so an upgrade from an earlier version cannot leave a private key world-readable. If a mode cannot be changed, startup fails rather than serving keys it cannot protect. A site directory that is deliberately shared with a backup group is fine at 0750: group access is left alone, world access is not.
• Nothing Vole writes is signed: not pushes, not approvals, not CI results. Vole relies on the SSH key at the connection, the checks before a push is accepted, and the audit log.
• The web interface is read-only for anonymous visitors. Passkey sign-in enables issue, comment, and approval writes; HTTP Git clone remains read-only. There is no password store. Restoring an old users.db should be followed by vole admin account revoke-sessions --all. The landing page introduction is not an exception: it is profile/README.md in the instance repository (instance_repo), edited by pushing like every other customization.
• Account names are permanent. They name contributor branches, so an upgrade from a build where names could change keeps each account’s current name and forgets the former ones.
• Open registration creates accounts without email verification. Leave open_registration off unless the instance is meant to accept public contributions.
• Remote administration uses the same SSH listener and key. Only active instance admins can run it, and every admin command is audited.
• Private repositories return 404 on the web and over HTTP, so visitors cannot confirm that they exist.
• Page-view metrics are daily totals keyed by account, plus one anonymous total. They store no path, address, referrer, or user agent; the admin-only status page uses them only to show how much the instance is read.
• Vole can obtain and renew TLS certificates itself when https_addr is set; otherwise put it behind a proxy that terminates TLS and keeps the Host header, so site routing still works. See Configuration § Automatic HTTPS with ACME.

Production baseline

A public instance is a web server, a Git server and a policy engine in one process. The settings below are the ones that matter on the day it is exposed; the linked sections explain each in full.

• Web behind TLS, on loopback. Leave https_addr empty, bind http_addr = "127.0.0.1:8080", and put a proxy in front that terminates TLS and passes the Host header through unchanged. Alternatively set https_addr with acme_email and acme_domains and let Vole do it. Set public_url to the canonical https:// address so links and passkeys agree with it. When the request arrives over TLS, or public_url is https://, Vole sends HSTS on every response, not only on the pages that use passkeys. See Configuration.
• Compression is already here. Forge pages, hosted sites, the stylesheet, scripts and source blobs are gzipped when the client asks for it, so a proxy in front should not compress them again: double compression costs CPU and hides the ETag revalidation that keeps the shared scripts and each Prism grammar to one download per browser. Archives, LFS objects, Git responses and ranged downloads are passed through untouched.
• Shared caches. If a caching proxy or CDN sits in front of the instance, it may keep public pages, static assets, and artifacts from public repositories. Private pages and downloads are marked private or no-store, so a cache that honours those directives will not serve them to an anonymous reader. Do not strip or rewrite those headers at the proxy.
• SSH firewalled. The SSH listener is the only write transport and the only place an account is authenticated. Expose it to the people who push, not to the internet at large, and leave it on a port other than 22 unless you also run sshd elsewhere.
• No development shortcuts. dev_only and dev_login are refused together with a public address, and an instance that needs neither should set neither. The same goes for open_registration, which is off by default.
• Quotas. Every repository is bounded, so one contributor or maintainer cannot make every page slow or fill the disk: repo_max_refs (500), repo_max_metadata_entries (2000) and repo_max_metadata_bytes (32 MiB) of documents on the metadata branch (issues, requests, comments, approvals, and release records), repo_max_bytes (5 GiB) of Git objects, and repo_max_lfs_bytes (10 GiB) of LFS objects and artifacts. The checks run in pre-receive, so a refused push moves no refs, and the object bound is enforced on upload. When one bites, the message names it: raise it in vole.toml, delete the branches or metadata that are not wanted, or run vole admin gc to drop unreachable objects. Per-account browser writes are budgeted for the untrusted (contribute) role too (contributor_writes_per_hour, 300); a person discussing never reaches it, and maintainers and admins are trusted roles that are not budgeted at all.
• Conservative limits. The defaults are chosen for a small instance: git_http_max_concurrent = 16, git_http_max_per_client = 8, ssh_max_connections = 64, ssh_max_git_processes = 16, ssh_handshake_timeout_seconds = 20, max_blob_bytes = 100 MiB, receive_max_bytes = 1 GiB, and a 600 s budget for one clone. Raise them with measurements, not with optimism; the read path is anonymous, so a limit that is too low is a slow clone and a limit that is missing is an outage.
• Bounded sign-in. The passkey ceremonies are the only anonymous endpoints that cost the instance work, so they are rate-limited per client address (20 begins a minute), per account name (10), and for registrations (5), and the in-flight ceremony map is capped at 4096 entries with a five-minute expiry. A refused client gets 429 with Retry-After. Behind a proxy, set git_http_client_ip_header so the budgets follow the real address. Invite tokens live in the URL fragment, so they never reach an access log; links in the old /-/invite/<token> shape are redirected into it. See Accounts § Browser sign-in.
• Bounded before authentication. The SSH listener refuses connections past ssh_max_connections before the handshake and drops a connection that has not finished its handshake in ssh_handshake_timeout_seconds, so an anonymous client cannot hold a file descriptor by opening a socket and saying nothing. After the handshake one connection may hold eight session channels, and the git commands they run share ssh_max_git_processes; an exec line longer than 4 KiB and a runner request line longer than 1 MiB are refused. Address-based backoff for repeated bad keys is a quota concern, not a transport bound, so it belongs with the abuse controls rather than here.
• Audited administration. Remote administration uses the same SSH listener and is restricted to instance admins. Ship the audit log off the host: local append-only is not tamper-evident. See Audit log and Architecture § Trust model.
• Backups you have restored once. users.db is not derived and not rebuildable; everything else either is, or can be re-pushed. Test the restore, not the backup. See Back up an instance.
• The service unit. The packaged unit drops all capabilities, sets UMask=0027, and hides other processes and devices. If you serve HTTPS or SSH on a port below 1024, grant CAP_NET_BIND_SERVICE with a drop-in rather than running the service as root. See Packaging.