vole

Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Stream run logs over the runner's SSH channel

5ece70a50ebda392626b92bdf29d28bc9b444954 · Keenan Nemetz · 10d ago · parent fdf2340f

Stream run logs over the runner's SSH channel

The runner wrote the log to a temp file and uploaded it as one LFS object
when the job ended, so a running job had no log to read and a log was stored
like a shared artifact even though it belongs to one attempt.

The runner sends 128 KiB chunks over the same runner channel that holds the
lease. The server appends them to data/logs/<repo>/<run>/<attempt>.log and
records the size while the run is still running. The finish record carries
the final size; the server checks the file length, syncs it, and stores the
result. A partial or failed stream leaves no log on the record.

Logs are no longer LFS objects: they are not shared, not content-addressed,
and the web serves the file directly, with range and step slices. GC no
longer treats them as LFS roots.

docs/runner.md +16−15

@@ -86,9 +86,9 @@ instance or kept on the runner, so each of those channels has a bound:
	86	86
	87	87	- Logs. One attempt's log is capped at 32 MiB. Past that the runner keeps the
	88	88	first 32 MiB and appends a line saying it was truncated, so a build that loops
	89		printing cannot fill the runner's disk or the instance's object store. The log
	90		is written straight to a file on the runner and uploaded once at the end; it is
	91		not held in memory and nothing is streamed to the server.
		89	printing cannot fill the runner's disk or the instance's log directory. The log
		90	is written straight to a file on the runner and streamed to the server in chunks
		91	while the job runs; it is not held in memory.
	92	92	- Artifacts. A workflow declares at most 32 patterns, and a run may publish at
	93	93	most 256 files totalling 4 GiB. The expanded set is checked before anything is
	94	94	uploaded; a pattern that matches a directory tree is the usual way to hit this.
@@ -247,24 +247,25 @@ Admins always count. For everyone else, a maintainer running CI on their laptop
	247	247	not turn a required check green. The result is still recorded, and the run page
	248	248	says whether it counts.
	249	249
	250		When a job finishes, the runner uploads its log and sends the attempt's record with
	251		its finish message on the same SSH channel that leased the job. Vole checks that the
	252		record names the account holding the lease, that it describes the job the queue
	253		handed out, and that every log and artifact pointer exists in the store. It then
	254		stores the attempt and the run's result in `data/runs.db` in one transaction. A
	255		tagged build that publishes release artifacts also pushes those pointers to the
	256		`vole` branch after the record is stored.
		250	While a job runs, the runner streams its log to the server in chunks over the same
		251	SSH channel that leased it. The server writes one file under `data/logs/` and the
		252	run page can open it before the job finishes. When the job ends, the runner sends
		253	the attempt's record with its finish message; Vole checks that the record names the
		254	account holding the lease, that it describes the job the queue handed out, and that
		255	every artifact pointer exists in the store. It then stores the attempt and the run's
		256	result in `data/runs.db` in one transaction. A tagged build that publishes release
		257	artifacts also pushes those pointers to the `vole` branch after the record is
		258	stored.
	257	259
	258	260	The runner did the work, so the runner reports the result. The SSH connection
	259	261	identifies the runner, and `data/audit.log` records the decision.
	260	262
	261	263	## Logs
	262	264
	263		The runner uploads the log before reporting the exit code. Vole does not stream
	264		live output while a step runs: a log assembled by the server would not be the
	265		runner's own record. Once the attempt finishes, the archived object is served
	266		directly and supports range requests, so the page does not read a large log into
	267		memory.
		265	The runner streams the log while the steps run. The server appends each chunk to
		266	the attempt's file, and the run page serves that file directly with range requests,
		267	so it does not read a large log into memory. The file is complete when the attempt's
		268	record is stored.
	268	269
	269	270	The stored attempt also names each step's byte range in that one log, so the run
	270	271	page can show a step's output without downloading the image build and other steps.