vole

Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

Stream run logs over the runner's SSH channel

5ece70a50ebda392626b92bdf29d28bc9b444954 · Keenan Nemetz · 11d ago · parent fdf2340f

Stream run logs over the runner's SSH channel

The runner wrote the log to a temp file and uploaded it as one LFS object
when the job ended, so a running job had no log to read and a log was stored
like a shared artifact even though it belongs to one attempt.

The runner sends 128 KiB chunks over the same runner channel that holds the
lease. The server appends them to data/logs/<repo>/<run>/<attempt>.log and
records the size while the run is still running. The finish record carries
the final size; the server checks the file length, syncs it, and stores the
result. A partial or failed stream leaves no log on the record.

Logs are no longer LFS objects: they are not shared, not content-addressed,
and the web serves the file directly, with range and step slices. GC no
longer treats them as LFS roots.

ci/agent/spool.go +8−10

@@ -5,18 +5,15 @@ package agent
	5	5	// It used to be a strings.Builder: the whole attempt in memory, then written to a
	6	6	// temp file, then read twice by the LFS transfer (once to hash, once to send). A
	7	7	// job that prints for an hour cost its runner an hour of RAM. The log is now a
	8		// bounded file on disk from the first byte: the step's output goes straight in,
	9		// the file is what gets uploaded, and the cap is where it is written rather than
	10		// where it is remembered.
	11		//
	12		// The log is still uploaded once, at the end, as an ordinary LFS object. Nothing
	13		// is streamed to the server: the content-addressed id cannot be known until the
	14		// last byte, so the only thing streaming would buy is a second protocol.
	15
		8	// bounded file on disk from the first byte: the step's output goes straight in, the
		9	// streamer reads the file as it grows, and the cap is where it is written rather
		10	// than where it is remembered.
	16	11	import (
	17	12	"fmt"
	18	13	"os"
	19	14	"sync"
		15
		16	"github.com/rodent-software/vole/ci/logs"
	20	17	)
	21	18
	22	19	const (
@@ -24,7 +21,7 @@ const (
	24	21	// hour of verbose build output is a few megabytes, and a job that prints more
	25	22	// than this is either looping or trying to fill the runner's disk. The cap is
	26	23	// large enough to hold the useful part of a very noisy build.
	27		MaxLogBytes int64 = 32 << 20
		24	MaxLogBytes = logs.MaxBytes
	28	25	// truncationNote is appended to a capped log so a reader is told why it ends
	29	26	// without the tool's summary line.
	30	27	truncationNote = "\n[log truncated: the attempt produced more than %s of output]\n"
@@ -97,7 +94,8 @@ func (s *spool) Len() int64 {
	97	94	return s.n
	98	95	}
	99	96
	100		// Path is the file to upload. seal must have run first.
		97	// Path is the file the streamer reads and Close removes. It is valid from the
		98	// moment newSpool created it.
	101	99	func (s *spool) Path() string {
	102	100	s.mu.Lock()
	103	101	defer s.mu.Unlock()