Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#84

Stream run logs over the SSH channel

merged · by keenan 10d ago · merged into main at 257e0d81 by keenan 10d ago

keenan opened this 10d ago

Why

A finished attempt’s log was written to the runner’s temp file and uploaded once, at the end, as an LFS object. That made a running job’s log invisible, made the log look like a shared content-addressed artifact when it belongs to exactly one attempt, and put the bytes in the wrong store. Now that CI history is out of Git, logs can be the instance’s own files.

What changed

  • The runner streams the log over the same vole-runner SSH channel that holds the lease, in 128 KiB chunks. The chunk carries its attempt and byte offset; the server refuses a gap or an oversized log.
  • The server appends each chunk to data/logs/<repo>/<run>/<attempt>.log and updates the attempt’s size while the job runs. The run page can open the file before the job finishes.
  • finish carries the final log size. The server checks the file is exactly that long, syncs it, and stores the result. A failed or partial stream is reported as no log: the runner drops the step ranges and the server removes the partial file.
  • The run page serves the file directly, with Range support and the existing per-step byte slices. The status page no longer classifies logs as LFS usage, and GC no longer treats them as LFS roots.
  • data/logs/ is created and hardened with the rest of the instance; backup and restore docs include it.

Tests

  • ci/logs: append, read, size, remove, offset gap, cap, path validation.
  • ci/runs: per-attempt log sizes and lease rules.
  • server integration: a real runner streams a log, the file is under data/logs, the served log matches the file, the stored attempt names the size, and step slices work.
  • Existing runner, SSH, web, and status tests updated.

Checks

  • go test -count=1 ./...
  • go test -race ./ci/logs ./ci/runs ./ci/agent ./server/ssh ./server/web
  • go vet ./...
  • vole workflow check
  • vole workflow run verify

Stacked change

This is based on ci-records-in-db (#83), not main, because it uses the database-only record path. Retarget it to main after #83 lands.

Deployment

No compatibility code ships. An existing local instance needs a one-time migration before this binary starts: copy each old run_logs.oid from data/lfs/repos/<repo>/<oid> to data/logs/<repo>/<run>/<attempt>.log, then rebuild run_logs with the new (run_id, attempt, size, created, updated) shape. Fresh instances need nothing.

Discussion (3)

keenan 10d ago

Rebased onto main after #83 landed; the request is now clean against main at 5ece70a. The migration note still applies: this build assumes run_logs is the new (run_id, attempt, size, created, updated) table and that logs are files under data/logs/. There is no compatibility path for LFS logs.

keenan 10d ago

Live instance migrated and deployed.

  • Copied 128 existing logs from data/lfs/repos/... to data/logs/<repo>/<run>/<attempt>.log (1.1 MiB), rebuilt run_logs with the new (run_id, attempt, size, created, updated) shape, and kept the old LFS objects for now (they are no longer roots).
  • Installed 7c94596, restarted the server and runner. Backups: runs-pre-logs-20261029-170032.db, runs-post-logs-20261029-170333.db, and logs-post-*.tar.gz under /home/keenan/vole-backups.
  • A real verify run (01M651PVA9149WK0J7RK81TWMK) streamed attempt 2 live: the running page had the job-output viewer, the log grew while the job ran, and the finished log was marked X-Log-Final: 1. Step accordions served their slices after finish.
  • Old run logs, including 01M64YZHENGAFTVB242T71JNCJ, now serve from data/logs with the final header. vole admin index rebuild leaves runs and run_logs intact.
  • The steps table is gone; the run page is one logs card with a job-output accordion and one accordion per step. Running steps say waiting rather than claiming to be the active one, because the runner does not report step boundaries until the record arrives.
keenan 10d ago

Final head 257e0d8; live instance deployed.

  • Run page is now one logs card: a job-output accordion plus one accordion per step. The steps table is gone.
  • The job output polls /log?attempt=N&after=<offset> once a second while the run is open and running, appends text nodes, follows only while the log is not final, and switches its state label to complete on X-Log-Final: 1. Step accordions load their slices once, on first open.
  • Running steps say waiting, not running: the runner does not report step boundaries until the record arrives. They become exact on refresh after finish; the job output is the live signal.
  • The old byte-count bar was removed; attempt chips appear only when there is more than one attempt.
  • Migration/deploy: 128 old LFS logs copied to data/logs, run_logs rebuilt, 257e0d8 installed, services restarted. The live verify run 01M651PVA9149WK0J7RK81TWMK streamed attempt 2 and finished with X-Log-Final: 1; old logs serve from files.
  • Backups: runs-pre-logs-20261029-170032.db, runs-post-logs-20261029-170333.db, logs-post-20261029-170333.tar.gz.

The endpoint and partial serving are covered by TestRunLogStreamsWhileTheJobRuns; the browser polling itself is syntax-checked but not driven by a headless browser test.