vole

Clone

Read only, no credentials:

https://rodent.software/git/vole.git

Write over SSH with a registered key:

git clone ssh://git@0.0.0.0:2222/vole.git

A free and open-source forge that lives in your Git repo.

Publish a robots.txt and sitemap for public pages

80300457bddaf6b90f472a336893ed5dee3c0a48 · Keenan Nemetz · 1d ago · parent c980fe53

Publish a robots.txt and sitemap for public pages

A crawler needs to know which paths are worth fetching and where the public pages are. robots.txt allows the public surface, closes the account, administration, write, fragment and Git paths, and points at a sitemap index. The index splits the instance pages from each public repository; a repository sitemap lists its overview, activity, code browser, issues, changes, releases and default-branch commits, with lastmod on detail pages.

Visibility is the anonymous read rule, not the viewer's session: a private repository is absent from the index and its sitemap is a 404 even for an admin. Account profiles stay out of the sitemap and out of robots.txt so their existing noindex request remains visible to crawlers.

server/routes.go +7

@@ -42,6 +42,12 @@ func (s Server) globalRoutes(mux http.ServeMux) {
	42	42	w := s.Web
	43	43	mux.HandleFunc("GET /{$}", w.IndexPage)
	44	44	mux.HandleFunc("GET /activity", w.InstanceActivity)
		45	// Crawlers discover the public repositories and pages through the sitemap.
		46	// robots.txt names the paths worth fetching and points at the index; the
		47	// per-repository files are below /r/{name}, where their URLs belong.
		48	mux.HandleFunc("GET /robots.txt", w.Robots)
		49	mux.HandleFunc("GET /sitemap.xml", w.SitemapIndex)
		50	mux.HandleFunc("GET /sitemap-pages.xml", w.SitemapPages)
	45	51	// Accounts are public on this instance: the directory lists everybody, and a
	46	52	// profile answers for any name in it. Their activity is not public — every
	47	53	// count on those pages follows the same read rule as the repository it came
@@ -123,6 +129,7 @@ func (s Server) repoRoutes(mux http.ServeMux) {
	123	129	mux.HandleFunc("GET /r/{name}", w.Overview)
	124	130	mux.HandleFunc("GET /r/{name}/counts", w.RepoCounts)
	125	131	mux.HandleFunc("GET /r/{name}/activity", w.Activity)
		132	mux.HandleFunc("GET /r/{name}/sitemap.xml", w.RepoSitemap)
	126	133
	127	134	// Code browsing: one route per view of a ref, with the path as a wildcard
	128	135	// because a branch name may contain "/" and so may a file path.