Sign in

Search issue titles and bodies, change requests and comments. Private repositories you cannot read are not included.
vole
Clone

Read only, no credentials:

https://rodent.software/vole.git

Write over SSH with a registered key:

git clone ssh://git@rodent.software:2222/vole.git

A free and open-source forge that lives in your Git repo.

#232

Publish robots.txt and a sitemap for public pages

merged · by keenan 1d ago · merged into main at 8916b12f by keenan 1d ago

keenan opened this 1d ago

Why

Search engines currently get no crawl guidance from the instance. They can spend requests on sign-in, settings, fragments and Git URLs, and they have no way to discover public issue, change and release pages as repositories grow.

What

  • GET /robots.txt allows the public surface, closes the account, administration, write, fragment and Git paths, and points at the sitemap index.
  • GET /sitemap.xml is a sitemap index over the instance pages and each public repository.
  • GET /sitemap-pages.xml lists the landing page and the instance activity feed.
  • GET /r/{name}/sitemap.xml lists the repository overview, activity, code browser, issue list and every issue, change list and every change request, release list and every tag, and commits on the default branch. Detail pages carry lastmod.

Privacy

Every URL follows the anonymous read rule, never the viewer’s session. A private repository is absent from the index, and /r/{private}/sitemap.xml is a 404 even for an admin. Account profiles stay out of the sitemap and out of robots.txt: they already carry <meta name="robots" content="noindex">, and a Disallow would hide that request from the crawler.

Checks

  • vole workflow run verify passes: format, vet, build, licenses, govulncheck, packages, deployment files, markdown links, and the race test suite.
  • New coverage in server/seo_test.go for the robots rules, the public/private split, the instance pages, and a repository sitemap containing an issue, a change request and a release.

Discussion (0)

No discussion yet.