Repo Renderer

# Repo Renderer

Repo Renderer takes a git repo and renders it into a purely static website so
you can host your code on the web without relying on someone else's git forge
(GitHub, Codeberg, Bitbucket, etc.) and without having to host your own code
forge (Gitea, Forgejo), which are significantly more resource-intensive than a
static site. If you configure Repo Renderer to re-render your repo as a git
hook, you can also host your own git backend that keeps the static web view up
to date by incrementally re-rendering files in your repo only when they change.

[![A repository page in the dark theme: the file drawer, language stats, file listing and README](docs/screenshots/repo.png)](docs/screenshots/repo.png)

| Syntax-highlighted files | Search, in the browser | The repository index |
|---|---|---|
| [![A Go source file with syntax highlighting](docs/screenshots/file.png)](docs/screenshots/file.png) | [![Search results for "RenderRepo"](docs/screenshots/search.png)](docs/screenshots/search.png) | [![The repository index, grouped into sections](docs/screenshots/index.png)](docs/screenshots/index.png) |


## Manifesto

The main reason I made this project is to allow me to keep hosting my own code
on my own website in an era when GitHub is increasingly enshittified by
Microsoft and badly behaved AI company web crawlers are increasingly hammering
sites like mine into the ground. In my personal experience, traffic from
Microsoft and Amazon has been especially badly behaved on this front.

I do *not* subscribe to the view that websites are only for humans or that
humans should need to prove their humanity through CAPTCHAs in order to use the
web. In the words of Michael Bolton:

> No way! Why should I \[do CAPTCHAs\]? He's the one who sucks.

I stand in favor of respectful web users, whether they have Javascript disabled,
whether they are using Tor, whether they are CURLing a site to view offline, or
whether they are writing a properly rate-limited Python script to crawl a site.
With that in mind, this project is an effort to make sure that even a
minimum-spec VPS (1 vCPU / 1GB RAM) is capable of hosting all your code without
forcing users to jump through hoops and without breaking a sweat as
badly-behaved crawlers hammer the site. And hopefully it offers a user
experience that is comparable or superior to sites like GitHub or Codeberg for
the casual code viewer.

## ⚠️ SLOP ALERT ⚠️

This project is heavily LLM-authored. I've spent a fair amount of time and
effort refining the design and looking over the code, but if LLMs are a
dealbreaker for you, this project probably isn't something you'll want to use.
I think it's appropriate to use an LLM to mitigate the harmful impact of LLM
scrapers on my personal site and to reduce my dependency on GitHub and
Microsoft's attempts to enclose the software commons. But that's just my
opinion and I can respect people who don't feel the same way.

## Overview

This is intended to be used for ultra-lightweight servers that may end up
getting absolutely hammered by abusive web crawlers. This project is built with
OpenBSD's `httpd(8)` in mind, which is a high-performance, lightweight,
high-security static HTTP server. Each page is rendered at most once per git
commit, and served statically to every visitor, with no server rendering taking
place on page views.

The renderer only shows the latest commit of each repository's default branch,
plus any other branches you put in your config. In order to save on resources,
the renderer does not show the full git history of the repository. Users who
care about the full git history can clone the repository and use `git` or their
favorite IDE to explore the history.

## Features

- **Repository index:** A listing of all your repositories, grouped into
  categories according to your config.
- **Repository overview:** the repository overview shows the file listing,
  README, and stats.
- **Tags:** Label repositories with tags like `lua` or `language` in the
  config. They show on the index and on each repository's page, for
  information only.
- **Lines of code counts:** Each repo shows a breakdown of lines of code,
  words of documentation, and bytes of assets.
- **Source code:** each file is viewable as statically rendered,
  syntax-highlighted pages. No javascript is required.
- **Custom language syntax highlighting:** A very wide number of languages
  are supported out of the box, but it's also possible to extend the renderer
  using syntax highlighting for custom programming language syntax.
- **Markdown rendering:** Markdown files are nicely rendered to make it
  easy to read your docs, with a separate tab to view the syntax-highlighted
  Markdown source code. Fenced code blocks get full syntax highlighting in both
  rendered Markdown and Markdown source code. Images and links also work
  properly.
- **File tree:** Each page shows a collapsible file drawer for quick and
  easy navigation of the files in the repo.
- **Source archive downloads:** Download the whole repo as a .tar.gz file,
  one archive for each tag and one for the latest code.
- **Super fast client-side search:** Seriously, you can just load the
  code into memory and do a string search to find stuff. It's orders of
  magnitude faster than you get with GitHub and you can use regexes.
- **Branch browsing.** You can list extra branches to render beside the
  default one (see [Browsing other branches](#browsing-other-branches)).
- **A sitemap:** Search engines get every page of each repository's
  default branch, dated by the last commit that changed it. Other
  branches, and repositories hidden from the index, are left out.
- **Private repositories:** A repository in no section of the index is
  private by default. With `private_htdocs` set, it's rendered to a
  private site at `/private/`, behind a password; without it, it isn't
  rendered at all (see [Private repositories](#private-repositories)).
- **A 404 page:** A missing address gets a page in the site's own style
  that links back to the index.
- **Light and dark themes:** The site follows your system setting, and a
  button lets you switch and remembers your choice. You can replace the
  built-in favicon in the config.
- **Incremental rendering:** A push re-renders only the pages that changed.
  It re-renders everything when the templates or config change, when someone
  rewrites history, when the set of rendered branches changes, when the
  repository's description changes, or when the last render failed or was
  cut short. That last case sweeps away whatever
  pages a failed render had already written, so reverting a bad commit, or
  force-pushing back to before it, leaves nothing of it on the site. The file
  drawer and the Download menu's list are documents of their own that every
  page shows in a frame, so adding a file rewrites only the drawer. The
  list names the latest commit, so every push rewrites it, and pushing a
  tag rewrites only that list.
- **Mirroring:** The post-receive hook can push each repository on to
  GitHub, Codeberg or anywhere else `git push` reaches (see
  [docs/openbsd.md](docs/openbsd.md#7-mirror-repositories-to-github-or-elsewhere-optional)).
- **A small security surface:** httpd serves the pages with a strict Content
  Security Policy (`default-src 'none'`), so the only scripts that run are
  the two the site serves itself, and no other site can show the pages in
  a frame (`frame-ancestors 'self'`). Raw files keep their real names and are
  served with a `sandbox` policy, which stops them from running as part of
  the site. The renderer validates every path and tag
  name before it writes anything, and it restricts its own filesystem access
  with `unveil` on OpenBSD.

Every page loads one small script for progressive enhancement:
[ui.js](internal/render/static/ui.js), which adds the theme toggle, a "Copy"
button for the clone command, Escape and click-away handling for the Download
menu, and fits the menu's frame to its contents. Without JavaScript, the site
is plain HTML: those controls disappear and the download list scrolls in a
fixed-height frame.

Code search also uses javascript, but because the site is purely static, there
is no fallback for browsers that don't support javascript (no server-side
search).

## The pieces

The renderer is one program that reads repositories and a config and
writes a site. Everything else is a small tool around it, which you can
use or leave out:

| Piece | Runs on | What it does |
|---|---|---|
| `repo-render` | the server | The renderer. It never touches the web server, user accounts or mirrors. |
| `contrib/post-receive` | the server, in each repository | The push hook: queues a render, and pushes on to the mirrors. |
| `contrib/new-repo` | the server | Creates or rewires a bare repository, linked to the hook. |
| [examples/httpd.conf](examples/httpd.conf) | the server | An OpenBSD httpd server block for the site, with its security headers. |
| `repo-render-preview` | your machine | A plain file server for looking at a rendered site locally. |
| `tools/create-repo` | your machine | Makes a working copy's repository on the server (and GitHub), adds the remote, pushes. |
| `tools/migrate-from-github` | your machine | Brings your GitHub repositories over to the server. |

[docs/openbsd.md](docs/openbsd.md) sets up a server with all of it.

## Building

On your machine:

```sh
make              # repo-render and repo-render-preview, for this machine
make preview      # render some repositories and serve them locally (below)
make check        # go vet, gofmt, shellcheck, the Go tests and the node tests
make fuzz         # fuzz the checks on pushed names and paths, 30s each (FUZZTIME)
make screenshots  # retake the README's screenshots of the preview site
make server       # dist/repo-render-openbsd-amd64.tar.gz, everything the server needs
```

You need Go 1.26 and `git`; `node` and `shellcheck` for `make check`;
`node` and `firefox` for `make screenshots`. `make server` builds for
OpenBSD on amd64; set `SERVER_OS` and `SERVER_ARCH` for another server.
The tarball holds `repo-render`, `new-repo`, the hook, the example
configs and an `install.sh` that puts them under `/usr/local`, so the
server doesn't need Go.

## What the renderer writes

The renderer reads the bare repositories in `git_dir` (as `<repo>.git`),
and writes:

```
<htdocs>/                        the website
    index.html                   repository index
    err/404.html, err/403.html   pages for a missing address and an empty
                                 directory, for the web server's error pages
    robots.txt                   keeps crawlers to the pages, off tarballs, raw files
                                 and the clone export, asks them to wait 10 seconds
                                 between fetches, and names the sitemap
    sitemap.xml                  sitemap index: one sitemap per repository on the index
    sitemap/<repo>.xml           every page of the default branch, dated by its last change
    ads.txt                      the spec's placeholder record: no one may sell ads
                                 on the site
    code/<repo>.html             a repository's root page
    code/<repo>/<path>.html      its files and directories
    code/<repo>@<branch>/…       extra branches (more below)
    raw/<repo>/<path>            raw files, byte for byte
    svg/<repo>/<path>.svg        SVGs prepared for embedding, with the theme
                                 colour applied
    snapshot/<repo>/<repo>@<commit>.tar.gz
                                 tarball of the latest commit (the Download menu's
                                 "latest"), named by the first 12 digits of its hash
    archive/<repo>/<repo>-<tag>.tar.gz    tarball of each tag
    iframes/<repo>/filetree.html the file drawer, one document framed by every page
    iframes/<repo>/downloads.html
                                 the download list: latest commit and tags, framed by
                                 the Download menu
    types/<repo>/<row>/<type>.html
                                 the files of one type, linked from the root
                                 page (row: code, docs or binary)
    search/<repo>/index.html     search page, at /search/<repo>/
    search/<repo>/corpus.json    the text the search page searches; the page adds
                                 ?v=<hash of it>, so it can be cached for good
    static/                      style.css, chroma.css, ui.js, search.js, favicon;
                                 pages (and style.css's icons) add ?v=<hash> to
                                 these, so an upgrade never pairs a page with a
                                 cached old copy, and they can be cached for good
    git/<repo>.git/              read-only clone export, served at /git/
<private_htdocs>/private/…       optional: the same layout for private repositories,
                                 to serve at /private/ behind a password; no sitemap,
                                 robots.txt, error pages or static/ of its own
<spool_dir>/queue                job queue: the hook adds to it, the renderer drains it
<state_dir>/                     render state and lock
```

Most files also get a `.gz` twin, for a server that serves precompressed
files. A directory has a page of its own, just like a file does:
`/code/foo/src` is the file `code/foo/src.html`. That way, a file named
`index` can live anywhere in a repository without conflicting with
anything. A web server that serves the site must map each page's URL to
its `.html` file; [examples/httpd.conf](examples/httpd.conf) does, and
so does `repo-render-preview`.

## Configuration

Give the config file with `--config FILE`, or set `REPO_RENDER_CONFIG`.
There is no default. `host` is required, and so are the four paths: where
the repositories are, where the site goes, and the renderer's queue and
state. A relative path is relative to the config file. This example shows
the settings you are most likely to change:

```yaml
host: code.example.com
git_dir: /var/git
htdocs: /var/www/htdocs/repo-render
spool_dir: /var/spool/repo-render
state_dir: /var/db/repo-render

site:
  title: Bruce's code
  tagline: Things I made
  footer: >-                   # optional HTML at the bottom of every page
    Built with <a href="/code/repo-renderer">Repo Renderer</a>.
  clone_url: "https://code.example.com/git/{repo}.git"
  unlisted: show               # repositories in no section go in a trailing section (default: private)
  unlisted_title: Other projects

sections:
  - title: Command line programs
    repos: [bp, bb, btui]
  - title: "[Tomo](https://tomo.example.com)"   # title and description are markdown
    description: The [Tomo](https://tomo.example.com) programming language and related projects.
    repos: [tomo, tomo-json]

lexers:
  tomo:                        # uses tomo.xml from lexer_dirs, or a chroma builtin
    globs: ["*.tm"]
    aliases: [tomo, tm]        # names you can use in ```fences

repos:
  tomo:
    branches: [dev]            # also render the dev branch
    tags: [language, tomo]     # shown on the index and the repository's page
```

The config is YAML. Write prose and HTML as a folded block (`>-`, then
indented lines): the lines are joined with spaces, and nothing inside
needs quoting or escaping. Quote short values that start with `*`, `{`
or `[`, like the globs and `clone_url` above.

A few more settings are worth knowing about:

- `style` and `style_light` choose the syntax highlighting colours for dark
  and light mode.
- `site.favicon` replaces the built-in icon. Point it at an `.svg`, `.png`,
  or `.ico` file.
- `log` is `syslog`, a file, or `""` (the default) for stderr only. The
  hook starts the renderer with no stderr, so set it on a server.
- `lexer_dirs` lists directories of your own `<name>.xml` syntax
  highlighters, searched in order.
- `git` is the git program, if it is not on the `PATH` the renderer runs
  with (cron's, say).
- `site.unlisted` decides what happens to a repository that is in no
  section. `private`, the default, keeps it off the public site: it goes
  on the private site, or nowhere when there is none (see
  [Private repositories](#private-repositories)), and whatever an earlier
  render left on the public site is removed on its next push or `render`.
  `show` lists it in a trailing
  section titled `site.unlisted_title`. `hide` renders it but leaves it off
  the index, so anyone with the URL can still read it. `repo-render check`
  lists the private repositories.
- Rendered markdown drops Pandoc title blocks (the leading `%` lines).
- `repo_max_bytes` (default 1 GiB, `0` for no cap) caps the total size of
  a commit's files. A branch whose tip is over it is not rendered: the
  site keeps its last render and the log says why. A tag over it gets no
  tarball. The other size limits apply per file, so without this a commit
  of many files just under them could fill the disk the whole site is on.
  The output takes a few times the cap (raw copies, pages and a tarball),
  and the cap applies to each branch and each tag separately.
- `repo_max_files` (default 25000, `0` for no cap) does the same for the
  number of files, symlinks and submodules in a commit. Each file becomes
  several files of output, so many tiny ones could use up the disk's
  inodes while staying far under `repo_max_bytes`.
- `archive_retention` (default `1h`) is how long a latest-commit tarball
  stays after a push replaces it. Its name never changes, so the web
  server can tell browsers to cache it for good. Keeping it a while means a page or
  menu loaded before the push still links to a file that exists. The
  renderer removes old ones when it next renders the repository. A
  tarball of a commit that a force push took off the branch goes at once.
  Tag tarballs follow their tags: each render remakes the tarball of a tag
  that has moved and removes that of a tag that is gone, even when the tag
  was changed in the bare repository, where no hook runs.

[examples/repo-render.yaml](examples/repo-render.yaml) documents every other
setting, including size limits and worker count. The renderer rejects
unknown keys. Run `repo-render check` after you edit the config.

The renderer bundles syntax highlighters for Tomo, BP, and Nomsu
([internal/highlight/lexers/](internal/highlight/lexers/)), so you only
need to add their entries under `lexers:` to the config. A lexer of the
same name in one of `lexer_dirs` overrides a bundled one.

Older versions had default paths, and took `listen`, `aliases`,
`htpasswd`, `ssh_key` and `mirrors`. The renderer now rejects those keys
and says where each went; [docs/upgrading.md](docs/upgrading.md) moves a
server over.

## Private repositories

A repository in no section is private (unless `site.unlisted` says
otherwise). Without `private_htdocs`, private repositories aren't rendered
at all. With it, they're rendered as a site of their own, in its
`private/` directory, to be served at `/private/` behind a password:

```yaml
private_htdocs: /var/www/private/repo-render
```

- `private_htdocs` must be outside `htdocs` (the renderer refuses
  otherwise), so a web server serving `htdocs` never serves it. Asking
  for the password is the web server's job: the private section of
  [examples/httpd.conf](examples/httpd.conf) does it for httpd, and
  [docs/openbsd.md](docs/openbsd.md#8-set-up-a-private-site-optional) sets it up.
- Private pages link only within `/private/`, carry `noindex`, are left
  out of the sitemap, and are cached by the browser alone (with
  examples/httpd.conf). Clone one with the login in the URL, or let git
  ask for it: `git clone https://bruce@code.example.com/private/git/REPO.git`.
- A repository that moves between the sites, because you added it to a
  section or took it out, is removed from the one it left on its next
  push or `render`.

## Browsing other branches

By default, the site shows each repository's default branch. To show others,
list them in the repository's config:

```yaml
repos:
  tomo:
    branches: [dev, my-name/feature-with-kebabs]
```

Write exact branch names. Patterns like `feature/*` are not supported, so
only branches you name appear on the site. Branch names can contain letters,
digits, `.`, `_`, and `-`, with `/` between parts. The same goes for
`branch`, which sets the default branch in place of the bare repository's
HEAD.

The default branch keeps the same URLs it always had. Each listed branch
gets its own copy of the site under `<repo>@<branch>`, with every `/` in the
branch name written as `:`:

```
/code/tomo/src/main.c                            the default branch
/code/tomo@dev/src/main.c                        the dev branch
/code/tomo@my-name:feature-with-kebabs/src/main.c
```

Raw files, images, search, and tarballs follow the same pattern
(`/raw/tomo@dev/…`, `/search/tomo@dev`, and so on). Each branch renders
incrementally, like a separate repository. A branch picker next to the
Download button lets visitors switch between them, and it links to the root
of the other branch. Search engines are asked not to index the non-default
branches. The clone export includes the listed branches too, so
`git clone -b dev` works. Each branch's Download menu offers that branch's
latest commit. Tags belong to the repository, so every branch lists the
same tags.

If you list a branch that does not exist, the renderer skips it until it
appears. If you remove a branch from the list, or delete it from the
repository, the renderer deletes its pages on the next run.

## Commands

```
repo-render --config FILE <command>
```

| Command | What it does |
|---|---|
| `kick` | Process the job queue while holding the render lock. The post-receive hook runs this. |
| `render [--full] REPO…` | Render the named repositories. `--full` ignores saved state and re-renders everything. |
| `render --all [--full]` | Render every repository in `git_dir`, and delete the output of any that is no longer there. To take a repository off the site, remove it from `git_dir` and run this, or make it private. |
| `index` | Rebuild only the index page and the static files. |
| `enqueue REPO REF OLD NEW` | Add a job to the queue. The post-receive hook runs this. |
| `check` | Check the config, lexers and templates, that git runs, and that the directories are there. Exits with an error if a render would fail. |

## Trying it locally

The quickest way is `make preview`. It clones (or fetches) the repositories
listed in `preview/repos.txt`, renders them with `preview/repo-render.yaml`
into `.preview/`, and serves the site at http://127.0.0.1:8080/. Edit those
two files to change what gets rendered; `make preview PREVIEW_LISTEN=:9000`
picks another address, and `make preview-clean` starts over from scratch.

To try it on a project of your own by hand, make a scratch directory with a config whose paths are relative to it, clone
a project into it, render it, and start the preview server:

```sh
mkdir -p /tmp/repo-render/git
cat > /tmp/repo-render/repo-render.yaml <<'EOF'
host: localhost
git_dir: git
htdocs: htdocs
spool_dir: spool
state_dir: db
site:
  unlisted: show
EOF
export REPO_RENDER_CONFIG=/tmp/repo-render/repo-render.yaml
git clone --bare ~/some/project /tmp/repo-render/git/project.git
./repo-render render --all
./repo-render-preview /tmp/repo-render/htdocs    # http://127.0.0.1:8080/
```

`repo-render-preview [--listen ADDR] [--private PRIVATE_HTDOCS] HTDOCS` is a
plain file server. It finds each page's `.html` file, serves
`private_htdocs` at `/private/`, and keeps the repositories' own HTML
from running as part of the site, and that's all: no caching rules, no
password. Don't put it on the internet.

To test the push path, set the scratch repository up with `new-repo`, and
push to it:

```sh
contrib/new-repo -c /tmp/repo-render/repo-render.yaml -p "$PWD/repo-render" /tmp/repo-render/git/project.git
git -C ~/some/project push /tmp/repo-render/git/project.git main
```

## Running it on a server

[docs/openbsd.md](docs/openbsd.md) sets up an OpenBSD server step by
step. In outline, whatever the system:

- **Repositories** are bare repositories in `git_dir`, made with
  `contrib/new-repo`. It refuses branch deletions over a push (git does
  not refuse tag deletions; the renderer removes a deleted tag's
  tarball), links the repository's `hooks/post-receive` to the one
  installed hook, and records the renderer's config, the `repo-render`
  to run and the mirrors in the repository's git config
  (`repo-render.config`, `repo-render.program`, `repo-render.mirror`),
  where the hook reads them. A repository's description is its
  `description` file.
- **On a push**, the hook queues a job for each branch and tag and starts
  `repo-render kick` in the background, which renders them under a lock,
  so a burst of pushes makes one render.
- **Mirrors** get the default branch (forced) and every tag after every
  push, in the background. Branches and tags that only the mirror has are
  left alone. A failure goes to syslog and is retried on the
  next push. ssh never prompts; set the repository's `core.sshCommand` to
  use a deploy key. Mirroring only sends: nothing made on a mirror comes
  back, and the next push overwrites it.
- **The web server** serves `htdocs` as static files. It must map each
  page's URL to its `.html` file (`/code/r/src` and `/code/r/src/` are
  `code/r/src.html`) and serve a directory's `index.html`. It should send
  the site's headers too: the pages carry no Content Security Policy of
  their own, and raw files keep their real names, so the `sandbox` policy
  on `/raw` and `/svg` is what stops them from running as part of the
  site. [examples/httpd.conf](examples/httpd.conf) does all of that for
  httpd, and the tests check it against a rendered site.

The renderer only ever reads the repositories and writes to the site,
the queue and its state, and on OpenBSD it restricts itself to those
(and its log) with unveil(2). Before it writes anything, it checks every
path and tag name from the repositories.

## Tests

`go test ./...` creates temporary bare repositories and pushes to them for
real, then checks the rendered output. It covers:

- the first render, and incremental renders for modified, added, and deleted
  files
- force pushes, config changes, and tags (including tag names with slashes),
  and that a tag push rewrites only the download list
- latest-commit tarballs: named by commit, linked only from the download
  list, and removed once replaced for longer than `archive_retention`, or
  at once when a force push drops their commit
- recovery from a hostile commit: after a revert or a force push, nothing
  it put on the site remains, even when rendering it failed partway or
  died, and tag tarballs follow tags moved or deleted on the server
- README selection and listing truncation
- markdown link rewriting and HTML escaping
- raw copies and themed SVGs
- symlinks and submodules
- the file drawer, and which pushes rewrite it
- index sections and branch views
- the clone export, which the tests clone back with git
- that httpd serving `examples/httpd.conf` serves every file of a rendered
  site, public and private, at the URL its pages link to, with the right
  headers, by a model of httpd (first match wins, a rewrite is looked up
  again) that matches locations with `internal/httpd`
- `new-repo` and the post-receive hook, run for real: what the hook
  queues and runs, and what it pushes to the mirrors
- the preview server's URL mapping
- the guarantee that a file page and a markdown fence highlight the same
  code identically
- falling back to plain text for highlighters that loop, crash, or run too
  long

`node --test test/*_test.js` tests the search matching logic and the "2 days ago" wording.