Standing up a real server

The recipe actually run against a Hetzner CPX12 (1 vCPU, 2 GB, x86_64), Debian 13, on 2026-09-15: get the code there, install, start, push from a laptop, check it, back it up, and know what to do when it breaks. Where a step has not been fully verified yet, that is said plainly rather than papered over.

Everything here is deploy/ made concrete for one box, one operator, one app (docs/design/production.md, tasks PROD-01..11 in TASKS.md). Four mechanisms outside Rails show up repeatedly below: fork (the kernel clones a running process; melee forks a warm process into a short-lived request child per HTTP request), Landlock (a Linux feature that lets a process narrow its own filesystem access to an allow-list), seccomp (a kernel filter on which syscalls a process may make at all), and the user namespace (unshare(CLONE_NEWUSER), letting a process remap its own uid as one more wall, on top of — not instead of — the real per-app uid). The full account of what they stop and don’t is Keeping it safe; this page is only the operating recipe.

What you need

• A Linux machine with root over SSH. Not the box your other projects share — this one runs sandboxed code.
• Kernel 6.7 or newer, with Landlock (ABI v4+) and cgroup v2. melee-server treats both as a hard requirement and refuses to start without them (What a server needs).
• Debian 13, not Ubuntu 24.04. Ubuntu’s default kernel.apparmor_restrict_unprivileged_userns=1 sysctl very likely blocks the sandbox’s drop-uid-then-unshare(CLONE_NEWUSER) sequence. Confirmed absent on this box’s Debian 13 (kernel 6.12.107+deb13-cloud-amd64) — the worry doesn’t even arise here — but still untested against any kernel that actually carries it, Ubuntu’s own included.
• Provider firewall: 22, 80, 443 in — and 443 out. Outbound is easy to forget, and it isn’t for GitHub (see below); it’s for apt, mise’s and Caddy’s own package repositories, all fetched over HTTPS. Found the hard way on this run: without it, install.sh doesn’t fail loudly — it just hangs on its first HTTPS fetch, and the fix was adding the outbound rule at the provider, not anything in the script.
• A DNS record for the app’s hostname, pointing straight at the box’s IP. Not through a CDN proxy: Caddy issues its own certificate with the HTTP-01 challenge on the first request it sees for that hostname, which has to reach the box itself. A proxy in front of the box would answer that challenge instead, or not forward it at all, and Caddy would never get a certificate. A wildcard record (*.apps.example.com) is worth setting up rather than one record per app: under the default routing shape the edge names no hostnames at all, so a wildcard plus a melee push is the whole of putting a new app online.

Getting the code there

The melee repository is private and this box carries no GitHub credentials, so install.sh’s own git clone step was skipped in favour of rsync-ing an existing checkout onto the box:

rsync -a --exclude='*.o' --exclude='*.a' --exclude='target' --exclude='build' \
  ./ root@<host>:/opt/melee/

Two things this run needed that a plain rsync doesn’t get right on its own:

• A worktree’s .git is a pointer file, not a repository — it names another checkout’s .git/worktrees/... path, which doesn’t exist on this box. It has to be replaced with a real, standalone clone’s .git directory before any git command on the box will work at all.
• chown -R root:root after, since rsync from a laptop leaves everything owned by that laptop’s uid, and install.sh — and melee-server itself — run as root.

With that done, install.sh is told the commit is already there:

MELEE_COMMIT="$(git rev-parse HEAD)" ./deploy/install.sh

install.sh checks whether $MELEE_CHECKOUT/.git is already at $MELEE_COMMIT before it does anything network-shaped, and skips the clone (and the vendor/spinel submodule update) entirely when it is — which is exactly the case here.

Installing

install.sh runs as root, once, and is safe to rerun. In order: kernel checks (version, Landlock — probed through the landlock_create_ruleset syscall itself, not a /sys file that turns out not to exist on any kernel — cgroup v2, the AppArmor userns sysctl); system packages and Caddy; mise and the exact Ruby/Rust mise.toml pins; the checkout (above); Spinel (make deps && make); melee-server and melee (cargo build --release); the home directory; the control-API token and the Caddy edge secret; and the systemd units, Caddyfile, logrotate config and backup timer, all read from the checkout’s own deploy/.

The two unit files are not installed verbatim. systemd rejects an ExecStart= line whose first word is a ${VAR} — it checks the executable against the literal unit-file text before any variable substitution happens — so melee-server.service and melee-backup.service write @MELEE_CHECKOUT@/@MELEE_RUBY@ as plain-text placeholders in the executable position, and install.sh substitutes them with sed before installing the unit, the same way it already does for logrotate.conf’s @MELEE_HOME@. Everything after the executable — ${MELEE_HOME}, ${MELEE_DOMAIN}, ${MELEE_APP_UID_RANGE}, ${MELEE_TRUSTED_PROXY}, ${MELEE_TZ} — stays a genuine systemd variable, read from /etc/melee/server.env at every start: those are what an operator is meant to edit later, and a systemctl restart melee-server alone is enough to pick them up.

On this box (1 vCPU, 1.9 GB) the whole run took 5 minutes 34 seconds: Spinel about 2 minutes 30 seconds, cargo build --release 2 minutes 29 seconds, no OOM despite the box’s small memory. Both the kernel checks and the home-directory layout checks passed cleanly.

What’s left is not automated — install.sh says so at the end — because it is genuinely operator judgment:

Review that file — the Caddyfile is rendered from it, so there is nothing to edit in /etc/caddy/Caddyfile itself — and confirm the firewall really is 22/80/443-only before starting anything. Start melee-server before reloading Caddy: under ondemand it is what answers the certificate question, and a name first asked for while it is down fails its handshake.

Starting

systemctl enable --now melee-server melee-backup.timer
systemctl restart caddy   # a full restart, not reload — the edge secret's EnvironmentFile is read at start

Caddy requested and got a certificate for volleyball.apps.ideasasylum.com on its first request, with DNS and outbound 443 both in place. Before trusting any of it, the gate is three checks, not a feeling:

journalctl -u melee-server -n 50   # no "--trusted-proxy is set without --edge-secret-file" warning
ss -ltnp                           # 22, 80, 443, loopback 8080/7070 — nothing on 2019
journalctl -u caddy | grep MELEE_EDGE_SECRET   # empty — the secret never reaches the journal

A warning at startup, or anything on 2019 (Caddy’s admin API, which should be a UNIX socket, not TCP, here), or a hit in the Caddy journal means the edge-secret protection (PROD-03, SEC-33) is not actually in place on this box, whatever the Caddyfile says.

Pushing from your laptop

The control API (--api) is loopback-only by design, so every command against production goes through an SSH tunnel. scripts/prod.sh, behind the mise run prod:* tasks, opens the tunnel for the length of one command and closes it again; the box it talks to is named in deploy/production.env:

mise run prod:push volleyball                  # deploy an app
mise run prod:melee volleyball logs -f         # any melee command: logs, stats, objects, env KEY VALUE
mise run prod:status                           # commit, Spinel pin, services, then `melee stats`
mise run prod:ssh                              # a root shell on the box

The app is a path, not just a name under examples/. volleyball above is shorthand for examples/volleyball, but anything that names a directory works, so an app that lives in its own repository somewhere else deploys through the same tasks and needs no tunnel, no MELEE_TOKEN and no --server of its own:

mise run prod:push ~/code/dinner-time
mise run prod:melee ~/code/dinner-time logs -f

Run them from this repository (that is where the tasks and the CLI are), and give the app’s directory. The app’s own melee.toml stays the development config: prod.sh overrides --server and --url for the one command.

The first time, it fetches the control-API token from the box to ~/.melee-production-token (mode 0600) and keeps using that file, handing it to the CLI through MELEE_TOKEN so it never appears on a command line. Keep that file outside any app directory — melee push uploads everything in the app directory, and the CLI refuses a token file inside it.

The tunnel’s local port is 7071, deliberately not the development default, so a hand-typed melee push that forgot --server reaches the development server (or nothing) rather than production. The first line every command prints is the destination:

--server overrides melee.toml: using control API at http://127.0.0.1:7071

Doing it by hand, which is all the script does:

ssh -L 7071:127.0.0.1:7070 <host>
MELEE_TOKEN="$(cat ~/.melee-production-token)" melee --server http://127.0.0.1:7071 push

The first push — a full Spinel compile of volleyball and the stdlib, on this box’s own CPU — took 34.9 seconds; a push after a Spinel pin move, which recompiles SQLite too, took 73 seconds.

Checking it works

robots.txt (served straight from public/, no warm process involved) came back fine on the very first try, before anything else did — which is the trap this run actually hit: it makes the sandbox look healthy from the outside even when it isn’t. It wasn’t, yet. The warm process’s first start on x86_64 died immediately with SIGSYS, because the seccomp allow-list in sandbox.rs was built from an arm64 strace (M0-11), and arm64 only ever has the *at/ppoll/dup3 forms of several calls and no arch_prctl at all — so six legacy x86_64 entry points glibc and the Spinel runtime still reach for were simply never in the list: access, arch_prctl, dup2, mkdir, poll, unlink, found one at a time by stracing volleyball outside the sandbox on the box itself, each the next syscall the warm process was killed on. Every real route 503’d until the fix (T-13: all six pushed onto the allow-list under #[cfg(target_arch = "x86_64")]) landed and was rebuilt on the box. What follows is what ran after that rebuild, not a plan for later:

curl -i https://volleyball.apps.ideasasylum.com/                 # 200
curl -i https://volleyball.apps.ideasasylum.com/robots.txt       # 200
curl -i -H 'Accept-Encoding: gzip' .../public/game.css           # gzip

• GET / → 200, with a session cookie carrying both Secure and HttpOnly.
• POST /games, with the page’s own CSRF token (not a stale or guessed one) → 303.
• The created game’s own page → 200.
• POST a point on it → 303, and the point shows up in the game’s history.
• robots.txt → 200; the CSS is served gzip-encoded.
• The negative control: a request straight to loopback (127.0.0.1:8080, bypassing Caddy entirely) with a spoofed X-Forwarded-Proto: https header and no X-Melee-Edge gets a cookie back without Secure. This is the proof that matters — not that --trusted-proxy is set, but that the edge secret, not just the source address, is what it actually checks (PROD-03, SEC-33); loopback is shared with every sandboxed app, so the address alone proves nothing.

Separately, sandbox::tests ran on the box itself as an unprivileged (non-root) user — 4 tests, all passing. That’s the half of PROD-06 the arm64 dev VM could assert the mechanism for but never the box’s own kernel: Landlock, seccomp and the uid drop, exercised for real, not by extension from another architecture.

Backups and restore

melee-backup.timer runs backup.sh daily (OnCalendar=daily, a 15-minute random delay so it doesn’t always land on the same second as anything else), as root, into dated directories under /var/backups/melee/<date>/ — a sqlite3 .backup of every app’s data plus its env and uid files, pruned after 14 days.

The timer running on schedule proves nothing by itself, so it was run by hand once rather than trusted on faith: melee-backup.service backed up all 3 of volleyball’s SQLite databases (the app’s own plus one per durable object — a game is a Durable instance with its own storage file). Then the drill in deploy/restore.md — stop the server, find the dated backup, restore the app’s data with its uid, restart, check the app — was followed for real, restoring into a scratch directory rather than over the live one: the restored database opened cleanly, and its melee_objects registry (docs/design/persistent.md) came back with 2 rows, matching what was actually live on the box. PROD-09’s own rule — nothing counts as “done” until a restore has actually been tried — is what this satisfies.

Upgrading

melee-server, the stdlib and the Spinel pin move together — a mismatch between them is a miscompile, not an error, because nothing checks them against each other yet. One command moves all three:

mise run prod:upgrade            # the commit you have checked out
mise run prod:upgrade <sha>      # or a specific one

It pushes the commit straight into the box’s checkout over SSH (the repository is private and the box has no GitHub credentials, so nothing is fetched there), checks it out, runs deploy/install.sh from that commit — which rebuilds Spinel only when the pin moved and never overwrites the files an operator edits under /etc/melee — restarts melee-server, and waits for /v1/health to answer. If the box’s checkout has local edits it refuses and says how to stash or discard them. When the Spinel pin moved it ends by telling you to:

Re-push every app. A deployed binary keeps running against whatever stdlib and Spinel commit it was built with — nothing invalidates it on its own — but it is not what the new server would build from the same source, and the next deploy of that app will be: mise run prod:push <app> for each.

Don’t restart the server on purpose while a push is in flight. A push’s activation is atomic — it only swaps the current symlink after a full, successful build — so one that loses its server partway through just leaves the previous release running rather than half-installing anything. That’s a safety net for an accident, not a reason to interleave an upgrade with a deploy; finish the push first.

When it breaks

A release that dies before answering its first request gets one immediate retry, then a 10-second cooldown where every request gets a 503 rather than a fresh spawn attempt each time — see Deploys and releases, “When a release won’t start” for the exact mechanics. The most common cause is MELEE_SESSION_SECRET missing, which shows up verbatim in the logs:

melee: MELEE_SESSION_SECRET is not set; refusing to start in warm mode

melee --server http://127.0.0.1:7071 logs -f   # the app's own log lines, mixed with the server's [melee] events
journalctl -u melee-server -f                   # the unit's own stdout/stderr — startup warnings, panics

What is not covered yet

• Multi-tenant hardening — a pid namespace and private /proc, the build running as its own uid rather than root, per-app credential hygiene across many apps. None of it is missing by accident; there’s one tenant here, and it’s the operator. See Keeping it safe, “What is not in place.”
• Packaging (P-03). This box still needs the whole repository checkout and toolchain, same as the dev machine — deploy/install.sh is a recipe, not an installer you hand to someone else.
• Signed builds (P-01). Nothing records what commit produced a given release binary or proves it wasn’t tampered with in transit; a rollback today is “re-push the old source,” not “restore a known-good artifact.”

Next

• Keeping it safe
• Logs, backups and upgrades