Standing up a real server

The recipe actually run against a Hetzner CPX12 (1 vCPU, 2 GB, x86_64), Debian 13, on 2026-09-15: get the code there, install, start, push from a laptop, check it, back it up, and know what to do when it breaks. Where a step has not been fully verified yet, that is said plainly rather than papered over.

Everything here is deploy/ made concrete for one box, one operator, one app (docs/design/production.md, tasks PROD-01..11 in TASKS.md). Four mechanisms outside Rails show up repeatedly below: fork (the kernel clones a running process; melee forks a warm process into a short-lived request child per HTTP request), Landlock (a Linux feature that lets a process narrow its own filesystem access to an allow-list), seccomp (a kernel filter on which syscalls a process may make at all), and the user namespace (unshare(CLONE_NEWUSER), letting a process remap its own uid as one more wall, on top of — not instead of — the real per-app uid). The full account of what they stop and don’t is Keeping it safe; this page is only the operating recipe.

What you need

• A Linux machine with root over SSH. Not the box your other projects share — this one runs sandboxed code.
• Kernel 6.7 or newer, with Landlock (ABI v4+) and cgroup v2. melee-server treats both as a hard requirement and refuses to start without them (What a server needs).
• Debian 13, not Ubuntu 24.04. Ubuntu’s default kernel.apparmor_restrict_unprivileged_userns=1 sysctl very likely blocks the sandbox’s drop-uid-then-unshare(CLONE_NEWUSER) sequence. Confirmed absent on this box’s Debian 13 (kernel 6.12.107+deb13-cloud-amd64) — the worry doesn’t even arise here — but still untested against any kernel that actually carries it, Ubuntu’s own included.
• Provider firewall: 22, 80, 443 in — and 443 out. Outbound is easy to forget, and it isn’t for GitHub (see below); it’s for apt, mise’s and Caddy’s own package repositories, all fetched over HTTPS. Found the hard way on this run: without it, install.sh doesn’t fail loudly — it just hangs on its first HTTPS fetch, and the fix was adding the outbound rule at the provider, not anything in the script.
• A DNS record for the app’s hostname, pointing straight at the box’s IP. Not through a CDN proxy: Caddy issues its own certificate with the HTTP-01 challenge on the first request it sees for that hostname, which has to reach the box itself. A proxy in front of the box would answer that challenge instead, or not forward it at all, and Caddy would never get a certificate.

Getting the code there

The melee repository is private and this box carries no GitHub credentials, so install.sh’s own git clone step was skipped in favour of rsync-ing an existing checkout onto the box:

rsync -a --exclude='*.o' --exclude='*.a' --exclude='target' --exclude='build' \
  ./ root@<host>:/opt/melee/

Two things this run needed that a plain rsync doesn’t get right on its own:

• A worktree’s .git is a pointer file, not a repository — it names another checkout’s .git/worktrees/... path, which doesn’t exist on this box. It has to be replaced with a real, standalone clone’s .git directory before any git command on the box will work at all.
• chown -R root:root after, since rsync from a laptop leaves everything owned by that laptop’s uid, and install.sh — and melee-server itself — run as root.

With that done, install.sh is told the commit is already there:

MELEE_COMMIT="$(git rev-parse HEAD)" ./deploy/install.sh

install.sh checks whether $MELEE_CHECKOUT/.git is already at $MELEE_COMMIT before it does anything network-shaped, and skips the clone (and the vendor/spinel submodule update) entirely when it is — which is exactly the case here.

Installing

install.sh runs as root, once, and is safe to rerun. In order: kernel checks (version, Landlock — probed through the landlock_create_ruleset syscall itself, not a /sys file that turns out not to exist on any kernel — cgroup v2, the AppArmor userns sysctl); system packages and Caddy; mise and the exact Ruby/Rust mise.toml pins; the checkout (above); Spinel (make deps && make); melee-server and melee (cargo build --release); the home directory; the control-API token and the Caddy edge secret; and the systemd units, Caddyfile, logrotate config and backup timer, all read from the checkout’s own deploy/.

The two unit files are not installed verbatim. systemd rejects an ExecStart= line whose first word is a ${VAR} — it checks the executable against the literal unit-file text before any variable substitution happens — so melee-server.service and melee-backup.service write @MELEE_CHECKOUT@/@MELEE_RUBY@ as plain-text placeholders in the executable position, and install.sh substitutes them with sed before installing the unit, the same way it already does for logrotate.conf’s @MELEE_HOME@. Everything after the executable — ${MELEE_HOME}, ${MELEE_DOMAIN}, ${MELEE_APP_UID_RANGE}, ${MELEE_TRUSTED_PROXY}, ${MELEE_TZ} — stays a genuine systemd variable, read from /etc/melee/server.env at every start: those are what an operator is meant to edit later, and a systemctl restart melee-server alone is enough to pick them up.

On this box (1 vCPU, 1.9 GB) the whole run took 5 minutes 34 seconds: Spinel about 2 minutes 30 seconds, cargo build --release 2 minutes 29 seconds, no OOM despite the box’s small memory. Both the kernel checks and the home-directory layout checks passed cleanly.

What’s left is not automated — install.sh says so at the end — because it is genuinely operator judgment:

Review that file, edit /etc/caddy/Caddyfile’s hostname and routing option, and confirm the firewall really is 22/80/443-only before starting anything.

Starting

systemctl enable --now melee-server melee-backup.timer
systemctl restart caddy   # a full restart, not reload — the edge secret's EnvironmentFile is read at start

Caddy requested and got a certificate for volleyball.apps.ideasasylum.com on its first request, with DNS and outbound 443 both in place. Before trusting any of it, the gate is three checks, not a feeling:

journalctl -u melee-server -n 50   # no "--trusted-proxy is set without --edge-secret-file" warning
ss -ltnp                           # 22, 80, 443, loopback 8080/7070 — nothing on 2019
journalctl -u caddy | grep MELEE_EDGE_SECRET   # empty — the secret never reaches the journal

A warning at startup, or anything on 2019 (Caddy’s admin API, which should be a UNIX socket, not TCP, here), or a hit in the Caddy journal means the edge-secret protection (PROD-03, SEC-33) is not actually in place on this box, whatever the Caddyfile says.

Pushing from your laptop

The control API (--api) is loopback-only by design — reach it by tunnelling in, on a port that is deliberately not the development default, so a dropped --server flag fails loudly instead of quietly hitting production:

ssh -L 7071:127.0.0.1:7070 <host>
scp <host>:/etc/melee/token ~/.melee-production-token && chmod 600 ~/.melee-production-token

Keep that file outside any app directory — melee push uploads everything in the app directory, and the CLI refuses a token file inside it.

MELEE_TOKEN="$(cat ~/.melee-production-token)" melee --server http://127.0.0.1:7071 push

The first line printed is the destination, before the token is even read:

--server overrides melee.toml: using control API at http://127.0.0.1:7071

The first push — a full Spinel compile of volleyball and the stdlib, on this box’s own CPU — took 34.9 seconds.

Checking it works

robots.txt (served straight from public/, no warm process involved) came back fine on the very first try, before anything else did — which is the trap this run actually hit: it makes the sandbox look healthy from the outside even when it isn’t. It wasn’t, yet. The warm process’s first start on x86_64 died immediately with SIGSYS, because the seccomp allow-list in sandbox.rs was built from an arm64 strace (M0-11), and arm64 only ever has the *at/ppoll/dup3 forms of several calls and no arch_prctl at all — so six legacy x86_64 entry points glibc and the Spinel runtime still reach for were simply never in the list: access, arch_prctl, dup2, mkdir, poll, unlink, found one at a time by stracing volleyball outside the sandbox on the box itself, each the next syscall the warm process was killed on. Every real route 503’d until the fix (T-13: all six pushed onto the allow-list under #[cfg(target_arch = "x86_64")]) landed and was rebuilt on the box. What follows is what ran after that rebuild, not a plan for later:

curl -i https://volleyball.apps.ideasasylum.com/                 # 200
curl -i https://volleyball.apps.ideasasylum.com/robots.txt       # 200
curl -i -H 'Accept-Encoding: gzip' .../public/game.css           # gzip

• GET / → 200, with a session cookie carrying both Secure and HttpOnly.
• POST /games, with the page’s own CSRF token (not a stale or guessed one) → 303.
• The created game’s own page → 200.
• POST a point on it → 303, and the point shows up in the game’s history.
• robots.txt → 200; the CSS is served gzip-encoded.
• The negative control: a request straight to loopback (127.0.0.1:8080, bypassing Caddy entirely) with a spoofed X-Forwarded-Proto: https header and no X-Melee-Edge gets a cookie back without Secure. This is the proof that matters — not that --trusted-proxy is set, but that the edge secret, not just the source address, is what it actually checks (PROD-03, SEC-33); loopback is shared with every sandboxed app, so the address alone proves nothing.

Separately, sandbox::tests ran on the box itself as an unprivileged (non-root) user — 4 tests, all passing. That’s the half of PROD-06 the arm64 dev VM could assert the mechanism for but never the box’s own kernel: Landlock, seccomp and the uid drop, exercised for real, not by extension from another architecture.

Backups and restore

melee-backup.timer runs backup.sh daily (OnCalendar=daily, a 15-minute random delay so it doesn’t always land on the same second as anything else), as root, into dated directories under /var/backups/melee/<date>/ — a sqlite3 .backup of every app’s data plus its env and uid files, pruned after 14 days.

The timer running on schedule proves nothing by itself, so it was run by hand once rather than trusted on faith: melee-backup.service backed up all 3 of volleyball’s SQLite databases (the app’s own plus one per durable object — a game is a Durable instance with its own storage file). Then the drill in deploy/restore.md — stop the server, find the dated backup, restore the app’s data with its uid, restart, check the app — was followed for real, restoring into a scratch directory rather than over the live one: the restored database opened cleanly, and its melee_objects registry (docs/design/persistent.md) came back with 2 rows, matching what was actually live on the box. PROD-09’s own rule — nothing counts as “done” until a restore has actually been tried — is what this satisfies.

Upgrading

melee-server, the stdlib and the Spinel pin move together — a mismatch between them is a miscompile, not an error, because nothing checks them against each other yet:

git -C /opt/melee fetch origin && git -C /opt/melee checkout --detach <new-sha>
git -C /opt/melee submodule update --init vendor/spinel
(cd /opt/melee/vendor/spinel && make deps && make)   # only if the Spinel pin moved
mise exec -- cargo build --release --manifest-path /opt/melee/supervisor/Cargo.toml
systemctl restart melee-server                        # SIGTERM: stops the app processes it owns first

Then re-push every app. A deployed binary keeps running against whatever stdlib and Spinel commit it was built with — nothing invalidates it on its own — but it is not what the new server would build from the same source, and the next deploy of that app will be.

Don’t restart the server on purpose while a push is in flight. A push’s activation is atomic — it only swaps the current symlink after a full, successful build — so one that loses its server partway through just leaves the previous release running rather than half-installing anything. That’s a safety net for an accident, not a reason to interleave an upgrade with a deploy; finish the push first.

When it breaks

A release that dies before answering its first request gets one immediate retry, then a 10-second cooldown where every request gets a 503 rather than a fresh spawn attempt each time — see Deploys and releases, “When a release won’t start” for the exact mechanics. The most common cause is MELEE_SESSION_SECRET missing, which shows up verbatim in the logs:

melee: MELEE_SESSION_SECRET is not set; refusing to start in warm mode

melee --server http://127.0.0.1:7071 logs -f   # the app's own log lines, mixed with the server's [melee] events
journalctl -u melee-server -f                   # the unit's own stdout/stderr — startup warnings, panics

What is not covered yet

• Multi-tenant hardening — a pid namespace and private /proc, the build running as its own uid rather than root, per-app credential hygiene across many apps. None of it is missing by accident; there’s one tenant here, and it’s the operator. See Keeping it safe, “What is not in place.”
• Packaging (P-03). This box still needs the whole repository checkout and toolchain, same as the dev machine — deploy/install.sh is a recipe, not an installer you hand to someone else.
• Signed builds (P-01). Nothing records what commit produced a given release binary or proves it wasn’t tampered with in transit; a rollback today is “re-push the old source,” not “restore a known-good artifact.”

Next

• Keeping it safe
• Logs, backups and upgrades