Logs, backups and upgrades

One NDJSON file per app that nothing rotates, a home directory nothing backs up, and an upgrade path you have to think about because the compiler is pinned.

Logs

<home>/logs/<name>.ndjson

One JSON object per line, holding three kinds of entry mixed together: the app’s own log.* events (each carrying a request id), anything the app wrote to stderr, and the server’s own events.

melee logs                 # the last 50 lines, formatted
melee logs --tail 500
melee logs -f              # follow

or read the file directly on the server — it is NDJSON, so jq works.

Nothing rotates them and nothing caps their size. An app logging in a loop will fill the disk. Until melee does this itself (it is open work), a logrotate entry with copytruncate is the pragmatic answer: melee holds the file open and appends, so truncating in place is safer than renaming out from under it.

There is no log level filter and no way to suppress a level — every line an app writes is written.

Backups

There is no backup and no replication. The whole state of the system is --home, and it is on one machine’s local disk.

What is in there:

Do not copy a live SQLite file with cp. The database is in WAL mode and a plain copy can catch it mid-write. Use SQLite’s own backup, which is consistent against a live database:

sqlite3 /var/lib/melee/apps/notes/data/app.sqlite ".backup '/backup/notes-$(date +%F).sqlite'"

Durable objects have a file each, under the app’s data directory — back the whole directory up the same way, per file.

The files are owned by the app’s uid and the directories are 0700, so this runs as root. That is the point: if you can read them without sudo, the isolation is not working.

Restoring is putting the files back and restarting the server. There is no import command.

Upgrading

Three things have versions and they have to move together: melee-server, the melee standard library, and the Spinel commit the library is pinned to. A mismatch is a miscompile, not an error — there is nothing today that checks them against each other, which is the sharpest edge on this page.

A safe-ish sequence:

1. Update the checkout on the server, submodule included.
2. Rebuild Spinel (make deps && make in vendor/spinel) and the binaries (cargo build --release).
3. Stop the server with SIGTERM so it stops the app processes it owns.
4. Start the new one.
5. Re-push every app. Deployed binaries were compiled against the old standard library and the old compiler. They keep running — nothing invalidates them — but they are not what the new server would build, and the next deploy of that app will be. Re-pushing makes the fleet consistent while you are watching rather than later when you are not.

Step 5 is the part it is tempting to skip. Whether skipping it is safe depends on what changed in the standard library, and nothing tells you.

Upgrading the Spinel pin deserves more care than upgrading melee: it is a whole-program compiler, the dialect’s traps are recorded against a specific commit, and the project’s own rule is to rebase the submodule deliberately rather than track master. Read docs/research/spinel.md and run the test suite before and after.

Restarts and reboots

A server restart loses nothing but warm processes. Apps restart on their next request, and scheduled work survives: pending timers are recorded in each app’s own database, and after a start or a deploy the server runs each app’s worker once to ask what it has pending, so a timer still fires even though nobody visited.

Stop with SIGTERM and give it time. The server’s orderly shutdown stops the app processes it owns; SIGKILL leaves them behind as strays that nothing will reap.

Disk

Three things grow and nothing prunes them:

• Release directories, one per successful deploy: source plus a binary of a couple of megabytes. Delete old ones by hand, keeping current and the one before it.
• Logs, as above.
• App data, which is the app’s business.

A failed build also leaves an un-activated release directory behind.

What you do not have

• No metrics: no request counters, no latency histogram, no per-app resource reporting. Logs and melee apps are the observability story.
• No alerting.
• No health check beyond GET /v1/health (which needs the token).
• No multi-machine anything: no replication, no failover, no placement.

For an app whose availability matters, that list is the argument for running it somewhere else.

Next

• Keeping it safe
• What melee cannot do