Monitoring

What the server counts about itself and every app, how to see it in a terminal with melee stats, and how to get it into a Grafana Cloud dashboard with one agent on the box. What is not measured is listed too.

melee-server keeps a small set of counters in memory: requests and how they ended, how long they took, how long they waited for a slot, process starts and stops and why, what is running right now, and what the kernel says each app is using. It serves them on the control API in two shapes: Prometheus text on GET /v1/metrics for a scraper, and JSON on GET /v1/stats for melee stats. Nothing runs inside the app binary: it is sealed by the sandbox and cannot load a metrics gem, so what the server sees from outside is what is measured. Two mechanisms outside Rails appear below: Prometheus text is a plain-text format (name{label="value"} 123, one line per number) that every metrics scraper reads, and a cgroup is a directory under /sys/fs/cgroup in which the Linux kernel enforces limits and keeps accounts (memory, CPU, process count) for every process placed in it; melee puts each app in /sys/fs/cgroup/melee/<app>, which is where the memory and CPU figures come from.

melee stats

From any app directory whose melee.toml names the server:

melee stats

server 0.1.0 up 3d, 1 app(s) deployed, 4 deploy(s) ok / 1 failed, 12 unrouted request(s)
APP                  WARM   WORKER   BUSY REQUESTS   FAILED     MEAN   STATIC   MEMORY PROCS
volleyball           up     -         0+0     1834        3    1.2ms      611    14.2M     2
BUSY is requests holding a slot + requests waiting for one; MEMORY and PROCS come from the app's cgroup (Linux, root).

Counts are since the server started; a restart zeroes them. The figures above are illustrative, not from a real run.

Reading /v1/metrics by hand

The two routes take the control token like every other route. They also take a second, optional token, configured with --metrics-token-file, which opens exactly these two routes and gets a 403 anywhere else. That is the token a scraper should hold, so that a leaked agent credential can read numbers but cannot deploy. The server refuses to start if the two files hold the same value. Both files are 0600 root; the server refuses a secret file that anyone else can read.

curl -s -H "Authorization: Bearer $(cat /etc/melee/metrics-token)" http://127.0.0.1:7070/v1/metrics

# HELP melee_apps_deployed Apps with a current release.
# TYPE melee_apps_deployed gauge
melee_apps_deployed 1
# HELP melee_requests_total Requests that reached the app path, by outcome.
# TYPE melee_requests_total counter
melee_requests_total{app="volleyball",outcome="2xx"} 1790
melee_requests_total{app="volleyball",outcome="3xx"} 41
...

A scrape reads counters, asks each pool what is running and reads five files from each app’s cgroup. It never starts a process and never writes anything, so polling it every fifteen seconds does not keep an idle app warm and does not touch the app’s directories. It also never waits for a process that is starting: a scrape that lands mid-start reports that app as not up. This is also why the number of durable objects per app is not in it: reading that would start the worker.

Setting up Grafana Cloud

The production recipe (deploy/install.sh) installs Grafana Alloy as the one agent on the box. It scrapes /v1/metrics with the metrics token, exports host metrics (CPU, memory, disk, network) and ships the systemd journals of melee-server, caddy and alloy to Loki, and sends all of it to your Grafana Cloud stack. Alloy runs as its own user and never reads /etc/melee/metrics-token itself: install.sh copies the value into /etc/melee/alloy-generated.env as MELEE_METRICS_TOKEN, a root-only file systemd hands to the service, and rewrites it on every run so the two cannot drift. The free tier (10,000 active series) is plenty: melee-server produces about 84 series per app and six for the server, plus a few hundred from the host exporter.

1. Create a Grafana Cloud account and a stack (any region). On the stack’s page at grafana.com, the Prometheus tile’s Details shows a remote-write URL and a numeric user id; the Loki tile’s Details shows a push URL (ending /loki/api/v1/push) and a user id of its own. Two URLs, two ids.

2. Make an API key: Security > Access Policies, a policy scoped to the stack with metrics:write and logs:write, then a token on it (older UI text calls this a MetricsPublisher API key). Copy it once; it starts glc_. One token serves both remote writes, and it is write-only: it can push samples and cannot read anything back.

3. Fill in /etc/melee/alloy.env on the box. install.sh wrote it with empty values, mode 0600 root:

   GRAFANA_CLOUD_PROM_URL=https://prometheus-xxx.grafana.net/api/prom/push
   GRAFANA_CLOUD_PROM_USER=123456
   GRAFANA_CLOUD_LOKI_URL=https://logs-xxx.grafana.net/loki/api/v1/push
   GRAFANA_CLOUD_LOKI_USER=654321
   GRAFANA_CLOUD_API_KEY=glc_...
   

   The two LOKI lines are optional: install.sh renders the journal-to-Loki half of the Alloy config only when GRAFANA_CLOUD_LOKI_URL is set, so you can start with metrics and add logs later by filling them in and rerunning install.sh (mise run prod:upgrade does that).

   Plain KEY=value, no quotes, nothing after the value on the line.

4. Start Alloy: systemctl enable --now alloy, or rerun install.sh, which enables it once it sees GRAFANA_CLOUD_PROM_URL filled in and leaves it disabled otherwise. journalctl -u alloy -n 50 should show it scraping without 401s; a 401 from 127.0.0.1:7070 means MELEE_METRICS_TOKEN in /etc/melee/alloy-generated.env no longer matches /etc/melee/metrics-token (rerun install.sh, which regenerates it), and a 401 from Grafana Cloud means the user id or key is wrong.

5. Import the dashboard: in Grafana, Dashboards > New > Import, upload deploy/grafana/melee.json from the checkout, and pick your stack’s Prometheus data source when asked. It has one variable, app, which defaults to every app.

Nothing above has been run against the production box yet (O-04 in TASKS.md); the configuration was written from Alloy’s documentation and unit-tested against the server, not against a live stack.

What each dashboard row shows

Stop reasons in full: idle (no request for --idle-secs), timeout (a request ran past --request-timeout-secs and the process group was killed), slot_error (a broken exchange on a slot, killed; a process that dies mid-request lands here or in exited depending on which the server noticed first, so read the two together as “died on its own”), replaced (the server stopped it: a deploy, melee env, melee restart or shutdown), exited (it ended on its own). Request outcomes: 2xx, 3xx, 4xx, 5xx from the app; timeout (504), busy (503), gone (502, the process died before answering), spawn_failed (502), cooling_down (503 after two failed starts) from the server.

What is not measured

• Durable-object calls. A request child calls the worker over a UNIX socket inside the sandbox that the server never sees, so there is no count of object calls and no per-object or per-method timing. The only object figure is melee_worker_admin_calls_total, which counts melee objects.
• Object count per app, because reading it would start the worker.
• Anything inside the app: individual routes, queries, template time. Outcomes are status classes, so a 404 and a 422 are both 4xx.
• The app’s own log lines. <home>/logs/<app>.ndjson is root-only and Alloy runs as alloy; only the three units’ journals reach Loki. melee logs reads the file as before.
• Alerting. Nothing pages you. The metrics to alert on are melee_warm_process_up at 0 while melee_requests_total grows under a server outcome, melee_app_oom_kills_total increasing, and disk from the host exporter (O-07).
• On macOS, or as a non-root user, the five cgroup metrics are absent and melee stats prints -.

The design, the full metric table and the series budget are in the repository’s docs/design/observability.md; the decision and the alternatives (AppSignal, self-hosted Prometheus, Netdata) are ADR 0018.

Next

• Logs, backups and upgrades
• Standing up a real server