Backups and monitoring

BackupsCopy Anchor Link

The optional backup service runs pg_dump every night at BACKUP_HOUR (UTC, default 3). It keeps 14 daily and 8 weekly dumps in BACKUP_DIR (default ./backups).

docker compose --profile backup up -d
docker compose run --rm backup --once        # take one now

Copy BACKUP_DIR off the machine with rsync, restic or object storage. A backup on the same disk as the database is not a backup.

RestoringCopy Anchor Link

Restore into an empty database, and test the procedure at least once before you need it:

docker compose stop web worker
docker compose exec db psql -U hub -d postgres -c 'DROP DATABASE hub WITH (FORCE)' -c 'CREATE DATABASE hub'
docker compose exec db psql -U hub -d hub -c 'CREATE EXTENSION timescaledb' -c 'SELECT timescaledb_pre_restore()'
docker compose exec -T db pg_restore -U hub -d hub --no-owner < backups/daily/hub-<stamp>.dump
docker compose exec db psql -U hub -d hub -c 'SELECT timescaledb_post_restore()'
docker compose start web worker

RetentionCopy Anchor Link

Health and metricsCopy Anchor Link

• GET /api/health answers 200 when the web app can reach the database, 503 otherwise. It needs no auth.
• Metrics are off until you set METRICS_TOKEN. Then both endpoints want Authorization: Bearer <METRICS_TOKEN>:
  • the web app's GET /metrics: ingest, live connections, pairing, the public API and data exports;
  • the worker's GET /metrics on port WORKER_METRICS_PORT (9464): job runs and durations, Discord re-verification, open play sessions and grace expiries.

Metric labels never contain user, account or device ids, names, IP addresses or coordinates.

The hub repository ships a Prometheus scrape example (ops/prometheus/scrape-example.yml), alert rules (ops/prometheus/alerts.yml) and a Grafana dashboard (ops/grafana/dashboards/hub-overview.json). Keep the scrape job names hub-web and hub-worker; the dashboard expects them.

curl -H "Authorization: Bearer $METRICS_TOKEN" http://127.0.0.1:9464/metrics

DecommissioningCopy Anchor Link

The admin decommission switch makes the hub answer 410 to the plugin, which disables the connection in every player's plugin for good. Players have to pair again to use another hub. Only use it when you shut the hub down permanently.