The Stack Grows a Safety Net — Five Weekly pg_dumps to R2, One pg17 Gotcha

Why Care?

A stack with no backups is a stack one deleted volume away from gone. Until today, the-water-foundation — four live tools, real config, a working data room — had zero backups. Now every database dumps weekly to storage that lives in a different blast radius than the thing it protects, and we didn't take that on faith: we forced live runs and read the archives back out of R2.

What's New?

• Five pg-dump-* cron services in the-water-foundation's Railway project (railwayapp-templates/postgres-s3-backups), one per database — twenty, outline, papermark, postiz, temporal — each SINGLE_SHOT_MODE, restart NEVER, cron 0 5 * * 5 (Fri 00:00 EST), dumping over the private network via reference-variable connection strings (no passwords duplicated).

• Per-client Cloudflare R2 bucket the-water-foundation as the target — deliberately not a Railway bucket in the same project. Files land as backups/<db>/<db>-<ISO8601>.tar.gz. All five verified present.

• restore-runbook.md — the download → gunzip → pg_restore path, plus the gotcha log below.

Gotchas (first execution)

1. pg_dump must be ≥ the server. The template ships pg16; postiz-postgres is Postgres 17, so pg16 wrote an empty archive. Fix: PG_VERSION=17 + NODE_VERSION=22 as build args on pg-dump-postiz (node:20-alpine is too old to carry a pg17 client; node:22-alpine isn't). The four pg16 DBs needed nothing.

2. Cron services don't run on deploy. They fire only at the scheduled time — a deploy just rebuilds and waits. To test, set the cron a few minutes out, bump CONFIG_EPOCH, wait, then restore 0 5 * * 5.

3. Verify by reading it back. A "SUCCESS" Railway deployment on a single-shot cron means "built + scheduled," not "dumped." The only proof that counts is the object in the bucket — listed via the R2 S3 endpoint.

What's Next

This is the first pillar of a stack-hygiene agent (report-only, scheduled) that will verify these backups ran, drill a restore, watch cert/deploy health, track version drift, and keep the Papermark fork synced — the self-maintaining layer the stack has been building toward.