Backups That Actually Restore: Weekly R2 Dumps for Both Client Stacks

Why Care?

Managed hosting means the client never thinks about their data surviving — that's the product. Both client Twenty stacks now dump their postgres to per-client Cloudflare R2 buckets on a weekly schedule, and the restore path isn't theoretical: we drilled it, hit one real snag, and wrote the working commands into the runbooks.

What's New?

• Per-client backup crons: a pg-dump-twenty service in each client's Railway project (Railway's own postgres-s3-backups template repo), single-shot mode + Railway cron, Fridays at midnight EST. pg_dump runs over the project's private network; the database never touches the internet.

• Per-client R2 buckets (reach-edu, palmer-ai) with bucket-scoped API tokens — the zero-shared-infrastructure rule (D2) now extends to backups, and a leaked token can only ever see one client's dumps. R2's zero egress means restores and drills are free; storage at CRM scale rounds to pennies.

• A restore drill that passed: dump pulled from R2 into a scratch postgres inside the project — 28 workspace tables, all six companies including the one written through the API hours earlier. Scratch services deleted after, per the drill's own hygiene.

• Three backup triggers, not one: the weekly cron; manual (railway redeploy on the backup service); and agent-initiated — the per-client twenty-interface skill now tells any agent session to OFFER a backup after ~10+ record creations. The ops discipline lives in the skill, not in anyone's memory.

The snags, honestly

1. The dump is a gzipped pg_dump tar; pg_restore refuses it compressed. gunzip first. This is exactly why drills exist.

2. railway ssh fails host-key verification in non-interactive shells — the drill ran as a throwaway runner service with its script in an env var instead.

3. Twenty's API playground mints 2-hour PLAYGROUND tokens that look like real API keys and then die. Real keys come from Settings → APIs. Documented so it never bites a client onboarding.

What's Next

Phase 4: Twenty's native MCP for real stakeholders — made dramatically easier by the day's best discovery, that the deployed instances are MCP OAuth-compliant out of the box (add connector URL → log in → done, desktop and mobile). Then the hub stopgap and the homebase connector-matrix spike.