the-gitfather

Off-site, Grandfather-Father-Son retained, monitored, restore-verified Postgres backups — driven by GitHub Actions, stored on Cloudflare R2.

_{The static backup-history dashboard — every backup over a 1-year window, with restore-verified drills, row-retirement columns, storage and estimated R2 cost. The tooltip splits what a week holds from what ran during it.}

A small, profile-driven tool any project can adopt: point it at a Postgres connection string and an R2 bucket and you get the off-site, immutable, restore-verified pillars of the 3-2-1-1-0 backup pattern, with a daily Slack status row and a static backup-history dashboard. The engine knows nothing about any one project; everything project-specific lives in a profiles/*.yaml file kept in your repo, and credentials come from the environment (GitHub secrets), never from the repo.

---

What you get

• Dumps at a per-project cadence (backups-per-day, default 3), promoted to GFS tiers — one dump per run, server-side copied into daily/weekly/monthly; retention and 14-day WORM immutability enforced by R2 lifecycle rules and bucket locks, not by code. → R2 buckets, retention, locks and tokens
• Backup-time integrity — a size floor, a pg_restore -l structural check, and a SHA-256 of the exact uploaded bytes, before a run is called good. → Verifying backups
• Daily durable verification — every durable object is hash-checked on write and full-restored (the freshest daily each day, weekly/monthly again at ~2 weeks), with row-count gates and a census floor that pages for anything the log names but the listing didn't return. → Verifying backups
• Staleness watchdog with self-heal — a 10-minute check, run outside GitHub in the Cloudflare Worker, that re-triggers a missed backup slot and pages when it can't — so an Actions outage is detected rather than silencing the detector. → scheduler/README.md
• Alerting that survives GitHub being the broken thing — one Slack message per day updated in place, a failure-only webhook, and an external dead-man's-switch. Every message is posted by the Worker, which holds the only Slack token; the jobs hold none, and just leave an outcome record. → Slack and alerting
• A static backup-history dashboard — a single self-contained page built from an append-only run-log in R2. No server, no database. → Backup-history dashboard
• A table archiver — move aged rows of an append-only table out of Postgres into one encrypted object per ISO week, then delete them only after a hash- and fingerprint-gated re-check. Optional, and it gets its own dashboard columns. → Archiving a table out of Postgres
• Client-side encryption — encryption: age, with a separate recipient for archives whose identity never enters CI. → Archiving
• 2-of-3 key escrow — npm run key-shares splits each decrypt key into three SLIP-39 word shares and prints one A4 card per holder; any two holders rebuild it, without meeting, and practice issues let you drill that at any time. → Key escrow
• A recovery kit — npm run recovery-kit stores everything a stranger needs to open the backups (the SLIP-39 spec and code, age, zstd, PostgreSQL's source) beside them, pinned and checksummed; an offline drill proves it complete, and the monthly manual drill runs on its tools. → Recovery kit guide (runbooks) · how it works
• A read-only preflight — npm run doctor -- all runs the real config schema plus live probes of every external client, and writes nothing. → Configuration & doctor
• Guarded R2 token rotation — npm run roll-r2 mates, connects, escrows and publishes a rolled credential in that order, and tracks when one is overdue. → Rotating an R2 token
• A Cloudflare scheduler — one Worker firing every client's workflows on time, instead of GitHub's best-effort cron. It is also the only thing that posts to Slack. → scheduler/

---

How it fits together

The reusable workflows live here; each consuming repo keeps a thin caller workflow that owns its secrets and passes the path to its own profile. Because this repo is public, the reusable workflows check out their own script code with no token. Because the profile lives in the caller repo, each reusable workflow checks out two things: the caller repo (for the profile) and this repo (for the scripts).

your-repo                              the-gitfather (this repo, public)
  pg-backup/myproject.yaml   ─────►     scripts/ + dashboard/
  .github/workflows/                   .github/workflows/  (reusable)
    pg-backup.yml ───────────uses────────►  pg-backup.yml
    pg-durable-verify.yml ───uses────────►  pg-durable-verify.yml   (daily; supersedes the weekly drill)
    pg-restore-drill.yml ────uses────────►  pg-restore-drill.yml    (optional once durable-verify is wired)
    pg-dashboard.yml ────────uses────────►  pg-dashboard.yml
    pg-archive.yml ──────────uses────────►  pg-archive.yml          (optional, weekly)

Scheduling, the watchdog and Slack live in the Cloudflare Worker. The caller workflows carry no schedule: cron. One free Worker:

• dispatches every client's backups and verifies on a punctual cadence;
• runs the staleness watchdog natively every 10 minutes against each client's bucket;
• is the only thing that posts to Slack.

Each run leaves an outcome record in its bucket, and its last step tells the Worker it finished (POST /notify, authenticated by the run's GitHub OIDC token). The backup job also publishes its profile's staleness: block to the bucket, so config flows GitHub → Cloudflare, never the reverse. See scheduler/README.md.

---

Where this fits: 3-2-1-1-0

The tool nails the back half (1-1-0) and supplies one off-site copy toward the front half — it does not by itself give you 3 independent copies on 2 distinct media. To earn the full 3-2-1-1-0, add a second, independent backup leg on a different medium / failure domain (e.g. a periodic pg_dump to local disk/NAS, or replicate the R2 bucket to another provider or region). The dump this tool already produces is the natural feed for it.

Digit by digit, and the threats each part is and isn't good for: Where this fits: 3-2-1-1-0 and threat model.

---

Quick start

The fastest path is to hand docs/setting-up-gitfather.md to an AI coding agent — a guided walkthrough that interviews you for each value, writes the profile + secrets, and finishes with a green doctor check.

By hand, in order:

1. Create the R2 bucket, lifecycle rules and locks (once, with account-level Cloudflare creds), and mint the scoped CI token. → R2 setup
2. Copy profiles/example.yaml into your repo as pg-backup/<name>.yaml and edit it. → Profile reference
3. Add the caller workflows — backup, dashboard, durable-verify, and optionally archive. → Wiring a consuming repo
4. Set the secrets and variables in your repo; secrets must be passed explicitly, not with secrets: inherit. → Secrets and variables
5. Add the client to the Worker's roster, with an R2 binding to its bucket and a slack block naming its channel, and invite the the-gitfather Slack app to that channel. → ROSTER
6. Preflight: npm ci && PROFILE=pg-backup/<name>.yaml npm run doctor -- all — read-only, safe against production creds. → doctor
7. Fire the backup once by workflow_dispatch and confirm three things: an object lands under <backup-prefix>/intraday/, a run appears in _log/, and the day's Slack row gets a 🖐️ ✅ tick. Then check that the dashboard renders.

---

Repository layout

the-gitfather/
  scripts/
    backup-pg-to-r2.ts        # dump → (encrypt) → upload intraday/ → promote to daily/weekly/monthly; records a SHA-256
    verify-durable-pg.ts      # DAILY: hash-check each durable object + restore freshest daily + re-restore aged weekly/monthly
    restore-drill-pg.ts       # pull newest → restore into a throwaway → assert row counts (exports drillObject)
    archive-table.ts          # move aged rows out of Postgres into per-ISO-week objects; prune, gated
    backfill-archive-sizes.ts # maintenance: fill missing object sizes into the archive index
    build-dashboard.ts        # render the static backup-history dashboard from the R2 run-logs
    doctor.ts                 # read-only preflight: same config schema + live client probes
    roll-r2-token.ts          # escrow a rolled/minted R2 token: verify → 1Password → GitHub secrets
    key-shares.ts             # 2-of-3 key escrow: split the age identities into SLIP-39 holder cards; recover
    recovery-kit.ts           # build / upload / verify the offline recovery kit stored beside the backups
    profile-export.ts         # emit the few profile values a CI bash step needs, as KEY=value
    runlog.ts                 # append-only run/verification log in R2 (the dashboard's source of truth)
    lib/                      # shared internals: config + profile schema, R2/Postgres clients, the outcome
                              #   record, scheduling, tier maths, log store, preflight probes (all .ts)
    __tests__/                # unit + bash-parity tests (node:test via tsx)
  dashboard/                  # the static page: template.html + heatmap.ts (SVG renderer) + theme.ts
  scheduler/                  # the Cloudflare Worker: schedules every client's workflows, runs the staleness watchdog, posts to Slack
  profiles/example.yaml       # copy this into YOUR repo and edit
  recovery-kit/               # the kit's pinned manifest, its build-tools.sh, and the offline drill
  docs/                       # the documentation linked below
  .github/
    workflows/                # REUSABLE (on: workflow_call): pg-backup, pg-durable-verify,
                              #   pg-restore-drill, pg-dashboard, pg-archive
    actions/setup-tools       # composite: pinned rclone (+ optional pg client)

Built as a GitHub-Actions toolkit (TypeScript run via tsx), but every script is runnable locally for testing.

---

Documentation

Document	What's in it
Setting up the-gitfather	The LLM-guided walkthrough: an interview, the files it writes, and a green doctor
Wiring a consuming repo	Every caller workflow, and the secrets/variables table
R2 buckets, retention, locks and tokens	The GFS tiers, the wrangler setup, and rotating the CI token
Verifying backups and restoring for real	The three integrity checkpoints, and the DR restore recipe
Archiving a table out of Postgres	The optional archiver: weeks, prune gating, keys, backfills
Key escrow: 2-of-3 recovery cards	Splitting the decrypt keys among three holders, the ceremony, practice drills, recovery
Recovery kit guide	Runbooks: a full restore, and the monthly check; keeping the kit current
The recovery kit: how it works	What the kit holds and why, pinning, storage and locking, the offline drill
Backup-history dashboard	What every cell means, what is published, how it's built
Slack and alerting	How a run reaches Slack, what posts where, the daily row, the failure webhook, the dead-man's-switch
Configuration, doctor, and troubleshooting	Profile reference, validation, preflight, local runs, symptoms
Where this fits: 3-2-1-1-0	The honest mapping, and the threat model
scheduler/README.md	The Cloudflare Worker: scheduler, staleness watchdog, Slack delivery, roster, cutover
profiles/example.yaml	The annotated profile — every knob, with its default

---

Development

npm ci
npm test          # unit + bash-parity tests (node:test via tsx). Needs rclone on PATH:
                  # the run-log tests drive the real append path through a `local` remote.
npm run typecheck # tsc --noEmit
npm run lint      # eslint

npm ci also points core.hooksPath at .githooks (the prepare script), so a pre-commit hook lints staged JS/TS and a lint error blocks the commit.

Every script runs locally against real credentials — see Running locally.

---

License

MIT — see LICENSE.