Phase 1.5 Batch 3 (M6): keep it running, back it up, get it back - #369
Merged
Conversation
Two closely related holes, both of the same shape: a rule that was true when it was written and had no way of staying true. **Every route is behind the gate, and now a test can say so (#275).** The gate was already one `onRequest` hook with an allow-list, which is the right shape, but every test of it named the routes it checked — so a route added in M7 could be public by oversight and nothing would fail. `buildApp` now records each route as Fastify registers it, on `app.lidRoutes`, and `coverage.test.ts` walks that table and asserts every entry is either gated or an exact member of the exported `PUBLIC_PATHS`. Nothing in a request path reads the table; it exists so a test can ask the running app what it actually serves. The hook's placement is load-bearing rather than tidy. `app.register()` defers until `ready()`, but the `app.get()` calls in `buildApp` run immediately, so an `onRoute` hook added by a test after `buildApp()` returns catches the nine route plugins and silently misses `/health` and all four static assets. Installing it before any registration is what makes the coverage real — a route-coverage test with a hole in it is worse than none, because it reports coverage it does not have. That was found by writing the test the wrong way first and mutating a route to be public: it stayed green. **`Cache-Control: private, no-store` is now one hook (#216).** Three route files set the header and four did not, and the four that did not were the oldest — the M1-M3 HTML pages. Per-route privacy headers always decay that way. An `onSend` hook sets the private default on anything that has not deliberately opted out, including the 404 page (which renders the rail, and the rail carries counts) and the 401 from `rejectUnauthenticated`. `hasHeader` rather than an unconditional set, because two responses have a different right answer: the sign-in pages send plain `no-store` (nothing about them is private *to* anyone; the point is that no cache holds a form with the previous email in it), and `/app.css` and the three scripts send `no-cache` — they are on the public allow-list, say nothing about the archive they style, and revalidating is what makes editing CSS and reloading feel instant. The six route files lose the header lines, not the reasoning: `media.ts` keeps the paragraph that first wrote the rule down, and now says why stating it per route is what left the older pages without it. No markup or CSS changed, so there is no render loop to report. Generated with Claude Code
Three things the archive did not have: a way to stay up, a copy, and a rehearsed way to return from that copy. Photo bytes are the only unrecoverable asset in this project and until now there was exactly one of them. **The server as a launchd LaunchAgent (#215).** `scripts/install-service.sh install|uninstall|status|restart`, writing `~/Library/LaunchAgents/`. It refuses rather than fights: if something is already listening on the port and there is no plist, it says which command to stop it with, because a `KeepAlive` job that cannot bind retries every ten seconds forever while looking installed. It also checks `data/owner.json` up front — the server's refusal to boot without credentials is correct, but far more legible now than in a log being rewritten every ten seconds. `bootstrap`/`bootout` and not the deprecated `load`/`unload`, because the modern verbs report a real exit status. Limitation, stated in `docs/OPERATIONS.md` and not worked around: **a LaunchAgent starts at login, not at boot.** After a reboot the archive is down until someone logs in. A LaunchDaemon would start at boot but runs as root with no login keychain, which is a larger change than #215 asked for. **A backup with two assets and two methods (#217).** `npm run backup`, into `LID_BACKUP_ROOT`, which has no default because a backup that quietly lands on the same disk as the thing it backs up looks like protection and is not. The database goes through SQLite's Online Backup API, never `cp` — a filesystem copy of a live WAL database can open cleanly and still be wrong. The photographs are content-addressed and immutable, so copy-if-missing is incremental by nature: no generations, no rotation, no rsync. Every run then opens the snapshot it just wrote, runs `integrity_check`, and compares four table counts, so "it produced a file" is never mistaken for "it produced a database". **Media is copied twice, before and after the snapshot, and that is the part the ticket does not ask for.** `ingest/index.ts` writes photo bytes (line 132) before committing the row that references them (line 138), so a photo arriving mid-backup would be a row in the snapshot pointing at a file the backup does not have — a restore with a broken image on a real day. The window is not hypothetical: backing up while a 60-photo ingest ran caught **five originals and two derivatives in the second pass**, and a restore from that snapshot found **112 of 112 photo files present**. Pass one alone would have left five dangling. **Restore, which is the only thing that proves any of the above (#218).** `npm run restore` assembles a copy and then checks four things. Three are cheap; the fourth is the one that matters — it walks every live `media_asset` and looks for its file on disk, because a restore can pass `integrity_check`, pass `foreign_key_check`, hold every row and still show grey boxes. It names the day with the most photos so that opening one page proves database, originals and derivatives all arrived together. **It refuses to write where an archive already exists and there is no `--force`**, because a restore that can overwrite is one typo away from replacing the real archive with a copy that is hours old. Verified against throwaway archives under `/tmp`, never the real `data/`. A backup taken while the app was serving under load: integrity ok, all four counts matched, app healthy after. 12/12 originals where filename = sha256 of content = source = restored, and 12 derivatives byte-identical under `cmp`. The scheduled job fired unattended off `StartCalendarInterval` with empty stderr. The restored copy, run on another port, rendered `/day/2026-08-29` with **100 images and 100/100 photo URLs returning 200**. Three bugs the rehearsal caught, all of which a reading would not have: - Derivatives got only one copy pass, so a photo ingested mid-backup had its original safe and its thumbnail missing. #217 names originals, but #218 asks a restored copy to *show* a day, and every photo on the Calendar and Day pages is a derivative — with no regenerate command in v0.1, skipping them restores grey boxes. - Drift was reported backwards: "the snapshot is behind by source_item 23 -> 24" about a snapshot one row *ahead*, because live counts were read before the snapshot rather than after. - Snapshots were left in WAL mode, so each one was three files while `restore.ts` copies only the `.sqlite`. Here the `-wal` was empty, but a populated one would have been silent data loss. Snapshots are now checkpointed and converted to `journal_mode=DELETE`, which costs nothing on restore because the app sets WAL on every open. `scripts/backup.test.ts` pins the ordering deterministically: five tests, all mutation-checked. Two mutations were MISSED and both are recorded rather than papered over. Reversing the count order still passed — `betweenPasses` fires before the database is opened, so a test built on it cannot see that bug; the vacuous test was deleted and replaced with a comment saying why. Removing the sidecar cleanup also changed nothing, because the `DELETE` conversion already removes them — and in the one case it would not have been dead code (the pragma failing) unlinking a populated `-wal` would have destroyed exactly what the backup exists to preserve. A cleanup that is redundant when things work and destructive when they don't is the wrong shape, so it now refuses instead. `data/owner.json` is deliberately not in the backup: it holds the password hash and the HMAC secret that signs every session, so including it would make a copy of the backup a way to sign in. Restore runs `set-password` instead, which rotates the session secret as a side effect. `docs/OPERATIONS.md` has the procedures and a "what is not covered" section: off-host copies, retention (snapshots accumulate; nothing prunes), Trash purge, and the boot-versus-login caveat. No markup or CSS changed, so there is no render loop to report. Generated with Claude Code
Three documents that were written in this tree and never committed, plus the `.mcp.json` that configures the browser-control server the render loop uses. - `docs/HANDOVER-MILESTONE-EXECUTION-2026-08-21-1829.md` — the M7-M19 ticketing and M0.1 session. - `docs/M0.1-HOST-INVENTORY.md` — #316's read-only reconnaissance of the shared Hetzner host: two other resident products, no Docker runtime at all, zero swap, and one shared tunnel ingress file. It supersedes several assumptions in `reference/HETZNER-SHARED-HOST-DEPLOYMENT-SPIKE.md`. - `docs/HANDOVER-PHASE-1.5-BATCH-3-2026-08-29-1159.md` — the handover this batch was built from. **This repository is public, so the shared host's IP address and the filename of the SSH key that reaches it are not in these files.** The handover now says to read the `brain` alias out of `~/.ssh/config` instead. That user has passwordless `sudo ALL:ALL`, which is precisely why its address does not belong in a public document alongside that fact. Separately and not fixed here, because it needs the owner: the same IP is already committed in `docs/IMPLEMENTATION-PLAN-POST-M6.md` on `main`. Rewriting `main` is forbidden, so removing it from history is not something to do quietly — the options are host-side. `.mcp.json` pins `chrome-devtools-mcp@1.1.1 --isolated` and contains no credentials or paths. Generated with Claude Code
This was referenced Aug 29, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch 3 is M6 (#215, #216, #217, #218) plus #275. Stacked on #368 so this PR
shows only Batch 3's three commits — GitHub will retarget it to
mainwhen #368merges.
Until now the archive had no way to stay up, no copy, and no rehearsed way back.
Photo bytes are the only unrecoverable asset in this project and there was
exactly one of them.
What is here
scripts/install-service.sh install|uninstall|status|restart.Cache-Control: private, no-storeas oneonSendhook rather thana line in each route, which is what left the M1-M3 HTML pages without it.
npm run backup: SQLite's Online Backup API for the database,copy-if-missing for the content-addressed photos, then verify what it wrote.
npm run restore: assemble a copy, check four things, print how torun it.
route is gated instead of naming the routes it happens to think of.
docs/OPERATIONS.md— the procedure, written for the person doing this at anunhelpful hour, including what is not covered.
Evidence
Everything below ran against throwaway archives under
/tmp, never the realdata/. The owner's instance stayed up on port 3000 and answered 200 before andafter every step.
ok, all four table countsmatched, app healthy after.
derivatives byte-identical under
cmp.StartCalendarInterval, empty stderr./day/2026-08-29with 100images and 100/100 photo URLs returning 200.
derivatives in the second media pass, and the restore found 112 of 112 photo
files present. Pass one alone would have left five dangling.
tsc --noEmitclean.Things worth arguing with
not ask for this. Ingest writes photo bytes before committing the row that
references them, so a photo arriving mid-backup is otherwise a row in the
snapshot pointing at a file the backup does not have.
originals, because those cannot be recreated. But Documented, rehearsed restore procedure #218 asks a restored copy to
show a day, every photo on the Calendar and Day pages is a derivative, and
there is no regenerate command in v0.1.
path/volume on the host at minimum", so the same device is a documented
compromise, and the owner is the one who knows whether
/Volumes/...is reallyanother disk.
--force. A restore that can overwrite is one typo awayfrom replacing the real archive with a copy that is hours old.
data/owner.jsonis not in the backup, so a copy of the backup is not away to sign in. Restore runs
set-password, which rotates the session secret.Limitations, not worked around
down until someone logs in, so Production build + process supervision #215's third criterion is not verifiable without
one. A LaunchDaemon would start at boot but runs as root with no login
keychain — a bigger change than the ticket asked for.
installwas not run on the real port 3000, because that would meanstopping the owner's hand-started server. That is the owner's call.
outside the host" is the Hetzner half; the app runs on the Mac.
report.
One thing for the owner, unrelated to the code
This repository is public and the shared Hetzner host's IP address is already
committed in
docs/IMPLEMENTATION-PLAN-POST-M6.mdonmain, next to the factthat its login has passwordless root. It has been kept out of the three documents
this PR adds, but rewriting
mainis forbidden, so removing it from history isnot something to do quietly. The fix is host-side.
🤖 Generated with Claude Code