Ops: email operator error alerts to support_filemill@keywind.cc #7

Backlog: Observability & ops

Problem

When something goes wrong (systemic job failure, delivery failure, panic, a stuck worker loop, a crash), nobody is notified. An unattended background service fails silently.

Proposed solution

Email operator alerts to support_filemill@keywind.cc (replaces the earlier support@keywind.cc and support@mill.keywind.cc). It's deliberately on keywind.cc, whose mail goes through Cloudflare Email Routing, not mill.keywind.cc, which is Mailgun's receiving domain. So each alert is exactly one Mailgun send and never re-enters Mailgun's inbound routing. No extra setup: keywind.cc's Cloudflare catch-all rule already delivers it. Requirement: one test send before relying on it (also shows whether alerts land in spam). The full design is in ERROR-ALERTING-PLAN.md, revised 2026-09-11 against the current code (supervisor, boot start, retention sweeps, sheets-link delivery).

Key points:

• New leaf package internal/alert: Reporter interface (no-op default), Mailer (satisfied by mailgun.Service.SendAlert) and Ledger (satisfied by store.Store). A package of its own because mailgun imports app and both need to report.
• Alert only on systemic failures. Split the two kinds of "job failed" in App.execute: a valid result.json with success:false is the sender's problem and never alerts. A timeout, crash, missing/invalid result, missing transformer, or success:true with a nonzero exit is systemic.
• Mandatory throttle, persisted in SQLite: 15-minute cooldown per category, global caps of 10 per hour and 20 per day (rolling 24h), suppressed counts in the next email. The daily cap is set by the Mailgun Free plan: 100 sends a day, shared with replies, and Mailgun rejects further sends past that. The hourly cap alone would allow 240 a day and lock out replies. Reaching the cap sends one final notice, which counts within the 20. Persisted because the supervisor restarts a crashing worker every ≤120s, and an in-memory throttle would reset each time.
• Crashes are reported by the next process. The supervisor passes FILEMILL_PREVIOUS_EXIT / FILEMILL_RAPID_RESTARTS to the restarted worker, which sends one restart alert, folded together with the count of jobs store.Open marks interrupted.
• Report never blocks: a buffered queue drained by a goroutine. A failed alert send is logged and dropped, never re-reported.

Alert sites

Intake 500s · wrong route (store(notify=)) · systemic job failures · job panics · reply send failing ≥5 min · MarkEmailDelivered failing after a successful send (immediate; resends every second, see #6) · sheets-link publish failures · orphaned Drive files · job-claim errors ≥1 min · both retention sweeps · restart after crash.

Deliberately not alerting: 401/400 webhook noise, unrouted/disallowed/benign mail.

What this can't cover

Startup fatal()s (no reporter yet), a failing Mailgun send (it's the alert channel), and a machine that's off. These are the heartbeat's job, #5.

Phases (one PR each; tests with fakes written first)

1. internal/alert core: throttle, queue, persisted Ledger. No behavior change.
2. Job taxonomy split in execute, plus panic recovery.
3. Mailgun sites and main.go wiring. send takes its subject verbatim; add SendAlert and alert_recipient config. Alerting goes live.
4. Crash reporting across restarts: supervisor env vars and the interrupted-job count; recover in the delivery and sweep loops.
5. Live verification against support_filemill@keywind.cc (after a test send).

About 3 days in total.

Decisions (defaults)

• Send from REPLY_FROM.
• Include the last 2 KB of transformer output.
• Heartbeat stays in Reliability: external heartbeat / dead-man's-switch for total-down detection #5.
• Do Reliability: at-least-once delivery can send a duplicate reply (#6) #6's retry cap before or alongside phase 3.

Config

alert_recipient: support_filemill@keywind.cc   # in the gitignored config/email.yaml; empty = disabled
# alert_cooldown_minutes: 15
# alert_max_per_hour: 10
# alert_max_per_day: 20      # Mailgun Free plan budget; revisit on a paid plan