Skip to content

fix(worker): add an alarm backstop to the registry sweeper - #30

Merged
crcatala merged 2 commits into
mainfrom
fix/alarm-backstops
Oct 27, 2026
Merged

crcatala merged 2 commits into
mainfrom
fix/alarm-backstops

Conversation

@crcatala-vps

@crcatala-vps crcatala-vps commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

flashlink's registry runs a "sweeper" on a Durable Object alarm: a timer that wakes it to delete expired files. Each run is a billed request plus a few storage writes. If a bug ever left something due "now", the sweeper would re-run back to back with nobody using flashlink. That's the kind of loop behind the well-known $10k–$34k Durable Object bills.

What changes:

  • Sweeper runs are now at least a minute apart. A looping sweeper costs at most 1,440 runs a day, about 1.5% of the free plan's daily requests (cents a month on Paid).
  • One place schedules the alarm, and a CI check fails if code anywhere else sets it.

Nothing changes for users. Expired links are already refused on every download, so the sweeper only frees storage, days after expiry by default. A minute's delay is invisible.

As decided, there's no kill switch: with a ceiling this low, deploying a fix is enough.

Technical details

  • packages/worker/src/registry.ts:
    • MIN_ALARM_GAP_MS = 60_000 and lastWakeAt (in memory, set at the start of alarm()).
    • ensureAlarm() and the sweeper's own reschedule are merged into scheduleAlarm(fired = false), which uses max(nextDue, lastWakeAt + gap). It keeps an earlier pending alarm unless called from the sweeper, whose alarm was just used.
    • In-memory state is enough: a loop can only come from the sweeper rescheduling itself, which happens in the run that records the time. An object evicted between runs starts from 0, which only lets an (already rate-limited) API call schedule sooner.
  • Gap choice (60 s, not the playbook's 2 s):
    • fastest deadlines: a link's purge after its grace period (default 7 days, minimum 1 s via PURGE_GRACE_SECONDS) and reaping abandoned uploads (15 min). None is user-facing: resolve enforces expiry and caps on every read;
    • 2 s would allow 43,000 runs a day for no benefit.
  • test/tooling/alarmCalls.test.ts:
    • a TypeScript AST scan of src/; only registry.ts#scheduleAlarm may call setAlarm;
    • workerd can't read source files, so it runs in node through vitest.tooling.config.ts, typechecked by the new tsconfig.node.json (Node types), and is excluded from tsconfig.json and the Workers vitest config;
    • test and typecheck scripts run both;
    • @types/node and typescript were added as worker devDependencies, at the versions already in the workspace.
  • test/helpers.ts resetState also clears lastWakeAt, since one registry instance serves every test.
  • Docs: docs/ARCHITECTURE.md gets a new "Alarm backstops" subsection under "Cost model", with the decision and the reason for no kill switch.

Test plan

  • pnpm verify (format, typecheck, tests: worker 101 + 2 tooling, was 98; core 26; cli 364; build)

  • test/alarms.test.ts, with Date faked inside the Durable Object and set a year ahead so real alarms never fire:

    • a looping sweeper (nextDue stubbed to "now") runs exactly 11 times in 10 minutes, each ≥60 s apart;
    • two links due 10 s apart are purged one minute apart, both within the gap;
    • a lone link's purge is scheduled exactly at expiry + grace.
  • Each guardrail reverted in turn, and a test fails each time:

    Reverted Failing test
    the gap looping, links due close together
    gap 61 s looping, links due close together
    gap +1 ms all three
    a direct setAlarm in alarm() the tooling scan
  • On the live deploy: upload a file with --ttl 1m, and with PURGE_GRACE_SECONDS small on a test deploy, check that wrangler tail shows one sweeper run per due cleanup and none while idle.

Review follow-up

Independent review (Grok, DeepSeek), fixed in 6c44128:

  • The live check expected a near-instant purge: verify-deployment.mjs now waits up to 150 s (was 90 s), and VERIFY_DEPLOYMENT.md and the old measurement in ARCHITECTURE.md say a purge can come up to a minute late.
  • The 1,440/day ceiling now has its qualifier: after an eviction, each API call can bring a sweep forward once.
  • New test: an API call right after a sweep still schedules a minute after it (fails without the gap on that path).
  • Test fix: the grace period is read from the config instead of hardcoded.

Worker tests: 102 + 2.

🤖 Generated with Claude Code

crcatala-vps and others added 2 commits October 27, 2026 15:53
The sweeper alarm is the one cost that needs no visitor: a bug that left
something due "now" would re-run it back to back, each run billed. Following
the studio playbook's alarm guardrails:

- MIN_ALARM_GAP_MS (60 s): the sweeper never runs within a minute of its last
  run, so a loop costs at most 1,440 runs a day. Nothing needs it sooner:
  expiry is enforced on read, so the sweeper only reclaims storage.
- ensureAlarm and the sweeper's own reschedule are merged into scheduleAlarm,
  the only place that sets the alarm; test/tooling/alarmCalls.test.ts (node,
  vitest.tooling.config.ts + tsconfig.node.json) fails CI otherwise.
- No kill switch, by decision: the ceiling is low enough.

Tests in test/alarms.test.ts; docs in ARCHITECTURE.md "Alarm backstops".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Christian Catalan <crcatala@gmail.com>
From review of #30:
- verify-deployment.mjs waits up to 150 s (was 90 s) for the sweeper, since a
  purge can now come up to a minute after it's due; VERIFY_DEPLOYMENT.md and
  the historical measurement in ARCHITECTURE.md say so.
- ARCHITECTURE.md: the 1,440/day ceiling is for the sweeper rescheduling
  itself; after an eviction each API call can bring a run forward once.
- Tests: an API call right after a sweep still schedules a minute after it;
  grace period read from the config instead of hardcoded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Christian Catalan <crcatala@gmail.com>
@crcatala
crcatala merged commit ea02441 into main Oct 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants