[ergofobe]

Closes #107.

The three chores were enqueued sixty seconds after boot and nothing enqueued them ever
again. On a server that stays up the trash was swept once, abandoned uploads were never
pruned, and reclaimStale — whose only recurring caller was the prune chore — never ran
again, so a worker that died mid-job left its row running for good.

What drives the cadence

A timer, not the chores. startMaintenance schedules a tick, and each tick

1. calls queue.reclaimStale(), then
2. asks for each of the three chores with enqueueUnique.

A self-chaining chore was tried in #92's PR and rejected, for a reason that still holds: a
chain lives in the queue and so can be broken by the queue, and the prune chore is the one
chain with no rescuer, because the thing that would rescue it is the thing it runs. A
death between its self-enqueue and its done write forks it permanently.

What rescues the rescuer: nothing has to. reclaimStale moves out of PRUNE_JOBS_JOB
and into the tick, so the recovery no longer depends on a job the queue has to keep alive.
A timer is outside the queue: no worker can strand it, and whatever kills the process that
owns it is also what starts the process that replaces it. The prune chore keeps only the
housekeeping — pruneCompleted and expired sessions — where losing an hour costs nothing.

When a chore throws

Deliberately: the successor is never the failing chore's responsibility, so a failure
cannot end the cadence. The queue's own backoff is the next run, and a tick declines to
add a second while a chore is queued or running. Once a chore has spent every attempt and
gone failed it is no longer pending, so the next tick simply starts it over. A sweep that
fails does not stop sweeping — that being #107 in miniature.

enqueueUnique is safe here in a way it was not for a chain, because a tick is not a
chain. Nothing is carried forward when a tick declines; the next one asks again.

What review turned up, and what it cost

Making reclaimStale recur means that for the first time it can meet a job that is
legitimately still working. started_at was stamped once at claim, so it dated the job
rather than its last sign of life, and anything slower than the fifteen-minute window — the
190 MB model download, a 4K transcode — would have been handed to a second worker while the
first was still inside it. Three commits deal with the consequences:

• A running job touches started_at every minute, so the window means silence, not age —
  which is what reclaimStale already said it meant. Written with Postgres's clock, and
  the cutoff it is compared against moves to Postgres's clock too.
• The beat is bounded (six hours). A heartbeat proves the process is alive, not the
  job; a fetch with no timeout would otherwise defend its row for ever and hold the name
  enqueueUnique keys on — The maintenance chores run once per boot and never again, so the trash is never swept #107 again, moved inside the heartbeat.
• Once the bound lets go of a hung job, two workers can hold one row, so beat, done and
  fail are all scoped to the attempt they belong to. A worker returning late no longer
  erases the run that replaced it.

stop() also waits for a tick already under way, so shutdown does not close the pool under
an open transaction.

Tests

All six cadence tests were written first and watched fail against the old behaviour (five
failed; the sixth — that a tick queues all three — is the one thing the boot-time enqueue
did do). The queue tests were written the same way.

• a chore runs, and another is scheduled
• a chore throws, and another is still scheduled — both while it is backing off (no
  duplicate) and after it has exhausted every attempt (a fresh one)
• a chore a dead worker left running is freed by a tick, with no help from the queue
• nothing is added beside a chore still queued or in flight
• the cadence goes on ticking until it is stopped
• a slow job is left alone while it is making progress, and let go of once past its lifetime
• a worker that returns late leaves the run that replaced it alone

Everything asserts on what the queue holds, never on a timer firing.

Verify

bun install && bun run verify && bun run --filter '@imogen/web' build — 963 pass, 0 fail,
web bundle built. No bun.lock diff. SDK pin untouched (v0.5.1).

Found, not fixed

• sweepTrash and pruneUploads are capped at 500 rows per run and neither asks for
  another pass when it fills the cap. With a strictly hourly cadence the cap is now the
  throughput ceiling: a ten-thousand-item trash backlog takes about twenty hours to
  reclaim, and nothing says so. Pre-existing, but the timer is what makes it visible.
• A tick's three enqueueUnique calls are sequential and unguarded, so a transient
  failure on the first costs the hour's other two. The reclaim has its own catch; the
  chores do not. Self-heals next hour.
• maintenance.stop() waits on an in-flight tick with no timeout, so a SIGTERM
  arriving while a tick is blocked on a wedged pool waits out the orchestrator's grace
  period instead of exiting.
• A server in a crash loop with an uptime under FIRST_TICK_DELAY_MS never ticks. The
  chores used to be rows a dead process left behind for the next boot; now they are not.
  The reclaim half is still covered by the boot-time reclaimStale in index.ts.

Review

code-review at medium, four rounds — one past the three-round cap, which I should flag.
Rounds 1–3 were fixed as found. Round 4 raised two more, both one-line corrections to the
commit round 3 had just produced (the heartbeat's clock, and the terminal-write ownership
guard) rather than a new design, so they were taken and the loop stopped there. Nothing was
redesigned twice. The four items above are what is left, and they are listed rather than
fixed because none of them blocks this change.

🤖 Generated with Claude Code