[ergofobe]

Closes #111. Closes #112. Both are blockers already on main from #108, found by an
independent confirming review of it. latest tracks main, so both are live for anyone
tracking it.

They are one change: both are consequences of the heartbeat, both live in jobs/queue.ts
(plus one line of index.ts), and the two significants from the same review are in the
same file and the same machinery.

What changed

#111 — the boot reclaim. The beat rewrites started_at every minute, so a crash
leaves rows whose last sign of life is under a minute old, and the fifteen-minute window
matched none of them. reclaimAllRunning() says what boot means: every running row, with
no clock in the query at all. Crash recovery goes back to instant from about 61 minutes,
and the comment above the call is true again.

#112 — the wedged worker. The lifetime bounded the row and not the worker, so a hung
handler kept its slot for ever and the released row let a second worker come and hang on
the same job — one slot per cycle until the pool was gone. The lifetime now races the
handler, and a beat that matches no row ends the wait too.

Significant: the retry backoff asks Postgres for the time. It wrote run_at from the
app's clock while claim compares it against the database's — the skew #108 moved the
reclaim cutoff to Postgres to remove, relocated rather than removed.

Significant: a write that matches nothing is no longer discarded. done, fail and
beat all filtered on the attempt and none looked at the result. The guard is now the row,
the attempt and status = 'running' — the attempt alone could not tell "somebody else has
it" from "nobody has it yet", because reclaimStale requeues without touching the attempt
count, so a late done marked a requeued job finished. Zero rows is logged, and a beat that
matches nothing stops the worker waiting, since the job is somebody else's by then.

What abandoning a handler does, and does not, do

A promise cannot be cancelled. Abandonment is the worker leaving, and nothing else:

• The handler goes on running until it returns or the process ends. It still holds its
  memory, its file handles and any child process it spawned.
• Nothing it writes afterwards lands: every write is scoped to the row, the attempt and
  running, and by then the row is failed. That is the reasoning Give the maintenance chores a cadence, and a rescuer the queue cannot strand #108 relied on, and it
  needed the running predicate to actually hold.
• The recovered thing is the worker slot, which is the part that was never coming back
  on its own. That is the invariant: a slot is always recoverable without restarting the
  process.

An abandoned job is failed, not requeued — the one place the queue gives up on a job it
could retry. Everywhere else a retry costs another run; here the run it would replace is
still going, so five attempts at a handler that hangs is five live copies of it. A job that
hung past its whole lifetime is for a person to look at.

Two edges that follow from that, neither hidden:

• A name that schedules itself — the maintenance chores, the walks — is no longer pending
  once its row is failed, so the next tick asks for it and a fresh copy runs beside the
  hung one. The alternative is a chore that stops happening, which was The maintenance chores run once per boot and never again, so the trash is never swept #107.
• An abandoned job lands in failed, which is the set the bulk "retry failed" sweeps back
  to queued. An operator clicking it while a hung handler is still live re-creates the
  second copy. Same family as Admin retry rewinds a running job's attempt count, so a worker can write over another worker's run #115.

Handler-side deadlines — the model fetch, the two Bun.spawns — are what would stop an
orphan holding real resources, and they are #116. The queue-level recovery does not depend
on them, and the next unbounded await would reintroduce #112 without it.

Tests

Written first, watched fail (6 red against main, two of them by timing out at five
seconds, which is the pool death itself):

• a boot reclaim that frees a row whose final beat was thirty seconds old, next to the
  hourly window that frees nothing;
• a pool of two, both workers inside handlers that never return, and an ordinary job that
  has to run anyway;
• a handler past its lifetime: the worker comes back, the handler has not, the row is
  failed with attempts to spare;
• a worker returning after the reclaim requeued its row, which must not mark it done;
• a worker whose job was given away mid-flight, which stops waiting and writes nothing.

bun install && bun run verify && bun run --filter '@imogen/web' build: 968 pass, 0 fail,
web bundle built. No bun.lock diff and the SDK pin is untouched.

Review

Three rounds at medium, which is the cap. Round 1: three mediums — the boot reclaim's
single-instance premise (now stated where it is relied on), the abandonment requeueing a
hung handler (now failed), and a write on a row already given away (now skipped). Round 2:
documentation that had stopped matching the code. Round 3 found one that mattered — the
abandonment write sat inside the handler's try, so a database blip under it fell into
fail and requeued the job the branch had just decided must never be retried — plus a beat
that could warn "taken away" on an ordinary success.

Round 3 was not clean, so this is not self-merged. Its findings are fixed in b321a29 and
verify is green, but the cap says three rounds and a fourth is exactly what went wrong in
#108. It wants a human's eye before it merges.

Found, not fixed

• Admin retry rewinds a running job's attempt count, so a worker can write over another worker's run #115 — admin retry rewinds a running job's attempt count, so a worker can write over
  another worker's run. It is what keeps the guard here evidence of ownership rather than
  proof of it, and the fix belongs in the admin service, not the queue.
• No handler-side deadlines: the model fetch and both ffmpeg/heif spawns can hang for ever #116 — no handler-side deadlines: the face-model fetch and both spawns can hang for ever.

Left as nits: reclaimStale's predicate has no supporting index now that it runs hourly,
and stop() still waits up to a whole lifetime for a wedged handler at shutdown.

🤖 Generated with Claude Code