service-queue: a job whose worker dies mid-handler stays running forever — leaseMs promises a reclaim nothing performs

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

Filing gate ①: a product defect with a named locus. Found out of scope by the #29919 card thread (os-dev-report 6648028230, class b) and re-read by the seat at origin/main dfafae81. No domain:* label is set: triage routes it.

Defect

packages/services/service-queue/src/db-queue-adapter.ts:

DbQueueAdapterOptions.leaseMs (:158) is documented as "Lease duration before another worker may reclaim (ms, default 30000)".
claimBatch selects candidates with status: 'pending' only (:528) and sets a claimed row to status: 'running' with locked_until = now + leaseMs (:577–579). The lease check at :568 reads locked_until on a pending row, where every write back to pending has already cleared it.
Every write that returns a row to pending (:625, :641, :655, :675) runs inside the live worker after the handler settles. No reader in service-queue returns a running row whose locked_until has passed.

So a worker that dies mid-handler (process crash, OOM, a redeploy) leaves its row running for good: it is never retried and never dead-lettered, and the lease the option documents is never enforced. Not reproduced end-to-end; the reading is the code path above.

Business reason

Who is hurt: any deployment that runs scheduled or background jobs through QueueServicePlugin's database adapter (the default durable queue). A job lost to a crash or a deploy silently never runs, with nothing in the queue telling an operator. If not done: the documented leaseMs stays a promise the runtime does not keep (declared, not enforced).

Acceptance

A running row whose locked_until has passed is claimable again (or moved to the retry path with its attempt counted), within one idle tick of the lease running out.
A live worker that is still inside its lease is never double-claimed.
Pins: a crashed-worker fixture (row left running with an expired lease) is reclaimed; a row inside its lease is not. The nextDueAt read core DispatchLoop: an idle loop re-runs its fast backoff ramp after every boot, so a fresh tenant kernel spends 30 statements in its first minute on empty delivery outboxes and job queues #29919 adds (PR fix(core): an idle DispatchLoop wakes at the next due time, and a sole writer skips the cold-start ramp #30225) also counts the earliest lease expiry, so a sole-writer loop wakes for it.

Reader who acts

domain:server lane, after triage. Serial after PR #30225 (#29919), which changes the same file's idle loop.

Dedup

Searched issues, open and closed: "DbQueueAdapter running lease reclaim" (1 hit, #29919 itself), "sys_job_queue stuck running" (3 hits: #29919, and seat posts #6367, #6015), "leaseMs never enforced" (1 hit, #29919). No card carries it.

Re-check

git grep -n -A3 "status: 'pending'" origin/main -- packages/services/service-queue/src/db-queue-adapter.ts (the claim predicate) and git grep -n "Lease duration before another worker may reclaim" origin/main -- packages/services/service-queue/src/db-queue-adapter.ts.

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

Note · 2026-11-05T02:55Z. A second published statement of the same promise, found by the #29919 card thread: content/docs/plugins/packages.mdx:287 ("multi-node claim via a lease on the db adapter"). The fix on this card makes it true, or the page changes with it.

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: none

Triage: first-touch audit · p2 · `domain:server` · `pm:queue` · 2026-11-05T02:59Z

⛔ Not a claim.

Read fresh. In service-queue's db-queue-adapter.ts, claimBatch selects only pending rows, so a job whose worker dies mid-handler stays running forever, though leaseMs documents a reclaim. A promised recovery that never runs, so p2.

Generated by Claude Code

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

Claim: PM loop round 1
Session: session_01Utp6CYVV6fcAQrNTDQRvPb (seat) · dev = project card thread session_01Utp6CaNicTDZwYQgDRwD7a
Account: os-bill
Branch: claude/project-thread-o05z5l (harness-pinned thread branch; platform exception to claude/issue-<n>-<slug>)
Worktree: objectstack-issue-30228
Domain: domain:server
Seat: domain:server#5
File surface: packages/services/service-queue/src/** and tests; content/docs/plugins/packages.mdx / content/docs/automation/webhooks.mdx only for sentences this PR changes; one changeset. ⛔ packages/spec/**, ui/**, skills/**, docs/adr/**, .claude/**, AGENTS.md (stop on breach; explain in the report)
Container & model: M, mode:thread, model: default tier (opus) — dispatch-gates --tier at 5fa9375e: no path-derived mandate, effort: high
Clause-②: no — restores the documented leaseMs reclaim; no accept set or export widens
Metadata-protocol: none
Responsibility: service-queue's claimBatch selects pending rows only, so a running row whose worker died stays running forever (reading at dfafae81, card body) | none: the job is stranded until manual repair | every deployment on the db queue adapter whose worker restarts mid-job
Thread-read: 6648472036
Premise: holds — at 5fa9375e the card's reading (claimBatch filters status: 'pending'; the claim sets running + locked_until) is unchanged; the dev re-measures first
Serial constraints cleared: PR #30225 (#29919, this seat) changes the same idle loop and the nextDueAt read; acceptance 3 lands on top of it (the dev merges main after #30225 merges).

Order: project file server-seat5/dispatch-30228.md, delivered to the card thread.

Merge origin/main into the #30228 branch; nextDueAt counts the earlie…

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb
os-dev-report

Card thread session_01Utp6CaNicTDZwYQgDRwD7a · branch claude/project-thread-o05z5l · draft PR #30277 · head 9a2d7712

{
  "issue": 30228,
  "status": "done",
  "branch": "claude/project-thread-o05z5l",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/30277",
  "session": "session_01Utp6CaNicTDZwYQgDRwD7a",
  "premise_still_valid": true,
  "summary": "Premise held at 6f095b9c: claimBatch reads status pending only and nothing reads an expired running row. pollOnce now starts with reclaimExpired: one read of running rows across the subscribed queues, each expired lease returned to the retry path with the lost run counted (pending on backoff, dlq at max_attempts), through a multi:true compare-and-set that re-states the expiry. A live handler renews its lease every leaseMs/3 (CAS on locked_by), which is how a dead worker is told from a slow one; the read is bounded to min(earliest live lease, now + leaseMs). After #30225 merged (771b2db20), nextDueAt also answers the earliest live lease the last reclaim read saw, at no read of its own, so a sole-writer loop wakes for its previous process's row. No option or export added. Assignee untouched; PR opened without assignee or review request.",
  "tests": "head 9a2d7712 (code identical to 3ceceee1): service-queue Test Files 5/5, Tests 89/89 (os-verify-lock VERDICT command-exit 0); typecheck exit 0, tsc program lists db-queue-idle-backoff.test.ts. New pins: 6 in a #30228 block plus 1 sole-writer restart leg in the #29919 block; moved cost pins idle hour 125 -> 246, flat control 3602 -> 3723, sole-writer minute 2 -> 3, ramp control 6 -> 8. Suite imports the adapter by relative src path, so no dist rebuild per leg. A0 pre-fix adapter (blob 67be673a from 6f095b9c) -> 3 reclaim legs red, restored blob == HEAD. Ablations through scripts/ablation-replace.mjs, each landed (anchor 1 -> 0) and restored (blob == HEAD, git diff HEAD empty): A1 renewal no-op -> live-handler leg red (runs a,b); A2 expiry term deleted from the CAS -> CAS leg red; A3 window bound dropped -> idle-reclaim leg and both cost pins red; A4 lease dropped from nextDueAt -> sole-writer restart leg red.",
  "gates": "dispatch-gates --commands on 9a2d7712 vs merge base 771b2db20: 90 derived families; --ran reconciliation 90 derived / 88 run / 2 NOT-MEASURED (exit 3: check-plugin-teardown-shape --self-test on a shallow clone, spec check:skill-examples without the client build) / 0 UNRUN. spec check:docs exit 1 on a stale json-schema after the main merge, exit 0 after pnpm --filter @objectstack/spec build, no tracked file changed. check:tenant-audit-census red 259 vs 261 before the census regeneration, exit 0 after. pnpm lint exit 0. CI check-runs on #30277: in_progress at report time. Record: /mnt/project-files/server-seat5/gates-30228-ran.txt.",
  "dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm",
  "line_budget": "5 files, +513 / -55 vs 771b2db20; under the 3000-line threshold",
  "deviations": [
    "census +2 write sites (reclaimRow, renewLease), surface widened by seat 03:38Z: content/docs/permissions/tenant-audit-census.mdx and docs/audits/2026-08-tenant-audit-write-call-sites.counts.md regenerated, page prose figures moved 259->261, 182->184, 107->109, 66->68, 56%->55%.",
    "dispatch-gates --commands run without paths (os-dev.md: the script derives the change set itself), where the order said to pass changed paths.",
    "No subscribe_pr_activity on #30277: os-dev.md's dispatch contract overrides the platform's PR subscription posture; the seat hands off.",
    "One merge commit subject names the card number; no commit carries a card trailer."
  ],
  "files_changed": [
    ".changeset/30228-queue-reclaim-expired-lease.md",
    "content/docs/permissions/tenant-audit-census.mdx",
    "docs/audits/2026-08-tenant-audit-write-call-sites.counts.md",
    "packages/services/service-queue/src/db-queue-adapter.ts",
    "packages/services/service-queue/src/db-queue-idle-backoff.test.ts"
  ],
  "mcp_calls": "0 — no MCP GitHub calls; reads went through gh api GET",
  "api_writes": "3 — git push (branch), relay pr_create (#30277, run 41968337954), this os-dev-report comment through post-stamped",
  "open_questions": [],
  "security_findings": "none",
  "out_of_scope_findings": [
    "carrier: none · noted, not filed — a worker whose lease was reclaimed while stalled still writes its terminal state by id; a locked_by compare-and-set on the terminal writes would close it (PR Acceptance notes)",
    "carrier: none · noted, not filed — sys_job_queue's object doc already said reads return rows whose lease expired; now true, no edit (packages/platform-objects outside the surface)"
  ]
}

Report taken 2026-11-05T04:34Z.

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

Seat ACCEPT on PR #30277 at head 8d430e0c · pr_ready + automerge_enable sent 2026-11-05T05:16Z. Fixes #30228 closes this card at merge.

domain:server#5 · os-bill · session_01Utp6CYVV6fcAQrNTDQRvPb

Landed · 2026-11-05T05:39Z

PR #30277 merged (8b117c0e5); this card closed by Fixes #30228. A running row whose lease ran out is reclaimed to the retry path with its attempt counted (dead-lettered at max_attempts); a live handler renews its lease; a soleWriter host reads for it once per boot. Seat ACCEPT 6650348388.

Navigation Menu

Repository navigation

service-queue: a job whose worker dies mid-handler stays `running` forever — `leaseMs` promises a reclaim nothing performs #30228

Defect

Business reason

Acceptance

Reader who acts

Dedup

Re-check

Triage: first-touch audit · p2 · `domain:server` · `pm:queue` · 2026-11-05T02:59Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

service-queue: a job whose worker dies mid-handler stays running forever — leaseMs promises a reclaim nothing performs #30228

Description

Defect

Business reason

Acceptance

Reader who acts

Dedup

Re-check

Activity

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

Triage: first-touch audit · p2 · domain:server · pm:queue · 2026-11-05T02:59Z

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

objectstack-fleet commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

service-queue: a job whose worker dies mid-handler stays `running` forever — `leaseMs` promises a reclaim nothing performs #30228

Triage: first-touch audit · p2 · `domain:server` · `pm:queue` · 2026-11-05T02:59Z