[timiturn3r]

fix(webhooks): lease heartbeat renewal and stale reclaim DLQ transition

Closes #1737

Summary of Changes

This PR resolves two webhook processor lease and concurrency defects in lib/webhooks/processor.ts:

1. Long-running handlers whose execution exceeded WEBHOOK_RETRY_CONFIG.leaseTimeoutMs (default 30s) could have their events reclaimed by another worker while the first execution was still in-flight, violating single-execution concurrency guarantees.
2. Crash-looping events whose workers terminated unexpectedly (e.g. OOM, process exit, mid-run deploy) were reclaimed without incrementing retryCount, causing them to re-execute indefinitely and never reach the Dead-Letter Queue (DLQ).

---

Failure Modes Explained

1. Handler Lease Expiry Re-run

When processWebhookEvent claimed an event, it recorded a single timestamp (updatedAt = now) and an expiration deadline (nextRetryAt = now + leaseTimeoutMs). While the handler executed, nothing extended this lease. If await handler(payload) ran longer than 30s, the event matched the stale-processing filter (updatedAt <= now - leaseTimeoutMs), allowing a second worker to acquire the claim and run the handler concurrently. Although the first worker's finalizer eventually returned conflict upon seeing an unexpected updatedAt, duplicate external side effects had already occurred.

2. Stale Reclaim Loop Without DLQ Transition

When an event caused an unhandled crash or process termination, handleWebhookProcessingFailure never ran. Subsequent worker cycles reclaimed the event by matching { status: 'processing', updatedAt: { lte: staleProcessingBefore } } and simply resetting updatedAt without bumping retryCount. Because retryCount never increased across abandoned attempts, the condition nextRetryCount > event.maxRetries was never satisfied, resulting in infinite recurring crashes.

---

Implementation Details

1. Heartbeat Lease Renewal (leaseTimeoutMs / 3):

   • While handler(payload) executes, a background interval triggers periodic lease renewals at Math.floor(leaseTimeoutMs / 3) (e.g., every 10s for default 30s lease; 33ms for 100ms test lease).
   • Each heartbeat executes an atomic conditional update matching { id: eventId, status: 'processing', updatedAt: currentToken }.
   • On success, currentToken updates to the new timestamp.
   • If renewal returns count === 0 (lease lost to another worker or manual intervention), the worker marks isOwner = false, stops the heartbeat, discards the handler result, and returns status: 'conflict' without modifying database state.
   • Both success and failure finalizers verify the renewed currentToken.

2. Atomic Stale Reclaim & DLQ Transition:

   • When claiming a stale processing event (updatedAt <= staleProcessingBefore):
     • If event.retryCount >= event.maxRetries, the event is transitioned directly to dlq with lastError = 'LEASE_EXPIRED', an audit event (webhook.dlq) is emitted via recordAuditEvent, and the handler is skipped.
     • If event.retryCount < event.maxRetries, the claim atomically increments retryCount: { increment: 1 } in the database, counting the abandoned attempt toward the retry limit.

3. Concurrency Documentation:

   • Updated docs/WEBHOOK_CONCURRENCY.md to document the heartbeat lease renewal mechanism and the reclaim-counts-as-attempt invariant.

---

Key Claim and Renewal Predicates

1. Active vs Stale Lease Detection

const staleProcessingBefore = new Date(
  now.getTime() - WEBHOOK_RETRY_CONFIG.leaseTimeoutMs
);

// If lease is still active, reject concurrent workers
if (event.status === 'processing' && event.updatedAt && event.updatedAt > staleProcessingBefore) {
  return {
    success: false,
    error: 'Event is already being processed',
    status: 'conflict',
    retryable: true,
    retryAfterMs: event.nextRetryAt
      ? Math.max(0, event.nextRetryAt.getTime() - now.getTime())
      : WEBHOOK_RETRY_CONFIG.leaseTimeoutMs,
  };
}

2. Stale Reclaim DLQ Transition

// Exceeded maxRetries -> move directly to DLQ
if (event.retryCount >= event.maxRetries) {
  const dlqUpdate = await prisma.webhookEvent.updateMany({
    where: {
      id: eventId,
      status: 'processing',
      updatedAt: { lte: staleProcessingBefore },
    },
    data: {
      status: 'dlq',
      retryCount: { increment: 1 },
      lastError: 'LEASE_EXPIRED',
      updatedAt: now,
    },
  });
  // ... record audit event 'webhook.dlq' ...
  return { success: false, error: 'LEASE_EXPIRED', status: 'dlq', retryable: false };
}

3. Stale Reclaim Atomic Increment

const claim = await prisma.webhookEvent.updateMany({
  where: {
    id: eventId,
    status: 'processing',
    updatedAt: { lte: staleProcessingBefore },
  },
  data: {
    status: 'processing',
    retryCount: { increment: 1 },
    updatedAt: now,
    nextRetryAt: new Date(now.getTime() + WEBHOOK_RETRY_CONFIG.leaseTimeoutMs),
  },
});

4. Heartbeat Lease Renewal

const renewal = await prisma.webhookEvent.updateMany({
  where: {
    id: eventId,
    status: 'processing',
    updatedAt: currentToken,
  },
  data: {
    updatedAt: nextToken,
    nextRetryAt: new Date(nextToken.getTime() + WEBHOOK_RETRY_CONFIG.leaseTimeoutMs),
  },
});

---

Verification & Test Results

All unit tests pass cleanly from the repo root via npx vitest run tests/unit/webhook-*.test.ts:

 RUN  v4.1.10 /home/timiturn3r/Documents/drips/Remitwise-Frontend

 ✓ tests/unit/webhook-concurrency.test.ts (7 tests) 21ms
   ✓ invokes a handler only for the worker that atomically claims the event
   ✓ does not overwrite a state transition when failure handling loses the processing claim
   ✓ allows only one concurrent DLQ replay and reports the loser as stale
   ✓ does not run a second handler while the lease is being renewed
   ✓ abandoned attempts count towards maxRetries and move to DLQ
   ✓ heartbeat stops after completion and issues no renewal updateMany
   ✓ lost lease discards result and writes no processed transition

 ✓ tests/unit/webhook-retry.test.ts (20 tests) 27ms
   ✓ WEBHOOK_RETRY_CONFIG defaults (4 tests)
   ✓ calculateNextRetryTime — exponential backoff (6 tests)
   ✓ handleWebhookProcessingFailure — DLQ transitions (4 tests)
   ✓ registerWebhookHandler (1 test)
   ✓ processPendingWebhooks (2 tests)
   ✓ getRetryPolicyInfo (3 tests)

 Test Files  2 passed (2)
      Tests  27 passed (27)
   Start at  22:10:03
   Duration  1.15s (transform 124ms, setup 330ms, import 143ms, tests 48ms, environment 1.42s)

TypeScript type checking (npx tsc --noEmit) passes with zero errors.