Skip to content

fix(ha): claim workflows on spawn + persist running to stop the "untriggered" steal (#89) - #98

Merged
gustavobertoi merged 2 commits into
mainfrom
fix/ha-untriggered-claim-on-spawn
Jun 4, 2026
Merged

gustavobertoi merged 2 commits into
mainfrom
fix/ha-untriggered-claim-on-spawn

Conversation

@gustavobertoi

Copy link
Copy Markdown
Collaborator

Closes the root cause of #89 (the e2e quarantine was only a workaround). One PR to main.

Root cause

On the 3-node HA stack, a freshly triggered workflow intermittently stayed untriggered forever:

  • the trigger path never set claimed_by and never persisted the untriggered → running
    transition (only journaled it), so the row sat at claimed_by=NULL, state=untriggered for the
    whole run;
  • a Postgres NOTIFY on the untriggered INSERT makes every node's claim actor sweep immediately,
    and the sweep claims claimed_by IS NULL AND state IN ('untriggered','running','sleeping') → another
    node stole and re-triggered it → a duplicate WorkflowHandler → stall;
  • recoverWorkflows (running/sleeping only) never healed it, and GET /v1/workflows/{id} reported
    the running workflow as untriggered.

Fix

  • ClaimRepository.ClaimWorkflow(nodeID, workflowID) — atomically claim a single workflow for
    the owning node (memory no-op → true; postgres UPDATE … RETURNING rows-affected).
  • WorkflowHandler.Init claims the workflow for this node on both the new-trigger and replay
    paths; on loss to another live node it bows out (signals the instance supervisor it's done
    here) so no duplicate runs. It then persists the running/sleeping state immediately.
  • The sweep's lease-expired branch and recoverWorkflows now include untriggered so a
    crash-stranded row is reclaimed/re-driven.
  • ADR-0018 amended with the ownership invariants.

Safety

Non-HA / memory: no behavior change (claim is a no-op; the claim actor + PgListener are HA-gated).
The e2e re-trigger quarantine stays as defense-in-depth until make ha-up runs confirm zero stuck
untriggered.

Verify

make lint 0 · make build ok · make test 701 pass · functional tests compile. Adds a memory
ClaimWorkflow unit test and a Postgres contract subtest (claims, idempotent for owner, blocks other
live nodes). The real 3-node assertion is the CI e2e tier (can't run a 3-node stack locally).

🤖 Generated with Claude Code

…ggered steal (#89)

On the multi-node HA stack a freshly triggered workflow intermittently stayed in
"untriggered" forever: the trigger path never set claimed_by and never persisted
the untriggered->running transition (only journaled it), so the row sat at
claimed_by=NULL, state=untriggered for the whole run. The NOTIFY-driven claim
sweep on other nodes then stole and re-triggered it, spawning a duplicate handler
and stalling the run; recovery (running/sleeping only) never healed it.

Fix:
- ClaimRepository.ClaimWorkflow(nodeID, workflowID) — atomically claim a single
  workflow for the owning node (memory no-op returns true; postgres UPDATE ...
  RETURNING rows-affected).
- WorkflowHandler.Init claims the workflow for this node on both the new-trigger
  and replay paths; on loss to another live node it bows out (tells the instance
  supervisor it's done here) so no duplicate runs. It then persists the
  running/sleeping state immediately so the DB row reflects reality.
- Claim sweep lease-expired branch and recoverWorkflows now include 'untriggered'
  so a crash-stranded row is reclaimed/re-driven.
- ADR-0018 amended with the ownership invariants.

Non-HA/memory: no behavior change (claim is a no-op, claim actor is HA-gated).
The e2e re-trigger quarantine stays as defense-in-depth until 3-node runs confirm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gustavobertoi
gustavobertoi merged commit cbfef75 into main Jun 4, 2026
5 checks passed
@gustavobertoi
gustavobertoi deleted the fix/ha-untriggered-claim-on-spawn branch June 4, 2026 06:19

This branch was previously deployed

1 inactive deployment
prod — dc818544 Deployed Jun 4, 2026 by gustavobertoi via pr-image #326
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant