Skip to content

fix(sdk): refuse a reordered serial resume instead of re-running a completed step - #685

Merged
miyaontherelay merged 5 commits into
mainfrom
fix/serial-replay-order-guard
Oct 26, 2026
Merged

miyaontherelay merged 5 commits into
mainfrom
fix/serial-replay-order-guard

Conversation

@miyaontherelay

@miyaontherelay miyaontherelay commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A resumed serial authored run could re-execute a completed step and still exit 0. This PR adds a fail-closed guard. It does not change any step id. Everything is in the SDK.

Mechanism

Serial step ids are <kind>-<N>. One ordinal counter is shared by every kind and assigned in call order (authored-flow-executor.ts nextId). Inside Promise.all, call order follows result-arrival order: each branch's next call is made when its previous step resolves. On resume, completed f.llm/f.agent results replay from the journal almost instantly. If the original run's results arrived in a different order (model latency), the branches take each other's ordinals.

  • Same-kind swap, different spec (for example run-3 re-requested with the other branch's command): the kernel's admission key is bound to the spec hash, so the resume fails with run_admission_conflict. That is fail-closed but spurious. It is the intermittent failure in resumes 'llm' branches without replaying completed effects.
  • Kind swap: run-3 and llm-3 hash to different admission keys. A resumed f.run that lands on the ordinal a completed f.llm holds gets a fresh key and runs again. Repro on main: effects L became LLZ, the model was called 4 times instead of 3, and flows resume exited 0 with completionReason: success. The result verifier only checks that ordinals are contiguous, and a swap keeps them contiguous.
  • f.human swap: the answer is keyed by wait id alone, and the lowered spec embeds the answer, not the question. A reordered resume handed human-3's recorded "yes" (given to "Ship the right release?") to "Delete the left bucket?", then re-asked the right question as human-4.

The fix

  • What is checked. On a re-dispatched attempt (dispatch.attempt > 1), the root driver (authored-root.ts) reads the root's authored-steps index and its wait.human ids once. It hands that snapshot to the body; dispatched children receive it too.
  • Who holds an ordinal (authored-serial-ordinals.ts):
    • every indexed <kind>-<N> step;
    • a parked f.human, from its wait.human id;
    • an unfinished f.dispatch, from any dispatch-N--* record of its child.
  • Refusal. A serial call whose ordinal is held by a different kind is refused before anything is admitted. This applies in both directions, and to helper ids and the complete-N marker. An f.human is refused when its journaled wait asked a different question or recipient.
  • Effects are indexed. Helper, MCP and plugin effects now record their completion on the root index like every other child, including MCP and plugin calls that fail with a diagnostic (authored-helper-effect.ts, authored-mcp.ts, authored-plugin-effect.ts). Their ordinals are visible. These are the same relayflows.authored-step.v1 records other steps already write. Flows that use these effects therefore get one more index record per call on a first run. Otherwise the guard only reads.
  • Hook watermarks. A replayed f.hook verdict restores the counter to the afterStep it recorded. That watermark also counts ordinals other branches took while the hook ran, so jumping past one gave that branch's completed step a fresh ordinal and re-ran it. A restore that would skip a journaled ordinal this attempt hasn't claimed is now refused, naming the hook, the ordinal and the journaled step.
  • One latch. The refusal is a single latch shared by the root and every dispatched child. It is checked when a step is declared and again when it starts, and by f.on and f.memory before they act. So an author catch cannot get anything else admitted: not a step saved before the refusal and awaited after it, not a child flow that is still running, not an activity, not a memory write.
  • Typed refusal. serial_replay_order_mismatch, exit 1. The message names the ordinal, the journaled step and kind, and the attempted kind, e.g. ordinal 3 is journaled as llm step "llm-3", but this resume reached it with run.
  • First attempts read nothing extra. Parallel roots are untouched; their ids come from the causal frontier.

The guard does not make a reordered resume succeed. It now fails closed: the root fails as step_failed, like today's run_admission_conflict, instead of re-running effects. Fix 2 makes it succeed.

Pre-existing caveats, also on main:

  • A resume past a hook verdict is refused by the kernel on the hook's own hook-N marker (run_admission_conflict, measured on unmodified main). Resuming past a hook that ran its own steps therefore already fails. The watermark guard now refuses that skip itself, a change of refusal kind that doesn't affect any working path. This needs its own fix.
  • From the final verification: a crashed-run resume that uses MCP fails with unsettled_derived_work on 2.0.55 and on main. I didn't reproduce it here.

Changed questions are refused. An f.human whose question or recipient text differs from the one it parked on is refused on every resume, for example a question built from the time of day. That is the same as an f.run whose command changes between attempts. It is documented in docs/SURFACE.md.

Gaps that remain for fix 2

  • In-flight steps. f.run, helper, MCP and plugin steps are indexed only at completion. An ordinal whose only holder was one of these, still in flight at the crash, is invisible to the guard. In that window a different-kind call can take the ordinal before a later ordinal is refused.
  • Identical specs. Same-kind swaps with identical specs are indistinguishable at the kernel, so the two calls' results can be exchanged.
  • Hooks. f.hook verdicts are keyed by step id plus hook name, not input. Two same-name hooks with different inputs can exchange verdicts. Closing that needs a hook-record format change.
  • Queued worker slot. An f.llm/f.agent that started before the refusal and is still waiting for a worker slot (--agent-capacity) is admitted when the slot frees. Its own ordinal passed the guard. The body-failure path closes the worker slots once the body settles. I removed the eager stop added in 8ec5a9b because it had no test, and in serial mode the deterministic runner has no queue to stop.

Impact

This affects hosted and self-hosted runs alike: both go through the same root driver. Exposed: serial roots, meaning any body with f.llm, f.agent, f.dispatch, f.human, helpers, budget, imports and so on, that runs effects as continuations inside Promise.all. Resume after a lost sandbox is the worst case, because replays return instantly while the original run waited on the model. Not exposed: parallel-eligible roots (causal-frontier ids), and serial bodies made only of f.run, whose children share one in-order daemon connection and so return in call order.

Follow-up: fix 2, stable per-branch ids (identityVersion 3)

  • New serial roots pin identityVersion: 3. Version 2 and legacy roots keep kind-N plus this guard, so no journaled id ever changes.
  • A v3 serial id is <kind>-after-<sha256(sorted causal predecessors)>-<ordinal within that frontier>. This is the parallel scheme generalized to every kind. The serial lifecycle already records each call's causal predecessors (today the index records after: ["llm-1"] on run-3). The ordinal counts calls declared against the same frontier, in synchronous declaration order, so result-arrival order never participates.
  • Where the frontier is unreliable (replaced intrinsics, unregistered aggregates, truncated graphs, timers, races), fall back to kind-N with this guard. In those cases the nondeterminism is the flow's own, and it is refused rather than repaired.
  • Bind f.human waits and hook records to a digest of the question or input. Index steps at admission, not completion.
  • The result verifier learns v3 ids the way it learned parallel ids.

Tests

All on Node v22.23.4. "Fails on revert" means: that one fix was reverted on its own, the CLI rebuilt, the test run and its failure recorded, then the file restored byte for byte (the diff sha matched before and after) and the test re-run green.

Test With the fix reverted Restored
live: f.run reaches a completed f.llm's ordinal resume ok:true, effects LLZ pass
live: f.llm reaches a completed f.run's ordinal resume ok:true pass
live: f.run reaches a completed f.agent's ordinal (agents in separate working directories, so they overlap) resume ok:true pass
live: f.human answer applied to a different question answer applied; parks again on human-4 pass
unit: f.run on an llm ordinal, refused before admission resolves success pass
unit: f.llm on a run ordinal, refused before preflight llm_cli_unresolved pass
unit: refusal stays latched through an author catch admitted pass
unit: completion marker on a journaled step's ordinal admitted pass
unit: f.human with a different question / recipient (asked and answered cases) parked / applied pass
unit: step declared before the refusal and awaited after it (8ec5a9b) run-1 admitted pass
review 2, item 1 live: f.run reaches a parked f.human's ordinal L re-ran (effects LL); question re-asked as human-4, exit 3 pass
item 2 live: f.run reaches an unfinished f.dispatch's ordinal L re-ran (CLL); refused only at ordinal 4 pass
item 2 unit: dispatch-1--run-1 claims ordinal 1; prefix parsing admitted / wrong map pass
item 3 live: f.run reaches a completed f.mcp call's ordinal (mock MCP server) L re-ran (LL); refused only at ordinal 4 pass
item 4 live: a running dispatched child stops when its parent refuses the child's next step ran (CDL) pass
item 4 unit: no f.memory.learn after a caught refusal finding learned pass
item 4 unit: no f.on subscription after a caught refusal subscription.open sent pass
final P2 unit (executor, two attempts): a replayed hook watermark skips the concurrent branch's run-3 second attempt admits run-2, hook-1, run-4, complete-5, re-running b pass
final P2 live (real extension through the plugin path): the same flow, crashed and resumed stopped only by the pre-existing hook-1 marker conflict, not by this guard pass
final P3 unit: completed helper effect indexed (slack mock, loopback) no index record pass
final P3 unit: completed plugin effect indexed no index record pass
final P3 unit: plugin call failed with a diagnostic indexed no index record pass
final P3 unit: MCP call failed with a diagnostic indexed no index record pass

These pin existing behaviour and pass with or without the guard:

  • live same-kind run/run, llm/llm and agent/agent swaps, refused by the kernel with run_admission_conflict;
  • a live in-order resume;
  • parallel roots ignore the snapshot;
  • an in-order re-dispatch writes byte-for-byte what the first run wrote (unit).

Cross-version:

  • 2.0.55: a serial root started by the published 2.0.55 CLI and resumed by this branch completes when resumed in order, and is refused when reordered (live, gated on RELAYFLOWS_PREVIOUS_CLI). On a live A/B against main, measured at ae2bf8f (first run plus in-order resume, all journals normalized; the later commits add only reads to this llm+run flow), both builds produce the same two variants. The second variant is concurrent branches appending two index records in the opposite order, which main also does (1 run in 20).
  • 2.0.50 compat set: RELAYFLOWS_PREVIOUS_CLI=<2.0.50 cli> over the parallel suites (authored-parallel-agents, -body, -eligibility, -llm, -run-live): 74 passed, 0 skipped. With 2.0.55 as the previous CLI, four of those gated cases fail by design, because they assert 2.0.50-only behaviour: an older CLI refusing a v3 root, and legacy root metadata without identityVersion.

Suites at 28b349b: the PR, parallel, lifecycle and authority files (serial-replay-identity, authored-serial-ordinals, authored-human, f-memory, authored-effect-index, authored-plugin-effect, mcp-lifecycle, mcp, authored-hooks, authored-root, authored-step-index, human-live, authored-activity, authored-parallel-*, authored-flow-lifecycle-executor, daemon-lifecycle, daemon-lifecycle-live, authored-node-authority, authored-surface-authority): 23 files, 383 passed, 8 skipped (the RELAYFLOWS_PREVIOUS_CLI-gated cases). The one failure in that run was the live hook test making the fixture manifest's network preflight call to api.github.com; it now runs offline and passes 3/3. npm run typecheck and typecheck:tests are clean.

Earlier suites (at 3c95275):

  • serial-replay-identity, authored-serial-ordinals, authored-human, f-memory: green.
  • Parallel, lifecycle (authored-flow-lifecycle-executor, daemon-lifecycle, mcp-lifecycle) and authority (authored-node-authority, authored-surface-authority) suites, plus authored-root: green.
  • npm test (full SDK gate: 4777 tests): 39 failed. 36 of them fail identically on unmodified main on this macOS host: hosted-base-snapshot, hosted-extension-isolation ("requires Linux"), hosted-extension-protocol, event-await-cli, det-withhold-live, babysitter-native-extension. The other three (authored-root source-authority, wrapper-exit-drain lease abort, live-kernel twenty-six-step) timed out while other jobs had the host at a load average of 48–66. Re-run in isolation on this branch, those three files pass 93/93, twice.
  • npm run typecheck, npm run typecheck:tests: clean.

No gate files are edited.

🤖 Generated with Claude Code


Agent Relay sessions

  • claude session af1462f7 · opened via gh pr create · last active 2026-10-26
  • claude session 28cd20e0 · contributor · ran gh pr commands · last active 2026-10-26

…mpleted step

Serial step ids are <kind>-<N> from one counter assigned in call order,
which inside Promise.all is result-arrival order. A resume replays
completed results in a different order, so branches can take each
other's ordinals. A same-kind swap with a different spec is already
refused by the kernel (run_admission_conflict). A kind swap is not:
run-3 and llm-3 are different admission keys, so a resumed f.run that
lands on a completed f.llm's ordinal ran again and the resume exited 0.
An f.human swap applied a recorded answer to a different question.

On a re-dispatched attempt the root driver now hands the body the root's
journaled step index. A serial call whose ordinal is journaled under a
different kind, or an f.human whose wait id was asked a different
question or recipient, is refused as serial_replay_order_mismatch before
anything is admitted. The refusal is latched against author catch.
Read-only: no step id changes, no new journal writes, first attempts read
nothing, parallel roots are untouched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 949e52a7-e36a-477f-8195-7581c2dd712e
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; no new actionable issue was found.

Summary

The PR refuses reordered serial resumes before they can reuse an ordinal for a different step kind.

  • Resumed serial flows stop when a saved step ordinal belongs to another kind.
  • Helper, MCP, and plugin calls now appear in the root step index.

miyaontherelay explicitly deferred stopping already-started model calls waiting for a worker slot. Their own ordinals passed the guard, and the eager stop lacked a dedicated test.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Serial root resumes] --> B[Read recorded steps and human wait ids]
  B --> C[Share snapshot with root and children]
  C --> D[Declare next step]
  D --> E{Ordinal belongs to another kind?}
  E -->|Yes| F[Set shared refusal]
  E -->|No| G[Check refusal again when step starts]
  G --> H[Admit or replay step]
  F --> I[Block later starts and fail root]
Loading

Reviews (5) · Last reviewed commit: "test(sdk): make the serial replay live t..." · Reviewed by Greptile

miyaontherelay and others added 4 commits October 26, 2026 05:06
A step declared before a serial_replay_order_mismatch and awaited after
it still started: each operation's start check only asked whether done()
had been called. Every operation now re-checks the refusal latch when it
starts, and the refusal closes queued worker admissions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nals; share the replay latch

Adversarial review of the serial replay guard:

- A parked f.human holds its ordinal before its answer is lowered into
  the step index. The root driver now adds the root's wait.human ids to
  the snapshot a re-dispatched body is checked against.
- An unfinished f.dispatch holds its ordinal from its child's first
  step. Any dispatch-N--* record now claims ordinal N for dispatch.
- Helper, MCP and plugin effects now index their completion on the root
  like every other child, so their ordinals are visible to a resume.
- The refusal latch is shared by the root and its dispatched children,
  so a child still running stops admitting, and f.on and f.memory check
  it before acting.
- Drop the queued-slot stops added in the previous commit: in serial
  mode the deterministic runner has no queue, and the body-failure path
  already closes worker slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ex failed effects

Final verification of the serial replay guard:

- A replayed f.hook verdict restores the counter to the watermark it
  recorded, which also counts ordinals other branches took while the hook
  ran. Jumping past one handed that branch's completed step a fresh
  ordinal and re-ran it. A restore that skips a journaled ordinal this
  attempt has not claimed is now refused as serial_replay_order_mismatch.
- MCP and plugin calls that fail with a diagnostic are indexed on the
  root like successful ones, so a caught failure's ordinal is guarded.
- Tests for helper and plugin indexing, which no suite covered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dependent

CI (linux-x64) failed three of the new live tests:

- The MCP call log was `${TMPDIR ?? '.'}/...`: with TMPDIR unset it was a
  relative path the mock server resolved against its own cwd. It now
  lives at an absolute path under the fixture root.
- The parked-human case could park before the slow branch wrote its
  effect. The parking branch now waits for that effect, so the journal
  holds both before the run stops.
- Teardown removed the fixture while a daemon was still flushing
  (ENOTEMPTY). Each fixture now SIGKILLs its daemon and waits for it to
  exit (a reaped or zombie process) before the directory is removed.

The original order no longer rests on a 1.5s model delay either: a
"slow" answer is held until the other branch has taken its next ordinal
and signalled `go`, so a loaded host cannot invert it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@miyaontherelay
miyaontherelay merged commit 7ac3030 into main Oct 26, 2026
9 checks passed
@miyaontherelay
miyaontherelay deleted the fix/serial-replay-order-guard branch October 26, 2026 06:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant