Skip to content

feat(agent): resume a suspended AgentRun through the runner surface - #2245

Open
nazq wants to merge 1 commit into
0xPlaygrounds:mainfrom
nazq:feat/runner-resume
Open

feat(agent): resume a suspended AgentRun through the runner surface#2245
nazq wants to merge 1 commit into
0xPlaygrounds:mainfrom
nazq:feat/runner-resume

Conversation

@nazq

@nazq nazq commented Aug 3, 2026

Copy link
Copy Markdown

Fixes #2244.

Related: #2116 (checkpoint / interrupt–resume operations), #2118 (interactive coding-agent roadmap). Complementary to #2121, which adds safe-boundary checkpoint capture for manually driven runs and explicitly lists runner-level durable checkpoint resumption as remaining work — this PR is that piece for the current runner, kept small and additive.

Motivation

AgentRun is a sans-IO, Serialize + Deserialize state machine, and its module docs advertise exactly the workflow servers need: serialize a run between steps (for example while tool calls are pending), persist it, and resume it later in another process. The capture half of that story works today by hand-driving next_step() / model_response() / tool_results().

The resume half is currently missing: the high-level driver (AgentRunner::run / stream) builds its AgentRun internally and has no way to accept a restored one. Anyone resuming a persisted run must hand-write the entire drive loop for the resume leg — and thereby loses the runner's hook stack, tool-server dispatch (including retrieval and MCP-registered tools), conversation-memory append, telemetry spans, and structured-output handling. In practice a process restart, migration, or approval-gate suspension leaves the second half of the run without those integrations.

This PR closes the loop with one additive entry point:

let run: AgentRun = serde_json::from_str(&persisted)?;
let response = agent.resume(run).run().await?;      // or .stream().await

Design

Public surface (additive only):

  • Agent::resume(run: AgentRun) -> AgentRunner<M> — mirrors Agent::runner(prompt).
  • AgentRunner::resume(agent, run) — mirrors AgentRunner::from_agent(agent, prompt).

Everything else on the runner keeps working: add_hook, max_turns, tool_concurrency, conversation / without_memory, and both run() and stream().

One drive loop, not a fork. run() and stream() already share the single engine (drive_agent), which takes an AgentRun as input. Resume reuses that seam: the runner carries an optional restored run, and both surfaces drive it through the identical loop, so hooks, tool execution, fail-closed semantics, memory append, and telemetry behave exactly as they do for a run that was never suspended. No second driver exists to drift.

Division of authority. The restored run is authoritative for loop state: prompt, accumulated history, pending tool calls, turn budget, invalid-tool-call retry budget, tool choice, and aggregated usage (a resumed run's final usage spans both processes). The agent is authoritative for the environment the run resumes into: model, hooks, tool server, preamble, request parameters, structured-output configuration, and memory. The run's budgets and tool-choice policy seed the runner on resume(), so builder overrides default to the suspended values and can still be changed before driving — e.g. agent.resume(run).max_turns(extended) deliberately raises an exhausted budget.

Mid-tool-batch re-entry. A run restored in the pending-tools state re-enters the drive loop directly on CallTools (the state machine already re-emits the pending calls idempotently, preserving internal_call_ids). Within one process the engine pins each turn's tool batch to the registry snapshot advertised to the model; those pinned handles cannot survive serialization, so a resumed batch takes a fresh snapshot and binds to the tool server's current registry. This is only permitted on the first step of a resumed drive — for every later step, a missing snapshot remains the driver-bug error it is today.

Memory semantics. Resume never loads conversation memory (the restored run carries its own history — loading would corrupt it), but a configured backend still receives the completed run's messages at Done, exactly once across the suspend/resume boundary: the suspending process never reached Done, so it never appended. without_memory() opts out, as usual.

Alternatives considered

  • Making the private drive loop public so callers can drive a restored run themselves. Rejected: it exposes a large internal surface (TurnSource, snapshot threading, span shaping) that would freeze implementation details into API, and every caller would still have to reassemble hooks/memory/telemetry correctly. resume() keeps one blessed driver.
  • A free-standing resume(agent, run) function. Equivalent power, but the runner-builder shape is the repo's idiom and gives the override story (max_turns, add_hook, …) for free.
  • Re-pinning the original turn's tool definitions on mid-batch resume. Not achievable — tool executors are not serializable — so the limitation is documented rather than worked around. The behavior (current-registry dispatch) is documented instead; the run still validates calls against the tool names that were advertised on the suspended turn.
  • Blocking memory append on resume (treating a restored run like caller-supplied explicit history, which bypasses memory entirely). Rejected: the restored run's messages are precisely this conversation's new messages, and appending once at completion is what would have happened without the interruption; skipping it would silently drop the exchange from memory.

Serialization caveats (documented on Agent::resume and the run module)

  • Run state embeds the full conversation accumulated so far; persisting it inherits the conversation's sensitivity.
  • The format is JSON-tolerant (serde defaults absorb absent fields) but carries no cross-version stability guarantee yet: resume with the same rig version that suspended the run.
  • Runs must be suspended at a step boundary (the state observed between next_step() calls, e.g. pending tool calls). A run snapshotted while a model response was outstanding cannot be advanced and fails with the existing protocol error.
  • The resuming agent should be configured equivalently to the one that started the run (tools, output mode, preamble); the run does not carry the environment.

Test coverage

New tests in rig-agent (all synthetic — MockCompletionModel, MockAddTool, CountingMemory; no network):

  • resumed_mid_tool_batch_run_completes_with_hooks — hand-drive a run to its tool boundary, serialize to JSON, drop it, restore, and resume via agent.resume(run).run(): the pending tool executes through the live tool server, ToolCall/ToolResult/CompletionCall hooks fire on the resumed leg, usage aggregates across the boundary, and the final history contains the pre-suspension turn.
  • resumed_run_streams_tool_activity_and_final_response — the same restored run through stream(): tool result surfaces as a stream item, hooks fire, final response arrives.
  • resume_seeds_and_overrides_the_suspended_turn_budget — an exhausted budget is preserved by default (MaxTurnsError) and extendable via .max_turns() before driving.
  • resumed_run_appends_to_memory_without_loading — zero loads, exactly one append, stored history equals the response messages.
  • resume_at_the_model_boundary_runs_like_a_new_run — a run serialized before its first model call resumes indistinguishably from a fresh run.

Validation: cargo fmt --check, cargo clippy -p rig-agent --all-targets --all-features (clean), cargo test -p rig-agent --all-features (492 lib tests + doctests, green) on the repo toolchain (1.94.0).

AgentRun serializes between steps, but the high-level driver could not
accept a restored run: resuming a persisted run meant hand-writing the
drive loop and losing the runner's hook, tool-server, memory, and
telemetry wiring.

Add Agent::resume(run) / AgentRunner::resume(agent, run), which hand a
deserialized run back to the single engine shared by run() and
stream(). The run supplies the loop state and seeds the runner's
budgets; the agent supplies the environment. A run restored with tool
calls pending binds them to the current tool registry (pinned per-turn
handles do not survive serialization); memory is appended at Done but
never loaded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(agent): resume a persisted AgentRun through the runner surface

1 participant