feat(agent): resume a suspended AgentRun through the runner surface - #2245
Open
nazq wants to merge 1 commit into
Open
feat(agent): resume a suspended AgentRun through the runner surface#2245nazq wants to merge 1 commit into
nazq wants to merge 1 commit into
Conversation
AgentRun serializes between steps, but the high-level driver could not accept a restored run: resuming a persisted run meant hand-writing the drive loop and losing the runner's hook, tool-server, memory, and telemetry wiring. Add Agent::resume(run) / AgentRunner::resume(agent, run), which hand a deserialized run back to the single engine shared by run() and stream(). The run supplies the loop state and seeds the runner's budgets; the agent supplies the environment. A run restored with tool calls pending binds them to the current tool registry (pinned per-turn handles do not survive serialization); memory is appended at Done but never loaded.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2244.
Related: #2116 (checkpoint / interrupt–resume operations), #2118 (interactive coding-agent roadmap). Complementary to #2121, which adds safe-boundary checkpoint capture for manually driven runs and explicitly lists runner-level durable checkpoint resumption as remaining work — this PR is that piece for the current runner, kept small and additive.
Motivation
AgentRunis a sans-IO,Serialize + Deserializestate machine, and its module docs advertise exactly the workflow servers need: serialize a run between steps (for example while tool calls are pending), persist it, and resume it later in another process. The capture half of that story works today by hand-drivingnext_step()/model_response()/tool_results().The resume half is currently missing: the high-level driver (
AgentRunner::run/stream) builds itsAgentRuninternally and has no way to accept a restored one. Anyone resuming a persisted run must hand-write the entire drive loop for the resume leg — and thereby loses the runner's hook stack, tool-server dispatch (including retrieval and MCP-registered tools), conversation-memory append, telemetry spans, and structured-output handling. In practice a process restart, migration, or approval-gate suspension leaves the second half of the run without those integrations.This PR closes the loop with one additive entry point:
Design
Public surface (additive only):
Agent::resume(run: AgentRun) -> AgentRunner<M>— mirrorsAgent::runner(prompt).AgentRunner::resume(agent, run)— mirrorsAgentRunner::from_agent(agent, prompt).Everything else on the runner keeps working:
add_hook,max_turns,tool_concurrency,conversation/without_memory, and bothrun()andstream().One drive loop, not a fork.
run()andstream()already share the single engine (drive_agent), which takes anAgentRunas input. Resume reuses that seam: the runner carries an optional restored run, and both surfaces drive it through the identical loop, so hooks, tool execution, fail-closed semantics, memory append, and telemetry behave exactly as they do for a run that was never suspended. No second driver exists to drift.Division of authority. The restored run is authoritative for loop state: prompt, accumulated history, pending tool calls, turn budget, invalid-tool-call retry budget, tool choice, and aggregated usage (a resumed run's final usage spans both processes). The agent is authoritative for the environment the run resumes into: model, hooks, tool server, preamble, request parameters, structured-output configuration, and memory. The run's budgets and tool-choice policy seed the runner on
resume(), so builder overrides default to the suspended values and can still be changed before driving — e.g.agent.resume(run).max_turns(extended)deliberately raises an exhausted budget.Mid-tool-batch re-entry. A run restored in the pending-tools state re-enters the drive loop directly on
CallTools(the state machine already re-emits the pending calls idempotently, preservinginternal_call_ids). Within one process the engine pins each turn's tool batch to the registry snapshot advertised to the model; those pinned handles cannot survive serialization, so a resumed batch takes a fresh snapshot and binds to the tool server's current registry. This is only permitted on the first step of a resumed drive — for every later step, a missing snapshot remains the driver-bug error it is today.Memory semantics. Resume never loads conversation memory (the restored run carries its own history — loading would corrupt it), but a configured backend still receives the completed run's messages at
Done, exactly once across the suspend/resume boundary: the suspending process never reachedDone, so it never appended.without_memory()opts out, as usual.Alternatives considered
TurnSource, snapshot threading, span shaping) that would freeze implementation details into API, and every caller would still have to reassemble hooks/memory/telemetry correctly.resume()keeps one blessed driver.resume(agent, run)function. Equivalent power, but the runner-builder shape is the repo's idiom and gives the override story (max_turns,add_hook, …) for free.Serialization caveats (documented on
Agent::resumeand therunmodule)next_step()calls, e.g. pending tool calls). A run snapshotted while a model response was outstanding cannot be advanced and fails with the existing protocol error.Test coverage
New tests in
rig-agent(all synthetic —MockCompletionModel,MockAddTool,CountingMemory; no network):resumed_mid_tool_batch_run_completes_with_hooks— hand-drive a run to its tool boundary, serialize to JSON, drop it, restore, and resume viaagent.resume(run).run(): the pending tool executes through the live tool server,ToolCall/ToolResult/CompletionCallhooks fire on the resumed leg, usage aggregates across the boundary, and the final history contains the pre-suspension turn.resumed_run_streams_tool_activity_and_final_response— the same restored run throughstream(): tool result surfaces as a stream item, hooks fire, final response arrives.resume_seeds_and_overrides_the_suspended_turn_budget— an exhausted budget is preserved by default (MaxTurnsError) and extendable via.max_turns()before driving.resumed_run_appends_to_memory_without_loading— zero loads, exactly one append, stored history equals the response messages.resume_at_the_model_boundary_runs_like_a_new_run— a run serialized before its first model call resumes indistinguishably from a fresh run.Validation:
cargo fmt --check,cargo clippy -p rig-agent --all-targets --all-features(clean),cargo test -p rig-agent --all-features(492 lib tests + doctests, green) on the repo toolchain (1.94.0).