fix(envoy-client): ack terminating stop commands so pegboard-envoy stops replaying them

[abcxff]

No description provided.

[abcxff]

Stack for rivet-dev/actors

• #5620 fix(container-runner): drain actors before announcing envoy …
• #5594 feat(container-runner): drain engine sleeps unless the idle …
• #5595 feat(container-runner): jitter idle timeout to avoid teardow …
• #5585 feat(container-runner): self-destroy on repeated actor start
• #5586 feat(container-runner): sleep on startup idle timeout
• #5589 chore(container-runner): tighten comments
• #5587 feat(container-runner): exit process when the last child sto …
• #5588 feat(container-runner): drain child before SIGTERM on engine …
• #5584 feat(container-runner): drain children and engine concurrent …
• #5565 fix(envoy-client): ack terminating stop commands so pegboard … 👈
• main

Get stack: forklift get 5565
Push local edits: forklift submit
Merge when ready: forklift merge 5565

_{change zrklppqy}

[railway-app]

🚅 Deployed to the actors-pr-5565 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
website	❌ Build Failed (View Logs)	Web	Aug 31, 2026 at 2:13 pm
kitchen-sink	😴 Sleeping (View Logs)	Web	Aug 28, 2026 at 5:26 am
mcp-hub	✅ Success (View Logs)	Web	Aug 12, 2026 at 7:34 pm
frontend-inspector	❌ Build Failed (View Logs)	Web	Aug 12, 2026 at 7:34 pm
ladle	❌ Build Failed (View Logs)	Web	Aug 12, 2026 at 7:34 pm
frontend-cloud	❌ Build Failed (View Logs)	Web	Aug 12, 2026 at 7:34 pm

[claude]

Review

Clean, well-motivated fix. Sourcing acks from processed_command_idx (which survives remove_actor) instead of the live ctx.actors map correctly closes the bug where a fast-stopping actor's checkpoint never made it into an ack before the entry was removed, causing pegboard-envoy to replay the stop forever. The three new tests cover the core scenarios well (immediate ack, retained-on-send-failure + replay retry, unknown-actor stop).

A few things worth a look:

1. The immediate ack sweeps the entire processed_command_idx map, not just the stop's own checkpoint.
batch_has_stop only gates whether to fire send_command_ack_inner(ctx, false), but that function still builds last_command_checkpoints from every entry in ctx.processed_command_idx (commands.rs:108). So any batch containing a stop, even a duplicate/replayed one, now also immediately acks and clears (server-side) the replay safety net for unrelated, just-processed CommandStartActors in the same batch, rather than waiting for the 5-minute periodic tick (ACK_COMMANDS_INTERVAL_MS). The existing TODO above (commands.rs:139-148) already documents a narrow race where an ack can be committed by pegboard-envoy before the local dedup map reflects it; this change increases how often that ack fires for freshly-started actors, shrinking the crash-recovery window (envoy process dies right after a start is acked but before the actor is durably initialized) compared to before. Worth confirming this is intentional/acceptable, since it's a meaningful behavior change beyond "ack stops promptly", or scope the immediate ack to just the stop checkpoints if unintended.

2. ActorEntry::last_command_idx is now dead.
After switching ack sourcing to processed_command_idx, nothing in the crate reads entry.last_command_idx anymore (still written in commands.rs:74 and via insert_actor's last_command_idx param, but never read, confirmed via grep across the crate). Since the field/param are pub, this won't trigger a dead_code warning, but it's vestigial after this refactor. Consider removing it (and the insert_actor parameter) unless it's meant to stay for a future purpose.

3. Minor comment accuracy nit.
The comment at commands.rs:90-92 ("Ack a stop immediately since its actor is removed before the periodic tick") implies removal happens synchronously with the stop command. Looking at events.rs:14-36, remove_actor is actually only called later, when the actor's own Stopped state-update event arrives with received_stop already set, a separate, later event. Not wrong in effect (removal can still race ahead of the 5-minute tick), but slightly imprecise about when/why; might be clearer as "...may be removed before the next periodic tick."

4. Test coverage suggestion.
Given point 1, a test pinning the "a stop in the batch also flushes unrelated pending checkpoints (e.g., a start for a different actor in the same or a prior batch)" behavior would help make that scope-widening explicit and regression-proof, since it's the main behavioral change here beyond the targeted stop-ack fix.

Nothing blocking, the core fix is sound and matches the stated goal (stop pegboard-envoy from replaying acked-but-forgotten stops). Points 1 and 2 are the ones I'd want a second opinion on before merging.

[claude]

Review

Solid, well-targeted fix. The root cause (stopped actors are removed from ctx.actors before the next periodic tick, so the live-actor scan that drives send_command_ack can never re-ack their stop, leaving pegboard-envoy to replay the stop command forever) is correctly diagnosed, and the fix sends an immediate, scoped ack right after processing a batch that contains stop commands.

Correctness

• Collecting stopped_actors from the raw batch before the dedup loop (commands.rs:23-27) is the right call: it ensures a replayed stop that gets skipped by dedup still triggers a re-ack, which is exactly what stop_ack_retried_via_replay_after_failed_send and unknown_actor_stop_is_acked verify.
• Not clearing processed_command_idx on the immediate stop ack (only the periodic tick clears it) is a deliberate, documented tradeoff that extends the existing ack-of-ack race noted in the pre-existing TODO comment (commands.rs:154-165) to removed actors. The comment was updated to call this out explicitly rather than leaving it implicit, which is good.
• The merge logic in send_command_ack (live actors union processed_command_idx, keyed by max index) is correct: it still recovers unconfirmed acks for live actors every tick, and separately covers stopped/removed actors via the dedup map, clearing both scopes together once the ack send succeeds.
• Keying by (actor_id, generation) throughout avoids cross-generation interference for restart/resurrection scenarios. Unknown-actor stops still populate processed_command_idx unconditionally (pre-existing behavior in the main loop), so they get acked too, matching unknown_actor_stop_is_acked.
• Net effect on the processed_command_idx map is a leak fix, not a new leak: previously entries for removed actors were never cleared at all (the old tick only scanned ctx.actors); now the periodic tick eventually clears them too.

Minor notes (non-blocking)

• send_stop_command_acks ignores the boolean returned by send_ack_checkpoints. That's fine given the "let a replay or the next tick retry" design, but a short comment noting the return value is intentionally discarded would save a future reader from wondering if that's an oversight.
• If a single batch ever contained two stop commands for the same (actor_id, generation) (shouldn't happen given monotonic indices, but not structurally prevented), stopped_actors would contain the pair twice, causing a harmless duplicate ack send. Not worth guarding against given the batch is engine-generated.

Tests

Good coverage of the actual regression: immediate ack on stop, retry-via-replay when the immediate send fails, the unknown-actor case (restart before the start command was ever locally observed), and confirming the periodic tick still re-acks live actors independently. The decode_ack_checkpoints helper correctly round-trips through the real versioned wire format rather than asserting on internal state, which keeps the test honest to the actual protocol boundary.

No security or performance concerns. This only touches an already-trusted internal envoy to pegboard-envoy path per the repo's trust boundary conventions, and the added HashMap allocations are bounded by batch and actor-map size and are local, not shared or concurrent, so the scc::HashMap guidance in CLAUDE.md doesn't apply here.

Re-verified against the current diff (single commit, unchanged since the prior review); same conclusions hold.