Skip to content

feat(runtime): implement mid-turn guidance injection with continuation pass - #629

Closed
sunheyi6 wants to merge 5 commits into
apache:mainfrom
sunheyi6:feat/mid-turn-guidance
Closed

feat(runtime): implement mid-turn guidance injection with continuation pass#629
sunheyi6 wants to merge 5 commits into
apache:mainfrom
sunheyi6:feat/mid-turn-guidance

Conversation

@sunheyi6

@sunheyi6 sunheyi6 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

This feature allows users to inject new steering instructions while the AI is still generating a response — the model will persistently perceive the guidance across subsequent steps, and auto-trigger a follow-up generation if the guidance arrives after a text-only reply.

Key Changes

Runtime Core

  • Switch pendingGuidance (one-shot) → injectedGuidance (persistent across steps)
  • Add continuation pass loop — guidance arriving after text-only reply triggers a follow-up model call
  • Expose injectGuidance() on RuntimeKernel and SessionManager
  • Add onThinkingComplete callback to ModelAdapter
  • Record guidance events to session store in real-time via agent-run.ts

Events

  • Add guidance event type to events.ts

UI/Materialize

  • materializeTurns supports multiple user/assistant messages per turn: first = prompt/answer, subsequent = steers / assistantFollowups
  • Chat view renders guidance injection badges and follow-up responses
  • Composer supports sending guidance mid-turn
  • New MessageCircleQuestion icon for guidance indicators

Desktop Integration

  • AppShell handles session-events for guidance, updates chat store
  • IPC exposes injectGuidance channel via main/preload
  • Composer CSS styles for guidance UI

Tests

  • Added guidance tests for composer-helpers, materialize, session-manager
  • Updated model-adapter and streaming-handoff test fixtures

Files Changed

25 files modified across 4 packages (core, runtime, ui, desktop)

@sunheyi6
sunheyi6 force-pushed the feat/mid-turn-guidance branch from 4b2b73e to 6aa0a9c Compare July 8, 2026 01:04

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I like the shape of the mid-turn guidance path. I found a few follow-ups that I think are worth fixing, but I’m leaving this as an approval rather than a blocker.

  • P2: The desktop renderer exposes sessions.injectGuidance, but AppShell does not pass onInjectGuidance into Composer. Because of that, the “Now” action in the queued composer path falls back to a normal send instead of steering the running turn. Wiring onInjectGuidance to window.maka.sessions.injectGuidance(activeId, text) should close the loop.
  • P2: shellRunContextSummary no longer gets merged into the current user turn tail. The helper and input contract are still there, but send() now only uses resolveTurnTailPrompt(). Restoring the joinPromptFragments([turnTail, shellRunContextSummary]) path would keep live shell context visible to the model.
  • P3: Guidance continuation calls are not counted in token_usage or recordLlmCall. That means the new follow-up generation path can underreport usage and cost. This can be handled either by accumulating continuation usage into the final usage event or by emitting separate usage for each continuation.

@Astro-Han

Copy link
Copy Markdown
Contributor

Also rebase is needed!

@sunheyi6
sunheyi6 force-pushed the feat/mid-turn-guidance branch 3 times, most recently from a57e495 to 860243f Compare July 8, 2026 10:57
@Astro-Han

Copy link
Copy Markdown
Contributor

CI failed and need repair.

sunheyi6 added a commit to sunheyi6/maka-agent that referenced this pull request Jul 8, 2026
…rializer

PR apache#629 regressed the per-step persistence and RuntimeEvent replay
materializer introduced by apache#633. Restore flushStep/currentStepMessageId
(one AssistantMessage per AI SDK step, signed thinking merged with its
tool call, orphan tool_results dropped) while keeping the guidance
continuation pass and standing-steer prepareStep.

@jackwener jackwener left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review done. The feature is worth having and the abort interplay + sessionSendChain serialization are solid, but there's one bug that blocks merge outright:

BLOCKER — guidance injected during a tool window corrupts next-turn replay (provider 400). The guidance event is appended to the ledger in stream order, so injecting while a tool is executing (or parked on a permission prompt — the single most natural moment to steer) lands it between function_call and function_response. On the next turn, materializeRuntimeReplayPlan hits the user-role guidance, flushes the pending calls via pushToolResults before the result has been iterated, and emits a tool_use with no matching tool_result → Anthropic 400, session wedged. Simplest fix: buffer guidance and append to the ledger only at the next step boundary (never between a call and its result); alternatively defer the user message in the replay materializer until buffered calls' results flush.

Also required:

  • RuntimeKernel.injectGuidance fans out to every active child subagent sharing the sessionId — one steer click injects into the parent AND all running subagents (duplicate 引导已注入 rows, stray user messages in subagent transcripts). Target the top-level run only.
  • The injection UI is unreachable exactly when steering matters most: during a permission park the status is waiting_for_user, isComposerResponseBusy says not-busy, so sends bypass injection and queue as a new turn. Treat waiting_for_user as busy for injection (after the blocker is fixed).
  • Continuation passes drop token usage entirely (the :1252 comment concedes it) — every pass re-sends the full history uncounted. Accumulate totalUsage across passes into the turn's token_usage.
  • guidanceConsumedCount is racy: a steer arriving during the last prepareStep's async body can be counted consumed without being delivered (and never triggers continuation). Track a monotonic delivered-index instead of a count both sides mutate.
  • Test coverage doesn't touch any hard path: the only runtime test uses a fake backend yielding a canned guidance event. Need real-AiSdkBackend (mock model) tests for: mid-tool-step injection → well-formed next-turn replay (no orphan tool_use); text-only turn + late steer → exactly one continuation; loop termination under repeated injection; abort during continuation.

Minor: hasSteer keys on user-row count alone, so a steer consumed inline (no continuation) still fragments a normal multi-step answer into 引导跟进 blocks — distinguish steer-with-continuation from steer-consumed-inline. And please drop the cosmetic quote-style comment churn + BrowserPanelFallback block move from the diff.

The core idea and most of the plumbing are right — fix the ledger-ordering blocker and the fan-out, add the replay tests, and this merges.

@sunheyi6
sunheyi6 force-pushed the feat/mid-turn-guidance branch from 2fa7cb6 to 273c387 Compare July 8, 2026 23:57
@sunheyi6
sunheyi6 requested a review from jackwener July 9, 2026 00:01
sunheyi6 added 2 commits July 12, 2026 11:53
…n pass

- Add mid-turn guidance injection in runtime (ai-sdk-backend, model-adapter, runtime-kernel, session-manager)
- Inject guidance persists across steps; continuation pass triggers a follow-up model call when guidance arrives after a text-only reply
- Expose injectGuidance() on RuntimeKernel and SessionManager; add injectGuidance to AgentBackend contract in @maka/core/backend-types
- Add guidance event type to events.ts; record guidance events to session store via agent-run.ts
- UI: materializeTurns surfaces mid-turn guidance as a 'steer' entry in the turn timeline (renders a '引导已注入' marker); chat-view renders it via TurnTimelineEntry
- Composer supports sending guidance mid-turn; desktop IPC exposes injectGuidance channel
- Tests for composer-helpers, materialize, session-manager
Blocker + required fixes (@jackwener, @Astro-Han):

- Buffer guidance events to step boundaries (finish-step / turn-end) so a
  steer injected during a tool window or permission park never lands between
  a tool_call and its tool_result — next-turn replay stays well-formed
  (no orphan tool_use). injectGuidance now only buffers; flushPendingGuidanceEvents
  emits at safe boundaries.
- Replace racy guidanceConsumedCount with a monotonic deliveredToModelIndex
  (advanced only when the slice is read) and a separate pendingGuidanceEventIndex.
- RuntimeKernel.injectGuidance targets the top-level run only — no fan-out to
  child subagents (was duplicating steers across subagent transcripts).
- Accumulate continuation-pass token usage into the turn's token_usage via
  sumNormalizedAiSdkUsage so follow-up passes no longer underreport usage/cost.
- isComposerResponseBusy treats waiting_for_user as busy so a send during a
  permission park steers the running turn instead of queueing a new one.
- AppShell wires onInjectGuidance to window.maka.sessions.injectGuidance(activeId, text).
- Restore shellRunContextSummary merge into the current user turn tail
  (joinPromptFragments([turnTail, shellRunContextSummary])).

UI: surface mid-turn guidance as a 'steer' entry in the turn timeline
(rendered via TurnTimelineEntry) instead of separate steers/assistantFollowups
arrays — a steer consumed inline no longer fragments a normal multi-step
answer; a steer followed by a text entry is a guidance continuation.

Tests: real-AiSdkBackend (mock model) tests for mid-tool-step injection
(no orphan tool_use), text-only turn + late steer (exactly one continuation),
repeated injection loop termination, and abort during continuation. Update
materialize + composer-helpers tests for the new model.
@sunheyi6
sunheyi6 force-pushed the feat/mid-turn-guidance branch from 273c387 to 3cab623 Compare July 12, 2026 03:56
sunheyi6 added 3 commits July 12, 2026 13:34
- ui/chat-view: guard timeline item .items access behind
  item.kind === 'tools' (union now includes 'steer', which has no
  .items member) — was a hard TS2339/TS7006 error.
- desktop/app-shell-session-events: fix misplaced brace so the
  'guidance' and 'default' cases are not swallowed inside the
  'complete' block (latent syntax error once ui build unblocks).
- desktop/streaming-handoff.test: drop the obsolete
  'assistant streaming handoff' describe block. It exercised
  drainAssistantStreamSlot / clearSettledAssistantStreamSlot /
  AssistantStreamSlot, all of which were removed from the
  codebase, so the suite no longer compiles.
…stability)

- ui: drop per-callsite strokeWidth on the guidance icons
  (MessageCircleQuestion in chat-view, GripVertical/ArrowUp/Pencil/
  Trash2 in composer). The icon + typography governance contract
  forbids fragmenting lucide's governed stroke.
- desktop test: the effect-stability harness bundles app-shell-effects
  with esbuild; the PR's new import of preservePendingOptimistic pulls
  @maka/ui -> lucide-react (CJS) into the bundle, whose
  require('react') breaks esbuild's shim. Externalize all node_modules
  (packages: 'external') so react/react-dom/lucide-react resolve at
  runtime instead.
@sunheyi6

sunheyi6 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

关闭原因 / Why this PR is being closed

This PR is superseded and closed in favour of #2222 — the same fix rebuilt on top of what upstream shipped since this PR was created.

Timeline:

Why not rebase this PR: its runtime (injectGuidance, guidance events, continuation accounting) would duplicate the steering mechanism upstream already ships in #1087/#1357. The reviewer feedback on this PR (buffer guidance to step boundaries, no subagent fan-out, waiting_for_user treated as busy, real-backend tests) is fully addressed in #2222 — using the upstream steering queue instead of a second, parallel mechanism.

#2222 reuses the upstream steering queue and adds only what was missing:

  • Desktop main sessions:send routes plain-text sends through runtime.steer (converged for chat/bot/voice/goal entrypoints), returning { ok, steered }.
  • Live-transcript rendering of the injected steer at its step boundary (via the live turn projection), including a bounded same-turn continuation pass for steers that arrive while the model is streaming its final text.
  • Composer queue UI (public Token chips, aligned to the input card width) with an immediate “立即发送” action and idle auto-drain.

Please direct further review/effort to #2222 (fixes #1954).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants