Skip to content

fix(agent-org): prevent Coordinator takeover and false delivery #995

Description

@ShiboSheng

Dependency and goal

Stack strictly on #763 / PR #991 at b96bd8b196e2c7fed6bcbcd05fd7261e3bf5e706, and land before #764. The Agent Org redesign rollout gate remains disabled by default until this fix and the later Group/projection gates pass.

This issue owns one safety correction: a Coordinator may orchestrate formal work, but must not poll, interrupt, cancel, take over, perform, and then self-certify the same work. It freezes four inseparable backend boundaries:

  1. Coordinator orchestration capability versus Member execution capability.
  2. Event-driven Coordinator waiting versus Provider/tool polling.
  3. Safe cancel/reassign handoff versus concurrent old/new side effects.
  4. Verified delivery evidence versus free-text completion claims.

The normative contract is the local design document docs/architecture/agent-org-long-lived-team-session-design.md, especially invariants 44–49 and §25.11A. Design remains the highest authority.

Evidence and root cause

The real-provider Team session Build Local Texas Holdem Game exposed the gap after PR8:

  • Coordinator performed 15 file-mutation tool calls while Implementer performed 25, so orchestration and execution were not actually separated.
  • Coordinator polled Task state 17 times with task_list, 5 times with task_get, and used 8 shell sleeps in a short interval instead of ending its Turn and waiting for durable events.
  • Coordinator cancelled all four formal Tasks while an Implementer was still producing workspace side effects, then continued the work itself.
  • Every observed Task cancellation was authored by Coordinator. The old TaskExecution continued writing after the durable Task had become Cancelled.
  • The run could remain at Finalizing or produce a success-like summary even though the formal board had no Completed Task/TaskOutput closure and all Tasks had been cancelled.

These are not prompt-only defects:

  • Coordinator is forbidden from owning a formal Task, but current tool assembly can still inherit ordinary SDE file/shell/browser capabilities.
  • The design previously rejected stale Task lifecycle/TaskOutput after cancel/reassign but explicitly allowed old workspace/external side effects to continue.
  • There is no hard rule that ends a Coordinator Turn after dispatch when no new work_revision or durable event exists.
  • Quiescence/Idle and a free-text completion call do not prove that the requested deliverable exists.

State after merge

  • A persisted coordinator Turn derives one CoordinatorOrchestration allowlist: bounded read, Task graph admin, Plan Approval, formal coordination, and completion request only.
  • Coordinator cannot write/move/delete files, run shell/process/tests/builds, operate browser/Computer Use, mutate external systems, use a generic delegate, own TaskExecution, or write TaskOutput.
  • After dispatch, a Coordinator Turn with no new durable event/revision terminates as waiting_for_org_event. Task, TaskOutput, Inbox, Plan, or user commits may coalesce into one new Root wake.
  • Cancelling or reassigning an in-progress Task atomically creates an execution handoff. A replacement remains Pending and cannot dispatch until the exact old runtime lease and Turn-owned processes are quiescent.
  • Only a backend-validated completion certificate may produce delivered. Idle, all-terminal Tasks, all-cancelled Tasks, Coordinator prose, and UI inference are not completion authority.

Implementation steps and authoritative entry points

1. Coordinator work profile

  • Derive CoordinatorOrchestration only from the persisted Agent Org Turn kind; do not persist a second role/capability flag.
  • Build a direct allowlist, not a denylist layered over the ordinary SDE registry.
  • Allow bounded transcript/Task/TaskOutput/Plan/Artifact/code reads, graph admin, Plan Approval, formal coordination, user reply, and completion request.
  • Exclude file mutation, arbitrary shell/process, test/build/deploy, browser/Computer Use, external mutation, generic delegate/subagent, TaskOutput write, and Task owner lifecycle.
  • Enforce the same decision in schema assembly, execute-time actor resolution, and the lowest Store/adapter boundary. Production Tauri, restart/recovery, debug/test entry points, and direct Tool construction use the same resolver.
  • Do not add a “read-only shell” classifier; arbitrary shell cannot be proven side-effect free from command text.

2. Event-driven Coordinator waiting

  • Persist the Coordinator trigger event and observed work_revision.
  • Permit one authoritative revision-keyed observation per trigger. Re-reading the same revision returns typed coordinator_no_new_work_facts guidance without repeated database queries.
  • When no committed fact changed, a no-progress breaker terminates the Turn with waiting_for_org_event.
  • Do not expose sleep, await-output, periodic Task polling, delayed self-message, per-Team timer, or periodic Provider wake.
  • Task/TaskOutput/Inbox/Plan event bursts coalesce to at most one Root wake. Working Watchdog remains a bounded repair safety net, not a model polling engine.

3. Safe Task cancel/reassign handoff

  • In one IMMEDIATE transaction, freeze the old in-progress Task, write structured cancel reason, create the Pending replacement when applicable, create its blocked assignment reference, and persist one TaskExecutionHandoffReceipt bound to old Task/owner/session/Turn/runtime lease/turn generation.
  • Receipt states are requested | yielding | released | timeout | unknown | failed; the receipt is not a new Task status.
  • Commit the fence before requesting Provider/process yield.
  • Every workspace-mutating TaskExecution tool revalidates Task/owner/Turn/lease/handoff immediately before its side effect.
  • Long-running processes are Turn-owned and stop as a process group.
  • Only after Provider terminal, exact release_if_current, zero Turn-owned processes, and no unknown external call may the receipt become Released and kick the existing replacement assignment/FIFO.
  • Record a 5-second SLO miss and enter durable Timeout/Unknown at 10 seconds. Timeout/Unknown keeps replacement blocked and waits for an explicit user resolution command; it never opens a second writer lane.
  • Restart resumes the same stop/wait receipt. It does not replay old or replacement side effects.

4. Verified completion

  • Treat a Coordinator completion call as a candidate, not authority.
  • In one IMMEDIATE transaction validate current Team, activation generation, work revision, all formal Turn/Inbox/recovery/handoff blockers, and the Task resolution closure.
  • A Task is satisfied only by Completed + TaskOutput, by a replaces_task_id chain ending in Completed + TaskOutput, or by a scoped user_scope_removed cancellation that cites the exact current-Team user EventStore event.
  • Every Task in the current work episode enters the closure by default. Do not add a model-controlled optional flag.
  • Persist one idempotent RunCompletionCertificate containing outcome, generation/revision, evidence Task/TaskOutput ids, bounded resolution links, validator version, and time.
  • delivered requires a complete closure and no blocker. Whole-episode user abandonment/Archive is cancelled; unresolved Failed/Unknown/needs-attention work is failed.
  • All-cancelled/all-failed/missing-output/stale-evidence/pending-handoff requests cannot become Delivered.
  • The final Coordinator summary references the committed certificate. Ordering remains certificate → assistant EventStore commit → Coordinator Turn terminal → Quiescence/Idle.

5. Wire, UI, observation, and recovery

  • Project Coordinator waiting_for_org_event separately from typing/running.
  • Project handoff waiting_for_previous_execution | released | timeout | unknown separately from Task status; do not optimistically show replacement Pending as running.
  • Show Delivered only with a certificate and bounded evidence. Cancelled/Failed/needs-attention must not use success copy.
  • PR8S only forbids uncertified success and projects certificate-backed outcomes. The unique FinalSummaryReceipt, bounded Finalizing lifecycle, terminal failure, and Retry belong to PR8F.
  • Record no-content metrics for tool denial, observation count, no-progress terminal, event-to-wake coalescing, old/new active workspace writers, handoff timing/outcome, certificate validation, and final commit ordering.

Scope boundaries

  • No Coordinator Task ownership or Coordinator TaskExecution.
  • No second Member runtime, scheduler, dispatcher, or workspace writer lane.
  • No new Task draining, revoked, waiting, waiting_for_user, or superseded state.
  • No rollback of side effects already committed before cancellation. Unknown non-idempotent outcomes remain visible and are not replayed automatically.
  • No arbitrary external exactly-once protocol.
  • No polling/background recovery loop.
  • No Group mention or peer Inbox source; feat(agent-org): [9/10] add multi-target Group ingress and linked Member Inbox #764 still owns those adapters.
  • Direct intervention, Team Pause/Archive, user Stop, and Task cancel keep distinct typed intents and receipts.
  • Transcript auto-scroll/typing-layout defects, repeated PR8 Return projection, and Planner plan-history visibility remain separately owned defects unless their producing invariant is directly changed here.

Failure scenarios that must be rejected or remain blocked

  • Coordinator receives a work tool through ordinary SDE fallback, stale registry cache, restart, direct Tool construction, or debug/test entry point.
  • Coordinator polls the same revision, sleeps, awaits output, self-wakes, or remains an active Provider with no new durable fact.
  • Cancel/replacement/handoff commits only partially.
  • Replacement starts before the old lease/process is released.
  • A stale old tool call produces a new side effect after the fence.
  • A stale release callback removes a newer runtime.
  • Timeout/Unknown automatically releases replacement or replays a non-idempotent side effect. “Keep stopped” cancels only the blocked replacement, does not restore the old Task, and leaves other Tasks running.
  • All Tasks Cancelled/Failed, missing TaskOutput, stale evidence, incomplete replacement closure, or pending handoff still produces Delivered.
  • Free text, Team Idle, all-terminal Tasks, or frontend state becomes success authority.
  • Ordinary SDE Send/Queue/Force Send/Stop/Resume/Compaction enters Agent Org tool/handoff/finality logic.

Acceptance

Deterministic owning-boundary tests

  • Coordinator schema/execute/adapter/Store allow-deny matrix, including restart, direct construction, and debug/test parity.
  • Trigger/revision observation idempotency, no-progress terminal, event burst coalescing, crash windows, and five-minute zero-wake/query/process checks.
  • Cancel/complete/reassign/Pause/Archive races; exact lease/generation CAS; process-group teardown; stale callback; timeout/unknown and explicit user resolution.
  • Active workspace-mutating TaskExecution count is always at most one across old/replacement owners.
  • Completion certificate all-valid/all-cancelled/all-failed/missing-output/stale-evidence/replacement/user-scope-removal matrices, response-loss replay, and EventStore/terminal ordering.
  • Canonical DDL/reset only while this rollout-gated data has no external compatibility promise; no ALTER TABLE or legacy probe unless preservation is explicitly requested.
  • Ordinary SDE has zero Agent Org context, receipt, query, listener, timer, or tool-profile change.

Real provider and packaged Tauri

Use a fresh isolated Team and a real provider to reproduce a realistic version of Build Local Texas Holdem Game:

  1. User asks Planner, Implementer, and Tester to build and verify a functioning local application.
  2. Coordinator creates and assigns formal Tasks, then becomes event-waiting; it never edits files, runs commands/tests, controls a browser, polls Tasks, or takes over.
  3. Implementer modifies the fixture and Tester runs the real app/tests. Tool trajectories and filesystem evidence belong to those Member TaskExecutions.
  4. While Implementer is deliberately slow, confirm five minutes of no Coordinator Provider/query/sleep activity until a durable progress/TaskOutput/Inbox event arrives.
  5. Cancel/reassign one in-progress Task through the real UI. The replacement stays Pending until the old runtime/process is released, and old/new active workspace writers never overlap.
  6. Exercise Timeout/Unknown resolution with visible Computer Use actions; debug endpoints may seed the isolated fault or read evidence only.
  7. Prove all-cancelled work cannot show Delivered; then complete a valid replacement closure with real TaskOutputs and prove exactly one certificate-backed delivery.
  8. Use Computer Use for every visible Send, Stop, Pause, Resume, Archive, Delete, confirmation, Session switch, and handoff-resolution action in the packaged Tauri App.
  9. Record database receipts, exact EventStore ordering, tool-schema/trajectory evidence, provider/runtime counts, process cleanup, screenshots, and ⌘5 request evidence.

Real-provider unavailability, quota exhaustion, or rate limiting is Blocked, not Pass.

Effort

Category P50 P90
Production 7,500 review lines 13,000 review lines
Tests / E2E / measurement 8,500 15,500
Old-path removal / locale / audit 1,500 3,000
Total 17,500 31,500
Substantive files about 95 about 145
Additional locale files up to 13 up to 13
Audit/evidence documents about 5 about 8

At 12,250 review lines (70% of P50), recheck scope and actual diff. If the forecast exceeds 31,500 review lines or 145 substantive files, sync Design/Issue and report before continuing. Above 47,250 review lines (P90 × 1.5), re-split the Impact Spike. If the work requires Group sources, a second runtime, a new Task state, external transaction compensation, more than three unplanned production files / 200 production lines, or total growth above 20%, stop and update the design and issue before implementation. Do not reduce authority, race, real-provider, packaged-app, SDE, or performance evidence to fit the estimate.

Plain-language contract

The Coordinator is the project manager, not a backup engineer. On a Timeout/Unknown handoff, Keep stopped cancels only that replacement while the rest of the Team continues. It can read the work, assign it, and verify real Member outputs. After assigning work it waits for an event instead of repeatedly asking whether the Member is done. If work must be reassigned, the old worker is safely stopped before the new worker starts. The Team may say “Delivered” only when the backend can point to completed Tasks and their actual outputs.

2026-08-27 completion candidate deadlock regression

真实 Provider Session PR8S TEST: Texas Holdem Game 暴露了 PR8S 阻断缺陷:4 个正式 Task 均为 completed 且 4 个 TaskOutput 均存在,work_revision=9、Coordinator observed revision=9,但 completion request=0、RunCompletionCertificate=0;最后两个 Coordinator Turn 都以 waiting_for_org_event 结束。EventStore 证明 Coordinator 两次准备交付,却按旧指令先调用 task_list,随后被同 revision 的 coordinator_no_new_work_facts 正确结束,org_run_complete 从未执行。Provider 网络告警之后已经恢复,不是根因。

修复边界:completion candidate readycertificate 已发布、Quiescence 可以进入 Idle 是不同事实。Candidate assessment 必须由唯一 completion owner 以纯读方式生成,并与证书 validator 复用 Task closure、TaskOutput、replacement、scope-removal、handoff 和非 Task blocker 逻辑;尚无 certificate 不能阻止 candidate ready,但仍继续阻止 Quiescence/Idle。原子 Coordinator Prompt snapshot 在 ready 时必须直接引导一次 org_run_complete,不得要求再次调用 task_list;同 revision no-change 与 waiting_for_org_event 保持不变。

工具契约同步拆分:task_list 提供 completion_candidate.state = ready | blocked | not_applicable | certified,Quiescence blockers 只解释证书发布后能否 Idle。No-change wire 必须显式投影为等待事件,不能显示成 0 tasks。Needs Attention 继续作为无证书时的安全状态;FinalSummaryReceipt、最终报告失败/Retry 和正式事件精确一次仍属于 PR8F / #997

回归验收:最后一个 Task 完成后,不发送纠正消息,Coordinator 从原子快照直接调用一次 org_run_complete;SQLite 恰好生成一张当前 generation/revision certificate;Overview 由 Needs Attention 自动变成 Delivered。Packaged Tauri 的所有可见路径使用 Computer Use,真实 Provider trajectory 不得出现完成前的强制 task_list,debug endpoint 只读取证据。

补充规模:新增 P50 1,700 / P90 3,400 review lines,约 12 / 19 个实质文件触达;当前实现快照约 9,724 review lines、126 个实质文件,PR8S 总量预测约 11,424 / 13,124 review lines、138 / 145 个实质文件。达到 12,250 review lines 或预测超过 145 个实质文件时重新统计并同步 Design/Issue。

2026-08-27 final implementation and real-provider evidence

第一版 packaged 回归进一步暴露了同一 owning boundary 的时序窗口:经过 Store 校验、已物化且未读的系统 MemberIdle 事件可以先于对应 Member TaskExecution Turn 的 terminal bookkeeping 被 Coordinator claim。修复只把该精确 Idle row 所指向的同一 Member、同一当前 TaskExecution Turn 投影为 terminal;伪造事件、陈旧事件、更新 Turn 或非匹配 Member 一律 fail closed。Candidate assessment、certificate validator 与 Quiescence 复用该有界投影,没有加入轮询、timer、后台扫描或 schema migration。

最终 packaged artifact 为 ORG2 Instance 2,executable SHA-256 为 3f208a2506e478c2b8071f2abc4aa71435094613019f918c7165976452ccc209。Computer Use 在全新隔离 ORGII_HOME/workspace 中使用真实 Provider orlando / GPT 5.6 Luna 完成自然 Texas Holdem 旅程:Planner、Implementer、Reviewer、Tester 共 4 个正式 Task 全部 completed 且 4 个 TaskOutput 存在;最后一个有效 Coordinator Turn 在 work_revision=9 直接调用 org_run_complete 恰好一次,且该最终 Turn 没有 task_list

SQLite 恰好生成一张 delivered certificate 73347e5e-965a-4039-80ff-2209e8b8d5e9。顺序证据为 certificate 06:41:51.617603Z → typed assistant publication binding 06:42:17.996650Z → Coordinator Turn terminal 06:42:18.067104Z → Run Idle 06:42:18.072905Z。Overview 由安全的 Needs Attention 自动变为 Delivered,显示 4 / 4 done 和 Coordinator inactive;没有用户纠正 completion prompt。

最终 agent_core 单线程全量测试为 3308 passed / 0 failed / 2 ignored;packaged focused WDIO completion regression 1 passed。当前提交总量为约 11,600 review lines(其中 308 行来自提交 hook 对 13 个 locale JSON 的机械格式化)、138 个实质文件,未达到 12,250 review-line 复核点和 145 文件 P90 闸门。

Provider 旅程中曾有一次临时 503,自动恢复后继续完成。另观察到模型对依赖关系的错误判断及无效 task_update/inbox_repair 参数,但 Store 全部 fail closed、零状态变更,且没有阻止证书闭环;它不属于本次证书死锁根因,作为 PR8F/watchdog 收敛观察项保留。

2026-08-27 Member coordination chatter regression

真实 Provider 手动测试发现,TaskExecution Member 会把“准备开始”“完成了哪些模块”“下一步做什么”和已经自行解决的小问题作为 plain 消息发给 Coordinator。每条消息都会持久化到 Inbox,并在同一事务中登记 Coordinator trigger;即使 UI 只显得啰嗦,后台仍可能浪费 Provider Turn、token 和调度延迟。

本补充修复保持 Task board 作为进度来源、TaskOutput 作为完成交付单。TaskExecution Member 只有在 Coordinator 确实需要行动时才能发送 plain 消息,并必须绑定当前精确 related_task_id 和 purpose:blocker、decision_required、material_change、risk 或 requested_reply。requested_reply 仅用于 Coordinator 明确要求的中途回复。Prompt 与工具说明明确禁止普通施工直播和重复完成消息。

Store 不分析正文,也不做关键词分类。它只在既有 receipt 事务内重新核对 persisted TaskExecution Turn、当前 generation、精确 Task、in_progress 状态、当前 Owner、Running Team 和 purpose 枚举。缺 Task 或 purpose 返回 typed guidance;伪造身份、stale Turn、未知 purpose 或失效 Task fail closed。所有失败均为零 Inbox、零 trigger 变化和零 wake。purpose 与 Task 只保留在工具调用、幂等 receipt/result 和无正文指标中;Inbox payload、SQLite schema、Tauri/UI/locale 均不变化。

本次不增加节流、timer、debounce、正文审核或 informational channel。合法 blocker/decision/risk 消息仍立即持久化并复用现有 trigger coalescing。已知取舍是模型仍可能把普通进度错标为 risk;本次用 Prompt、真实 Provider 回归和无正文指标约束,而不以语义误杀真正协调消息。

自动化验收要求覆盖五种 purpose、缺失/错 Task、terminal/reassigned/stale/Paused/Archived、幂等与 burst coalescing,并证明拒绝路径不写 Inbox、不改变 trigger sequence、不调用 wake hook。Packaged Tauri 继续使用 Computer Use 和真实 Provider orlando / GPT 5.6 Luna;自然工作流不得反复直播进度,隔离 blocker 必须恰好送达并唤醒 Coordinator。

规模:本补充实际改动约 490 review lines、8 个既有生产/测试文件,无 schema/UI/locale 文件,处于 P90 500 行内;PR8S 总量约 12,090 review lines,仍低于 12,250 行复核点。

2026-08-27 final coordination-noise evidence and accepted boundary

用户确认接受少量 TaskExecution 可见 narration。它只作为 Member Session 的 transcript 展示,不属于正式协调消息;PR8S 的硬验收边界是这些文字本身不写 Agent Org Inbox、不登记 Coordinator trigger、也不启动 Coordinator Provider Turn。本补充不增加 UI 隐藏、EventStore 缓冲、正文分类、timer 或 informational channel。

最终 packaged artifact 为 ORG2 Instance 92,executable SHA-256 为 ec2c9e747e634039b605d094e8039d0f1bb53b5ba1bab81a16d062a2723a23bc。Computer Use 在隔离 ORGII_HOME 与 workspace 中使用真实 Provider orlando / GPT 5.6 Luna 驱动 Build Tested Texas Holdem App。Implementer 的可见进度 narration 没有产生任何成功的 Member plain org_send_message receipt,也没有产生 Member plain Inbox row。Reviewer 的两条实质 correctness 发现以 purpose=material_change 和同一精确 Task id 合法写入两次、读取两次,并即时触发 Coordinator 处理,证明合法协调通道没有被误杀。四个 invalid plain 调用保持 invalid_params,未成为已接受协调消息。

Provider 旅程遇到外部 HTTP 429/503、反复文件编辑失败及一次 packaged 进程退出;重启同一隔离数据后 durable run 完整,最终通过真实 Team Overview Pause,故不为本次长旅程声明 certificate 或 Delivered。既有自然真实 Provider 证书旅程仍证明最终交付路径。本次自动化结果:agent_core 3313 passed / 0 failed / 2 ignored;session_persistence 46 passed;warnings denied Clippy、TypeScript typecheck、Rust format check、git diff check 均通过;focused rendered completion regression 1 passed。

最终 diff 为 12,230 review lines,141 个实质文件加 13 个 locale 文件,仍低于 12,250 行复核闸门和 145 个实质文件 P90。

Metadata

Metadata

Assignees

Labels

Effort: HighagentAgent runtime, behavior, memory, providers, or orchestrationbugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions