Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
d80fb7e
feat(sdk): compose virtual machine execution profiles
KaiEureka Oct 24, 2026
1d29126
feat(workflows): configure guest-authored child execution defaults
KaiEureka Oct 24, 2026
6cef0ef
docs(environments): explain public virtual machine execution profiles
KaiEureka Oct 24, 2026
3411b96
fix(environments): allow configured VM cleanup to finish
KaiEureka Oct 24, 2026
064ecb9
fix(cli): retain configured cleanup grace after interruption
KaiEureka Oct 24, 2026
5463483
fix(runtime): honor configured scheduler cleanup on cancellation
KaiEureka Oct 24, 2026
16ed3ef
test(environments): cancel live guest commands after explicit readiness
KaiEureka Oct 24, 2026
0bf3e80
fix(workflows): preserve entry isolation with unused VM profiles
KaiEureka Oct 24, 2026
f2246f1
fix(runtime): expose configured capabilities to model callers
KaiEureka Oct 24, 2026
c50e7cd
docs: report model-facing capability audit
KaiEureka Oct 24, 2026
009c849
fix(context): preserve private role prompts during discovery
KaiEureka Oct 24, 2026
8a58961
fix(tools): describe active execution restrictions to the model
KaiEureka Oct 24, 2026
4fe8af2
fix(context): describe the effective inherited isolation
KaiEureka Oct 24, 2026
3b6906d
docs: use an English translation link in the VM example
KaiEureka Oct 24, 2026
6adcb78
docs(examples): use an English translation link label
KaiEureka Oct 24, 2026
00d4489
fix(runtime): align model-visible tools with configured capabilities …
KaiEureka Oct 24, 2026
8c77cc1
merge: include reviewed VM lifecycle and recovery fixes
KaiEureka Oct 24, 2026
b3a88b0
chore: include final VM transfer ownership fixes
KaiEureka Oct 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions docs/llm-discoverability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Model-facing capability audit

[Chinese](llm-discoverability.zh-CN.md)

This October 24, 2026 audit follows a capability from its implementation through
the role configuration, model-visible instructions, tool schema, execution, and
returned observations. It covers the runtime features already merged through
PR 244 and the VM execution series through PR 247.

The audit identifies nine families of defects. They include missing discovery
information and instructions that advertise operations unavailable in the current
configuration. Fixes derive model-facing information from the configuration that
actually supplies the tools and execution environments.

## Confirmed findings and repairs

| ID | Normal trigger and consequence | Repair | Executable evidence |
| --- | --- | --- | --- |
| D1 | A Team member can launch an authored Workflow with an isolation string, but receives neither the configured choices nor the authoring interface. Selecting a VM or custom provider requires knowledge outside the request. | `manage_workflow` advertises the configured isolation names and default. Its `describe` action returns strategy semantics, inherited tool permissions, launch prerequisites, and an executable authoring example. | A public Team call inspects configured VM names without allocating a guest. Another call discovers the example, writes the source, starts the real Python child, and collects two agent results. |
| D2 | `run_workflow` exposes names but describes every input as an object usually containing a goal. Weave requires groups and checks; Reconstruct requires a task integration. | `run_workflow(action="describe", name=...)` returns the callable's input instructions and phases. Built-in instructions state their required inputs and representative calls. | A public Team discovers a custom function's input and executes it successfully. All three built-ins expose their required input terms without executing their bodies or spending model tokens. |
| D3 | An open topology omits the configured role catalog. Restricted roles can be told to use coordination tools they lack, while prebuilt teams receive spawning guidance even though their seats already exist. | Team context lists reachable configured role names, mentions granted coordination tools, and explains the prebuilt roster. Each role retains its own private prompt. | Tests capture the public model request for a prebuilt Team, inspect an open topology, check a message-only role, and ensure another role's prompt stays private. |
| D4 | A caller can supply `UseSkillTool` to an Agent or Workflow without a Team context builder. The tool requires a catalog name but never supplies the catalog. | The tool's own schema carries current skill names and descriptions. An empty catalog is identified explicitly. | A test discovers a skill name solely from the schema and loads its exact body. Catalog refresh and protocol conversion preserve the choices. |
| D5 | `FileWriteTool(allow_create=False)` still advertises whole-file creation. Child-run inheritance can also restrict an already-created tool after construction. | The exported schema reflects the current permission and advertises targeted replacement only when creation is disabled. Runtime write checks remain in place. | A test selects the advertised mode and performs a real replacement. A separate test tightens a tool after creation and confirms other instances retain their own modes. |
| D6 | Base/Single2's fixed prompt claims six tools and a shell even for `tools="read"` or caller-supplied tool sets. | The profile refers to the supplied tool definitions, makes shell instructions conditional, and preserves its task, safety, and delivery instructions. | A public read-only Agent request contains the actual read tools and no unconditional claim of shell availability. |
| D7 | A headless Analyst is told to call `ask_user` before planning, although that tool is removed when there is no human input channel. | Clarification guidance uses the tool when available and describes how to report a required unresolved decision. | A headless default Analyst has conditional instructions and no unavailable question tool. |
| D8 | A serialized Team's message acknowledgement requires `submit`, even when the sender has no such tool. The sender needs to finish its turn before the recipient can run. | The acknowledgement explains that a final response without tool calls yields the turn, with `submit` as an available-tool alternative. | Message execution returns an actionable instruction for a sender with no submit tool. Existing serialized scheduler tests continue to exercise delivery after the sender yields. |
| D9 | A headless local session advertises a Bash or adoption tool as executable even when that tool requires an OS process sandbox and the current environment has none. | The outgoing tool description states the current execution restriction. The execution check remains authoritative, and the tool instance is unchanged. | A public Agent avoids the blocked operation after reading the actual schema. The authorized local-shell control runs a real command; a direct blocked call still refuses execution. |

## Audited surfaces

The inventory contains 19 core model-callable tools and four example-specific
tools. Tool wrappers that preserve another tool's interface were checked at the
delegation boundary.

| Surface | Scope and outcome |
| --- | --- |
| Workspace tools | Bash, file reads, file edits, patch application, Git diff, and commit adoption. Parameter names were compared with the arguments consumed by execution. D5 and D9 cover configured edit and execution permissions. |
| Team coordination | Spawn, spawn with review, messages, roster lookup, child adoption, and submit. D3 and D8 cover runtime-specific guidance. Existing topology enforcement and asynchronous result delivery are retained. |
| Human input and skills | Question-tool availability and skill catalog delivery. D4 and D7 cover the gaps. Skill bodies continue to be returned unchanged. |
| Workflow entry points | Registered and member-authored Workflows, discovery, input guidance, isolation selection, lifecycle actions, and result collection. D1 and D2 cover the gaps. |
| Structured completion | `structured_output` and `submit_findings` already publish their schemas and receive explicit terminal instructions from the runtime. Existing validation and forced-submission behavior are retained. |
| Agent instructions | Analyst, Coder, Tester, generic role, Base/Single2, and Workflow role composition. D3, D6, and D7 address the configuration-dependent contradictions. |
| Context and transport | Context assembly, skill refresh, workflow discovery wrappers, and Chat Completions, Responses, and Anthropic tool conversion. The adapters preserve the new names and descriptions. |
| Bundled workflows | Duo, Weave, and Reconstruct expose input instructions through discovery. Their solving, selection, and verification algorithms remain unchanged. |
| IO2Code example | `query_oracle` and `check_candidate` publish task-specific input formats and return query or validation feedback. Active and passive modes supply their corresponding instructions and tools. |
| ARC-Bench example | `checkpoint` and `run_acceptance` expose states, scopes, and argument shapes. Requirement identities are supplied by the task and progress context. Existing acceptance semantics are retained. |

## Using the discovery interfaces

Call `manage_workflow` with `action="describe"` before authoring a child. The
response contains the configured strategy names, inherited default, permitted
child tool names, the subset constructible with `opencollab.tools.builtin_tools`,
and the callable entry shape. The example can be written to a Python file and
launched with an `args` object containing `tasks`.

Strategy availability is reported as `checked_on_allocation`. A configured Docker
provider or VM resource describes an execution option; the provider performs
connection, capacity, and allocation checks when the option is used. Discovery
performs no hardware probe and acquires no container or VM. The current launch
permission reflects the member's Bash permission and execution environment.

Call `run_workflow` with `action="describe"` and a registered name to inspect its
inputs. Then invoke the same tool with `args` and the default `run` action. Custom
Workflow authors put caller-facing input instructions in the function docstring;
discovery retains them through the module wrapper.

## Configuration boundaries

Model routes, request capacity, environment provisioning, and runtime defaults
remain programmer-owned settings. This change exposes information needed to use
existing agent actions. A new role-specific environment authorization policy or
a live hardware inventory would require its own behavior and API design.

Configured strategies continue through the existing execution checks. Child tools
retain the parent tool restrictions. Describing a capability leaves models,
budgets, hardware allocation, and candidate state unchanged.

## Verification

The focused regressions are in
[`test_model_discoverability.py`](../tests/interfaces/test_model_discoverability.py).
They use scripted model responses to drive real SDK sessions, tool execution,
and an owned Python child process. Provider conversion tests inspect the outgoing
schemas for Chat Completions, Responses, and Anthropic.

Run them with `uv run pytest -q tests/interfaces/test_model_discoverability.py`.
The pull request records the full-suite, architecture, dependency, packaging,
and CI results for its final revision.
58 changes: 58 additions & 0 deletions docs/llm-discoverability.zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# 模型能力告知审计

[English](llm-discoverability.md)

这次审计沿着功能实现、角色配置、模型实际收到的提示与工具定义、调用执行、结果返回逐段检查。范围覆盖已经通过 PR 244 合入的运行能力,以及 PR 247 所在的虚拟机支持系列。

确认的问题分为九类,既包括已经实现的功能缺少发现入口,也包括提示要求模型使用当前配置中没有的操作。修复从实际配置生成模型需要的信息。

## 确认的问题与修复

| 编号 | 正常触发方式与影响 | 修复内容 | 执行依据 |
| --- | --- | --- | --- |
| D1 | Team 成员可以给自编 Workflow 填写隔离字符串,却没有收到已配置选项和编写接口说明,选择 VM 或自定义环境需要额外知识 | `manage_workflow` 展示实际注册的隔离名字与默认值,增加 `describe`,返回含义、继承工具权限、启动条件和可执行范例 | 公开 Team 调用可以发现 VM 选项且没有申请机器,另一个调用根据返回范例写入源码、启动真实 Python 子进程并收回两个 Agent 的结果 |
| D2 | `run_workflow` 只列出名字,输入提示笼统地说通常传 goal,Weave 实际需要任务组与验证命令,Reconstruct 需要任务集成 | 增加按名字读取输入说明的 `describe`,补齐三个内置 Workflow 的必需输入和用法 | 公开 Team 先读取自定义函数的输入说明再成功执行,三个内置 Workflow 都可在零模型调用下查询说明 |
| D3 | 开放拓扑遗漏已配置角色目录,受限角色可能被要求调用未授权的协调工具,预建 Team 又收到继续创建成员的提示 | 上下文列出可达角色名字,只提及已配置的协调工具,并说明预建成员已经存在,各成员保留自己的私有提示 | 检查开放拓扑、仅能投信的角色、公开 SDK 发出的预建 Team 模型请求,以及其他角色提示的隔离 |
| D4 | 单 Agent 或 Workflow 可以直接使用技能工具,但缺少 Team 的上下文构造器时,模型拿不到技能名字与用途 | 技能工具自身的 schema 带上当前目录,空目录也明确说明 | 从 schema 发现技能并加载原文,另行检查目录更新和三种协议转换 |
| D5 | 禁止创建文件时,文件工具仍向模型展示 create 模式,子运行继承权限后也可能出现相同矛盾 | 工具导出的 schema 根据当前权限展示可执行模式,原写入检查继续生效 | 从展示的模式选择操作并完成真实文本替换,验证运行中收紧权限及实例相互独立 |
| D6 | Base 与 Single2 固定声称提供六个工具和 Shell,实际只读配置或自定义工具集合可能没有这些能力 | 根据实际工具定义使用操作,Shell 指令改为有条件使用,保留任务、安全和交付要求 | 捕获公开只读 Agent 的模型请求,确认工具和提示一致 |
| D7 | 无人工输入渠道的 Analyst 仍被要求先调用 ask_user,该工具实际已被移除 | 提示在工具可用时提问,对必需但尚未解决的决定说明缺项 | 检查默认无人工输入 Analyst 的工具集合与提示 |
| D8 | 串行 Team 的投信回执要求调用 submit,但发送者可能没有该工具,接收者又需要等待发送者结束当前轮次 | 明确可以用没有工具调用的最终回复结束轮次,submit 作为工具存在时的另一种方式 | 对没有 submit 的发送者执行投信,验证回执可以指导其让出执行机会 |
| D9 | 无人值守的本地 Session 可能展示 Bash 或采纳工具,但工具要求进程沙箱,当前环境又没有满足这一条件 | 发给模型的工具说明明确当前执行限制,原执行检查继续生效,工具实例保持原样 | 公开 Agent 从工具定义得知限制,授权本地 Shell 的对照能执行真实命令,直接调用受限工具仍被拒绝 |

## 扫描范围

清单覆盖 19 个核心模型工具和 4 个示例专用工具。对于复用其他工具接口的包装器,核对其转发行为。

| 模块 | 检查内容与结果 |
| --- | --- |
| 工作区工具 | Bash、文件读取与修改、补丁、Git diff、提交采纳,逐项对照声明参数和执行实际读取的参数,配置权限问题归入 D5 与 D9 |
| Team 协作 | 创建成员、带审核的创建、投信、成员状态、子任务采纳和提交,配置相关提示问题归入 D3 与 D8,原拓扑约束及异步交付继续保留 |
| 人工输入与技能 | 提问工具的可用性、技能目录和正文交付,问题归入 D4 与 D7,技能正文继续原样返回 |
| Workflow 入口 | 已注册与成员自编 Workflow 的发现、输入、隔离选择、生命周期和结果收集,问题归入 D1 与 D2 |
| 结构化完成 | structured_output 与 submit_findings 已提供 schema,运行时也有明确的最终提交指令,原验证行为继续保留 |
| Agent 提示 | Analyst、Coder、Tester、通用角色、Base、Single2 和 Workflow 角色组合,配置相关矛盾归入 D3、D6、D7 |
| 上下文与模型协议 | 上下文构造、技能更新、Workflow 发现包装,以及 Chat Completions、Responses、Anthropic 的工具转换,新增选项经过转换后仍完整 |
| 内置 Workflow | Duo、Weave、Reconstruct 的输入说明通过发现入口提供,解题、选择和验证算法保持原样 |
| IO2Code 示例 | query_oracle 与 check_candidate 提供任务输入格式和执行反馈,主动与被动模式分别配置对应提示和工具 |
| ARC-Bench 示例 | checkpoint 与 run_acceptance 提供状态、范围和参数,任务与进度上下文提供需求编号,原验收行为继续保留 |

## 修复后的使用方式

成员编写 Workflow 前可以调用 `manage_workflow` 的 `describe`。返回信息包括实际注册的隔离选项、继承默认值、子 Agent 可用工具、可通过公开工具工厂构造的子集,以及函数入口和范例。模型可以把范例写成 Python 文件,再传入包含 tasks 的参数对象运行。

环境选项的状态明确写为分配时检查。已配置的 Docker 后端或虚拟机资源表示存在相应执行选项,连接、容量和实际分配由执行时的后端检查。发现操作不会启动容器或领取虚拟机。当前启动权限依据成员的 Bash 权限和执行环境计算。

调用已注册 Workflow 前,可以给 `run_workflow` 传入名字和 `describe` 动作,读取该函数的输入说明,再带 args 执行。自定义 Workflow 将调用者需要的输入说明放进函数 docstring,模块发现包装会保留这些信息。

## 程序配置边界

模型路由、请求并发额度、环境供应和运行默认值继续由程序员配置。这次修复补充现有 Agent 操作所需的发现信息。按角色限制隔离方式的授权策略,以及实时硬件目录,需要各自设计行为和接口。

环境申请继续经过已有检查,子工具继续继承父工具限制。查询说明保持模型调用、预算、硬件分配和候选状态不变。

## 验证方式

定向回归位于 [test_model_discoverability.py](../tests/interfaces/test_model_discoverability.py)。测试使用脚本化模型响应驱动真实 SDK、工具执行与受管理的 Python 子进程,并检查三种模型协议中的工具定义。

运行命令为 `uv run pytest -q tests/interfaces/test_model_discoverability.py`。修复 PR 会记录最终提交对应的完整测试、架构检查、依赖检查、安装包和 CI 结果。
Loading
Loading