Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,4 +336,6 @@ python -m http.server 8899 --directory demo/site # 本地预览

默认保持逐步规划;启用 `--execution-mode stage` 后,由慢模型规划阶段、快模型连续选择动作,
规则验证优先,仅在必要时调用模型验证或重规划。
`--execution-mode goal` 可启用带任务记忆的局部连续执行,阶段结束再独立 Verify;
保留 A11y/CDP 语义融合及 JEV 服务恢复,默认 step/stage 行为不变。
配置、执行边界、开发入口和验收方法见 [阶段执行说明](docs/STAGE_EXECUTION.md)。
17 changes: 16 additions & 1 deletion docs/RESULT_DIR_CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,12 @@ OSWorld 2.1 显式配置替代 judge/用户模拟器时,`code_version.json.res
| `result.txt` | 分数(`score\n`) | 同上 |
| `episode.json` | `{task_id, harness, stop_reason, steps, status(scored\|failed_unscored), score, exception_type, error?, error_traceback?}` | episode 终态(必写)。`error`/`error_traceback` 仅异常题有:异常消息与栈尾几行(脱敏)——只记类型事后没法定位 |
| `timing.json` | `{wall_seconds, status}` | episode 终态 |
| `task_memory.json`(goal) | 最新任务记忆:要求、事实、资料覆盖、交付单元、成果、失败与当前阶段 | goal 事件排水及终态 |
| `evidence/`(goal) | 阶段验证引用的截图/CLI 全文正本;临时目录清理后仍可读取 | 阶段验收/记忆落盘 |

`timing.json` 另外提供 `environment_seconds`、`model_usage_by_role`、`fast_model`,分别记录环境
分段时间、逐角色请求时间/token、排序次数/耗时/图像使用次数。缺失 usage 不补零冒充已测值,
另记 `calls_missing_usage`。原字段及 `browser/` 正本规则保持不变。

## 步号语义(demo 侧对齐依据)

Expand All @@ -63,7 +69,10 @@ OSWorld 2.1 显式配置替代 judge/用户模拟器时,`code_version.json.res
|------|--------|---------|
| `task_requirements` | verifier.task_requirements | episode 开始时抽取任务需求(1-12 条) |
| `plan_step` | planner.plan_step | 每步规划(截图+可见控件+需求+近期动作→下一步决策) |
| `plan_stage` | stage_planner.plan_stage | 仅 stage 模式:初始阶段、阶段完成或请求修复时生成阶段计划 |
| `plan_stage` | stage_planner.plan_stage / goal_planner.plan_goal | stage/goal 的阶段开始或修复;goal 允许读取已存资料后继续规划 |
| `verify_stage` | goal_verifier.verify_goal | goal 阶段结束时独立验证或提取资料增量;明确执行失败的本地检查不算模型调用 |
| `ground_visual` | grounding.ground_visual | goal 按需生成当前截图的有限候选,不直接执行慢模型坐标 |
| `gui_choice` | planner_choice.PlannerChoicePolicy | 仅显式启用 planner 选择后端时,在相同候选与截图上选 ID;独立记录时间/token,不冒充 JEV 排序分数 |
| `verify_observation` | verifier.verify_observation | 每步动作执行后独立验证(BEFORE/AFTER 截图→achieved/not_achieved/uncertain)|

`verify_observation` 不信任 planner 的自我宣称:它拿前后截图+预期变化独立裁决,驱动 recent 历史、no_action 连击计数、toggle 检查、以及终态双确认(terminal_claim 模式下逐条核验 requirements + 输出文件存在性)。
Expand All @@ -80,6 +89,12 @@ OSWorld 2.1 显式配置替代 judge/用户模拟器时,`code_version.json.res
| `stage_local` | 消费已生成的阶段计划;含 stage_goal、milestone 和当帧重绑定后的 plan,不代表新的模型调用 |
| `stage_progress` | 里程碑推进,含 stage_goal、milestone、验证 source |
| `stage_replan` | 阶段修复请求,含原因及当前里程碑;target_unbound 时另含 `unbound_proposals`、`available_targets`、`recovery_scopes`;不等于调用已经完成 |
| `task_memory` / `memory_update_rejected` | goal 的完整记忆快照/拒绝理由;不是规划模型自行宣称的完成 |
| `goal_action` | 真实回执、当前控件、画面变化与局部效果;effect_verified=false,不等于业务验收通过 |
| `stage_verified` | goal 的阶段裁决、证据引用、结束原因与用时,区分阶段成功和单元完成 |
| `observation_refresh` | 刷新观察而未执行动作,action_replayed=false;不增加动作步数 |
| `recovery` / `goal_stalled` | goal 的视觉候选补全/无可执行进展终止原因 |
| `environment_timing` | setup/observe/execute/export/evaluate 分段耗时 |
| `action`(kind 字段) | runner 写的执行记录 `{step, kind: "action", action: {…协议动作…}}` |

stage 模式的 planner 事件只在真实规划调用时写入,增加 mode、reason、seconds;
Expand Down
14 changes: 13 additions & 1 deletion infra/config.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,11 @@
"endpoint_file": "/mnt/shared-storage-gpfs2/evobox-share-gpfs2/xuyuanxun/shared_resources/sa_cua/services/qwen_jev.json",
"timeout_seconds": 300
},
"planner": {
"backend": "planner",
"endpoint": "",
"timeout_seconds": 300
},
"kev": {
"backend": "kev",
"endpoint": "",
Expand All @@ -63,7 +68,14 @@
},
"execution": {
"mode": "step",
"stage_max_actions": 24
"stage_max_actions": 24,
"recovery_max_calls": 2,
"recovery_max_seconds": 300,
"verify_max_tokens": 16384,
"verify_reasoning_effort": null,
"verify_max_chars": 48000,
"image_quality": 85,
"max_request_bytes": 2097152
},
"budget": {
"max_steps": 30,
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ name = "tandem"
version = "0.1.0"
description = "Tandem:快慢双模 CUA harness(主体)+ tandem_eval 集群测评薄适配层"
requires-python = ">=3.10"
# 运行时依赖白名单见 AGENTS.md。当前为空:M0 尚无消费者;Pillow 在 M1 截图处理落地时加入。
dependencies = []
# 截图处理使用仓库白名单内的 Pillow。
dependencies = ["Pillow"]

[project.optional-dependencies]
dev = ["pytest>=7", "ruff>=0.4"]
Expand Down
26 changes: 22 additions & 4 deletions scripts/run_experiment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,21 @@ EXPERIMENT=""
TASKS=""
CONCURRENCY=4
SLOW_MODEL="kimi-k3"
FAST_MODEL=""
SLOW_KEY_IDS="1,2,3,4"
MAX_STEPS=500
MAX_WALL=14400
EXECUTION_MODE=stage
STAGE_MAX_ACTIONS=10
DRY_RUN=0

usage() {
cat <<'EOF'
Usage:
bash scripts/run_experiment.sh \
--experiment NAME --tasks ID1,ID2,... [--concurrency N] \
[--slow-model NAME] [--slow-key-ids 1,2,3,4] \
[--slow-model NAME] [--fast-model NAME] [--slow-key-ids 1,2,3,4] \
[--execution-mode step|stage|goal] [--stage-max-actions N] \
[--max-steps N] [--max-wall SECONDS] [--dry-run]

Experiment names use lowercase letters, digits and hyphens only.
Expand All @@ -33,9 +37,12 @@ while [[ $# -gt 0 ]]; do
--tasks) TASKS="$2"; shift 2 ;;
--concurrency) CONCURRENCY="$2"; shift 2 ;;
--slow-model) SLOW_MODEL="$2"; shift 2 ;;
--fast-model) FAST_MODEL="$2"; shift 2 ;;
--slow-key-ids) SLOW_KEY_IDS="$2"; shift 2 ;;
--max-steps) MAX_STEPS="$2"; shift 2 ;;
--max-wall) MAX_WALL="$2"; shift 2 ;;
--execution-mode) EXECUTION_MODE="$2"; shift 2 ;;
--stage-max-actions) STAGE_MAX_ACTIONS="$2"; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;;
Expand All @@ -50,6 +57,10 @@ if [[ -z "$TASKS" || "$CONCURRENCY" -lt 1 || "$MAX_STEPS" -lt 1 || "$MAX_WALL" -
usage >&2
exit 2
fi
if [[ ! "$EXECUTION_MODE" =~ ^(step|stage|goal)$ || "$STAGE_MAX_ACTIONS" -lt 1 ]]; then
usage >&2
exit 2
fi

EXPERIMENT_ROOT="${TANDEM_EXPERIMENT_ROOT:-$REPO/rjob_logs}"
EXPERIMENT_DIR="$EXPERIMENT_ROOT/$EXPERIMENT"
Expand Down Expand Up @@ -77,6 +88,7 @@ print(json.loads(Path(sys.argv[1]).read_text())["log"]["results_root"])
PY
)
export STAGING EXPERIMENT DRIVER_DIR TASKS CONCURRENCY SLOW_MODEL SLOW_KEY_IDS MAX_STEPS MAX_WALL
export EXECUTION_MODE STAGE_MAX_ACTIONS FAST_MODEL
python3 - <<'PY'
import json
import os
Expand All @@ -90,9 +102,12 @@ args = {
"tasks": os.environ["TASKS"].split(","),
"concurrency": int(os.environ["CONCURRENCY"]),
"slow_model": os.environ["SLOW_MODEL"],
"fast_model": os.environ["FAST_MODEL"] or None,
"slow_key_ids": [int(item) for item in os.environ["SLOW_KEY_IDS"].split(",")],
"max_steps": int(os.environ["MAX_STEPS"]),
"max_wall_seconds": float(os.environ["MAX_WALL"]),
"execution_mode": os.environ["EXECUTION_MODE"],
"stage_max_actions": int(os.environ["STAGE_MAX_ACTIONS"]),
"watch_timeout_seconds": float(os.environ["MAX_WALL"]) + 600,
"git_describe": subprocess.check_output(
["git", "describe", "--always", "--dirty"], text=True
Expand Down Expand Up @@ -123,14 +138,17 @@ run_task() {
local run_id="${EXPERIMENT}-${task}"
local driver_log="${DRIVER_DIR}/${task}.driver.log"
local rc=0
local fast_args=()
if [[ -n "$FAST_MODEL" ]]; then fast_args=(--fast-model "$FAST_MODEL"); fi

tandem-eval rjob run \
env PYTHONPATH=src python3 -m tandem_eval rjob run \
--run-id "$run_id" \
--benchmark osworld21 \
--tasks "$task" \
--execution-mode stage \
--stage-max-actions 10 \
--execution-mode "$EXECUTION_MODE" \
--stage-max-actions "$STAGE_MAX_ACTIONS" \
--slow-model "$SLOW_MODEL" \
"${fast_args[@]}" \
--slow-key-ids "$SLOW_KEY_IDS" \
--max-steps "$MAX_STEPS" \
--max-wall "$MAX_WALL" \
Expand Down
31 changes: 25 additions & 6 deletions src/tandem/components.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""组件契约(依赖注入边界,无插件框架、无隐藏分发)。
在旧九组件上增加 semantic_enricher,统一 slow planner 与 fast selector 看到的元素语义。
grounder(visual fallback)默认 None,session 侧判空跳过。compile_gui 产出协议动作。"""
semantic_enricher 统一 planner 与 selector 的元素语义;goal 额外注入任务记忆、阶段验证与
按需视觉补全。step/stage 装配不变。compile_gui 只产出协议动作,不调用 VM。"""

from __future__ import annotations

Expand All @@ -11,12 +11,17 @@
from .config import SessionConfig
from .gui.candidates import ActionCandidateBuilder
from .gui.compile import gui_action
from .gui.grounding import ground_visual
from .gui.interaction import GuiInteractionController
from .gui.proposals import focused_candidates
from .gui.semantics import enrich_targets
from .gui.targets import A11yTargetBuilder
from .memory.state import TaskMemory
from .models.jev import JevOmniPolicy
from .planning.goal_planner import goal_requirements, plan_goal
from .planning.planner import plan_step
from .planning.stage_planner import plan_stage
from .verification.goal_verifier import verify_goal
from .verification.verifier import task_requirements, verify_observation


Expand All @@ -32,31 +37,45 @@ class Components:
focus_candidates: Callable
grounder: Callable | None
compile_gui: Callable
memory_factory: Callable | None = None
stage_verifier: Callable | None = None
set_deadline: Callable | None = None
interaction_factory: Callable = GuiInteractionController


def build_default(*, client, endpoint: str, config: SessionConfig,
image_stage: tuple[str, str] | None = None, endpoint_provider=None) -> Components:
if config.fast_backend == "kev":
if config.fast_backend == "planner":
from .models.planner_choice import PlannerChoicePolicy

selector = PlannerChoicePolicy(client)
elif config.fast_backend == "kev":
from .models.kev import KevPolicy

selector = KevPolicy(endpoint, timeout_seconds=config.fast_timeout_seconds)
else:
selector = JevOmniPolicy(endpoint, timeout_seconds=config.fast_timeout_seconds,
image_stage=image_stage, endpoint_provider=endpoint_provider)
image_stage=image_stage, endpoint_provider=endpoint_provider,
joint_actions=config.execution_mode == "goal")
return Components(
planner=partial(
plan_goal if config.execution_mode == "goal" else
plan_stage if config.execution_mode == "stage" else plan_step,
client,
allow_cli=config.allow_cli and config.route_policy != "gui",
route_policy=config.route_policy,
),
verifier=partial(verify_observation, client),
requirements=partial(task_requirements, client),
requirements=partial(
goal_requirements if config.execution_mode == "goal" else task_requirements, client),
selector=selector,
target_builder=A11yTargetBuilder(),
semantic_enricher=enrich_targets,
candidate_builder=ActionCandidateBuilder(),
focus_candidates=focused_candidates,
grounder=None,
grounder=partial(ground_visual, client) if config.execution_mode == "goal" else None,
compile_gui=gui_action,
memory_factory=TaskMemory if config.execution_mode == "goal" else None,
stage_verifier=partial(verify_goal, client) if config.execution_mode == "goal" else None,
set_deadline=getattr(client, "set_deadline", None),
)
23 changes: 18 additions & 5 deletions src/tandem/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,36 @@ class SessionConfig:
cli_timeout_seconds: int = 90
execution_mode: str = "step"
stage_max_actions: int = 24
goal_recovery_max_calls: int = 2
goal_recovery_max_seconds: float = 300.0 # Checked before another grounding call, not an HTTP interrupt.
slow_max_tokens: int = 16384
slow_verify_max_tokens: int = 16384
slow_verify_reasoning_effort: str | None = None
stage_verify_max_chars: int = 48000
slow_timeout_seconds: float = 120
slow_direct_api: bool = False
slow_reasoning_effort: str | None = None # None=服务端默认;off=enable_thinking:false
fast_backend: str = "jev" # "jev"(qwen-jev-choice /rank)| "kev"(Kev System One)
slow_image_quality: int = 85
slow_max_request_bytes: int = 2 * 1024 * 1024
fast_backend: str = "jev" # jev / kev;planner 显式复用慢模型 API 选择候选。
fast_timeout_seconds: float = 300

def __post_init__(self):
if self.execution_mode not in {"step", "stage"}:
raise ValueError("execution_mode must be step or stage")
if self.execution_mode not in {"step", "stage", "goal"}:
raise ValueError("execution_mode must be step, stage or goal")
if type(self.stage_max_actions) is not int or self.stage_max_actions < 1:
raise ValueError("stage_max_actions must be a positive integer")
if type(self.goal_recovery_max_calls) is not int or self.goal_recovery_max_calls < 1:
raise ValueError("goal_recovery_max_calls must be a positive integer")
if self.goal_recovery_max_seconds <= 0:
raise ValueError("goal_recovery_max_seconds must be positive")
if self.route_policy not in {"auto", "gui", "cli"}:
raise ValueError("route_policy must be auto, gui or cli")
if not 1 <= self.cli_timeout_seconds <= 120:
raise ValueError("cli_timeout_seconds must be within 1..120 (协议上限)")
if self.fast_backend not in {"jev", "kev"}:
raise ValueError("fast_backend must be jev or kev")
if self.fast_backend not in {"jev", "kev", "planner"}:
raise ValueError("fast_backend must be jev, kev or planner")
if self.slow_reasoning_effort not in {None, "off", "low", "medium", "high", "max"}:
raise ValueError("slow_reasoning_effort must be null/off/low/medium/high/max")
if self.slow_verify_reasoning_effort not in {None, "off", "low", "medium", "high", "max"}:
raise ValueError("slow_verify_reasoning_effort must be null/off/low/medium/high/max")
10 changes: 6 additions & 4 deletions src/tandem/gui/candidates.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@
"double_click": "The next useful step is to open one observed file or list item with a double click.",
"set_slider": "The next useful step is to set a visible slider to the required value.",
"keypress": "The next useful step is one supported keyboard navigation or submission command, not text replacement.",
"scroll": "A required control is outside the current viewport, so scrolling is the next useful step.",
"wait": "The required control is still loading, absent, or disabled, so waiting is the only useful next step.",
"scroll": "Scroll within the foreground view to reveal more controls or content.",
"wait": "Wait briefly, then observe the current UI again.",
"done": "Every requirement of the current local GUI goal is visibly satisfied.",
"visual_fallback": "The required interaction depends on visual or spatial information not represented by the semantic targets.",
"return_to_planner": "The current UI state contradicts the assumptions of the current stage and requires replanning.",
Expand Down Expand Up @@ -103,7 +103,8 @@ def build(
def relevance(target: UITarget) -> tuple[int, int]:
words = set(re.findall(r"[a-z0-9]+", target.name.lower()))
contextual_words = set(
re.findall(r"[a-z0-9]+", " ".join((target.name, *target.parent_context)).lower())
re.findall(r"[a-z0-9]+", " ".join(
(target.name, *target.parent_context, target.nearby_text)).lower())
)
return (
len(words & hint_words) * 4 + len(contextual_words & context_words),
Expand Down Expand Up @@ -149,7 +150,8 @@ def candidates(operation: str, values: list[UITarget], description: str) -> list
id=f"{operation}:{target.id}",
operation=operation,
target_id=target.id,
description=description.format(role=target.role.replace("_", " "), name=target.name),
description=description.format(role=target.role.replace("_", " "), name=target.name)
+ (f" Context: {target.nearby_text}" if target.nearby_text else ""),
source=target.source,
metadata={"semantic_key": target.semantic_key},
)
Expand Down
Loading
Loading