diff --git a/AGENTS.md b/AGENTS.md index 49bab53..49a8662 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,7 @@ window → indexed elements → operation + target → guarded execution → obs - One request chooses the operation, every target head and risk on the same observed state. Consume only the matching head. The model may choose only observed targets and supported operations. - Never replay a mutation. Consume its decision before input and record delivery before post-action observation. Unknown delivery stops with history intact. Preserve returned focus facts even if read-back fails. - Capture-time uniqueness and a fresh target census guard execution. Unknown identities, incomplete reads and process replacement refuse input. Preserve secure classification before value reads, redacted history, separate AX deadlines and bridge replacement after transport failure. -- Keep the action space small. INCREMENT/DECREMENT once changed Calculator from 6/6 to 1/8 despite identical observations; they stay unoffered. Measure any new operation against tasks that do not need it. +- Keep the action space small. Measure any new operation against tasks that do not need it. - Preserve separate content/control publication budgets. A content-heavy window must not hide its toolbar. Static text names come from their values. Inspect traversal depth before declaring an Electron app inaccessible. - SELECT needs observed options. Do not expand it based on an unmeasured assumption that closed menus expose choices. Popup/menu-opening presses can activate apps and stay withheld. - No token-probability reconstruction, extra planner or new compatibility framework without a concrete product requirement. diff --git a/README.md b/README.md index 4940de6..94b74a7 100644 --- a/README.md +++ b/README.md @@ -13,9 +13,9 @@ Give it one app and one natural-language goal. Each observation produces a numbe Requires macOS, Python 3.11+, Swift and Accessibility permission for the terminal or app launching the bridge. No Screen Recording permission is used. ```sh -uv sync +uv sync --extra agent scripts/build.sh -uv run --env-file .env open-computer-use inspect --no-open +uv run --extra agent --env-file .env open-computer-use inspect --no-open ``` Open the inspector at `http://127.0.0.1:8767`. It shows the observed element table, pending decision, risk and executed history. The server binds to loopback and requires its startup token for changes. @@ -35,13 +35,15 @@ DECISION_MODEL=deepseek-flash DECISION_API_KEY=your-key ``` +The model-driven Agent and Inspector use the optional `agent` extra (`pip install "open-computer-use[agent]"` or `uv sync --extra agent`). Their public Python API is unchanged. + The optional Jev backend answers structured choice heads instead of writing JSON. Set `JEV_API_KEY`, optionally `JEV_BASE_URL` and `JEV_MODEL`. Jev cannot generate field values; `TEXT_MODEL_API_KEY`, `TEXT_MODEL_BASE_URL` and `TEXT_MODEL` configure that helper, falling back to the decision backend settings. The product reads only user-configured environment variables. The chat backend supports JSON schema and JSON object responses. `DECISION_RESPONSE_FORMAT=json_schema` or `json_object` overrides provider detection; `DECISION_REASONING=default` keeps the provider's reasoning behavior. No token logprobs are requested. ## From a coding agent -A coding agent can be the decision model itself. `install.sh` compiles the bridge, installs `open-computer-use` as an editable uv tool and copies [the skill](skill/open-computer-use/SKILL.md) into `~/.claude/skills` and `~/.codex/skills` when they exist. No decision backend or key is involved. +The native CLI has no Python runtime dependencies and loads neither the model client nor the Inspector. A coding agent can be the decision model itself. `install.sh` compiles the bridge, installs `open-computer-use` as an editable uv tool and copies [the skill](skill/open-computer-use/SKILL.md) into `~/.claude/skills` and `~/.codex/skills` when they exist. No decision backend or key is involved. ```sh open-computer-use observe Calculator # header with fingerprint, then one element per line @@ -67,7 +69,7 @@ with Agent("TextEdit", f"Replace the document with exactly '{sentence}'.", verif print(agent.state["verified"]) ``` -`verify` receives the final observation: `True` confirms the goal, `False` rejects `DONE`, and `None` means the result could not be judged. Without it, `DONE` is the model's claim. +`verify` receives the final observation: `True` confirms the goal, `False` rejects `DONE`, and `None` means the result could not be judged. Without it, `DONE` is the model's claim. `verified` checks the goal only; it does not certify the whole background interval. An optional `focus_check` callback connects the caller's existing full-task monitor: `True` means clear so far, `False` a recorded activation, and `None` unavailable. A violation or unavailable configured monitor stops before further input. The caller must keep monitoring through cleanup before accepting the task. Consequential operations are held at `risk >= 0.5`. Inspect the decision before calling `agent.approve_pending()`, or supply an `approve(decision)` callback. The rating shares the decision request; it is the same model's judgment, not an independent safety reviewer. @@ -85,11 +87,15 @@ Consequential operations are held at `risk >= 0.5`. Inspect the decision before There is one native executor. Activation flags, Codex-host execution, mouse input, dragging, screenshots and visual fallback have been removed. A canvas without usable accessibility controls is outside scope; it cannot be made a successful background task by taking the screen. +Known `took_focus` facts stop the Agent, including facts returned with post-dispatch errors. CLI `act` returns `focus_violation` and exit code 3; this is never a retry instruction. Other exits are 0 for a delivered command, 1 for refusal/staleness, and 2 for uncertain or unconfirmed delivery. None establishes whole-task success. + Native APIs can still cause app-specific side effects. `took_focus` records observed endpoint changes only. Full-run acceptance independently listens for macOS application-activation events, including a transient activation followed by a return to the original app. A run that changes the requested value but steals focus fails acceptance. Bringing the previous app back does not undo that failure. ## Delivery and identity -Every result and bridge exception carries `disposition`: +Every action result and bridge exception carries `disposition`. Action replies missing or contradicting it are protocol failures: the bridge is reset, the in-flight mutation remains uncertain, and no input is replayed. `AXBRIDGE_BIN` overrides must implement the current protocol. + +The delivery states are: | Delivery | Evidence | Retry | | --- | --- | --- | @@ -110,7 +116,7 @@ Secure fields are classified before value reads; their values and descendants ar One app per run, including its accessible windows and open menus. Publication budgets and identity census limits are separate; incomplete observations are explicit. A run stops at 40 operations, 80 decisions, three consecutive unchanged operations, or before delivering the same action a third time on the same observed state. WAIT is exempt from repeat stopping. Existing history supplies the count; a stop is blocked, never task success. Observed state can hide real progress, and changing or unobserved state can evade this heuristic. Model task success varies independently of reliable execution. -`examples/calculator.py` and `examples/textedit.py` read their intended results back from the app. An endpoint sample showing the app inactive is not proof it stayed inactive throughout. Focus acceptance requires the independent event record over the same task interval. +`examples/calculator.py` delegates to the shared owned-instance acceptance runner; supply `--max-calls`, `--usage-file` and `--output` for your approved batch. Its optional `--goal` and `--expect` still customize the single task. `examples/textedit.py` edits an already-open disposable document and reports its goal reading only. An endpoint sample showing the app inactive is not proof it stayed inactive throughout. Focus acceptance requires the independent event record over the same task interval. [The background acceptance runner](evals/README.md) retains the 16 legacy task definitions, reads real app results independently and requires a complete activation-event record through cleanup. It preserves already-running user apps and records unavailable cases explicitly; historical scores are not current results. diff --git a/docs/design.md b/docs/design.md index 7b1bb32..a7db634 100644 --- a/docs/design.md +++ b/docs/design.md @@ -43,3 +43,7 @@ One request chooses the operation, a speculative target for each operation, fiel The inspector displays the table, pending decision, risk and executed history. It binds to loopback, checks Host/Origin, requires a startup token and serializes commands. It has no activation control. Its `DONE` state remains a model claim unless an outcome check supplied evidence. Budgets remain 40 operations, 80 decisions and three consecutive unchanged actions. History also records each pre-action fingerprint and editing parameters. Two delivered matching state/action pairs block a third input, including A → B → A cycles; refusals do not count and WAIT is exempt. The check runs before field-value generation, uses the existing redacted history path, and rechecks freshness before refusing. No parallel counter or planner state is added. A blocked repetition does not establish task completion or progress in hidden fields. + +The native CLI and optional model Agent share execution and delivery semantics, not a claim of whole-task completion. A delivered command is not a completed goal; `verified` is a goal reading, not full-interval focus evidence. Known endpoint focus facts stop the Agent and survive dispatch errors; CLI actions report them as `focus_violation` (exit 3). A caller may pass the existing full-task observer as `focus_check`, checked before a command, after prediction/value preparation and after the command. False or unavailable evidence stops further input. The acceptance owner continues observing through cleanup; it alone can establish the complete interval. No new session service or second activation observer is introduced. + +The core package has no runtime dependencies. Agent imports are lazy, and only the optional `agent` extra loads HTTP/model and Inspector code. The separate model choices, speculative heads and Jev text helper remain in that optional layer: their decision contract is unchanged in this cleanup. Replacing that contract or dropping standalone Agent mode remains a separate measured product decision. diff --git a/evals/README.md b/evals/README.md index 3e77e43..31ab1f4 100644 --- a/evals/README.md +++ b/evals/README.md @@ -5,7 +5,7 @@ The 16 task goals and result checks come from the legacy evals branch. Its scrip Approve a usage-billed batch before running it. Select tasks explicitly and run only one writer for a shared usage file: ```sh -uv run --env-file .env python -m evals.run \ +uv run --extra agent --env-file .env python -m evals.run \ --tasks calc-multiply textedit-edit --reps 1 \ --max-calls 20 --usage-file /absolute/path/batch.json \ --output /absolute/path/new-acceptance-folder @@ -18,6 +18,7 @@ A pass requires all of the following: - The independent app check is false after setup and true after the real Agent run. No expected answer is injected by a scripted solve. Calculator reads its display, TextEdit its document, Settings its pane title (or a recognized detail heading while the title is unavailable), Safari its window title, and Dictionary its definition text. Finder reads the exact owned window's target directory, so row selection cannot satisfy opening a folder. - System Settings navigation tasks hold operations other than native row selection, scrolling, waiting and Escape, using the existing approval callback; they never approve unrelated preference changes. - The Agent ended with `done`. `DONE` with a false reading is recorded separately as `false_done`. +- The same observer supplies `Agent.focus_check`, so an activation during model prediction or field-value preparation can withhold the next input. An unavailable configured monitor also stops. - The activation-event observer started before app setup and ended after cleanup. **Any activation event invalidates this run**, including during cleanup. An unrelated human app switch is conservatively inconclusive; the runner does not label it task-caused or silently exclude it. It never restores foreground focus. - The owned instance/window was confirmed closed. Missing observer endpoints, native errors and cleanup failures invalidate acceptance. diff --git a/evals/run.py b/evals/run.py index cd53b7b..edd08b1 100644 --- a/evals/run.py +++ b/evals/run.py @@ -1,5 +1,6 @@ """Real model tasks, independent result readings, and lifetime-wide focus events.""" import argparse +from dataclasses import replace import json import queue import subprocess @@ -67,6 +68,12 @@ def acquire(self): if event["event"] in {"unavailable", "error", "eof"}: raise RuntimeError(event.get("detail", "Ownership monitor ended before acquiring an instance")) + def focus_clear(self): + if (not self.events or self.events[0]["event"] != "ready" or self.process.poll() is not None + or any(e["event"] == "error" for e in self.events)): + return None + return not any(e["event"] == "activated" for e in self.events) + def close(self): self.process.stdin.close() # The native owner closes only what it created, before its stopped event. @@ -128,7 +135,7 @@ def phase(name): if record["before"] is not False: raise RuntimeError("The initial result check was not false after setup") phase("agent") - with Agent(str(pid), task.goal, menus=False, risk_threshold=0, + with Agent(str(pid), task.goal, menus=False, risk_threshold=0, focus_check=lease.focus_clear, approve=lambda d: task.app != "System Settings" or d["operation"] in { "CLICK", "SCROLL_UP", "SCROLL_DOWN", "WAIT", "PRESS_ESCAPE"}) as agent: try: @@ -177,11 +184,15 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--tasks", nargs="+", choices=[t.id for t in TASKS], required=True) parser.add_argument("--reps", type=int, default=1) + parser.add_argument("--goal", help="Override the goal for a single selected task") + parser.add_argument("--expect", help="Override its independent expected result") parser.add_argument("--max-decisions", type=int, default=16) parser.add_argument("--max-calls", type=int, required=True, help="Approved total batch cap, including prior stages") parser.add_argument("--usage-file", type=Path, required=True, help="Shared cumulative batch counter; run one writer at a time") parser.add_argument("--output", type=Path, required=True, help="New artifact directory outside the checkout") options = parser.parse_args() + if (options.goal is not None or options.expect is not None) and len(options.tasks) != 1: + parser.error("Goal/result overrides require exactly one task") if min(options.reps, options.max_calls, options.max_decisions) < 1: parser.error("Budgets and repetitions must be positive") root = Path(__file__).resolve().parents[1] @@ -202,6 +213,8 @@ def main(): continue if budget.data["calls"] >= options.max_calls: return + task = replace(task, goal=options.goal if options.goal is not None else task.goal, + expected=options.expect if options.expect is not None else task.expected) record = run_once(task, binary, output / f"{rep + 1:02d}-{task.id}", options.max_decisions) stream.write(json.dumps(record, ensure_ascii=False) + "\n"); stream.flush() print(json.dumps({k: record[k] for k in ("task", "passed", "status", "before", "after", "monitor_complete", "cleaned", "focus", "error")}), flush=True) diff --git a/examples/calculator.py b/examples/calculator.py index f1c3059..a8f9a09 100644 --- a/examples/calculator.py +++ b/examples/calculator.py @@ -1,102 +1,12 @@ #!/usr/bin/env python3 -"""Six presses in a window that stays behind everything else. +"""Calculator uses the shared full-lifetime acceptance runner and owned instance. -Calculator is a good honest test: two dozen elements, every one of them a -plain button with a real label, no text to generate, and a result that is -either right or wrong. The agent has to pick 1, 2, ×, 3, 4, = in that order -and then recognise that it is finished. - - uv run --env-file .env python examples/calculator.py - -Nothing is brought to the front. Check with `background=True` in the output, -or just keep working while it runs. +Supply --max-calls, --usage-file and --output for your approved model-call batch. +Optional --goal and --expect keep the single-task example configurable. """ +import sys -import argparse -import subprocess -import time - -from open_computer_use import Agent -from open_computer_use.desktop import Desktop - -GOAL = "Compute 12 times 34 and leave the result on the display." -EXPECTED = "408" - - -def clear(app): - """Start from a known display, and from a window that only shows this run. - - Calculator keeps a visible tape of earlier calculations, and All Clear does - not clear it. Left alone it accumulates across runs until the window is - showing the expected answer before the agent has pressed anything — which - is the state this example is supposed to be proving the absence of. The - button is 'All Clear' only when the display is already clear. - """ - desktop = Desktop(app, menus=True) - try: - page = desktop.observe() - if any(m["label"] == "View > Hide History" for m in page.get("menus", [])): - desktop.act({"operation": "MENU", "op": "MENU", "menu": "View > Hide History"}, page) - page = desktop.observe() - button = next((e for e in page["elements"] if e["label"] in ("All Clear", "Clear")), None) - if button: - desktop.act({"operation": "PRESS", "op": "PRESS", "path": button["path"], "expect": "Clear"}, page) - finally: - desktop.close() - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--goal", default=GOAL) - parser.add_argument("--expect", default=EXPECTED) - options = parser.parse_args() - - # macOS terminates an idle background app on its own, so make sure it is up. - subprocess.run(["open", "-g", "-a", "Calculator"], check=True) - time.sleep(2.0) - clear("Calculator") - - # The loop asks this before accepting DONE, so the model cannot declare - # victory over a display that does not say what it should. Only the last - # line counts: anywhere in the window is not the same as on the display, - # and a check that accepts anywhere is one a leftover from the previous - # run can satisfy without this run doing anything at all. - def reached(page): - lines = [line for line in page["text"].replace("‎", "").splitlines() if line.strip()] - return bool(lines) and options.expect in lines[-1] - - started = time.perf_counter() - with Agent("Calculator", options.goal, verify=reached) as agent: - shown = 0 - for state in agent.run(): - for step in state["history"][shown:]: - print( - f'{step["elapsed_ms"]:>6} ms {step["operation"]:<6} {step["label"]:<14}' - f' model {step["latency_ms"]:>4} ms risk {step["risk"]:.2f}' - ) - shown = len(state["history"]) - final = agent.state - total = round((time.perf_counter() - started) * 1000) - - decisions = [d["latency_ms"] for d in final["decisions"]] - model_ms = sum(decisions) - print(f'\n{final["status"]} after {len(final["history"])} operations, {total} ms') - print(f" {len(decisions)} model calls, {model_ms} ms of it waiting on the model " - f"({model_ms / total * 100:.0f}% of the wall clock)") - print(f' outcome check: {final["verified"]} took the screen: {final["took_focus"]}') - - # Independent verification: read the display, not the model's DONE. - desktop = Desktop("Calculator") - try: - page = desktop.observe() - display = page["text"].replace("‎", "") - print("\nindependent check") - print(f' display : {display!r}') - print(f' contains {options.expect:<22}: {options.expect in display}') - print(f' the app never came to the front : {not page["active"]}') - finally: - desktop.close() - +from evals.run import main if __name__ == "__main__": - main() + main(["--tasks", "calc-multiply", *sys.argv[1:]]) diff --git a/open_computer_use/__init__.py b/open_computer_use/__init__.py index 270cfb1..778b049 100644 --- a/open_computer_use/__init__.py +++ b/open_computer_use/__init__.py @@ -1,7 +1,13 @@ -"""A macOS computer-use agent with a dynamic, indexed action space.""" +"""Native background desktop operations; the model-driven Agent is optional.""" -from .agent import Agent from .desktop import Desktop, StaleWindow __all__ = ["Agent", "Desktop", "StaleWindow"] __version__ = "0.1.0" + + +def __getattr__(name): + if name == "Agent": + from .agent import Agent + return Agent + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") diff --git a/open_computer_use/actions.py b/open_computer_use/actions.py index 715a8fa..f402e68 100644 --- a/open_computer_use/actions.py +++ b/open_computer_use/actions.py @@ -23,14 +23,7 @@ "SELECT": "Choose a value from a pop-up button's menu.", "MENU": "Run a menu-bar command by name, without opening the menu first.", "CLICK": "Select an observed row through the app's native selection API.", - # INCREMENT and DECREMENT are deliberately absent. The bridge derives and - # executes both, and adding them here is one line — it was added, measured, - # and taken out again. Calculator publishes them on "Show Sidebar" and - # "Mode", so offering them put two more heads in every answer and two more - # ways to be wrong in a task that needs neither. On "compute 12 times 34", - # against deepseek-flash: 6 of 6 runs correct in 6 operations without them, - # 1 of 8 with them, the rest wandering into the 40-operation budget. The - # element table was identical in both. See "What is not offered". + } diff --git a/open_computer_use/agent.py b/open_computer_use/agent.py index 82157a1..75640d4 100644 --- a/open_computer_use/agent.py +++ b/open_computer_use/agent.py @@ -10,7 +10,8 @@ UnknownOutcome, disposition, ) -from .model import action_space, choose, field_context, field_text +from .actions import action_space +from .model import choose, field_context, field_text from .questions import MAX_DECISIONS, MAX_STALE_RETRIES, MAX_STEPS, RISK_THRESHOLD @@ -74,6 +75,7 @@ def __init__( risk_threshold=RISK_THRESHOLD, menus=True, verify=None, + focus_check=None, ): goal = goal.strip() if isinstance(goal, str) else "\n".join(goal).strip() if not goal: @@ -84,6 +86,9 @@ def __init__( # True to confirm, False to reject it, None if it cannot be judged. # Without one, DONE is only ever the model's own claim. self.verify = verify + # The caller owns the complete task interval, including setup/cleanup. + # True is clear so far, False a recorded activation, None unavailable. + self.focus_check = focus_check self.pending_text = None # (operation, path) pairs this app has refused on the window as it # now stands. Cleared the moment the window changes, because a @@ -128,6 +133,8 @@ def _elapsed(self): return round((time.perf_counter() - self.state["started_at"]) * 1000) def command(self, name, body=None): + if self._focus_failed(): + return self.snapshot() if name == "tick": self._tick() elif name == "predict": @@ -136,12 +143,32 @@ def command(self, name, body=None): self._act(body or {}) else: raise ValueError(f"Unknown command {name!r}") + self._focus_failed() return self.snapshot() + def _focus_failed(self, decision=None): + if self.focus_check is None: + return False + try: + clear = self.focus_check() + except Exception: + clear = None + if clear is True: + return False + self.state.update(status="blocked", decision=None, note=( + "Activation occurred during the monitored interval; stopped without further input." + if clear is False else "The focus monitor is unavailable; stopped without further input.")) + if decision: + self._record(decision, outcome="not attempted", disposition=disposition("not_delivered"), + detail=self.state["note"], window_changed=False) + return True + def _record(self, decision, text=None, **details): state = self.state action = decision["action"] or {} state["elapsed_ms"] = self._elapsed() + if details.get("took_focus"): + state.update(took_focus=True, note="Foreground focus changed to the target app. Stopped; background acceptance failed.") state["history"].append({ "step": len(state["history"]) + 1, "operation": decision["operation"], @@ -264,6 +291,8 @@ def _act(self, body): operation = decision["operation"] if operation in {"DONE", "BLOCKED"}: + if self._focus_failed(decision): + return if not self.desktop.fresh(page): state["status"] = "ready" raise StaleWindow("The window changed since the decision. Choose again.") @@ -337,6 +366,8 @@ def _act(self, body): state["note"] = detail return + if self._focus_failed(decision): + return if operation == "WAIT": time.sleep(0.15) result = {"detail": "waited", "mechanism": "sleep", "disposition": disposition("not_delivered")} @@ -351,6 +382,7 @@ def _act(self, body): text=text, secure=secure, outcome="executed, unconfirmed" if certain else "unknown", + took_focus=getattr(error, "took_focus", False), disposition=error.disposition, detail=str(error), ) @@ -373,6 +405,7 @@ def _act(self, body): text=text, secure=secure, outcome="failed", + took_focus=getattr(error, "took_focus", False), disposition=error.disposition, detail=str(error), path=action.get("path") if action else None, @@ -388,8 +421,8 @@ def _act(self, body): return state["elapsed_ms"] = self._elapsed() stuck = self._stalled() - state["status"] = "blocked" if stuck else "ready" - if stuck: + state["status"] = "blocked" if stuck or state["took_focus"] else "ready" + if stuck and not state["took_focus"]: state["note"] = f"{operation} was refused: {error}" return self.pending_text = None @@ -415,8 +448,6 @@ def _act(self, body): window_changed=None, executed_ms=self._elapsed(), ) - if result.get("took_focus"): - state["took_focus"] = True try: changed = self._observe() except BridgeError as error: @@ -432,7 +463,7 @@ def _act(self, body): ) # Three operations that changed nothing visible is a stuck policy. stuck = self._stalled({"WAIT", "SCROLL_UP", "SCROLL_DOWN"}) - state["status"] = "blocked" if stuck else "ready" + state["status"] = "blocked" if stuck or state["took_focus"] else "ready" return def run(self): diff --git a/open_computer_use/axbridge.swift b/open_computer_use/axbridge.swift index 7967913..53a3a56 100644 --- a/open_computer_use/axbridge.swift +++ b/open_computer_use/axbridge.swift @@ -207,6 +207,7 @@ enum Delivery: String { struct BridgeError: Error { let message: String var delivery: Delivery = .notDelivered + var tookFocus = false } func mutationError(_ result: AXError, _ message: String) -> BridgeError { @@ -556,11 +557,6 @@ func operations(for element: AXUIElement, role: String, surface: AXUIElement?, e if !secureField(element), settable(element, kAXSelectedTextRangeAttribute) { out.append("SELECT_TEXT") } } if role == kAXPopUpButtonRole || role == "AXMenuButton" { out.append("SELECT") } - if available.contains(kAXIncrementAction) { out.append("INCREMENT") } - if available.contains(kAXDecrementAction) { out.append("DECREMENT") } - // Only where the accessibility API offers nothing. CLICK is aimed at a - // rectangle rather than addressed to an element, so it is the weaker way to - // reach anything that can also be pressed. if out.isEmpty, named, clickableContentRoles.contains(role) { out.append("CLICK") } // AXShowMenu is deliberately not offered. It works, but showing a menu // requires macOS to activate the app, so the one operation that opens a @@ -1139,7 +1135,7 @@ func execute(_ request: [String: Any]) throws -> [String: Any] { throw BridgeError(message: "act needs app and op") } guard ["PRESS", "CLICK", "TYPE_TEXT", "PASTE_TEXT", "SELECT_TEXT", "KEY_CHORD", - "SELECT", "MENU", "KEY", "SCROLL_UP", "SCROLL_DOWN", "INCREMENT", "DECREMENT"].contains(op) else { + "SELECT", "MENU", "KEY", "SCROLL_UP", "SCROLL_DOWN"].contains(op) else { throw BridgeError(message: "unsupported background operation \(op)") } let app = try runningApp(appName) @@ -1157,9 +1153,14 @@ func execute(_ request: [String: Any]) throws -> [String: Any] { dispatchStarted = false defer { dispatchProcess = nil } let before = frontmostPID() - let result = try executeAction(request, app: app, op: op) - try requireProcess() - return result.merging(focusFacts(before, app)) { _, fact in fact } + do { + let result = try executeAction(request, app: app, op: op) + try requireProcess() + return result.merging(focusFacts(before, app)) { _, fact in fact } + } catch var error as BridgeError { + error.tookFocus = focusFacts(before, app)["took_focus"] as? Bool ?? false + throw error + } } func executeAction(_ request: [String: Any], app: NSRunningApplication, op: String) throws -> [String: Any] { @@ -1274,12 +1275,6 @@ func executeAction(_ request: [String: Any], app: NSRunningApplication, op: Stri throw mutationError(result, "background click failed: \(describe(result))") } return ["detail": current, "mechanism": target.attribute] - case "INCREMENT", "DECREMENT": - // Actions the element published about itself. Asking the app to step - // its own control moves nothing on screen and aims at nothing. - let action = op == "INCREMENT" ? kAXIncrementAction : kAXDecrementAction - try perform(element, action) - return ["ok": true, "detail": current, "mechanism": action] case "KEY_CHORD": guard let key = request["key"] as? String else { throw BridgeError(message: "KEY_CHORD needs a key") } let editingKeys = Set("cmd+a cmd+c cmd+x cmd+v cmd+z cmd+shift+z cmd+left cmd+right left right up down shift+left shift+right alt+left alt+right shift+alt+left shift+alt+right delete forwarddelete tab shift+tab return escape".split(separator: " ").map(String.init)) @@ -1544,7 +1539,7 @@ func handle(_ request: [String: Any]) -> [String: Any] { let delivery: Delivery = method == "act" && dispatchStarted && error.delivery == .notDelivered ? .deliveredUnverified : error.delivery return ["id": id, "ok": false, "error": error.message, - "acted": delivery != .notDelivered, "disposition": delivery.disposition] + "disposition": delivery.disposition, "took_focus": error.tookFocus] } catch { let delivery: Delivery = method == "act" ? .uncertain : .notDelivered return ["id": id, "ok": false, "error": "\(error)", "disposition": delivery.disposition] diff --git a/open_computer_use/cli.py b/open_computer_use/cli.py index a476d2d..1ab5037 100644 --- a/open_computer_use/cli.py +++ b/open_computer_use/cli.py @@ -17,7 +17,6 @@ import time from contextlib import closing -from . import demo from .actions import EDITING_KEYS, action_space from .desktop import Bridge, BridgeError, Desktop, Executed, StaleWindow, UnknownOutcome @@ -104,10 +103,14 @@ def act(options): # Executed is the certain-delivery subclass. Either way: never retry blind. outcome = {"outcome": "executed, unconfirmed" if isinstance(error, Executed) else "unknown", "delivery": error.disposition["delivery"], + "took_focus": getattr(error, "took_focus", False), "detail": f"{error} Read the window before sending anything else."} code = 2 except BridgeError as error: - outcome, code = {"outcome": "refused", "delivery": error.disposition["delivery"], "detail": str(error)}, 1 + outcome, code = {"outcome": "refused", "delivery": error.disposition["delivery"], + "took_focus": getattr(error, "took_focus", False), "detail": str(error)}, 1 + if outcome.get("took_focus"): + outcome["outcome"], code = "focus_violation", 3 emit({"operation": options.operation, "label": action.get("label", options.operation), **outcome}) try: show(desktop.observe()) @@ -135,11 +138,12 @@ def main(argv=None): run.add_argument("--length", type=int, help="SELECT_TEXT UTF-16 length") commands.add_parser("inspect", help="the local inspector for the built-in agent", add_help=False) options, rest = parser.parse_known_args(argv) - if options.command == "inspect": - return demo.main(rest) - if rest: + if rest and options.command != "inspect": parser.error(f"unrecognized arguments: {' '.join(rest)}") try: + if options.command == "inspect": + from .demo import main as inspect + return inspect(rest) if options.command in {"status", "apps"}: with closing(Bridge()) as bridge: emit(bridge.call(options.command)) diff --git a/open_computer_use/demo.py b/open_computer_use/demo.py index 70e62f2..414c2cc 100644 --- a/open_computer_use/demo.py +++ b/open_computer_use/demo.py @@ -31,6 +31,7 @@ def serialise(agent): decision = state.get("decision") return { "status": state["status"], + "note": state.get("note"), "app": state["app"], "goal": state["goal"], "window": state["window"], @@ -164,7 +165,7 @@ def log_message(self, *_args): def main(argv=None): parser = argparse.ArgumentParser(prog="open-computer-use inspect", description="Local inspector for open-computer-use") - parser.add_argument("--port", type=int, default=int(os.environ.get("OPEN_COMPUTER_USE_PORT", os.environ.get("JEV_CU_PORT", 8767)))) + parser.add_argument("--port", type=int, default=int(os.environ.get("OPEN_COMPUTER_USE_PORT", 8767))) parser.add_argument("--no-open", action="store_true") options = parser.parse_args(argv) server = HTTPServer(("127.0.0.1", options.port), Handler) diff --git a/open_computer_use/desktop.py b/open_computer_use/desktop.py index bcc300a..68b9e92 100644 --- a/open_computer_use/desktop.py +++ b/open_computer_use/desktop.py @@ -117,6 +117,7 @@ def _readline(self, timeout): return line def call(self, method, timeout=20, mutating=False, **body): + mutating = mutating or method == "act" with self.lock: if self.closed: raise BridgeError("the accessibility bridge is closed") @@ -136,7 +137,7 @@ def call(self, method, timeout=20, mutating=False, **body): raise OSError("incomplete request write") self.process.stdin.flush() answer = json.loads(self._readline(timeout)) - if not isinstance(answer, dict) or answer.get("id") != request["id"]: + if not isinstance(answer, dict) or type(answer.get("id")) is not int or answer["id"] != request["id"]: raise ValueError("the reply does not match the request") if not isinstance(answer.get("ok"), bool): raise ValueError("the reply has no outcome") @@ -144,12 +145,12 @@ def call(self, method, timeout=20, mutating=False, **body): raise ValueError("the reply has no result") envelope = answer if not answer["ok"] else answer["result"] reported = envelope.get("disposition") - if "disposition" in envelope and ( - not isinstance(reported, dict) or reported.get("delivery") not in { + if mutating or "disposition" in envelope: + delivery = reported.get("delivery") if isinstance(reported, dict) else None + if (not isinstance(delivery, str) or delivery not in { "not_delivered", "delivery_uncertain", "delivered_unverified", "delivered_verified" - } - ): - raise ValueError("the reply has an invalid delivery outcome") + } or reported != disposition(delivery)): + raise ValueError("the reply has no valid delivery disposition; rebuild the bridge") except (OSError, EOFError, TimeoutError, ValueError) as error: self._stop() failure = UnknownOutcome if mutating else BridgeError @@ -160,18 +161,12 @@ def call(self, method, timeout=20, mutating=False, **body): if not answer["ok"]: message = answer.get("error", "unknown bridge error") delivery = answer.get("disposition", {}).get("delivery") - if delivery == "delivery_uncertain": - raise UnknownOutcome(message) - if delivery == "delivered_unverified" or answer.get("acted"): - raise Executed(message) - raise BridgeError(message) - result = answer["result"] - if mutating: - # Older AXBRIDGE_BIN overrides have no structured delivery field. - result.setdefault("disposition", disposition( - "delivered_verified" if result.get("verified") is True else "delivered_unverified" - )) - return result + failure = (UnknownOutcome if delivery == "delivery_uncertain" else + Executed if delivery in {"delivered_unverified", "delivered_verified"} else BridgeError) + error = failure(message) + error.took_focus = answer.get("took_focus") is True + raise error + return answer["result"] def close(self): with self.lock: @@ -189,9 +184,6 @@ def close(self): "MENU": 220, "SELECT": 220, "PRESS": 60, - # Stepping a control is the app changing its own value, done immediately. - "INCREMENT": 60, - "DECREMENT": 60, "TYPE_TEXT": 25, "PASTE_TEXT": 40, "KEY_CHORD": 80, diff --git a/open_computer_use/model.py b/open_computer_use/model.py index 7068b06..1df1210 100644 --- a/open_computer_use/model.py +++ b/open_computer_use/model.py @@ -11,7 +11,12 @@ import os import time -import httpx +try: + import httpx +except ModuleNotFoundError as error: + if error.name != "httpx": + raise + raise RuntimeError("The model-driven Agent needs the optional dependencies: install open-computer-use[agent].") from None from .actions import ( action_space, head_properties, object_format_instructions, diff --git a/open_computer_use/static/app.js b/open_computer_use/static/app.js index 9ed76ad..39d293c 100644 --- a/open_computer_use/static/app.js +++ b/open_computer_use/static/app.js @@ -9,8 +9,8 @@ async function api(path, body) { : {}; const response = await fetch(path, options); const payload = await response.json(); - if (payload.error) banner(payload.error); - else banner(null); + banner(payload.error || payload.note || (payload.status === "done" + ? "The model finished. Full-run background acceptance is unverified here." : null)); return payload; } @@ -114,7 +114,7 @@ function render(state) { renderElements(state); renderDecision(state); renderHistory(state); - $("status").textContent = state.status; + $("status").textContent = state.status === "done" ? "Model finished" : state.status; $("status").className = "pill " + state.status; $("elapsed").textContent = state.elapsed_ms ? `${state.elapsed_ms} ms elapsed` : ""; const live = !["idle", "done", "blocked"].includes(state.status); diff --git a/pyproject.toml b/pyproject.toml index 8718f76..40750e8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,10 @@ description = "A macOS computer-use agent that chooses instead of generating. Na readme = "README.md" license = "MIT" requires-python = ">=3.11" -dependencies = ["httpx[http2]>=0.28,<1"] +dependencies = [] + +[project.optional-dependencies] +agent = ["httpx[http2]>=0.28,<1"] [project.urls] Homepage = "https://github.com/max1874/open-computer-use" @@ -22,7 +25,7 @@ packages = ["open_computer_use"] artifacts = ["open_computer_use/bin/axbridge"] [dependency-groups] -dev = ["ruff>=0.14,<1"] +dev = ["ruff>=0.14,<1", "open-computer-use[agent]"] [tool.ruff] line-length = 120 diff --git a/skill/open-computer-use/SKILL.md b/skill/open-computer-use/SKILL.md index 71aa805..82829e0 100644 --- a/skill/open-computer-use/SKILL.md +++ b/skill/open-computer-use/SKILL.md @@ -54,7 +54,7 @@ The target app must be running. Start one without bringing it forward with ``` The first output line says what happened; the rest is the new table. 3. **Verify** from that table: the value, checked state, selection or window - you intended. A delivered operation is not a finished task. + you intended. A delivered operation is not a finished task. `active: false` and `took_focus: false` are endpoint samples, not proof about the complete task interval. Whole-task acceptance needs an independent result reading and continuous activation events from setup through cleanup. | Operation | Does | |---|---| @@ -83,7 +83,7 @@ an element listed with `"identity_proven": false` cannot be acted on. value back, `delivered_unverified` otherwise. - `unknown` or `executed, unconfirmed` (exit code 2): input may have landed. Never send it again blind; read the table and decide from what is there. -- `took_focus: true` means the app came forward. That breaks the background +- `focus_violation` (exit code 3) preserves the delivery outcome but requires stopping; do not retry it. `took_focus: true` means the app came forward. That breaks the background promise: stop and tell the user. ## Ask before diff --git a/tests/runtime_contracts.mjs b/tests/runtime_contracts.mjs index f33c8c5..357bf1a 100644 --- a/tests/runtime_contracts.mjs +++ b/tests/runtime_contracts.mjs @@ -6,7 +6,7 @@ const source = await readFile(new URL('../open_computer_use/axbridge.swift', imp test('the native bridge has no activation, cursor delivery or screenshot escape hatch', () => { for (const forbidden of [/\.activate\s*\(/, /cghidEventTap/, /mouseEventSource/, - /case "activate"/, /case "capture"/, /case "DRAG"/, /CGPreflightScreenCaptureAccess/]) { + /case "activate"/, /case "capture"/, /case "DRAG"/, /case "INCREMENT"/, /case "DECREMENT"/, /CGPreflightScreenCaptureAccess/]) { assert.doesNotMatch(source, forbidden); } assert.match(source, /event\.postToPid\(pid\)/); @@ -14,7 +14,7 @@ test('the native bridge has no activation, cursor delivery or screenshot escape }); test('native row clicks cannot fall through to a physical pointer route', () => { - const click = source.slice(source.indexOf(' case "CLICK":'), source.indexOf(' case "INCREMENT"')); + const click = source.slice(source.indexOf(' case "CLICK":'), source.indexOf(' case "KEY_CHORD"')); assert.match(click, /guard let target = backgroundClickTarget/); assert.match(click, /throw BridgeError/); assert.match(click, /setAttribute\(target.element, target.attribute, target.value\)/); diff --git a/tests/test_agent.py b/tests/test_agent.py index eea6041..79c0434 100644 --- a/tests/test_agent.py +++ b/tests/test_agent.py @@ -150,3 +150,47 @@ def test_wait_remains_available_on_an_unchanged_state(self): self.assertEqual(result["status"], "ready") self.assertEqual(len(result["history"]), 4) self.desktop.act.assert_not_called() + + def test_focus_violation_stops_even_when_the_app_can_still_be_read(self): + self.desktop.act.return_value["took_focus"] = True + verify = Mock(return_value=True) + with Agent("Scratch", "Edit message", verify=verify) as agent: + states = list(agent.run()) + self.assertEqual(len(states), 1) + self.assertEqual(states[0]["status"], "blocked") + self.assertTrue(states[0]["took_focus"]) + self.assertIsNone(states[0]["verified"]) + self.assertEqual(states[0]["history"][0]["disposition"], disposition("delivered_unverified")) + self.assertIn("focus", states[0]["note"]) + self.desktop.act.assert_called_once() + verify.assert_not_called() + + def test_a_post_dispatch_error_keeps_its_focus_violation(self): + error = UnknownOutcome("reply unconfirmed") + error.took_focus = True + self.desktop.act.side_effect = error + with Agent("Scratch", "Edit message") as agent: + result = agent.command("tick") + self.assertEqual(result["status"], "blocked") + self.assertTrue(result["took_focus"]) + self.assertTrue(result["history"][0]["took_focus"]) + self.assertEqual(result["history"][0]["disposition"], disposition("delivery_uncertain")) + self.desktop.act.assert_called_once() + + def test_an_activation_during_prediction_refuses_input(self): + monitor = Mock(side_effect=[True, False, False]) + with Agent("Scratch", "Edit message", focus_check=monitor) as agent: + result = agent.command("tick") + self.assertEqual(result["status"], "blocked") + self.assertEqual(result["history"][0]["disposition"], disposition("not_delivered")) + self.assertFalse(result["took_focus"]) + self.desktop.act.assert_not_called() + + def test_a_missing_monitor_cannot_silently_become_clear(self): + with Agent("Scratch", "Edit message", focus_check=lambda: None) as agent: + with patch("open_computer_use.agent.choose") as choose: + result = agent.command("tick") + self.assertEqual(result["status"], "blocked") + self.assertEqual(result["history"], []) + choose.assert_not_called() + self.desktop.act.assert_not_called() diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..c2ebb4e --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,46 @@ +"""CLI delivery facts are independent of the optional model loop.""" +import contextlib +import io +import json +from pathlib import Path +import subprocess +import sys +from types import SimpleNamespace +import unittest +from unittest.mock import Mock, patch + +from open_computer_use import cli +from open_computer_use.desktop import UnknownOutcome, disposition + + +class CLIContracts(unittest.TestCase): + def test_core_imports_do_not_load_optional_dependencies(self): + # No site-packages: a CLI import must not need the installed Agent extra. + root = str(Path(__file__).resolve().parents[1]) + subprocess.run([sys.executable, "-S", "-c", + "import sys; sys.path.insert(0, sys.argv[1]); import open_computer_use.cli; " + "assert not {'httpx', 'open_computer_use.agent', 'open_computer_use.demo'} & sys.modules.keys()", + root], check=True) + + def test_focus_violation_is_terminal_but_does_not_erase_delivery(self): + for failed_reply in (False, True): + with self.subTest(error=failed_reply): + desktop = Mock() + desktop.observe.return_value = {"fingerprint": "observation"} + desktop.act.return_value = {"disposition": disposition("delivered_unverified"), "took_focus": True} + if failed_reply: + error = UnknownOutcome("post-action reply unavailable") + error.took_focus = True + desktop.act.side_effect = error + options = SimpleNamespace(app="Scratch", fingerprint="observation", operation="PRESS", text=None) + output = io.StringIO() + with patch("open_computer_use.cli.Desktop", return_value=desktop), patch( + "open_computer_use.cli.resolve", return_value={"label": "Continue"} + ), patch("open_computer_use.cli.show"), contextlib.redirect_stdout(output): + code = cli.act(options) + result = json.loads(output.getvalue()) + self.assertEqual(code, 3) + self.assertEqual(result["outcome"], "focus_violation") + self.assertEqual(result["delivery"], "delivery_uncertain" if failed_reply else "delivered_unverified") + desktop.act.assert_called_once() + desktop.close.assert_called_once() diff --git a/tests/test_contracts.py b/tests/test_contracts.py index 7711bf6..350a821 100644 --- a/tests/test_contracts.py +++ b/tests/test_contracts.py @@ -46,8 +46,8 @@ def test_option_refs_are_prefixed_without_changing_native_choice(self): self.assertEqual(decision["action"]["path"], "0.1") self.assertEqual(decision["action"]["option"], 0) - def test_retired_pointer_operations_cannot_be_selected(self): - for op in ("DRAG", "DRAG_POINT", "CLICK_POINT", "TYPE_KEYS"): + def test_retired_operations_cannot_be_selected(self): + for op in ("DRAG", "DRAG_POINT", "CLICK_POINT", "TYPE_KEYS", "INCREMENT", "DECREMENT"): with self.subTest(operation=op), self.assertRaises(ValueError): self.decide({"operation": op, "drag_target": "1", "drag_destination": "2"}) @@ -135,3 +135,36 @@ def test_lost_mutation_reply_is_not_replayed_on_a_fresh_helper(self): self.assertEqual(marker.read_text().splitlines(), ["delivered"]) finally: bridge.close() + + + def test_missing_or_contradictory_delivery_never_becomes_a_safe_retry(self): + for reported in (None, {"delivery": "delivered_unverified", "retry": "safe"}, + {"delivery": [], "retry": "unsafe"}): + with self.subTest(disposition=reported), tempfile.TemporaryDirectory() as directory: + marker = Path(directory) / "deliveries" + helper = Path(directory) / "bridge" + helper.write_text(f"""#!{sys.executable} +import json, sys +from pathlib import Path +for line in sys.stdin: + request = json.loads(line) + result = {{'pong': True}} + if request['method'] == 'act': + with Path({str(marker)!r}).open('a') as output: + output.write('delivered\\n') + result = {{'verified': True}} + reported = {reported!r} + if reported is not None: result['disposition'] = reported + print(json.dumps({{'id': request['id'], 'ok': True, 'result': result}}), flush=True) +""") + helper.chmod(0o700) + with patch("open_computer_use.desktop.binary", return_value=str(helper)): + bridge = Bridge() + try: + with self.assertRaises(UnknownOutcome) as caught: + bridge.call("act") + self.assertEqual(caught.exception.disposition["retry"], "unsafe") + self.assertEqual(bridge.call("ping"), {"pong": True}) + self.assertEqual(marker.read_text().splitlines(), ["delivered"]) + finally: + bridge.close() diff --git a/uv.lock b/uv.lock index ea3ada7..3c64349 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 3 +revision = 5 requires-python = ">=3.11" [[package]] @@ -110,20 +110,27 @@ wheels = [ name = "open-computer-use" version = "0.1.0" source = { editable = "." } -dependencies = [ + +[package.optional-dependencies] +agent = [ { name = "httpx", extra = ["http2"] }, ] [package.dev-dependencies] dev = [ + { name = "open-computer-use", extra = ["agent"] }, { name = "ruff" }, ] [package.metadata] -requires-dist = [{ name = "httpx", extras = ["http2"], specifier = ">=0.28,<1" }] +requires-dist = [{ name = "httpx", extras = ["http2"], marker = "extra == 'agent'", specifier = ">=0.28,<1" }] +provides-extras = ["agent"] [package.metadata.requires-dev] -dev = [{ name = "ruff", specifier = ">=0.14,<1" }] +dev = [ + { name = "open-computer-use", extras = ["agent"] }, + { name = "ruff", specifier = ">=0.14,<1" }, +] [[package]] name = "ruff"