feat: ref-based interaction tools (fill, hover, press_key, select, check, scroll) by markcallen · Pull Request #3 · everydaydevopsio/pilot

markcallen · 2026-08-27T00:38:11Z

Summary

6 new interaction tools that work with element refs from browser_snapshot, enabling fully selector-free workflows:
- browser_fill — replace form field value with React/Vue/Svelte-compatible events
- browser_hover — move mouse to element by ref or coordinates
- browser_press_key — named keys and modifier combos (Control+a, Meta+c, etc.)
- browser_select — select <option> by value, label, or index
- browser_check — toggle checkbox/radio with optional target state
- browser_scroll — scroll element into view, by direction, or absolute position
Retrofitted existing tools: browser_click and browser_type now accept ref alongside selector/coordinates
Shared infrastructure: ref-resolver (scroll into view → box model → remote object) and interaction/events (native value setter + event dispatch for framework compatibility)

Test plan

All 204 unit tests pass (existing tests unmodified)
Smoke test: 25 tools registered, 29/29 passed
Build: clean
Lint: 0 errors
Prettier: clean
E2E tests with real browser (CI)

🤖 Generated with Claude Code

Add 6 new interaction tools that work with element refs from browser_snapshot, enabling selector-free workflows: - browser_fill: replace form field value with React/Vue/Svelte- compatible events (native value setter + input/change dispatch) - browser_hover: move mouse to element by ref or coordinates - browser_press_key: named keys (Enter, Tab, Escape, Arrow*, F1-F12) and modifier combos (Control+a, Meta+c, Shift+Enter) - browser_select: select <option> by value, label, or index - browser_check: toggle checkbox/radio with optional target state - browser_scroll: scroll element into view, by direction, or absolute Retrofit existing tools: - browser_click: now accepts ref alongside selector/coordinates - browser_type: now accepts ref alongside selector Shared infrastructure: - ref-resolver: resolve ref → scroll into view → box model coords → remote object ID - interaction/events: framework-compatible value setting and event dispatch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The click tool now accepts ref in addition to selector/coordinates, so the error message changed from "selector or x/y" to "ref, selector, or x/y". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds ref-based (snapshot-driven) browser interaction capabilities to the MCP server, enabling selector-free automation by resolving element refs to backend node IDs, coordinates, and remote objects via CDP.

Changes:

Introduces six new MCP interaction tools (browser_fill, browser_hover, browser_press_key, browser_select, browser_check, browser_scroll) plus shared ref-resolution/event-dispatch infrastructure.
Extends existing browser_click and browser_type to accept ref in addition to selectors/coordinates.
Registers the new tools server-side and updates the canonical expected-tool list and one e2e assertion.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/mcp/tools/names.ts	Adds new interaction tool names to the canonical expected tool list.
src/mcp/tools/interaction.ts	Registers the six new interaction MCP tools and defines their parameter shapes.
src/mcp/tools/browser.ts	Extends `browser_click` / `browser_type` tool schemas and routing to support `ref`.
src/mcp/server.ts	Registers the new interaction tools with the MCP server.
src/commands/type.ts	Adds ref-based focus support for typing.
src/commands/select.ts	Implements select-option behavior using resolved element refs and injected JS.
src/commands/scroll.ts	Implements scrolling by ref/direction/absolute coordinates via CDP.
src/commands/press_key.ts	Implements key/modifier combo parsing and CDP key event dispatch.
src/commands/hover.ts	Implements hover via ref resolution or explicit coordinates.
src/commands/fill.ts	Implements framework-friendly form filling via native value setter + events.
src/commands/click.ts	Adds ref-based coordinate resolution for clicking.
src/commands/check.ts	Implements checkbox/radio toggling via injected JS and ref resolution.
src/browser/interaction/ref-resolver.ts	Adds shared ref resolver (scroll-into-view → box model → remote object).
src/browser/interaction/events.ts	Adds shared field value setter and event dispatch helpers.
e2e/mcp/browser-tools.test.ts	Updates the `browser_click` missing-params error expectation to include `ref`.

Suppressed comments (2)

src/commands/click.ts:67

The new ref-based click path (params.ref) is not covered by the existing unit tests in test/commands/click.test.ts (which only cover selector and x/y). Please add a unit test that passes a mocked ElementRefMap and asserts resolveRef coordinates are used.

  if (params.ref !== undefined && refMap) {

src/commands/type.ts:39

The new ref-based focus path (params.ref) is not covered by the existing unit tests in test/commands/type.test.ts. Adding a test that supplies a mocked ElementRefMap and asserts DOM.focus is invoked via focusRef would help lock in the new behavior.

  if (params.ref && refMap) {

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

markcallen · 2026-08-27T01:20:11Z

+  if (params.ref !== undefined && refMap) {
+    const resolved = await resolveRef(client, refMap, params.ref);
+    x = resolved.x;
+    y = resolved.y;
+  } else if (params.selector !== undefined) {
    const pos = await waitForSelector(


Fixed — executeClick now throws "Element ref map is required when using ref parameter" when ref is provided without refMap, instead of silently falling through to undefined coordinates.

markcallen · 2026-08-27T01:20:13Z

+  if (params.ref && refMap) {
+    const resolved = await resolveRef(client, refMap, params.ref);
+    await focusRef(client, resolved.backendNodeId);
+  } else if (params.selector) {
    await focusSelector(client, params.selector);
  }


Fixed — executeType now throws the same guard error when ref is provided without refMap.

markcallen · 2026-08-27T01:20:16Z

+    async (params) => {
+      const { client, refMap } = requireContext(context);
+      const result = await executeScroll(client, refMap, params);


Fixed — browser_scroll now validates that at least one of ref, direction, or x/y is provided and returns an error otherwise.

markcallen · 2026-08-27T01:20:17Z

+  server.tool(
+    'browser_fill',
+    'Fill a form field by element ref. Replaces the entire field value and fires input/change events compatible with React, Vue, and Svelte. Use browser_snapshot first to get refs.',
+    fillShape,


Acknowledged — e2e tests for all new interaction tools are planned for Phase 11 (agent E2E suite) per the implementation plan.

markcallen · 2026-08-27T01:20:19Z

+  const { result } = await client.Runtime.callFunctionOn({
+    objectId: resolved.objectId,
+    functionDeclaration: `function(desiredState) {
+      var tag = this.tagName.toLowerCase();
+      var type = (this.type || '').toLowerCase();
+      if (tag !== 'input' || (type !== 'checkbox' && type !== 'radio')) {
+        throw new Error('Element is not a checkbox or radio button');
+      }
+      var current = this.checked;
+      var target = desiredState !== null ? desiredState : !current;
+      if (current !== target) {
+        this.click();
+      }
+      return this.checked;
+    }`,
+    arguments: [{ value: params.checked ?? null }],
+    returnByValue: true,
+    awaitPromise: false
+  });
+
+  if (result.subtype === 'error' || result.className === 'Error') {
+    throw new Error(result.description ?? 'Failed to toggle checkbox');
+  }


Fixed — now captures exceptionDetails from Runtime.callFunctionOn and uses exceptionDetails.exception.description for reliable error messages.

markcallen · 2026-08-27T01:20:20Z

+  const { result } = await client.Runtime.callFunctionOn({
+    objectId: resolved.objectId,


Fixed — same exceptionDetails handling applied to executeSelect.

- click/type: fail fast when ref provided without refMap instead of silently falling through to undefined coordinates - hover: remove unused selector field from HoverParams - scroll: validate that at least one of ref/direction/x/y is provided - check/select: use exceptionDetails from Runtime.callFunctionOn for reliable error reporting instead of checking result.subtype E2E tests for new interaction tools are planned for Phase 11. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

markcallen and others added 2 commits August 26, 2026 17:37

fix: update e2e test to match new click error message

8329b5a

The click tool now accepts ref in addition to selector/coordinates, so the error message changed from "selector or x/y" to "ref, selector, or x/y". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

markcallen requested a lite review from Copilot August 27, 2026 01:05

Copilot started reviewing on behalf of markcallen August 27, 2026 01:06 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

markcallen merged commit 738eec1 into main Aug 27, 2026
4 checks passed

Navigation Menu

Uh oh!

feat: ref-based interaction tools (fill, hover, press_key, select, check, scroll) - #3

feat: ref-based interaction tools (fill, hover, press_key, select, check, scroll)#3
markcallen merged 3 commits into
mainfrom
feat/element-interaction

markcallen commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

markcallen Aug 27, 2026

Uh oh!

markcallen Aug 27, 2026

Uh oh!

Uh oh!

markcallen Aug 27, 2026

Uh oh!

markcallen Aug 27, 2026

Uh oh!

markcallen Aug 27, 2026

Uh oh!

markcallen Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		const { result } = await client.Runtime.callFunctionOn({
		objectId: resolved.objectId,

Uh oh!

Conversation

markcallen commented Aug 27, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

markcallen Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

markcallen Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

markcallen Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

markcallen Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

markcallen Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

markcallen Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants