Skip to content

Plugins: a zeron.tool answer can show the agent images - #1362

Closed
hoangvu12 wants to merge 1 commit into
zeronsh:zeron/zgxfrom
hoangvu12:plugin-tool-images
Closed

hoangvu12 wants to merge 1 commit into
zeronsh:zeron/zgxfrom
hoangvu12:plugin-tool-images

Conversation

@hoangvu12

@hoangvu12 hoangvu12 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

A plugin tool has no way to show the agent a picture. Whatever run returns reaches the agent as one JSON text block, so a base64 PNG arrives as a long string the model reads as characters. We're building a design tool as a Luau plugin, and its agent needs to see the screens it draws.

Now a tool result with a top-level imageContent array goes out as MCP image content, after the rest of the result as text:

return {
	screen = "Home",
	imageContent = { { type = "image", data = png_base64, mimeType = "image/png" } },
}

The change is in the MCP server's tools/call handler (crates/mcp/src/jsonrpc.rs), so it applies to any tool. Plugins are the ones that need it, because their answers pass through unchanged. Each entry must be a PNG, JPEG, WebP or GIF of at most 5 MB. An entry that fails this is left out, and a last text block tells the agent which one and why, so one bad image can't make the model's API reject the whole turn. Images also stop before the reply reaches the server's 8 MB cap (MAX_REPLY_BYTES), since cap_reply would otherwise cut the reply to text and lose every image without saying so.

Such a result goes without structuredContent. When it's there, Codex gives the model only structuredContent and drops the images, and Claude Code shows it in place of the text blocks, which would hide the "left out" note. Results without an imageContent array go out exactly as before, structuredContent included.

I checked this against the agents' own clients with a small MCP server that returns a text block and an image, with and without structuredContent. Claude Code 2.1.310 and OpenCode 1.18.30 saw the image both ways. Codex 0.128.0 sends a non-null structuredContent alone (models.rs, as_function_call_output_payload), so it needs the image result without it. Devin's client passed the image to its model, but its free model has no vision.

Documented in apps/zgx/docs/plugins.md (Tools for agents). The SDK's zeron.luau header is left alone, since its copies in every plugin must match it byte for byte (sdk_copies_match_canonical).

Testing

Four new tests in jsonrpc.rs call a plugin tool through tools/call against an in-memory engine. They check that images arrive as image content with the rest as text and no structuredContent, that bad entries (wrong shape, invalid base64, unsupported type, over 5 MB) are left out and named, that images past the 8 MB reply cap are left out and the reply goes whole, and that results without an image array are unchanged.

cargo test -p zeron-mcp --lib: 74 passed, 0 failed. rustfmt --check and cargo clippy -p zeron-mcp --tests show nothing new. The SDK and its 16 copies in plugins-builtin and plugins-dev are unchanged and still identical, as sdk_copies_match_canonical requires.

Open questions

  • The 5 MB cap and the four types match Anthropic's image limits. Other providers differ.
  • The zeron.tool line in the SDK's zeron.luau header could mention imageContent too, at the next SDK sync.

@hoangvu12
hoangvu12 changed the base branch from main to zeron/zgx November 12, 2026 07:09
A plugin tool's answer reached the agent as one JSON text block, so a base64 PNG was a long string the model read as characters. Now a tool result with a top-level `imageContent` array goes out as MCP image content, after the rest of the result as text. Each entry must be a PNG, JPEG, WebP or GIF of at most 5 MB, and images stop before the reply reaches the 8 MB cap; one left out is named in a last text block. Such a result goes without `structuredContent`: Codex shows the model only that (no images) when it is there, and Claude Code shows it in place of the texts.
@hoangvu12

Copy link
Copy Markdown
Contributor Author

Folded into #1392, which combines our plugin pull requests into one for review. This change is its commit 439a369.

@hoangvu12 hoangvu12 closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant