Tune prompts for review mode

[michaeljguarino]

This will hopefully help the agent chose appropriately when reviewing prs

Test Plan

Test environment: https://console.your-env.onplural.sh/

Checklist

• I have added a meaningful title and summary to convey the impact of this PR to a user.
• If required, I have updated the Plural documentation accordingly.
• I have added tests to cover my changes.
• I have deployed the agent to a test environment and verified that it works as expected (required only when changing agent code).

Plural Flow: console

[soffi-ai]

Soffi AI Summary

This PR improves the AI workbench's agentic behavior when handling pull request review tasks. The core motivation is to help the orchestrator agent correctly identify and delegate PR review work to the coding subagent using a dedicated "review mode," rather than treating reviews as out-of-scope or routing them through integration tools.

Key changes:

• Prompt tuning for review mode: The job.md.eex orchestrator prompt now conditionally renders review-specific guardrails when a job has @review set — instructing the agent to launch the coding subagent in review mode, not to re-use integration tools for posting reviews, and not to approve/reject PRs.
• Removal of the unused Plan subagent: The plan subagent (both its Elixir module, prompt, and tool schema) has been entirely removed since it was never used in practice, reducing dead code and potential confusion.
• Review mode surfaced in coding subagent description: The Subagents tool now passes job context so the coding subagent's description advertises review mode when a review job is active.
• Dedicated coding agent review tool schema: A new coding_agent_review.json tool definition is added, providing the agent a distinct schema for invoking the coding subagent in review mode.
• GitHub Checks integration for PR reviews: AgentRun gains a check_id field (with migration), and the PR review flow across GitHub, GitLab, Bitbucket (cloud and datacenter), and Azure DevOps now supports creating/updating check runs or equivalent status reports alongside the review comment, so review results are surfaced as CI status signals.
• PR review summary prompt refresh: The agent_review_summary.md.eex and agent_review_check.md.eex templates are updated to produce cleaner, more structured review output.

Changes

PR review mode for AI workbench agent

• Initial prompt tuning for review mode: conditionally renders review-specific instructions in the orchestrator job prompt so the agent correctly delegates PR review to the coding subagent rather than treating it as out of scope. (d517386)
• Cleans up the summary comment in the review prompt and removes the Plan subagent entirely — its Elixir module, prompt template, tool schema, and tests are all deleted since the subagent was never used in practice. (617c599)
• Adds GitHub Checks (and equivalent SCM status) integration to PR review runs: introduces a check_id field on AgentRun (with a migration), wires check creation and completion through GitHub, GitLab, Bitbucket Cloud, Bitbucket Datacenter, and Azure DevOps PR implementations, and adds a dedicated coding_agent_review.json tool schema so the coding subagent can be invoked explicitly in review mode. (d3450c3)

---

Updated: 2026-09-01 00:18 UTC

Deploy in Soffi

[greptile-apps]

Greptile Summary

The PR tunes workbench prompts and tool metadata so review-enabled jobs consistently direct coding agents to review the pull request’s head branch.

• Adds review-aware coding-agent schemas, descriptions, and subagent prompts.
• Propagates review capability through workbench job and coding-agent initialization.
• Formats reviewed-file summaries as a collapsible Markdown table.
• Adds focused coverage for review-mode selection, prompt rendering, subagent descriptions, and summary escaping.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
lib/console/ai/tools/workbench/coding_agent.ex	Selects review-specific tool metadata when enabled while retaining runtime validation for required review inputs.
lib/console/ai/tools/workbench/subagents.ex	Makes the coding-subagent description aware of the current job’s review capability.
lib/console/ai/workbench/engine.ex	Propagates the job’s review capability into orchestrator prompts and subagent metadata.
lib/console/ai/workbench/subagents/coding.ex	Supplies an explicit review flag when rendering the coding-subagent system prompt.
priv/prompts/workbench/coding.md.eex	Provides review-specific mode and pull-request input guidance only for review-enabled jobs.
priv/prompts/workbench/job.md.eex	Directs review-enabled orchestration toward coding-agent review mode and avoids duplicate review publication.
priv/tools/workbench/coding_agent_review.json	Defines the model-facing tool schema for jobs where pull-request review mode is available.
priv/pr/agent_review_summary.md.eex	Replaces the file-summary list with a collapsible Markdown table using escaped cell values.

_{Reviews (2): Last reviewed commit: "Tune prompts for review mode" | Re-trigger Greptile}

[michaeljguarino]

plural deploy this

[plural-copilot]

GitOps deployment PR: https://github.com/pluralsh/plrl-up-demos/pull/2760

This deploys source PR head 754bf8cd7ba81c82668e22a177a5773b192568d8 (sha-754bf8c) to the demo environment.

[michaeljguarino]

plural review this

[plural-copilot]

Plural Summary

Medium-depth review found two regressions: the revised review-summary template conflicts with its accompanying assertion and will fail that test, and the runtime currently permits review plus followup, which can check out a branch different from the required PR-head branch while still publishing to the supplied PR. Review-mode propagation and removal of the active Plan subagent paths otherwise looked complete.

Grade: B

Confidence is high for the two reported regressions because they are directly implied by the changed template/test and the reviewed runtime dispatch paths. I did not execute tests under the strict read-only constraint, so runtime/environmental coverage is limited to static inspection.

Files changed (24)

Filename	Summary
assets/src/components/ai/chatbot/toolCallDisplay.tsx	Removes the stale Plan display-title override.
lib/console/ai/tools/workbench/coding_agent.ex	Selects review-aware schemas/descriptions, gates review mode, validates review inputs, and creates agent runs.
lib/console/ai/tools/workbench/plan.ex	Deletes the unused Plan Workbench tool.
lib/console/ai/tools/workbench/subagents.ex	Makes coding-subagent descriptions conditional on job review capability.
lib/console/ai/workbench/engine.ex	Removes Plan subagent dispatch and propagates review capability to the orchestrator prompt.
lib/console/ai/workbench/subagents/coding.ex	Propagates review capability into the coding-subagent prompt.
lib/console/ai/workbench/subagents/plan.ex	Deletes the unused Plan subagent implementation.
lib/console/deployments/pr/review.ex	Adds table-cell escaping for pipe/newline handling in review summaries.
lib/console/schema/workbench_job.ex	Adds the shared coding_review?/1 capability predicate.
priv/pr/agent_review_summary.md.eex	Reformats review summaries with grade and expandable file table.
priv/prompts/workbench/coding.md.eex	Adds conditional review-mode instructions for coding subagents.
priv/prompts/workbench/job.md.eex	Adds conditional orchestrator guidance for PR review tasks.
priv/prompts/workbench/plan.md	Deletes the unused Plan prompt.
priv/tools/workbench/coding_agent.json	Restricts the standard coding-agent schema to analyze/write modes.
priv/tools/workbench/coding_agent_review.json	Adds the review-enabled coding-agent schema and review-specific parameter guidance.
priv/tools/workbench/plan.json	Deletes the unused Plan tool schema.
test/console/ai/tools/workbench/coding_agent_test.exs	Adds coverage for review schema, validation, prompting, and run creation.
test/console/ai/tools/workbench/subagents_test.exs	Adds conditional coding-subagent review-description coverage.
test/console/ai/workbench/engine_test.exs	Removes Plan-subagent engine expectations.
test/console/ai/workbench/subagents/plan_test.exs	Deletes tests for the removed Plan subagent.
test/console/deployments/pr/impl/github_test.exs	Updates expected review-summary heading presentation.
test/console/deployments/pr/review_test.exs	Adds table formatting/escaping coverage, but contains a heading assertion inconsistent with the template.
test/console/schema/workbench_job_test.exs	Adds coding_review?/1 predicate coverage.
test/test_helper.exs	Removes the deleted Plan subagent Mimic registration.

spurious review due to a bug

[michaeljguarino]

plural deploy this

[plural-copilot]

Demo deployment pin created for image sha-4fc03ff: https://github.com/pluralsh/plrl-up-demos/pull/2761

[michaeljguarino]

plural review this

[michaeljguarino]

plural deploy this

[plural-copilot]

Demo deployment GitOps PR: https://github.com/pluralsh/plrl-up-demos/pull/2762

Pins the console image to sha-617c599 from source commit 617c599e74bae06ecf5352c082be9c229aab9ab9.

[michaeljguarino]

plural review this

[michaeljguarino]

plural deploy this

[plural-copilot]

GitOps deployment PR: https://github.com/pluralsh/plrl-up-demos/pull/2763

It pins the demo Console image to sha-d3450c3 for source commit d3450c3163bad372e83b14bb1efd7537a823381a.