[abhinav-pola]

TL;DR

Fixes the two Devin Review findings on openrouter-web#50950: sandbox agents now honor providerOrder, and toolcall_schema_fuzz scores a 400/422 as request_rejected only when a provider rejected the request.

What changed?

• AgentCliOpts.providerOrder is passed as TB_OPENROUTER_PROVIDER_ORDER to the pi before_provider_request extension. The extension sets provider.order and keeps any other provider preferences the caller sent. terminal_bench, deep-swe and swe-atlas pass providerOrder through for Ori agents. Agents that don't forward request plugins fail with a SolverError, which matches how they already handle models.
• toolcall_schema_fuzz scores a 400/422 as request_rejected only when error.metadata.provider_name in the error body is a non-empty string. A 400/422 that names no provider, such as OpenRouter rejecting an out-of-range max_tokens, now aborts the run instead of producing a 0% score.

Why?

• Agent benchmarks accepted providerOrder but dropped it before sending requests.
• A setting that was invalid for the whole run made every sample request_rejected.

How to test

The error shapes below were captured live today:

# An upstream schema rejection has provider_name "Azure" and is scored as request_rejected
curl -s https://openrouter.ai/api/v1/chat/completions -H "Authorization: Bearer $OPENROUTER_API_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"deepseek/deepseek-v4-flash","messages":[{"role":"user","content":"call f"}],"tools":[{"type":"function","function":{"name":"f","parameters":{"type":"object","properties":{"a":{"type":"strng"}}}}}],"provider":{"only":["azure"],"allow_fallbacks":false}}'
# A run-level rejection has provider_name null and aborts the run
curl -s https://openrouter.ai/api/v1/chat/completions -H "Authorization: Bearer $OPENROUTER_API_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"deepseek/deepseek-v4-flash","messages":[{"role":"user","content":"hi"}],"max_tokens":99999999}'

Benchmark impact

• toolcall_schema_fuzz: rejections by upstream providers are scored as before. Rejections that name no provider now abort the run instead of being scored.
• Agent benchmarks change only when providerOrder is set.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/72a3f13c847645c5b06306d88cec089d
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/72a3f13c847645c5b06306d88cec089d?variant=devin
Requested by: @abhinav-pola

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Abhinav

SYSTEM:
<latest_message>
Abhinav Pola (U090K0G7JF3) [ts=1791572889.759909]: @Devin move the tool call fuzz dataset to the public repo and sync it back to the subtree
</latest_message>

=== BEGIN THREAD HISTORY (in #agents-benchmarks) ===
Abhinav Pola (U090K0G7JF3) [ts=1791572889.759909]: @Devin move the tool call fuzz dataset to the public repo and sync it back to the subtree
=== END THREAD HISTORY ===
Channel ID: C0BAKP8P5C3
Thread URL: https://openrouter.slack.com/archives/C0BAKP8P5C3/p1791572889759909?thread_ts=1791572889.759909&amp;cid=C0BAKP8P5C3

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

[devin-ai-integration]

Devin Review found 1 potential issue.