chore: update ChatGPT shipment test case - #340
Conversation
Co-authored-by: Akshay Dodeja <dodeja@gmail.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "description": "List actively tracked shipments without nested containers.", | ||
| "user_prompt": "Show the first 10 shipments where shipping-line tracking has not stopped. Do not include nested containers.", | ||
| "description": "List the 10 most recently updated shipments without nested containers.", | ||
| "user_prompt": "Show my 10 most recently updated shipments. Don’t include nested containers.", |
There was a problem hiding this comment.
When ChatGPT evaluates this case, list_shipments cannot fulfill “most recently updated” because the endpoint exposes no update-time sort and orders results by creation date. A conforming invocation therefore returns the most recently created shipments, causing this case to validate an inaccurate response or fail a correct implementation.
Prompt To Fix With AI
This is a comment left during a code review.
Path: chatgpt-app-submission.json
Line: 151
Comment:
**Unsupported update-time ordering**
When ChatGPT evaluates this case, `list_shipments` cannot fulfill “most recently updated” because the endpoint exposes no update-time sort and orders results by creation date. A conforming invocation therefore returns the most recently created shipments, causing this case to validate an inaccurate response or fail a correct implementation.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5326f208a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "description": "List actively tracked shipments without nested containers.", | ||
| "user_prompt": "Show the first 10 shipments where shipping-line tracking has not stopped. Do not include nested containers.", | ||
| "description": "List the 10 most recently updated shipments without nested containers.", | ||
| "user_prompt": "Show my 10 most recently updated shipments. Don’t include nested containers.", |
There was a problem hiding this comment.
When an older shipment has been updated after newer shipments, this test cannot return the requested account-wide top 10: the list_shipments schema in packages/mcp/src/server.ts exposes no updated-time or sort argument, while the source-of-truth GET /shipments operation in docs/openapi.json orders results by creation date. Fetching a page of 10 and sorting it locally would still omit recently updated records outside that page, making the submission case unsatisfiable or encouraging an inaccurate response; use the API's creation-date ordering or restore a supported filter-based case.
AGENTS.md reference: AGENTS.md:L27-L27
Useful? React with 👍 / 👎.
![vorflux[bot]](https://avatars.githubusercontent.com/in/2483280?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary
Reviewed — found 1 issue. This PR updates the ChatGPT submission test for list_shipments; I reviewed the changed prompt and expected output against the tool’s supported ordering behavior.
Findings
chatgpt-app-submission.json
- The test requires update-time ordering that
list_shipmentsdoes not support.
Verdict
| "description": "List actively tracked shipments without nested containers.", | ||
| "user_prompt": "Show the first 10 shipments where shipping-line tracking has not stopped. Do not include nested containers.", | ||
| "description": "List the 10 most recently updated shipments without nested containers.", | ||
| "user_prompt": "Show my 10 most recently updated shipments. Don’t include nested containers.", |
There was a problem hiding this comment.
list_shipments cannot order by update time; /shipments is ordered by creation date and exposes no sort parameter. Please change this prompt and expected output to "most recently created shipments," or add supported update-time ordering before using this test case.
2 participants
Summary
list_shipmentsarguments and pagination behaviorVerification
jq empty chatgpt-app-submission.jsonNeed help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Greptile Summary
This PR revises ChatGPT app-submission case 3 to request ten recent shipments without nested containers.
Confidence Score: 4/5
The test case should be corrected before merging because it requests shipment ordering that the invoked tool cannot produce.
The changed prompt asks for the most recently updated shipments, but
list_shipmentsexposes no update-time sort and the backend contract orders shipments by creation date.Files Needing Attention: chatgpt-app-submission.json
Important Files Changed
Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "chore: update ChatGPT shipment test case" | Re-trigger Greptile