Skip to content

chore: update ChatGPT shipment test case #340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dodeja merged 1 commit into from
Aug 21, 2026
+3 −3
Merged

chore: update ChatGPT shipment test case #340

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions chatgpt-app-submission.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,11 +147,11 @@
"expected_output_url": null
},
{
"description": "List actively tracked shipments without nested containers.",
"user_prompt": "Show the first 10 shipments where shipping-line tracking has not stopped. Do not include nested containers.",
"description": "List the 10 most recently updated shipments without nested containers.",
"user_prompt": "Show my 10 most recently updated shipments. Don’t include nested containers.",

@greptile-apps greptile-apps Bot Aug 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Unsupported update-time ordering

When ChatGPT evaluates this case, list_shipments cannot fulfill “most recently updated” because the endpoint exposes no update-time sort and orders results by creation date. A conforming invocation therefore returns the most recently created shipments, causing this case to validate an inaccurate response or fail a correct implementation.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: chatgpt-app-submission.json
Line: 151

Comment:
**Unsupported update-time ordering**

When ChatGPT evaluates this case, `list_shipments` cannot fulfill “most recently updated” because the endpoint exposes no update-time sort and orders results by creation date. A conforming invocation therefore returns the most recently created shipments, causing this case to validate an inaccurate response or fail a correct implementation.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Codex Fix in Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use an ordering supported by list_shipments

When an older shipment has been updated after newer shipments, this test cannot return the requested account-wide top 10: the list_shipments schema in packages/mcp/src/server.ts exposes no updated-time or sort argument, while the source-of-truth GET /shipments operation in docs/openapi.json orders results by creation date. Fetching a page of 10 and sorting it locally would still omit recently updated records outside that page, making the submission case unsatisfiable or encouraging an inaccurate response; use the API's creation-date ordering or restore a supported filter-based case.

AGENTS.md reference: AGENTS.md:L27-L27

Useful? React with 👍 / 👎.

@vorflux vorflux Bot Aug 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list_shipments cannot order by update time; /shipments is ordered by creation date and exposes no sort parameter. Please change this prompt and expected output to "most recently created shipments," or add supported update-time ordering before using this test case.

"file_attachment_urls": null,
"tools_triggered": "list_shipments",
"expected_output": "Returns up to 10 shipments matching tracking_stopped=false without nested container relationships, with pagination details.",
"expected_output": "Returns up to 10 of the most recently updated shipments with a page size of 10, include_containers=false, no carrier, status, or port filters, and no nested containers. Mentions pagination if present.",
"expected_output_url": null
},
{
Expand Down
Loading