Skip to content

Expand /evaluate response definition: formatted content and case matching - #38

Open
m-messer wants to merge 3 commits into
mainfrom
31-expand-evaluate-response-definition
Open

m-messer wants to merge 3 commits into
mainfrom
31-expand-evaluate-response-definition

Conversation

@m-messer

Copy link
Copy Markdown
Contributor

Summary

  • Adds a reusable FormattedContent schema (paths/shared/schemas/FormattedContent.yml) representing a submission's normalized value in multiple, service-defined formats keyed by an arbitrary format id (e.g. latex/sympy for MATH, formatted/ast for CODE) — resolves the response-format ambiguity raised in Response Definition #31 (Lambda Feedback's responseSympy/responseLatex pattern) without hardcoding any specific format names.
  • Wires it into Feedback.response, and adds Feedback.matchedCase (index into a service-defined case-matching list, e.g. Lambda Feedback's cases config) — both grounded in real example responses from a Lambda-Feedback-compatible service (compareExpressions).
  • Adds a mathExampleResponse example to evaluate.yml demonstrating both new fields in matched and unmatched states.
  • Doc-only clarification on PreSubmissionFeedback.yml / EvaluateRequest.yml pointing out that pre-submission preview content comes back via Feedback.response, not the request-side toggle schema.
  • Fixes a pre-existing stray-character typo in EvaluateHealthResponse.yml that was breaking YAML parsing and blocking lint.

Explicitly out of scope (by design): no formal Cases/Case schema — case-based matching stays a Lambda-Feedback-specific, informal convention under the free-form configuration object.

Closes #31.

Test plan

  • npm run lint passes (0 errors; only pre-existing unrelated warnings remain)
  • npm run bundle succeeds; confirmed FormattedContent and the new fields resolve correctly in dist/openapi.yml

🤖 Generated with Claude Code

https://claude.ai/code/session_017yjEAviVJipdWqJ9RgjnSm

…matching (#31)

Adds a reusable FormattedContent schema for representing a submission's
normalized value in multiple, service-defined formats (e.g. latex/sympy for
MATH, formatted/ast for CODE), wired into Feedback.response. Also adds
Feedback.matchedCase for services with case-based matching. Resolves the
response-format ambiguity raised in issue #31.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017yjEAviVJipdWqJ9RgjnSm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Response Definition

3 participants