Provide preceding conversation context when grading a selected turn #253

Problem or motivation

A response may depend on information supplied or corrected earlier in a conversation.

For example:

1. The user requests a booking on Monday.
2. The assistant asks for missing details.
3. The user supplies those details and changes the day to Tuesday.
4. The assistant presents the completed plan using Tuesday.

A rubric applied to the completed response should be able to verify that the latest preference was honored. If the evaluator sees only the current prompt and response, it may treat earlier details as unsupported or miss that a value was corrected.

Proposed solution

Support grading a selected turn with the relevant preceding conversation available as context, while keeping the rubric’s target turn explicit.

This should allow evaluators to:

• Use facts, answers, and corrections established earlier.
• Keep clarification requirements distinct from completion requirements.
• Exclude later turns when judging an earlier response.
• Preserve property IDs, scores, and explanations associated with the evaluated turn.

The exact representation can follow the planned turn-aware expectations format.

Alternatives considered

Repeating conversation history inside each prompt changes the evaluation input and adds bookkeeping. Whole-conversation rubrics are useful for overall behavior, but individual response properties still need the context available when that response was produced.

Additional context

Related: #229 documents the existing per-invocation behavior. Its maintainer review also discusses planned turn-aware expectations.

If this use case is already covered by that work, please link the relevant issue or PR. Small clarification, completion, and correction examples would help verify the intended behavior.

Human confirmation

• I am a human (not a bot, agent, or AI) filing this issue.