Support user-defined rubrics over a complete conversation #252

Problem or motivation

Some expectations describe behavior across a conversation rather than the quality of one response.

For example, a booking assistant may need to collect missing information, honor a later correction, and complete the task without repeating questions that have already been answered.

Grading completion requirements independently on every turn can penalize a correct clarification request. Grading only the last response can miss behavior elsewhere in the conversation.

Proposed solution

Provide a supported way to author rubric properties for a complete conversation and evaluate them against the available user messages, assistant responses, and relevant tool evidence.

Useful properties could include:

• Clarifies missing information before completing the task.
• Honors the user’s latest correction.
• Completes the requested outcome once the necessary information is available, without re-asking resolved questions.

Results should preserve each property’s ID, score, and explanation, and clearly identify its conversation-wide scope. Missing grading results should remain distinguishable from properties that were evaluated and failed.

Alternatives considered

Per-turn rubrics remain useful for individual responses. Custom evaluators can assess broader behavior, but a supported conversation-level path would make these expectations easier to author and review consistently.

Additional context

Related: #132, #229, #236, and #243.

ADK already provides a rubric-based multi-turn trajectory evaluator, which may be useful here. The public interface can follow the project’s intended direction.

Human confirmation

• I am a human (not a bot, agent, or AI) filing this issue.