Repository navigation
fix(harness): caller asks again in its next turn when a question gets a general answer - #142
Closed
KarthikAvinashFI wants to merge 1 commit into
Closed
KarthikAvinashFI wants to merge 1 commit into
KarthikAvinashFI wants to merge 1 commit into
Conversation
… a general answer
Member
Author
|
Moved into #140, which now carries this caller rule change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When the agent answered the simulated caller's question with a general remark and moved on to its own question, the caller went along and never asked again. Rule 12e told it to ask for an unanswered part "once ... and only then close", so the follow-up was held back until the end of the call and was usually lost. A caller that lets the question go hides the agent's non-answer from the evals.
What changed
src/fi/alk/harness/simulator_voice.pySIMULATOR_INSTRUCTIONS: an unanswered part or a general-remark answer is asked for once in the caller's next turn, even when the agent has moved on; the caller answers what the agent asked and asks its own question in the same turnBehavior callouts
Verification
Local calls on the guest ride booking scenario
pin-corporate-event-expiration-query, where the caller asks whether its guest PIN is single-use or valid for several rides and the agent answers with a general remark about where the PIN comes from.Both after-calls went on to a confirmed booking.
Tests
No new tests: the change is instruction wording. The harness suite (
tests/harness) andtests/test_simulator_voice_instructions.pygive the same failures as ondev.Scope
Out of scope: a persona-level exception for idle or distracted callers, and an eval that scores whether the main question was resolved; tracked separately.
Linear
Linear: TH-8365