Skip to content

fix(harness): caller asks again in its next turn when a question gets a general answer - #142

Closed
KarthikAvinashFI wants to merge 1 commit into
devfrom
fix/th-8365-caller-returns-to-unanswered-question
Closed

KarthikAvinashFI wants to merge 1 commit into
devfrom
fix/th-8365-caller-returns-to-unanswered-question

Conversation

@KarthikAvinashFI

Copy link
Copy Markdown
Member

Why

When the agent answered the simulated caller's question with a general remark and moved on to its own question, the caller went along and never asked again. Rule 12e told it to ask for an unanswered part "once ... and only then close", so the follow-up was held back until the end of the call and was usually lost. A caller that lets the question go hides the agent's non-answer from the evals.

What changed

File Change Preserved
src/fi/alk/harness/simulator_voice.py Rule 12e in SIMULATOR_INSTRUCTIONS: an unanswered part or a general-remark answer is asked for once in the caller's next turn, even when the agent has moved on; the caller answers what the agent asked and asks its own question in the same turn The rest of 12e (bring the question back after the agent's own question, a redirect is not an answer, ask once for the next step) and every other rule

Behavior callouts

  • The caller now raises an unanswered question in the next turn instead of near the end of the call, so calls where the agent sidesteps a question show the follow-up early in the transcript.

Verification

Local calls on the guest ride booking scenario pin-corporate-event-expiration-query, where the caller asks whether its guest PIN is single-use or valid for several rides and the agent answers with a general remark about where the PIN comes from.

Run What the caller did after the general answer Result
Before (dev rule) Gave the PIN and moved on to the trip; never asked about PIN validity again Question dropped
After, call 1 Next turn (1:22): "I do have the PIN, I just wanted to know if it's single-use or good for multiple rides? But anyway, the PIN is seven six eight two." Asked again in the next turn
After, call 2 Next turn (1:39): "Wait, I do have the PIN, but is it single-use or can I use it for multiple rides?" The agent answered that the organization decides, and the caller accepted Asked again in the next turn

Both after-calls went on to a confirmed booking.

Tests

No new tests: the change is instruction wording. The harness suite (tests/harness) and tests/test_simulator_voice_instructions.py give the same failures as on dev.

uv run pytest tests/harness tests/test_simulator_voice_instructions.py

Scope

Out of scope: a persona-level exception for idle or distracted callers, and an eval that scores whether the main question was resolved; tracked separately.

Linear

Linear: TH-8365

@KarthikAvinashFI KarthikAvinashFI self-assigned this Oct 6, 2026
@KarthikAvinashFI

Copy link
Copy Markdown
Member Author

Moved into #140, which now carries this caller rule change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant