Skip to content

fix(harness): apply an edited max_turns, require everything=true to drop every scenario, and import fi.simulate on the callback adapter's main thread - #128

Merged
KarthikAvinashFI merged 7 commits into
devfrom
fix/th-8142-guard-suite-wipe
Oct 13, 2026
Merged

KarthikAvinashFI merged 7 commits into
devfrom
fix/th-8142-guard-suite-wipe

Conversation

@KarthikAvinashFI

@KarthikAvinashFI KarthikAvinashFI commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Part of TH-8142.

Why

  • An edited max_turns was accepted by the platform editor but chat runs ignored it, because only background noise and persona fields were overlaid onto the scenario a run plays.
  • A single misread builder turn could call drop_scenario('*') and replace a whole suite with what came after it.
  • Every hosted chat run of a callback-style agent failed before the first turn with Plugins must be registered on the main thread: the callback adapter imported fi.simulate on a request thread, and with livekit installed that import registers plugins, which livekit allows only on the main thread. Voice agents do not use this adapter.

What changed

  • max_turns is now one of the editable fields applied to the scenario a run plays.
  • Dropping every scenario needs an explicit everything=true; without it the tool refuses and says so. After an explicit drop-all, the tool says the emptied suite is not saved until a scenario is submitted, and the write-scenarios skill text matches.
  • The callback adapter imports fi.simulate at module level, so it runs on the main thread at process start.

Tests

  • tests/harness/test_scenario_edits.py: an edited max_turns reaches the scenario.
  • On a stored run whose scenario was sealed with max_turns 8 and edited to 3, the chat runner's scenario reads 3 on this branch and 8 on dev.
  • tests/test_harness.py: drop-all is refused without everything=true and accepted with it.
  • tests/harness/test_callback_http_adapter.py: runs the adapter as a bundle does and sends one request; fails before the fix when livekit is installed, passes after.
  • tests/harness + tests/test_harness.py: the same failures as on dev (database-backed and prompt-budget tests), nothing new.

@KarthikAvinashFI KarthikAvinashFI self-assigned this Sep 30, 2026
@KarthikAvinashFI
KarthikAvinashFI changed the base branch from main to dev October 6, 2026 08:44
@KarthikAvinashFI KarthikAvinashFI changed the title fix(harness): require everything=true before dropping every scenario fix(harness): apply an edited max_turns to the scenario a run plays and require everything=true to drop every scenario Oct 8, 2026
@KarthikAvinashFI
KarthikAvinashFI marked this pull request as ready for review October 9, 2026 04:42
@KarthikAvinashFI KarthikAvinashFI changed the title fix(harness): apply an edited max_turns to the scenario a run plays and require everything=true to drop every scenario fix(harness): apply an edited max_turns, require everything=true to drop every scenario, and import fi.simulate on the callback adapter's main thread Oct 9, 2026
Base automatically changed from dev to main October 9, 2026 10:02
@KarthikAvinashFI
KarthikAvinashFI changed the base branch from main to dev October 9, 2026 10:40
Base automatically changed from dev to main October 12, 2026 11:25
@KarthikAvinashFI
KarthikAvinashFI changed the base branch from main to dev October 12, 2026 17:46
@KarthikAvinashFI
KarthikAvinashFI merged commit 13f66d5 into dev Oct 13, 2026
4 checks passed
@KarthikAvinashFI
KarthikAvinashFI deleted the fix/th-8142-guard-suite-wipe branch October 13, 2026 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants