Skip to content

[BUG] phishing_v1 cassette replay is time-dependent: 4 items miss after some hours #346

Closed

Description

@gapilongo

Found 2026-11-09 while merging FIX-2 (#345). tests/integration/eval/test_cassettes.py::test_the_committed_cassettes_replay_offline_like_a_live_run[phishing_v1] passed earlier today and fails on main at fc0cf4f at 19:24 with no code change. phish-003, -004, -006 and -008 raise CassetteMiss in triage.assess (fast tier): the triage input hash differs from the recorded one.

Cause (suspected): a wall-clock value, such as the email's age or the time since it was received, goes into the triage prompt for email alerts.

Fix: derive any time-relative prompt input from the alert or eval clock, not the wall clock, so replays are deterministic. Then re-record only if the prompt content changes, and add a test that replays at two different wall-clock times.

This blocks every merge: safe_merge runs tests/integration/eval.

Field Value
Tier T1
Priority Must

Activity

  1. added a commit that references this issue on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions