Repository navigation
CommonTools: screen events for autonomous programs (what the screen is doing, since when) - #313
Conversation
Shared screen events and their timed log, the Switch HOME check in front of a game's observer, the transition tracker with per-game fade profiles, the scene judges and the interrupt classifier, taken as they are at nav/story-steps fd603ca. None of it uses a story type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…ta-free executor) # Conflicts: # SerialPrograms/cmake/SourceFiles.cmake
…tion) # Conflicts: # SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.cpp
…/screen-state # Conflicts: # SerialPrograms/cmake/SourceFiles.cmake
…reen-state Brings in #312's typed step args (StoryArgs gave way to StoryActionArg there) and #45/#325's split of StoryLog, AutoStoryStats and the range options. Conflicts were only the file and test registration lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
The newer code-health rules (header symbol counts, no char* / char[]) failed 19 times against nav/story-engine. This is the #313 part of #122's 7d04d13, so the fix lands in the PR that owns these files: declarations move unchanged into headers by concern, each included by the header they came from (ScreenEvents: event types, fades, event log; Interrupts: warp judge; Transitions: fade profiles; AutoStory_Tests: the screen-state suites). The *_name() helpers return std::string and test rows hold std::string. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…ts) into nav/story-steps #313 now carries its own part of 7d04d13's header splits, so those files come in unchanged; ScreenStory_Tests.h keeps this branch's two extra suites. Registration lists keep this branch's order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
add_tests_AutoStory() also registers the screen story suites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
String building uses std::format; add_fade_evidence, first_beat and run_tracker take their output reference last. Pokemon_ScreenStory_Tests.h is replaced by Pokemon_ScreenEvents_Tests.h, Pokemon_Interrupts_Tests.h and Pokemon_Transitions_Tests.h; each suite includes only its own header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…reen-state Interrupts, Transitions, SceneJudges, FadeProfiles and WarpJudge are only used by the tile games; they come back in the 2D core PR (nav/grid-screen-state). The warp-seen and talk-over cases leave Pokemon_ScreenEvents_Tests with them and return there as Pokemon_SceneJudges_Tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
# Conflicts: # SerialPrograms/Source/Pokemon/AutoStory/Pokemon_AutoStory_Tests.h
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…t retry loop (#391) # Conflicts: # SerialPrograms/Source/Pokemon/AutoStory/Pokemon_AutoStory_Tests.h
The screen events (what the screen shows, a watcher that logs them in order, fades) are not AutoStory- or Pokemon-specific, so they move to CommonTools/ScreenEvents in namespace PokemonAutomation. The one Switch-specific piece, SwitchHomeObserver, moves next to HomeMenuDetector in NintendoSwitch/Inference/NintendoSwitch_HomeScreenObserver. Each header includes only what it uses; describe(ScreenEvents) moves to ScreenEventTypes.h next to the type. The tests register from the unit test runner instead of add_tests_AutoStory(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…oves to the segment report Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…onformance run Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…st-case) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine
left a comment
There was a problem hiding this comment.
Reviewed SHA: e699019
Verdict: blocked by 2 findings
Verdict
NEEDS_REWORK
Scope: this PR's own diff (origin/nav/story-engine...e6990190f, 11 files, +792), infra bar. The tip has since moved to 929298e, a base merge only (.github/workflows/story-data.yml, devtools/story-data-tests/README.md); the PR's own diff is unchanged, so this review holds for it. Out of scope as briefed: the Story data codec CI failure (#312), AutoStory *_Tests.cpp code health (#312), missing CommandLineTests images. Line endings: all added lines are CRLF. Code health on this diff: 1 failure (finding 1 below), the rest advisory.
Blocking
SerialPrograms/Source/CommonTools/ScreenEvents/ScreenEvents_Tests.cpp:135-148: the test cases don't follow the name / input / expectation shape. The name sits insideScreenEventsInput,ScreenEventsExpectationis empty, and every check is hidden inside astd::functionbody. This is the one code-health failure the PR adds (test-caseat line 148). Make the table data: for example, rows of (name, input event sets / pixel colours, expected describe() text / WarpFade / stop pattern), one test class per kind, and the name outside the input.- Comments about lab runs and PRs, in infra headers:
CommonTools/ScreenEvents/ScreenFades.h:24-28("upstream PokemonAutomation#1717", "BDSP's Oreburgh Gate, mean 46, lab-B 2026-11-08", "Check against the Gate's frames on the console before tightening") andNintendoSwitch/Inference/NintendoSwitch_HomeScreenObserver.h:22-25("BDSP, lab-B 2026-11-09"). Keep the why ("a dim, textured cave floor is not a fade"; "a game's detectors may read HOME's plain screen as a quiet overworld"). Move the dates, console names and PR numbers to the PR description or commit message.
Should-fix
CommonTools/ScreenEvents/ScreenEventTypes.h:37:SCREEN_EVENT_COUNT = 11is a hand-kept count with nothing tying it to the enum. A 12th event would silently be masked off byfrom_bits()(ScreenEvents.cpp:60), left out ofdescribe()(:86), and never logged byScreenEventLog::record()(:117). Addstatic_assert((size_t)ScreenEvent::SWITCH_HOME_MENU + 1 == SCREEN_EVENT_COUNT);next to the enum, or derive the count from a last enumerator.NintendoSwitch/Inference/NintendoSwitch_HomeScreenObserver.cpp:27-29:make_overlays()draws only the game's boxes, notm_home's, so the overlay doesn't show the HOME check that runs on every frame.HomeMenuDetector::make_overlays()exists, so callm_home.make_overlays(items);as well.CommonTools/ScreenEvents/ScreenEvents.cpp:115-150:ScreenEventLoghas no test. Nothing checks the rose/fell edges, the first frame's rises, the capacity drop and itsdropped()count, or thedescribe(log)text.test_watchernever readswatcher.log(). These are the parts the order-reading judges depend on. Add table cases for them (a small capacity makes the drop easy to hit).CommonTools/ScreenEvents/ScreenEventTypes.h:34(SWITCH_HOME_MENU), withBATTLE_STARTED/BATTLE_ENDED(:30-31) andWarpFade(:43, "a door's black, a cave mouth"): the PR description says CommonTools gets "nothing story- or Pokemon-specific", but the shared enum names a Nintendo Switch screen and Pokemon/overworld ideas, and CommonTools is console-agnostic. Either reword the description and the header comment to say this is the vocabulary for Switch overworld games, or keep the console screen out of the shared enum. For example, make the HOME wrapper return a generic "game not on screen" event, or return HOME as a separate result fromSwitchHomeObserver.
Nits
ScreenEventLog.h,ScreenEventTypes.handScreenFades.hare all implemented inScreenEvents.cpp. One.cppper header (ScreenEventLog.cpp,ScreenFades.cpp, ...) makes each one easier to find.NintendoSwitch_HomeScreenObserver.h:26: the file is HomeScreenObserver but the class isSwitchHomeObserver. Use one name for both.ScreenEvents.cpp:105: the nested ternary (sum <= ... ? BLACK : sum >= ... ? WHITE : NONE) reads better as twoifs, likefade_in()just above it.ScreenEventTypes.h:43:enum class WarpFadehas no: uint8_t, whileScreenEventandScreenEventEdgedo.
Checked and fine: ScreenEventWatcher locking (atomics for seen/stopped_on, mutex for the log, log() returns a copy). VisualInferenceCallback never passes a null snapshot (its base implementation dereferences it), and SwitchHomeObserver guards anyway. {:+} on the millisecond count. from_bits masking. The registration in ComputerPrograms_UnitTestRunner.cpp and SourceFiles.cmake. Nothing in this PR calls ScreenEventObserver yet (the description says AutoStory and PLZA build on it); that is fine for a stacked fork PR but worth a line in any upstream handoff.
…eview) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
… on #312, ScreenEvents case names Conflicts: ScreenEvents_Tests.cpp takes #313's case struct (name outside the input); the story-data README keeps this branch's reader-only text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
… step Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…and #313 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
|
Review follow-up: ee797a4 puts the case name outside its input (code health now 0), a44ab05 takes the lab runs and PR numbers out of the ScreenFades/HomeScreenObserver comments. Should-fix 3-6 (SCREEN_EVENT_COUNT assert, HOME overlay, a ScreenEventLog test, the Pokémon events in the shared enum) are left for follow-ups. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…merged, review fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…merged, review fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…merged, review fixes Conflict: the engine-runs test table takes the engine's case shape (name outside the input); the breakpoint rows are written in it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…rged, review fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Depends on #312.
Autonomous programs need to know what the screen is doing and since when: a fade, a dialogue box, a battle starting, the screen going still. This adds that vocabulary to CommonTools/ScreenEvents (the event set, a run_until watcher that logs every rise and fall in order, and fade reading), with nothing story- or Pokémon-specific in it, and puts the one Switch-specific piece,
SwitchHomeObserver(a HOME menu check in front of a game's observer), in NintendoSwitch/Inference next to HomeMenuDetector. AutoStory and PLZA build on it. Tested with the CommonTools::ScreenEvents unit tests.🤖 Generated with Claude Code
https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss