Skip to content

CommonTools: screen events for autonomous programs (what the screen is doing, since when) - #313

Merged
dayfine merged 37 commits into
nav/sharedfrom
nav/screen-state
Nov 14, 2026
Merged

dayfine merged 37 commits into
nav/sharedfrom
nav/screen-state

Conversation

@dayfine

@dayfine dayfine commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Depends on #312.

Autonomous programs need to know what the screen is doing and since when: a fade, a dialogue box, a battle starting, the screen going still. This adds that vocabulary to CommonTools/ScreenEvents (the event set, a run_until watcher that logs every rise and fall in order, and fade reading), with nothing story- or Pokémon-specific in it, and puts the one Switch-specific piece, SwitchHomeObserver (a HOME menu check in front of a game's observer), in NintendoSwitch/Inference next to HomeMenuDetector. AutoStory and PLZA build on it. Tested with the CommonTools::ScreenEvents unit tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

Shared screen events and their timed log, the Switch HOME check in front
of a game's observer, the transition tracker with per-game fade profiles,
the scene judges and the interrupt classifier, taken as they are at
nav/story-steps fd603ca. None of it uses a story type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine and others added 9 commits November 9, 2026 20:24
…ta-free executor)

# Conflicts:
#	SerialPrograms/cmake/SourceFiles.cmake
…tion)

# Conflicts:
#	SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.cpp
…/screen-state

# Conflicts:
#	SerialPrograms/cmake/SourceFiles.cmake
…reen-state

Brings in #312's typed step args (StoryArgs gave way to StoryActionArg
there) and #45/#325's split of StoryLog, AutoStoryStats and the range
options. Conflicts were only the file and test registration lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
The newer code-health rules (header symbol counts, no char* / char[])
failed 19 times against nav/story-engine. This is the #313 part of
#122's 7d04d13, so the fix lands in the PR that owns these files:
declarations move unchanged into headers by concern, each included by
the header they came from (ScreenEvents: event types, fades, event log;
Interrupts: warp judge; Transitions: fade profiles; AutoStory_Tests: the
screen-state suites). The *_name() helpers return std::string and test
rows hold std::string. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 10, 2026
…ts) into nav/story-steps

#313 now carries its own part of 7d04d13's header splits, so those
files come in unchanged; ScreenStory_Tests.h keeps this branch's two
extra suites. Registration lists keep this branch's order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine and others added 6 commits November 10, 2026 14:06
add_tests_AutoStory() also registers the screen story suites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
String building uses std::format; add_fade_evidence, first_beat and
run_tracker take their output reference last. Pokemon_ScreenStory_Tests.h
is replaced by Pokemon_ScreenEvents_Tests.h, Pokemon_Interrupts_Tests.h
and Pokemon_Transitions_Tests.h; each suite includes only its own header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…reen-state

Interrupts, Transitions, SceneJudges, FadeProfiles and WarpJudge are only used
by the tile games; they come back in the 2D core PR (nav/grid-screen-state).
The warp-seen and talk-over cases leave Pokemon_ScreenEvents_Tests with them
and return there as Pokemon_SceneJudges_Tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine and others added 3 commits November 13, 2026 13:57
…t retry loop (#391)

# Conflicts:
#	SerialPrograms/Source/Pokemon/AutoStory/Pokemon_AutoStory_Tests.h
The screen events (what the screen shows, a watcher that logs them in
order, fades) are not AutoStory- or Pokemon-specific, so they move to
CommonTools/ScreenEvents in namespace PokemonAutomation. The one
Switch-specific piece, SwitchHomeObserver, moves next to HomeMenuDetector
in NintendoSwitch/Inference/NintendoSwitch_HomeScreenObserver.

Each header includes only what it uses; describe(ScreenEvents) moves to
ScreenEventTypes.h next to the type. The tests register from the unit
test runner instead of add_tests_AutoStory().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
@dayfine dayfine changed the title AutoStory: screen state and interrupt tracking (split from #122, layer 2 of 3) CommonTools: screen events for autonomous programs (what the screen is doing, since when) Nov 13, 2026
dayfine and others added 7 commits November 13, 2026 15:36
…oves to the segment report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…onformance run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: e699019
Verdict: blocked by 2 findings

Verdict

NEEDS_REWORK

Scope: this PR's own diff (origin/nav/story-engine...e6990190f, 11 files, +792), infra bar. The tip has since moved to 929298e, a base merge only (.github/workflows/story-data.yml, devtools/story-data-tests/README.md); the PR's own diff is unchanged, so this review holds for it. Out of scope as briefed: the Story data codec CI failure (#312), AutoStory *_Tests.cpp code health (#312), missing CommandLineTests images. Line endings: all added lines are CRLF. Code health on this diff: 1 failure (finding 1 below), the rest advisory.

Blocking

  1. SerialPrograms/Source/CommonTools/ScreenEvents/ScreenEvents_Tests.cpp:135-148: the test cases don't follow the name / input / expectation shape. The name sits inside ScreenEventsInput, ScreenEventsExpectation is empty, and every check is hidden inside a std::function body. This is the one code-health failure the PR adds (test-case at line 148). Make the table data: for example, rows of (name, input event sets / pixel colours, expected describe() text / WarpFade / stop pattern), one test class per kind, and the name outside the input.
  2. Comments about lab runs and PRs, in infra headers: CommonTools/ScreenEvents/ScreenFades.h:24-28 ("upstream PokemonAutomation#1717", "BDSP's Oreburgh Gate, mean 46, lab-B 2026-11-08", "Check against the Gate's frames on the console before tightening") and NintendoSwitch/Inference/NintendoSwitch_HomeScreenObserver.h:22-25 ("BDSP, lab-B 2026-11-09"). Keep the why ("a dim, textured cave floor is not a fade"; "a game's detectors may read HOME's plain screen as a quiet overworld"). Move the dates, console names and PR numbers to the PR description or commit message.

Should-fix

  1. CommonTools/ScreenEvents/ScreenEventTypes.h:37: SCREEN_EVENT_COUNT = 11 is a hand-kept count with nothing tying it to the enum. A 12th event would silently be masked off by from_bits() (ScreenEvents.cpp:60), left out of describe() (:86), and never logged by ScreenEventLog::record() (:117). Add static_assert((size_t)ScreenEvent::SWITCH_HOME_MENU + 1 == SCREEN_EVENT_COUNT); next to the enum, or derive the count from a last enumerator.
  2. NintendoSwitch/Inference/NintendoSwitch_HomeScreenObserver.cpp:27-29: make_overlays() draws only the game's boxes, not m_home's, so the overlay doesn't show the HOME check that runs on every frame. HomeMenuDetector::make_overlays() exists, so call m_home.make_overlays(items); as well.
  3. CommonTools/ScreenEvents/ScreenEvents.cpp:115-150: ScreenEventLog has no test. Nothing checks the rose/fell edges, the first frame's rises, the capacity drop and its dropped() count, or the describe(log) text. test_watcher never reads watcher.log(). These are the parts the order-reading judges depend on. Add table cases for them (a small capacity makes the drop easy to hit).
  4. CommonTools/ScreenEvents/ScreenEventTypes.h:34 (SWITCH_HOME_MENU), with BATTLE_STARTED / BATTLE_ENDED (:30-31) and WarpFade (:43, "a door's black, a cave mouth"): the PR description says CommonTools gets "nothing story- or Pokemon-specific", but the shared enum names a Nintendo Switch screen and Pokemon/overworld ideas, and CommonTools is console-agnostic. Either reword the description and the header comment to say this is the vocabulary for Switch overworld games, or keep the console screen out of the shared enum. For example, make the HOME wrapper return a generic "game not on screen" event, or return HOME as a separate result from SwitchHomeObserver.

Nits

  1. ScreenEventLog.h, ScreenEventTypes.h and ScreenFades.h are all implemented in ScreenEvents.cpp. One .cpp per header (ScreenEventLog.cpp, ScreenFades.cpp, ...) makes each one easier to find.
  2. NintendoSwitch_HomeScreenObserver.h:26: the file is HomeScreenObserver but the class is SwitchHomeObserver. Use one name for both.
  3. ScreenEvents.cpp:105: the nested ternary (sum <= ... ? BLACK : sum >= ... ? WHITE : NONE) reads better as two ifs, like fade_in() just above it.
  4. ScreenEventTypes.h:43: enum class WarpFade has no : uint8_t, while ScreenEvent and ScreenEventEdge do.

Checked and fine: ScreenEventWatcher locking (atomics for seen/stopped_on, mutex for the log, log() returns a copy). VisualInferenceCallback never passes a null snapshot (its base implementation dereferences it), and SwitchHomeObserver guards anyway. {:+} on the millisecond count. from_bits masking. The registration in ComputerPrograms_UnitTestRunner.cpp and SourceFiles.cmake. Nothing in this PR calls ScreenEventObserver yet (the description says AutoStory and PLZA build on it); that is fine for a stacked fork PR but worth a line in any upstream handoff.

dayfine added a commit that referenced this pull request Nov 14, 2026
… on #312, ScreenEvents case names

Conflicts: ScreenEvents_Tests.cpp takes #313's case struct (name outside the input); the story-data README keeps this branch's reader-only text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 14, 2026
@dayfine

dayfine commented Nov 14, 2026

Copy link
Copy Markdown
Owner Author

Review follow-up: ee797a4 puts the case name outside its input (code health now 0), a44ab05 takes the lab runs and PR numbers out of the ScreenFades/HomeScreenObserver comments. Should-fix 3-6 (SCREEN_EVENT_COUNT assert, HOME overlay, a ScreenEventLog test, the Pokémon events in the shared enum) are left for follow-ups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
@dayfine
dayfine changed the base branch from nav/story-engine to nav/shared November 14, 2026 21:56
@dayfine
dayfine merged commit 4f15782 into nav/shared Nov 14, 2026
13 checks passed
dayfine added a commit that referenced this pull request Nov 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 14, 2026
…merged, review fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 14, 2026
…merged, review fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 14, 2026
…merged, review fixes

Conflict: the engine-runs test table takes the engine's case shape (name outside the input); the breakpoint rows are written in it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 14, 2026
…rged, review fixes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 14, 2026
AutoStory: review fixes on the shared engine (#312, #313, #332, #474, #479)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant