Skip to content

AutoStory: replay recorded console frames through the real watchers - #306

Draft
dayfine wants to merge 159 commits into
nav/grid-story-stepsfrom
autostory/frame-replay
Draft

dayfine wants to merge 159 commits into
nav/grid-story-stepsfrom
autostory/frame-replay

Conversation

@dayfine

@dayfine dayfine commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Depends on #362 (which replaced #122) (base nav/grid-story-steps); the diff shows only this change.

Replays recorded console frames through the real watchers and screen-event observers on a virtual clock, so a console failure can become a deterministic test.

  • Pokemon_FrameReplay.h/.cpp: ReplayFrameSource serves a recording (a folder of <ms>.png, the captured-transitions convention) by virtual time; replay_until() feeds a window of it to VisualInferenceCallbacks as wait_until would, and replay_events() turns it into a ScreenEventLog for judge_warp() / classify_interrupt(). One thread, no real time.
  • Recordings stay out of the repo: tests read CommandLineTests/<game>/FrameReplay/<case>/ and skip when it is missing. The header gives the ffmpeg line for cutting an error report's Video.mp4 to that form.
  • Synthetic tests include the late-watcher case: a BlackScreenOverWatcher started after a door's black never fires, which is one way a walk reports "no fade" after a real warp.
  • make_UNIT_TESTS_ALL() loses a blank line to stay at the 50-line limit.

Testing

Pokemon:: suites (FrameReplay, Interrupts, ScreenEvents, Transitions, ...) in the headless SerialProgramsTests, in a scratch merge with upstream/headless-tests: 208 passed, 0 failed, 2 skipped (captured transitions, no images locally). Code health: 0 failures.

🤖 Generated with Claude Code

https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

dayfine and others added 13 commits October 29, 2026 13:56
Games open the Switch's own keyboard for names (e.g. BDSP's player name).
keyboard_enter_code() types on it blind from an assumed starting key, so it
can't recover from a press the keyboard ignored while sliding in, or start
from wherever the keyboard left its cursor.

SystemKeyboardDetector recognizes the keyboard (OK blue, or grey while the
field is empty), find_system_keyboard_cursor() reads the key under the cursor,
and system_keyboard_field_has_text() says whether the field is empty. Unit
tests cover all three on six reference screenshots (CommandLineTests,
NintendoSwitch/SystemKeyboardDetector).

type_on_system_keyboard() in FastCodeEntry reads the cursor before each key
and moves it with keyboard_get_path(), now declared in the header, over
KEYBOARD_POSITIONS, so it shares the key grid and the wrap-around moves with
blind code entry and also supports AZERTY.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
A frame source serves a recording's frames ("<ms>.png") by virtual time,
and replay_until / replay_events feed them to VisualInferenceCallbacks
or a ScreenEventObserver at the inference period on one thread, so a
console failure can become a deterministic test of the real detectors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…stration function at 50 lines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine and others added 4 commits November 9, 2026 11:02
The late-watcher case showed a fade missed by a watch that starts after
the hold; it now replays the walk's watch (WatchAlong over the hold, then
the same watcher on) and checks it sees a door's black that is over before
a 2000 ms hold ends, while a watcher started at the end of the hold still
misses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
@dayfine
dayfine changed the base branch from nav/story-steps to nav/grid-story-steps November 11, 2026 07:33
dayfine and others added 11 commits November 11, 2026 07:01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Move the game-agnostic part of SV's checkpoint_reattempt_loop and
checkpoint_save into Pokemon/AutoStory/Pokemon_CheckpointRetry: the retry
policy, reset counter and StoryLog lines, with the game's run, report,
reset and give-up as callables. SV keeps its saves and messages and its
historical counts (102 attempts, report on the 11th failure).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
The tutorial and main story checkpoint dropdowns are now the shared
StoryRangeOptions, added under the same keys (START_CHECKPOINT_TUTORIAL,
END_CHECKPOINT_TUTORIAL, START_CHECKPOINT_MAINSTORY, END_CHECKPOINT_MAINSTORY)
over the same databases, so saved settings load unchanged; the end stays
unlocked while running. The start-before-end check is the range's. The
checkpoint loops log the shared attempt and reset lines, and the run logs
the shared checkpoint start line.

Rebased onto autostory/story-range-options (#325) and squashed: it
uses only the shared range options (#325) and log lines (#45), not #312.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…tory-options

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Conflicts in the test registration: the base now registers shared suites through add_tests_AutoStory() with one header per suite, so FrameReplay gets Pokemon_FrameReplay_Tests.h (as in #307) and registers there; SourceFiles.cmake keeps both lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
… for the sims

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine and others added 29 commits November 13, 2026 15:34
StoryRunCounts and record_story_counts() move here from #45, which has no
caller of them; SegmentReport::story_counts() is their first user. Their
tests move with them as the Pokemon::StoryRunCounts suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…moves here from #45

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

# Conflicts:
#	SerialPrograms/Source/Pokemon/AutoStory/Pokemon_StoryRunCounts_Tests.cpp
#	SerialPrograms/Source/Pokemon/AutoStory/Pokemon_StoryRunCounts_Tests.h
…o the segment report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

# Conflicts:
#	SerialPrograms/Source/Pokemon/AutoStory/Pokemon_AutoStory_Tests.h
#	SerialPrograms/cmake/SourceFiles.cmake
…oves to the segment report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…ve to CommonTools/ScreenEvents

Conflicts: the AutoStory test list and SourceFiles.cmake keep this
branch's suites; the screen event files and AutoStoryStats tests take
nav/screen-state's moves (CommonTools/ScreenEvents, StoryRunCounts).
Includes of the old Pokemon_ScreenEvents*.h paths follow the move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
ScreenObservation now takes ScreenEvent and ScreenEvents from
CommonTools/ScreenEvents/ScreenEventTypes.h instead of its own copies.
Renames: TEXT_BOX -> DIALOG_BOX, BATTLE -> BATTLE_STARTED, CAN_MOVE ->
PLAYER_CAN_MOVE, STILL -> SCREEN_STILL; ScreenEvents::add(e) ->
set(e, true). The shared describe() takes the separator the engine's
had, defaulting to ", ". screen_observation() keeps the same mapping:
the engine's events only, and a choice box also counts as a dialog box.
The FRLG observer and the interrupt/judge code drop the Pokemon::
prefix and include ScreenFades.h for fade_in()/fade_events().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Full lines go to the log; the video overlay shows each without its
leading "[Tag] ". Table tests in Pokemon_StoryLog_Tests.cpp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Screen events moved to CommonTools/ScreenEvents: FrameReplay includes
them from there. Pokemon_AutoStory_Tests.h: the engine's suites
(CheckpointRetry; AutoStoryStats tests are gone) plus FrameReplay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
The event line moves to the story engine that writes it, and the segment
reload line goes: a reload is a checkpoint reset, written by
checkpoint_reset_line with the attempt limit. The header says what a
segment, a checkpoint and an event are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

# Conflicts:
#	SerialPrograms/Source/VideoGames/PokemonSV/Programs/AutoStory/PokemonSV_AutoStoryTools.cpp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
event_log_line moves here from Pokemon_StoryLog, which keeps the
checkpoint lines the engine and SV share. The runner's other checkpoint
lines (failed at, done) go through checkpoint_log_line. A segment's
reload is a checkpoint reset: it writes checkpoint_reset_line with the
attempt limit, and the status sink shortens lines with story_short_line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

# Conflicts:
#	SerialPrograms/Source/Pokemon/AutoStory/Pokemon_StorySegments.cpp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…1879)

* HttpServer: add a minimal Qt-free HTTP/1.1 server on AbstractServerSocket

A minimal HTTP/1.1 server for a single local client that issues one
request at a time (e.g. an MCP client), built on the socket library
instead of Qt:

- One connection at a time (the server socket's model); requests are
  processed in order, so pipelined bytes simply wait in the buffer.
- The per-connection task does all parsing and writing; the socket's
  receive callbacks only append data and flip flags. The user handler
  may block on another thread (e.g. the program thread) and can poll
  HandlerContext::connection_alive() to bail out of long-running work;
  responses are discarded if the connection dies meanwhile.
- Content-Length bodies only (chunked -> 411), sizes capped
  (8 KB request line, 64 KB headers, 1 MiB body, 2 MiB buffer),
  "Expect: 100-continue" answered, strict request/header validation
  (400), responses always carry Content-Length.
- start()/stop() are fully synchronous from the caller's point of view
  once on_listen_finished has fired: stop() closes the connection,
  waits for the in-flight handler, and joins the socket's worker
  threads, so the server can be started again afterwards.

No TLS, no upgrades, no method routing or auth: the user handler owns
all of that. 10 loopback tests (raw ClientSocket clients) cover the
happy path, keep-alive, pipelining, Connection: close, 400/411/413,
100-continue, disconnect and stop() during a handler, and
second-client rejection.

Part of the groundwork for the in-app MCP server; stacks on the
ServerSocket factory alias PR.

* HttpServer: fix CI build failures

- Rename ParseState::ERROR to ParseState::ERROR_RESPONSE: ERROR is a
  macro on Windows (winerror.h), which broke the qualified name on
  MSVC and clang-cl (C2589).
- Keep the AsyncTask handle from ThreadPool::dispatch in the
  stop-during-handler test: dispatch is [[nodiscard]] and clang's
  -Wunused-result (CI's -Werror) rejects discarding it. Its destructor
  also waits for the trivial release task to finish.

* HttpServer: inherit from AbstractHttpServer; address review

Split the request/response types, the handler contract, and the
listener events out into AbstractHttpServer (stacked on the
extract-abstract-http-server PR); HttpServer now implements that
interface, with the types re-exported under its own name.

Fixes from the review:

- Deadlock in on_connect(): the connection task handle was moved into
  m_connection_task while holding m_lock, and destroying the old handle
  blocked on the old task, whose final cleanup takes m_lock. Task
  handles now live in m_tasks under a dedicated m_task_lock (the socket
  library's m_task_lock pattern): on_connect() dispatches outside
  m_lock and never waits on an old task, so a new client is no longer
  held up while the previous handler is still running.
- Deadlock in stop(): the socket was destroyed while holding m_lock,
  and ~ServerSocket joins the socket's worker threads, whose callbacks
  take m_lock. The socket is now moved out under m_lock and destroyed
  after releasing it.
- Busy-spin on partial requests: the wait predicate was
  !buffer.empty(), which is permanently true once a partial request is
  buffered. It now requires the buffer to have grown past what the
  parse just saw, plus the dead/stop flags.
- Chunked requests: 501 Not Implemented instead of 411 (RFC 9112).
- The 413 check at parse time was unreachable (an incomplete request
  is always smaller than the buffer cap); the 2 MiB buffer cap is now
  enforced when data is appended instead, closing the connection.
- end_connection() only closes the socket's connection while it still
  belongs to this connection, so a task noticing its own connection
  died late cannot close the next client's connection.
- Documented that start()/stop() are not thread-safe with each other.

Test: new case that a connection flooding past the buffer cap is
closed without a response while the server keeps listening; chunked
case now expects 501.

* HttpServer: close the reconnect write race from the review

The socket's connection thread sets the state back to LISTENING before
it reports the disconnect, so a fast reconnect can be accepted before
the old connection's on_disconnect callback runs. Two consequences:

- The old connection's task could then write its final response through
  socket->send() into the new client's socket (send() has no
  per-connection identity; it writes whatever the socket is connected
  to). on_connect() now marks the connection it replaces dead (and
  notifies it) before the new one takes over, so the old task's
  post-handler check discards its response.
- on_disconnect() marked m_connection dead, which in that window is the
  new client's connection: it would kill the new connection task. It
  now only does so while the socket is not already CONNECTED to the
  next client.

Test: a client disconnects while its handler is still blocked on a
gate; a second client connects (replacing the first in the server)
before the gate is released. The first handler's late response must be
discarded, and the second client's stream must contain only its own
response.

* HttpServer: address the second review round

Five items from the fresh review:

- [P1] stop() could hang waiting for a handler: the socket suppresses
  its disconnect callback for a close_connection() close, so a handler
  polling connection_alive() kept seeing true while shutdown waited
  for it. stop() now marks the active connection dead before it waits.
  The stop-during-handler test now polls connection_alive() after the
  gate, which would spin forever without the mark.
- [P1] A response could reach a different client after reconnection:
  every write now carries the connection's socket id
  (AbstractServerSocket::connection_id(), new), so the socket refuses
  a write whose connection was replaced. This closes the last of the
  'task outlives the connection' windows; the id check and the
  transport selection happen under the socket's lock.
- [P2] Expect: 100-continue stalled clients: the interim response was
  sent only after the complete request (including body) was parsed,
  which a client waiting for it before sending the body never sees.
  Parsing now reports a separate NEED_BODY state (headers complete and
  valid, body pending), and the 100 is answered at that point. The
  test now performs the real two-phase handshake.
- [P2] Complete headers bypassed the 64 KiB limit: the in-progress
  check only fires while the terminator is still missing. The
  completed header boundary is now checked against the limit too, with
  a test sending a ~75 KiB header section in one piece.
- [P2] The test client's receive buffer was a data race: 'received'
  and the error strings are now protected by the waiter's mutex, and
  assertions use synchronized snapshots (same for the server
  listener's listen error).

* Cleanup

---------

Co-authored-by: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com>
Co-authored-by: Alexander Yee <a-yee@u.northwestern.edu>
The SystemKeyboardDetector screenshots are on the fork's CommandLineTests
main, not upstream's, so the image tests could not load them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
# Conflicts:
#	SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.cpp
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants