Skip to content

Lab mGBA slice 1: MgbaLink connection and video source - #493

Merged
dayfine merged 6 commits into
lab/toolingfrom
lab/mgba-slice1
Nov 15, 2026
Merged

dayfine merged 6 commits into
lab/toolingfrom
lab/mgba-slice1

Conversation

@dayfine

@dayfine dayfine commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Base: lab/tooling. The diff shows only this change. Slice 1 of docs/design/Lab-mGBA-Console.md (agents branch); protocol and bridge are agents #491.

The lab app had no way to see an mGBA instance. This adds MgbaLink::Connection (Qt-free MgbaLink v1 client over ClientSocket: HELLO, FRAME, KEYS with LATE, BYE on close, one command at a time, a non-blocking abort()) and VideoSource_MgbaLink, which polls FRAME, composes the 240x160 picture nearest-neighbour into GAME_BOX of a 1920x1080 frame (x=180..1740, y=5..1045) and stamps each frame from its frame number, never later than the time it was read and never backwards. In Developer mode it shows up as "mGBA (127.0.0.1:47801)" in the video source list (loopback hosts only). The source drops its frame when the bridge goes away and reports frames to the FPS readout and stream history (it reconnects itself every 2 s, so it opts out of the session no-video watchdog); destroying it aborts the connection instead of waiting out a reply timeout.

Tested with the unit suites Controllers::MgbaLink (line framing, replies, frame stamps, sessions against a fake bridge incl. busy, single-use open and abort) and CommonFramework::VideoSource_MgbaLink (compose geometry, address), run locally on macOS; draft CI skips them, so the Ubuntu and Windows default workflows are dispatched on the branch.

MgbaLink_Connection speaks MgbaLink v1 to bridge.lua over a ClientSocket
(HELLO, FRAME, KEYS with LATE, BYE on close). VideoSource_MgbaLink polls
FRAME, composes the 240x160 picture nearest-neighbour into GAME_BOX of a
1920x1080 frame and stamps it from the frame number. Developer mode lists
it as a video source. Unit tests: framing, replies, LATE, frame clock,
compose geometry, and a session against a fake bridge.

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: 06d50dd

Verdict

NEEDS_REWORK

Scope: Controllers/MgbaLink/*, VideoSource_MgbaLink, and the small edits to VideoSourceOption.cpp, VideoSourceSelectorWidget.cpp, VideoSourceDescriptor.h, the test runner and SourceFiles.cmake. Held to the infra bar. Not counted as findings (already noted): the fps constant living in MgbaLink_Protocol.h, and the PROTOCOL.md wording on ERR busy and reply timeouts.

The basics hold up. The code is Qt-free in Controllers/, uses no std::thread, std::mutex or std::filesystem, runs its threads from GlobalThreadPools, and uses the Mutex/ConditionVariable wrappers. The socket lifetime is right: m_socket is the last member, remove_listener blocks on any callback still running, and ClientSocket::close() is the non-blocking variant, so calling it from on_receive_data is safe. Code health: 0 failures (advisories listed under Low).

CI: draft PR jobs were all skipped. A dispatched Ubuntu Default run at this SHA built the code and passed every Controllers::MgbaLink and CommonFramework::VideoSource_MgbaLink case. It failed on one unrelated case, the fork-only image PokemonFRLG::WildEncounterReader - dark_eng_chansey.png (FileException). Windows (WinSocket fake-bridge path, MSVC /W4 /WX) and Mac have not run.

High

H1. The frame clock stamps frames in the future and jumps backwards (VideoSource_MgbaLink.cpp:168-181, FrameClock::resync). You asked whether the re-anchor is sound. It is sound when mGBA falls behind (paused or slow): stamps lag at most 500 ms and then jump forward, and they stay monotonic. It is not sound when mGBA runs ahead:

  • mGBA synced to a 60 Hz display runs about 0.46% faster than 59.7275 fps. Its stamps creep up to 500 ms past the time the app read the frame, about 108 s after each anchor. Then every re-anchor moves the stamp back by about 500 ms. With fast-forward this happens over and over.
  • A frame stamped later than the moment the app read it claims to come after events it actually predates, for example a button press issued after the read. Back-steps also break consumers that diff timestamps. VisualInferencePivot keys on last_timestamp, and stability and black-screen watchers measure durations between frames. StreamHistoryTracker trims by timestamp.

Fix: the read time now is a hard upper bound for when the frame happened, so never stamp later than it. Use stamp = min(clock.timestamp(frame), now), or re-anchor whenever the stamp is ahead of now and keep the 500 ms tolerance only for lag. Also keep stamps monotonic: no stamp earlier than the previous one. Add an "mGBA running ahead" case to the resync table. Today it only covers "small drift" and "paused".

H2. A dead or frozen bridge looks like live video (VideoSource_MgbaLink.cpp:129-136, poll_loop).

  • snapshot_recent_nonblocking(min_time) ignores min_time and always returns m_last. VideoFeed.h says to return a null snapshot when nothing is >= min_time.
  • m_last is never cleared when the connection drops or the reconnect loop is running. Inference keeps processing the last frame indefinitely.
  • The source never calls report_source_frame(), and it is built with allow_watchdog_reset = false. So the VideoSession FPS readout, StreamHistorySession (error-report video) and the no-video watchdog never see this source. The only sign of a dead bridge is the orange log line.

Fix: honour min_time, clear m_last (or mark it stale) on disconnect, and report each new frame through report_source_frame() so FPS and stream history work.

Medium

M1. Destroying the source can block the UI for about 10 s (~VideoSource_MgbaLink). The destructor sets m_stopping and joins the poller. The poller may be inside connection.open() (connect timeout 5 s, then HELLO 5 s) or frame() (5 s), and then ~MgbaLink_Connection sends BYE with another 5 s timeout. VideoSession::set_source/reset destroy the source on the caller's thread, so switching away from an unresponsive bridge freezes the UI. Fix: give the connection a non-blocking abort() that sets m_disconnect_error, notifies m_cv and calls m_socket.close(). Have the destructor call it on the active connection, which it holds under m_lock, and skip BYE when stopping.

M2. open() can't be called twice (MgbaLink_Connection.cpp:52-64). m_connect_error is never reset before m_socket.connect(). A second open() on the same object sees the old "", treats the connection as made before the socket connects, and then fails with "Unable to send HELLO". m_replies from the old session also survives. No caller does this today: poll_loop builds a fresh connection each time. Either reset both under m_lock before connect(), or say in the header that open() is called once per object.

M3 (question). The video source takes the bridge's only client slot. The header says the bridge "holds one client at a time", and VideoSource_MgbaLink "polls FRAME on its own connection". It also reconnects every 2 s, so it grabs the slot back as soon as any other client leaves. How will slice 2's controller (KEYS/GRANT) connect? If the plan is one shared connection, say so in the header so slice 2 doesn't stack a second client on this one.

M4 (question). Does the bridge answer FRAME while mGBA is paused? If its Lua socket handling runs from a frame callback, a paused mGBA never replies. Then FRAME times out every 5 s, the loop reconnects, and the "mGBA paused re-anchors" path never runs in practice. Please confirm against bridge.lua. The answer changes how much H1's lag path matters.

Low / nits

  • FrameReply::frame/width/height have no default member initializers (AGENTS.md: "Initialize every scalar"). width/height are parsed but never checked against the PNG that is loaded.
  • std::format advisories: 17 in MgbaLink_Connection.cpp and the tests, for example fail("Unable to connect to " + m_host + ":" + std::to_string(m_port) + ": " + error).
  • Test failure messages don't say what was expected and what was read. Test_Resync returns only resynced={}. Test_Session returns only the count and the last result. Test_Address doesn't print the parsed result.
  • When no bridge is running, poll_loop logs an orange line every 2 s indefinitely. Log the first failure and then only on state changes.
  • Developer-mode gating is only in the selector list. VideoSourceOption::load_json restores a saved MgbaLink selection outside developer mode too.
  • compose_mgba_frame doesn't check that box fits inside canvas. A wrong MgbaComposeGeometry writes out of bounds in pixel(). Throw InternalProgramError on a bad geometry.
  • The FRAME path comes from the peer and is opened as given. For a non-loopback host, the path refers to a file on the other machine. Consider refusing non-loopback hosts in slice 1.
  • MgbaLink::MgbaLink_Connection repeats the namespace in the class name. MgbaLink::Connection reads better (optional).
  • PR description says "CI runs them". Draft CI skipped them, and only the dispatched Ubuntu run did.

- FrameClock::stamp(): never stamps a frame later than its read time
  (re-anchors when mGBA runs ahead), keeps stamps monotonic; only lag
  re-anchors are logged. Table cases for running ahead and a frame going
  back.
- VideoSource_MgbaLink: snapshot_recent_nonblocking() honours min_time;
  the last frame is dropped when the connection ends; each new frame goes
  through report_source_frame() and the widget reports rendered frames;
  the no-video watchdog is no longer opted out.
- Connection::abort(): wakes open() or a pending command at once and skips
  BYE; the source's destructor aborts the active connection so it never
  waits out a 5 s timeout. open() is single-use (InternalProgramError).
- Lows: FrameReply defaults, PNG size checked against FRAME, std::format,
  test messages print expected vs read, repeated retry errors logged once,
  saved mGBA selection ignored outside developer mode, box-in-canvas check,
  loopback hosts only, MgbaLink_Connection -> MgbaLink::Connection.

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: bd6f9f7

Verdict

NEEDS_REWORK

This re-review covers the fix commit bd6f9f7 against my review of 06d50dd. All earlier findings are fixed or answered. The one item left is new. It comes from following my H2 note about the watchdog, so part of it is on me. It is a one-line fix.

CI: Windows Default (MSVC /W4 /WX) and Ubuntu Default ran at this SHA. Both built the code, and all 38 Controllers::MgbaLink and 24 CommonFramework::VideoSource_MgbaLink cases passed, including the new stamp, abort and single-use cases. Each run failed only on PokemonFRLG::WildEncounterReader - dark_eng_chansey.png (FileException). That failure comes from the base and is not counted here. Mac has not run. Code health: 0 failures. The remaining advisories are function length, the public-method count, and run_session's parameter order in the tests.

Earlier findings

  • H1 (clock ahead): fixed. FrameClock::stamp re-anchors when time > now and clamps to the previous stamp, so stamps are never later than the read time and never go backwards. After a lag re-anchor, time = now, which is at or after every earlier stamp, so the clamp only matters for a frame number that goes back. The table now has a case for mGBA running ahead and one for a frame number going back.
  • H2 (dead bridge looks live): fixed. snapshot_recent_nonblocking honours min_time. run_connection clears m_last on both exit paths. publish_frame calls report_source_frame, and the widget reports rendered frames. The QImage deleter keeps the snapshot's pixels alive, which is correct. Watchdog opt-in: see M5.
  • M1 (destructor blocks): fixed. The destructor sets m_stopping and aborts m_connection under one m_lock. run_connection checks m_stopping and publishes the pointer under that same lock, so an abort can't be missed. The pointer is cleared before the Connection is destroyed, on both paths. abort() doesn't hold the connection's lock while it calls m_socket.close(), so there is no lock-order problem with wait_for_reply. The base VideoSource sanitizer outlives the poller join.
  • M2 (open twice): fixed. m_opened makes a second call throw InternalProgramError, and a test covers it.
  • M3 (client slot): answered. The header now says slice 2 shares this connection.
  • M4 (FRAME while paused): still unanswered. It matters less now, because M5 resets the source either way.
  • Lows: all fixed: defaults, PNG size check, std::format, expected-vs-read messages, log once, developer-mode gate in load_json, box-in-canvas check, loopback only, rename.

Medium

M5 (new). With the watchdog on, a paused mGBA or a missing bridge rebuilds the source every 5 s (VideoSource_MgbaLink.cpp:160, VideoSource(logger, true)). AUTO_RESET_SECONDS defaults to 5 on Mac and Windows. The watchdog only resets when no new frame has been reported. That happens in three cases:

  • mGBA is paused and the bridge answers FRAME: the frame number doesn't change, so poll_connected reaches continue and reports nothing.
  • mGBA is paused and the bridge doesn't answer: FRAME times out after 5 s.
  • No bridge is running.

In each case, every 5 s VideoSession::on_watchdog_timeout logs a red "No video detected for 5 seconds..." line and calls reset(). That destroys the source and builds a new one, which also:

  • blanks the video on each reset;
  • drops the connection (and in slice 2, the shared controller connection with it);
  • creates a new RepeatSuppressingLogger each time, so the "polling ..." line and the connect failure come back every 5 s, which undoes the log-once fix.

The source already reconnects itself every 2 s, so a session reset adds nothing. Fix: pass allow_watchdog_reset = false again. FPS readout and stream history still work through report_source_frame. Update the PR description, which says frames go to "the no-video watchdog". If you want to keep the watchdog, re-report the last frame stamped now while the frame number doesn't change, which is what a capture card does on a still screen. Then also stop poll_connected from skipping the report on a repeated frame.

Low / nits

  • A small step back in frame number freezes stamps (FrameClock::stamp). After a LOAD a few frames back (under max_lag), new frames clamp to the last stamp until their mapped time passes it. For about the size of the step back, several different frames carry the same timestamp. Consumers that key on timestamps, such as VisualInferencePivot's last_timestamp, may treat them as one frame. This is harmless in slice 1, since nothing sends LOAD yet. One option for slice 2: re-anchor at (now, frame) whenever the frame number goes back. A larger step back is logged as "mGBA paused or slow", which is wrong for a LOAD.
  • When mGBA runs faster than 59.7275 fps, every frame re-anchors. That is the case when it syncs to a 60 Hz display. Each stamp is then the read time, and the frame-number clock adds nothing. That is correct, but the header says frames are "stamped with the time its frame number maps to". Consider saying that this holds only while mGBA is not ahead.
  • A FRAME size mismatch is thrown with &m_logger, not the repeat-suppressing logger. A bridge that keeps sending a wrong size logs it every 2 s.

…e once

A paused mGBA or a missing bridge reports no new frames, so the session
watchdog rebuilt the source every 5 s. The source reconnects itself every
2 s, so it passes allow_watchdog_reset = false. The FRAME size mismatch now
goes through the repeat-suppressing logger, and the header says frames are
stamped from their frame number only while mGBA is not running ahead.
@dayfine

dayfine commented Nov 15, 2026

Copy link
Copy Markdown
Owner Author

Fixed in 1d3d6a9.

  • M5: VideoSource_MgbaLink passes allow_watchdog_reset = false again; the source reconnects itself every 2 s. FPS readout and stream history still go through report_source_frame. The PR description no longer mentions the watchdog, and the header says why the source opts out.
  • Low (fps above 59.7275): the header now says frames are stamped from their frame number only while mGBA is not running ahead, and with the read time when it is.
  • Low (size mismatch): poll_connected takes the repeat-suppressing logger and throws the FRAME size mismatch with it, so a bridge that keeps sending a wrong size logs it once.
  • Low (step back freezes stamps): left for slice 2, where LOAD arrives.

Code health: 0 failures. Ubuntu Default and Windows Default dispatched at 1d3d6a9.

# Conflicts:
#	SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.cpp
@dayfine
dayfine marked this pull request as ready for review November 15, 2026 00:56
@dayfine
dayfine merged commit 5101913 into lab/tooling Nov 15, 2026
2 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant