Repository navigation
Lab mGBA slice 1: MgbaLink connection and video source - #493
Conversation
MgbaLink_Connection speaks MgbaLink v1 to bridge.lua over a ClientSocket (HELLO, FRAME, KEYS with LATE, BYE on close). VideoSource_MgbaLink polls FRAME, composes the 240x160 picture nearest-neighbour into GAME_BOX of a 1920x1080 frame and stamps it from the frame number. Developer mode lists it as a video source. Unit tests: framing, replies, LATE, frame clock, compose geometry, and a session against a fake bridge.
dayfine
left a comment
There was a problem hiding this comment.
Reviewed SHA: 06d50dd
Verdict
NEEDS_REWORK
Scope: Controllers/MgbaLink/*, VideoSource_MgbaLink, and the small edits to VideoSourceOption.cpp, VideoSourceSelectorWidget.cpp, VideoSourceDescriptor.h, the test runner and SourceFiles.cmake. Held to the infra bar. Not counted as findings (already noted): the fps constant living in MgbaLink_Protocol.h, and the PROTOCOL.md wording on ERR busy and reply timeouts.
The basics hold up. The code is Qt-free in Controllers/, uses no std::thread, std::mutex or std::filesystem, runs its threads from GlobalThreadPools, and uses the Mutex/ConditionVariable wrappers. The socket lifetime is right: m_socket is the last member, remove_listener blocks on any callback still running, and ClientSocket::close() is the non-blocking variant, so calling it from on_receive_data is safe. Code health: 0 failures (advisories listed under Low).
CI: draft PR jobs were all skipped. A dispatched Ubuntu Default run at this SHA built the code and passed every Controllers::MgbaLink and CommonFramework::VideoSource_MgbaLink case. It failed on one unrelated case, the fork-only image PokemonFRLG::WildEncounterReader - dark_eng_chansey.png (FileException). Windows (WinSocket fake-bridge path, MSVC /W4 /WX) and Mac have not run.
High
H1. The frame clock stamps frames in the future and jumps backwards (VideoSource_MgbaLink.cpp:168-181, FrameClock::resync). You asked whether the re-anchor is sound. It is sound when mGBA falls behind (paused or slow): stamps lag at most 500 ms and then jump forward, and they stay monotonic. It is not sound when mGBA runs ahead:
- mGBA synced to a 60 Hz display runs about 0.46% faster than 59.7275 fps. Its stamps creep up to 500 ms past the time the app read the frame, about 108 s after each anchor. Then every re-anchor moves the stamp back by about 500 ms. With fast-forward this happens over and over.
- A frame stamped later than the moment the app read it claims to come after events it actually predates, for example a button press issued after the read. Back-steps also break consumers that diff timestamps.
VisualInferencePivotkeys onlast_timestamp, and stability and black-screen watchers measure durations between frames.StreamHistoryTrackertrims by timestamp.
Fix: the read time now is a hard upper bound for when the frame happened, so never stamp later than it. Use stamp = min(clock.timestamp(frame), now), or re-anchor whenever the stamp is ahead of now and keep the 500 ms tolerance only for lag. Also keep stamps monotonic: no stamp earlier than the previous one. Add an "mGBA running ahead" case to the resync table. Today it only covers "small drift" and "paused".
H2. A dead or frozen bridge looks like live video (VideoSource_MgbaLink.cpp:129-136, poll_loop).
snapshot_recent_nonblocking(min_time)ignoresmin_timeand always returnsm_last.VideoFeed.hsays to return a null snapshot when nothing is>= min_time.m_lastis never cleared when the connection drops or the reconnect loop is running. Inference keeps processing the last frame indefinitely.- The source never calls
report_source_frame(), and it is built withallow_watchdog_reset = false. So the VideoSession FPS readout,StreamHistorySession(error-report video) and the no-video watchdog never see this source. The only sign of a dead bridge is the orange log line.
Fix: honour min_time, clear m_last (or mark it stale) on disconnect, and report each new frame through report_source_frame() so FPS and stream history work.
Medium
M1. Destroying the source can block the UI for about 10 s (~VideoSource_MgbaLink). The destructor sets m_stopping and joins the poller. The poller may be inside connection.open() (connect timeout 5 s, then HELLO 5 s) or frame() (5 s), and then ~MgbaLink_Connection sends BYE with another 5 s timeout. VideoSession::set_source/reset destroy the source on the caller's thread, so switching away from an unresponsive bridge freezes the UI. Fix: give the connection a non-blocking abort() that sets m_disconnect_error, notifies m_cv and calls m_socket.close(). Have the destructor call it on the active connection, which it holds under m_lock, and skip BYE when stopping.
M2. open() can't be called twice (MgbaLink_Connection.cpp:52-64). m_connect_error is never reset before m_socket.connect(). A second open() on the same object sees the old "", treats the connection as made before the socket connects, and then fails with "Unable to send HELLO". m_replies from the old session also survives. No caller does this today: poll_loop builds a fresh connection each time. Either reset both under m_lock before connect(), or say in the header that open() is called once per object.
M3 (question). The video source takes the bridge's only client slot. The header says the bridge "holds one client at a time", and VideoSource_MgbaLink "polls FRAME on its own connection". It also reconnects every 2 s, so it grabs the slot back as soon as any other client leaves. How will slice 2's controller (KEYS/GRANT) connect? If the plan is one shared connection, say so in the header so slice 2 doesn't stack a second client on this one.
M4 (question). Does the bridge answer FRAME while mGBA is paused? If its Lua socket handling runs from a frame callback, a paused mGBA never replies. Then FRAME times out every 5 s, the loop reconnects, and the "mGBA paused re-anchors" path never runs in practice. Please confirm against bridge.lua. The answer changes how much H1's lag path matters.
Low / nits
FrameReply::frame/width/heighthave no default member initializers (AGENTS.md: "Initialize every scalar").width/heightare parsed but never checked against the PNG that is loaded.std::formatadvisories: 17 inMgbaLink_Connection.cppand the tests, for examplefail("Unable to connect to " + m_host + ":" + std::to_string(m_port) + ": " + error).- Test failure messages don't say what was expected and what was read.
Test_Resyncreturns onlyresynced={}.Test_Sessionreturns only the count and the last result.Test_Addressdoesn't print the parsed result. - When no bridge is running,
poll_looplogs an orange line every 2 s indefinitely. Log the first failure and then only on state changes. - Developer-mode gating is only in the selector list.
VideoSourceOption::load_jsonrestores a saved MgbaLink selection outside developer mode too. compose_mgba_framedoesn't check thatboxfits insidecanvas. A wrongMgbaComposeGeometrywrites out of bounds inpixel(). ThrowInternalProgramErroron a bad geometry.- The FRAME path comes from the peer and is opened as given. For a non-loopback host, the path refers to a file on the other machine. Consider refusing non-loopback hosts in slice 1.
MgbaLink::MgbaLink_Connectionrepeats the namespace in the class name.MgbaLink::Connectionreads better (optional).- PR description says "CI runs them". Draft CI skipped them, and only the dispatched Ubuntu run did.
- FrameClock::stamp(): never stamps a frame later than its read time (re-anchors when mGBA runs ahead), keeps stamps monotonic; only lag re-anchors are logged. Table cases for running ahead and a frame going back. - VideoSource_MgbaLink: snapshot_recent_nonblocking() honours min_time; the last frame is dropped when the connection ends; each new frame goes through report_source_frame() and the widget reports rendered frames; the no-video watchdog is no longer opted out. - Connection::abort(): wakes open() or a pending command at once and skips BYE; the source's destructor aborts the active connection so it never waits out a 5 s timeout. open() is single-use (InternalProgramError). - Lows: FrameReply defaults, PNG size checked against FRAME, std::format, test messages print expected vs read, repeated retry errors logged once, saved mGBA selection ignored outside developer mode, box-in-canvas check, loopback hosts only, MgbaLink_Connection -> MgbaLink::Connection.
dayfine
left a comment
There was a problem hiding this comment.
Reviewed SHA: bd6f9f7
Verdict
NEEDS_REWORK
This re-review covers the fix commit bd6f9f7 against my review of 06d50dd. All earlier findings are fixed or answered. The one item left is new. It comes from following my H2 note about the watchdog, so part of it is on me. It is a one-line fix.
CI: Windows Default (MSVC /W4 /WX) and Ubuntu Default ran at this SHA. Both built the code, and all 38 Controllers::MgbaLink and 24 CommonFramework::VideoSource_MgbaLink cases passed, including the new stamp, abort and single-use cases. Each run failed only on PokemonFRLG::WildEncounterReader - dark_eng_chansey.png (FileException). That failure comes from the base and is not counted here. Mac has not run. Code health: 0 failures. The remaining advisories are function length, the public-method count, and run_session's parameter order in the tests.
Earlier findings
- H1 (clock ahead): fixed.
FrameClock::stampre-anchors whentime > nowand clamps to the previous stamp, so stamps are never later than the read time and never go backwards. After a lag re-anchor,time = now, which is at or after every earlier stamp, so the clamp only matters for a frame number that goes back. The table now has a case for mGBA running ahead and one for a frame number going back. - H2 (dead bridge looks live): fixed.
snapshot_recent_nonblockinghonoursmin_time.run_connectionclearsm_laston both exit paths.publish_framecallsreport_source_frame, and the widget reports rendered frames. TheQImagedeleter keeps the snapshot's pixels alive, which is correct. Watchdog opt-in: see M5. - M1 (destructor blocks): fixed. The destructor sets
m_stoppingand abortsm_connectionunder onem_lock.run_connectionchecksm_stoppingand publishes the pointer under that same lock, so an abort can't be missed. The pointer is cleared before theConnectionis destroyed, on both paths.abort()doesn't hold the connection's lock while it callsm_socket.close(), so there is no lock-order problem withwait_for_reply. The baseVideoSourcesanitizer outlives the poller join. - M2 (open twice): fixed.
m_openedmakes a second call throwInternalProgramError, and a test covers it. - M3 (client slot): answered. The header now says slice 2 shares this connection.
- M4 (FRAME while paused): still unanswered. It matters less now, because M5 resets the source either way.
- Lows: all fixed: defaults, PNG size check,
std::format, expected-vs-read messages, log once, developer-mode gate inload_json, box-in-canvas check, loopback only, rename.
Medium
M5 (new). With the watchdog on, a paused mGBA or a missing bridge rebuilds the source every 5 s (VideoSource_MgbaLink.cpp:160, VideoSource(logger, true)). AUTO_RESET_SECONDS defaults to 5 on Mac and Windows. The watchdog only resets when no new frame has been reported. That happens in three cases:
- mGBA is paused and the bridge answers FRAME: the frame number doesn't change, so
poll_connectedreachescontinueand reports nothing. - mGBA is paused and the bridge doesn't answer: FRAME times out after 5 s.
- No bridge is running.
In each case, every 5 s VideoSession::on_watchdog_timeout logs a red "No video detected for 5 seconds..." line and calls reset(). That destroys the source and builds a new one, which also:
- blanks the video on each reset;
- drops the connection (and in slice 2, the shared controller connection with it);
- creates a new
RepeatSuppressingLoggereach time, so the "polling ..." line and the connect failure come back every 5 s, which undoes the log-once fix.
The source already reconnects itself every 2 s, so a session reset adds nothing. Fix: pass allow_watchdog_reset = false again. FPS readout and stream history still work through report_source_frame. Update the PR description, which says frames go to "the no-video watchdog". If you want to keep the watchdog, re-report the last frame stamped now while the frame number doesn't change, which is what a capture card does on a still screen. Then also stop poll_connected from skipping the report on a repeated frame.
Low / nits
- A small step back in frame number freezes stamps (
FrameClock::stamp). After a LOAD a few frames back (undermax_lag), new frames clamp to the last stamp until their mapped time passes it. For about the size of the step back, several different frames carry the same timestamp. Consumers that key on timestamps, such asVisualInferencePivot'slast_timestamp, may treat them as one frame. This is harmless in slice 1, since nothing sends LOAD yet. One option for slice 2: re-anchor at(now, frame)whenever the frame number goes back. A larger step back is logged as "mGBA paused or slow", which is wrong for a LOAD. - When mGBA runs faster than 59.7275 fps, every frame re-anchors. That is the case when it syncs to a 60 Hz display. Each stamp is then the read time, and the frame-number clock adds nothing. That is correct, but the header says frames are "stamped with the time its frame number maps to". Consider saying that this holds only while mGBA is not ahead.
- A FRAME size mismatch is thrown with
&m_logger, not the repeat-suppressing logger. A bridge that keeps sending a wrong size logs it every 2 s.
…e once A paused mGBA or a missing bridge reports no new frames, so the session watchdog rebuilt the source every 5 s. The source reconnects itself every 2 s, so it passes allow_watchdog_reset = false. The FRAME size mismatch now goes through the repeat-suppressing logger, and the header says frames are stamped from their frame number only while mGBA is not running ahead.
|
Fixed in 1d3d6a9.
Code health: 0 failures. Ubuntu Default and Windows Default dispatched at 1d3d6a9. |
# Conflicts: # SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.cpp
Base:
lab/tooling. The diff shows only this change. Slice 1 of docs/design/Lab-mGBA-Console.md (agents branch); protocol and bridge are agents #491.The lab app had no way to see an mGBA instance. This adds
MgbaLink::Connection(Qt-free MgbaLink v1 client overClientSocket: HELLO, FRAME, KEYS with LATE, BYE on close, one command at a time, a non-blockingabort()) andVideoSource_MgbaLink, which polls FRAME, composes the 240x160 picture nearest-neighbour into GAME_BOX of a 1920x1080 frame (x=180..1740, y=5..1045) and stamps each frame from its frame number, never later than the time it was read and never backwards. In Developer mode it shows up as "mGBA (127.0.0.1:47801)" in the video source list (loopback hosts only). The source drops its frame when the bridge goes away and reports frames to the FPS readout and stream history (it reconnects itself every 2 s, so it opts out of the session no-video watchdog); destroying it aborts the connection instead of waiting out a reply timeout.Tested with the unit suites
Controllers::MgbaLink(line framing, replies, frame stamps, sessions against a fake bridge incl. busy, single-use open and abort) andCommonFramework::VideoSource_MgbaLink(compose geometry, address), run locally on macOS; draft CI skips them, so the Ubuntu and Windows default workflows are dispatched on the branch.