Repository navigation
Conversation
Games open the Switch's own keyboard for names (e.g. BDSP's player name). keyboard_enter_code() types on it blind from an assumed starting key, so it can't recover from a press the keyboard ignored while sliding in, or start from wherever the keyboard left its cursor. SystemKeyboardDetector recognizes the keyboard (OK blue, or grey while the field is empty), find_system_keyboard_cursor() reads the key under the cursor, and system_keyboard_field_has_text() says whether the field is empty. Unit tests cover all three on six reference screenshots (CommandLineTests, NintendoSwitch/SystemKeyboardDetector). type_on_system_keyboard() in FastCodeEntry reads the cursor before each key and moves it with keyboard_get_path(), now declared in the header, over KEYBOARD_POSITIONS, so it shares the key grid and the wrap-around moves with blind code entry and also supports AZERTY. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
9c06250 to
d1b5bdf
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
…tem keyboard 01.00 starts on the HOME screen with PLZA's tile selected. Its first event, launch_from_home, launches the selected tile without moving the cursor (closing it first if it is running: X, then A on the close dialog), then waits for the language list. The old start closed the game through a helper that presses DOWN/UP/HOME, which put the cursor on the first tile and launched FireRed. The launch takes the first screen's detector, so it can move to the shared core for BDSP and FRLG. new_game_language only works the language list. The name screen waits for the Switch system keyboard (#62's detector) and types with type_on_system_keyboard(); the look picker is pressed blind and logged as not checked. Each intro screen, the confirm menu and the frame after "Ready!" are dumped to DebugDumps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine
left a comment
There was a problem hiding this comment.
Reviewed SHA: 6b5da81
Reviewed to the infra bar, as upstream would read it before the handoff. The typer and detector code at this tip is byte-identical to 97a09b4, the commit run on the console on 2026-11-13, so those results apply: PLZA passed in English x3, Spanish, German, Italian, Japanese and French x2; one French run failed "cursor did not reach 'u'"; Korean failed twice.
Verdict
NEEDS_REWORK
Blocking
-
The cursor can't be read on column 11 or row 4, and an unreadable cursor is never moved (
NintendoSwitch_SystemKeyboardDetector.cppfind_system_keyboard_cursor,NintendoSwitch_SystemKeyboardTyping.cppmove_cursor). The search area is x 0.02-0.86, y 0.44-0.82. That leaves out the backspace/Return/OK column and the globe/shift/ABC/#+=/accents/Space row. When a read fails,move_cursoronly waits, so after 10 tries it throws. The header says the typer "works wherever the keyboard left its cursor", and the keyboard does remember its cursor. So if the last use ended on OK (a person pressing A on OK), the next call fails. A wrap move can land there too: right from!goes onto OK, which spans rows 3-4.keyboard_get_pathalready handles column 11. Read column 11 (and row 4, or at least detect it), or after a failed read press up/left to bring the cursor back onto the grid. Add screenshots with the cursor on OK and on Space. -
The detector's white key-panel box crosses the row 0/row 1 key boundary (
m_key_panel(0.15, 0.555, 0.65, 0.020), checked withis_white, max stddev sum 10). With the cursor on any of2..0orw..p, part of its cyan border lies inside the box. I estimate about 2% cyan, which puts the red stddev near 25. I worked this out from the screenshot geometry and did not run it. Both positive test images have the cursor on1, which is outside the box, so the tests can't catch this. It would bite any caller that waits onSystemKeyboardWatcherwhen the keyboard opens a second time with the cursor remembered, for example a rival name typed after the player name. Move the box to a part of the panel no cursor border reaches, and add a test image with the cursor on e.g.t. -
The tests should use
ImageUnitTestand cover more than the default cursor position. The sibling detector tests (Test_CheckOnlineDetector) deriveImageUnitTest. That base class runs the colour-distortion augmentation, which is exactly what the absolute cyan thresholds (red < 150 && green > 200 && blue > 190) and the OK colour ratio need, and it setsm_memory.Test_SystemKeyboardDetectorderivesUnitTest, loads the image itself and leavesm_memoryunset. Both keyboard images also use the same cursor(0,0)from BDSP. Before upstream, add: the cursor mid-grid (item 2), the cursor on column 10, on OK and on Space (item 1), an AZERTY/French frame, and a PLZA frame, since PLZA is the game it was run on. -
Code health reports 13 failures. They are
namespace-closein all four new files, plusheader-commentonSystemKeyboardWatcher(h:41) andadd_tests_SystemKeyboardDetector(h:60). PR_WORKFLOW requires every finding fixed before a PR goes upstream, the advisorystring_concatones included (det.cpp:95,130-135, typing.cpp:70,91).
Should-fix
-
The French failure can't be diagnosed from what is logged. The run typed
Land then stopped on3. Nothing logs the cursor it read on each try or the presses it sent. The error message does not give the last position read or the target either. The likely cases call for different fixes: the keyboard ignored the presses, the reader picked up some other cyan, or a stale frame made the loop oscillate. Logread (r,c) -> target (r,c), pathon each try, and write the message ascursor at (0,2) '3', target (1,6) 'u' after 10 tries. Keep "cursor unreadable" separate from "cursor read but not moving". -
Typed characters and Shift are never checked. Only cursor moves are verified. A dropped
AorLCLICKtypes a wrong name with no error, and the code assumes Shift is off, so a keyboard that opens auto-capitalised or in caps lock inverts the case. Check that the field changed after eachA(for example, its text grew), or say plainly in the header that this is not covered. -
Check every character before the first press, and throw an internal error for a bad one. An unsupported character in the middle of the string throws after the first half is already typed. It is a caller bug, so it should not use
SEND_ERROR_REPORT;keyboard_enter_codeusesNO_ERROR_REPORT, andInternalProgramErrorwould fit too. -
Replace the output parameters and sentinels with a returned value.
find_system_keyboard_cursor(screen, int& row, int& col)returns a bool and writes twoint&. The typer then casts them touint8_t, and the test uses-1with a separatekeyboardbool to mean "no keyboard". Returnstd::optional<...>of a row/col struct, and give the test an optional expected cursor. -
system_keyboard_field_has_text()can misread a busy background as text. It treats stddev sum > 30 over the field as text, but the game shows through the dimmed field. A busy PLZA background could pass for text. The caller would then press B on an empty field and close the keyboard, which is the hazard the function's own comment warns about. Count bright (white) text pixels instead. Also, neither this function norSystemKeyboardDetector/Watcherhas a caller in this PR. Upstream will ask what uses them, so link the first caller (PLZA/BDSP name entry) or name it in the description. -
The typer decodes the action bits itself and hard-codes its timings.
((uint8_t)action & 0x80)repeats the existingis_wrap(action).& 0x7fcopies the enum's encoding, so put that in a helper inCodeEntryTools.h.MOVE_HOLD/WRAP_RELEASE/... and the unnamed100ms, 250msforLCLICKdon't use theCODEBOARD_ENTRY_*delays (wired/wireless, Switch 1/2). Either say why the feedback loop needs its own fixed values, or use the settings. Every screenshot is a Switch 1 capture, so the header should say Switch 2 is untested.
Korean: out of scope for the first upstream version
The Korean keyboard (~/BDSPLab/fails/plza-ko-keyboard.png) adds a candidate strip (< ... >) above the keys, which moves the whole grid down about 0.08 of the screen height. Its letter keys are Hangul jamo, so neither the detector boxes nor KEYBOARD_POSITIONS fit it. Typing a Latin name on it would also need the globe key to switch layouts first. The detector returning false is the safe way to fail. Keep it that way. Name the supported keyboards in the detector/typer header (Latin QWERTY/AZERTY; console-tested in EN/ES/DE/IT/FR/JA), and state that Korean, and probably Chinese (untested), are not supported. The caller's error should say so rather than time out without a reason.
API shape and diff hygiene
- Exposing
keyboard_get_path()is fine. It already had external linkage, and the declaration and its comment are accurate. In that header, use the full-path include style of its siblings ("NintendoSwitch/Programs/FastCodeEntry/NintendoSwitch_CodeEntryTools.h"). - Upstream needs the
ComputerPrograms_UnitTestRunner.cppchange. Upstream registers image tests in that file, as it does foradd_tests_CheckOnlineDetector. - Handoff: this branch is one commit plus 13 merges of fork
main, which is 39 commits ahead of upstreammain. Cut the handoff as one commit onupstream/main. The PR diff applies cleanly there (git apply --check). - The description is out of date: it says this version "hasn't been run on the console yet", but the console runs above used identical typer code.
- CRLF, no tabs, file headers, include guards and brace style are all fine.
Nits
#include <map>in the typer (it usesstd::mapdirectly).- Name the
LCLICKtimings like the other constants (SHIFT_HOLD/SHIFT_RELEASE). - Give the OK colour ratio
{0.172, 0.276, 0.552}and the grey range600, 690a short "measured on ..." comment.
Not checked: CI and a local build; no compile was needed for these claims.
…ecked typing Review of #62: - The cursor reader covers the right column (backspace, Return, OK) and the bottom row; the typer steps back onto the character grid from there. - The detector's key-panel check uses two strips (rows 0/1 and 2/3), so a cursor anywhere can only spoil one of them. - system_keyboard_field_has_text() reads the backspace key (dark with text), not the field over the game's background. - The typer checks every character before the first press (UserSetupError), logs the cursor read and the moves of each try, names the last cursor and the target in its error, and checks that each key press adds white pixels to the field. - Presses use the code entry hold for Switch 1; the read-back waits stay fixed. - read_system_keyboard_cursor() returns std::optional<SystemKeyboardKey>. - Tests derive ImageUnitTest, add six Legends: Z-A frames (en/fr/de/it/ja/es) and list the screens still to capture, which are added once their file exists. - Headers state the supported keyboards (Latin QWERTY/AZERTY), Korean and Chinese out of scope, Switch 2 untested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…sor frames The cursor border pulses between bright cyan and a darker teal (0, 185, 210); the reader only matched the bright phase, so a cursor on '/' read as nothing. Widen the colour test (red < 130, green > 160, blue > 170, green - red > 60), which still leaves out the OK button and ABC underline blue. Add tests on Legends: Z-A console frames with the cursor on T, /, !, 0, -, backspace, Return, OK, globe, the N-accent key and Space. AZERTY and Korean stay planned: the layout follows the system language, not the game's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…s (function length) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Conflict: NintendoSwitch_SystemKeyboardDetector.{h,cpp} were added on both
sides (lab copy vs core #62). Kept core's version; typing moved to
Programs/FastCodeEntry/NintendoSwitch_SystemKeyboardTyping.h. SourceFiles.cmake:
dropped the lab entry in the core library list, kept core's main-list entries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…_keyboard autostory/engine (core #62) moved type_on_system_keyboard out of SystemKeyboardDetector.h into FastCodeEntry/NintendoSwitch_SystemKeyboardTyping.h. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Conflicts: the system keyboard detector takes #62's version (its typing is FastCodeEntry::type_on_system_keyboard, which NewGame now calls); the unit test runner keeps both include sets; SourceFiles lists the RNG handoff once. RouteIO includes Pokemon_RouteLog.h, which no header brings in any more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
The SystemKeyboardDetector screenshots are on the fork's CommandLineTests main, not upstream's, so the image tests could not load them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…-keyboard-v2) into plza/newgame
…62's typer - Launch: no blind A presses after the user select; wait up to 60 s for the language list (about 15 s of logos on the console, 2026-11-15). - Language list: D-pad taps 600 ms apart. - Look picker: tilt left (150 ms) until a highlight reads, since the first tilt lands on a different portrait each launch; A only once the target reads. - Name: type_on_system_keyboard() (switch/system-keyboard), Simplified Chinese switched to letters first; + until Confirm Choices shows (Chinese takes one + to accept the text, one to submit). - Drop the blind name typer (PokemonLZA_AutoStoryNameKeys) and its tests; new_game_problem() checks the name against the typer's key map. - Revert this PR's CommandLineTests pin bump. - Tests on the 2026-11-15 frames: language list, look highlight (10), keyboard counter (10), the system keyboard reader (9), Confirm Choices (12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Base:
main, no parent PR; the diff shows only this change. First caller: Legends: Z-A AutoStory's name entry (plza/story-base,PokemonLZA_AutoStoryScenes.cpp). Upstream handoff branch:upstream/system-keyboard-v2(one commit on upstreammain); fixtures: dayfine/CommandLineTestsswitch/system-keyboard-v2.Games open the Switch's own keyboard for names, and
keyboard_enter_code()types on it blind from an assumed starting key, so it can't start from wherever the keyboard left its cursor or notice a press the keyboard ignored.SystemKeyboardDetector/Watcher: the key panel, the bar below it and the OK button; the panel check uses two strips so the cursor can't hide the keyboard.read_system_keyboard_cursor(): the key under the cursor asstd::optional<SystemKeyboardKey>, including the right column (backspace, Return, OK) and the bottom row.system_keyboard_field_has_text(): reads the backspace key (dark with text), so the game behind the field doesn't matter.FastCodeEntry::type_on_system_keyboard(): checks the whole string first (UserSetupError), then per key reads the cursor, moves it alongkeyboard_get_path()(now declared in its header), logs the read and the moves, and checks that the press added to the field. Presses use the Switch 1 code entry hold.Testing
NintendoSwitch::SystemKeyboardDetectorimage tests (ImageUnitTest, 23 screenshots: BDSP, Legends: Z-A in six languages, the cursor on 11 other keys incl. the right column, the bottom row and column 10, four look-alikes) pass with--augment-color-distortions. The cursor-on-/frame failed first: the border pulses to a darker teal, now matched. AZERTY and Korean tests run once their screenshot is in CommandLineTests (the layout follows the console's system language).🤖 Generated with Claude Code
https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss