Skip to content

Switch: read the on-screen keyboard and type on it with feedback - #62

Draft
dayfine wants to merge 22 commits into
mainfrom
switch/system-keyboard
Draft

dayfine wants to merge 22 commits into
mainfrom
switch/system-keyboard

Conversation

@dayfine

@dayfine dayfine commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Base: main, no parent PR; the diff shows only this change. First caller: Legends: Z-A AutoStory's name entry (plza/story-base, PokemonLZA_AutoStoryScenes.cpp). Upstream handoff branch: upstream/system-keyboard-v2 (one commit on upstream main); fixtures: dayfine/CommandLineTests switch/system-keyboard-v2.

Games open the Switch's own keyboard for names, and keyboard_enter_code() types on it blind from an assumed starting key, so it can't start from wherever the keyboard left its cursor or notice a press the keyboard ignored.

  • SystemKeyboardDetector/Watcher: the key panel, the bar below it and the OK button; the panel check uses two strips so the cursor can't hide the keyboard.
  • read_system_keyboard_cursor(): the key under the cursor as std::optional<SystemKeyboardKey>, including the right column (backspace, Return, OK) and the bottom row.
  • system_keyboard_field_has_text(): reads the backspace key (dark with text), so the game behind the field doesn't matter.
  • FastCodeEntry::type_on_system_keyboard(): checks the whole string first (UserSetupError), then per key reads the cursor, moves it along keyboard_get_path() (now declared in its header), logs the read and the moves, and checks that the press added to the field. Presses use the Switch 1 code entry hold.
  • Supported: the Latin keyboard (QWERTY; AZERTY shares the grid but has no screenshot yet). Korean and Chinese are out of scope, Switch 2 is untested; the headers say so.

Testing

  • NintendoSwitch::SystemKeyboardDetector image tests (ImageUnitTest, 23 screenshots: BDSP, Legends: Z-A in six languages, the cursor on 11 other keys incl. the right column, the bottom row and column 10, four look-alikes) pass with --augment-color-distortions. The cursor-on-/ frame failed first: the border pulses to a darker teal, now matched. AZERTY and Korean tests run once their screenshot is in CommandLineTests (the layout follows the console's system language).
  • On console B (Switch 1, 2026-11-13), the typer at 97a09b4 entered the player's name in Legends: Z-A in English x3, Spanish, German, Italian, Japanese and French x2; one French run failed with "cursor did not reach 'u'", which the new per-try log would explain; Korean failed as expected. This revision changes the reader, the press hold and adds the field check, and has not been run on the console yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss

Games open the Switch's own keyboard for names (e.g. BDSP's player name).
keyboard_enter_code() types on it blind from an assumed starting key, so it
can't recover from a press the keyboard ignored while sliding in, or start
from wherever the keyboard left its cursor.

SystemKeyboardDetector recognizes the keyboard (OK blue, or grey while the
field is empty), find_system_keyboard_cursor() reads the key under the cursor,
and system_keyboard_field_has_text() says whether the field is empty. Unit
tests cover all three on six reference screenshots (CommandLineTests,
NintendoSwitch/SystemKeyboardDetector).

type_on_system_keyboard() in FastCodeEntry reads the cursor before each key
and moves it with keyboard_get_path(), now declared in the header, over
KEYBOARD_POSITIONS, so it shares the key grid and the wrap-around moves with
blind code entry and also supports AZERTY.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
@dayfine
dayfine force-pushed the switch/system-keyboard branch from 9c06250 to d1b5bdf Compare October 29, 2026 20:57
dayfine and others added 3 commits October 30, 2026 07:38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
@dayfine dayfine added detector Screen detectors and readers game:shared Shared or game-neutral code labels Nov 1, 2026
dayfine and others added 6 commits November 2, 2026 07:01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine and others added 4 commits November 9, 2026 07:01
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
dayfine added a commit that referenced this pull request Nov 13, 2026
…tem keyboard

01.00 starts on the HOME screen with PLZA's tile selected. Its first event,
launch_from_home, launches the selected tile without moving the cursor
(closing it first if it is running: X, then A on the close dialog), then
waits for the language list. The old start closed the game through a helper
that presses DOWN/UP/HOME, which put the cursor on the first tile and
launched FireRed. The launch takes the first screen's detector, so it can
move to the shared core for BDSP and FRLG.

new_game_language only works the language list. The name screen waits for
the Switch system keyboard (#62's detector) and types with
type_on_system_keyboard(); the look picker is pressed blind and logged as
not checked. Each intro screen, the confirm menu and the frame after
"Ready!" are dumped to DebugDumps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E

@dayfine dayfine left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed SHA: 6b5da81

Reviewed to the infra bar, as upstream would read it before the handoff. The typer and detector code at this tip is byte-identical to 97a09b4, the commit run on the console on 2026-11-13, so those results apply: PLZA passed in English x3, Spanish, German, Italian, Japanese and French x2; one French run failed "cursor did not reach 'u'"; Korean failed twice.

Verdict

NEEDS_REWORK

Blocking

  1. The cursor can't be read on column 11 or row 4, and an unreadable cursor is never moved (NintendoSwitch_SystemKeyboardDetector.cpp find_system_keyboard_cursor, NintendoSwitch_SystemKeyboardTyping.cpp move_cursor). The search area is x 0.02-0.86, y 0.44-0.82. That leaves out the backspace/Return/OK column and the globe/shift/ABC/#+=/accents/Space row. When a read fails, move_cursor only waits, so after 10 tries it throws. The header says the typer "works wherever the keyboard left its cursor", and the keyboard does remember its cursor. So if the last use ended on OK (a person pressing A on OK), the next call fails. A wrap move can land there too: right from ! goes onto OK, which spans rows 3-4. keyboard_get_path already handles column 11. Read column 11 (and row 4, or at least detect it), or after a failed read press up/left to bring the cursor back onto the grid. Add screenshots with the cursor on OK and on Space.

  2. The detector's white key-panel box crosses the row 0/row 1 key boundary (m_key_panel(0.15, 0.555, 0.65, 0.020), checked with is_white, max stddev sum 10). With the cursor on any of 2..0 or w..p, part of its cyan border lies inside the box. I estimate about 2% cyan, which puts the red stddev near 25. I worked this out from the screenshot geometry and did not run it. Both positive test images have the cursor on 1, which is outside the box, so the tests can't catch this. It would bite any caller that waits on SystemKeyboardWatcher when the keyboard opens a second time with the cursor remembered, for example a rival name typed after the player name. Move the box to a part of the panel no cursor border reaches, and add a test image with the cursor on e.g. t.

  3. The tests should use ImageUnitTest and cover more than the default cursor position. The sibling detector tests (Test_CheckOnlineDetector) derive ImageUnitTest. That base class runs the colour-distortion augmentation, which is exactly what the absolute cyan thresholds (red < 150 && green > 200 && blue > 190) and the OK colour ratio need, and it sets m_memory. Test_SystemKeyboardDetector derives UnitTest, loads the image itself and leaves m_memory unset. Both keyboard images also use the same cursor (0,0) from BDSP. Before upstream, add: the cursor mid-grid (item 2), the cursor on column 10, on OK and on Space (item 1), an AZERTY/French frame, and a PLZA frame, since PLZA is the game it was run on.

  4. Code health reports 13 failures. They are namespace-close in all four new files, plus header-comment on SystemKeyboardWatcher (h:41) and add_tests_SystemKeyboardDetector (h:60). PR_WORKFLOW requires every finding fixed before a PR goes upstream, the advisory string_concat ones included (det.cpp:95,130-135, typing.cpp:70,91).

Should-fix

  1. The French failure can't be diagnosed from what is logged. The run typed L and then stopped on 3. Nothing logs the cursor it read on each try or the presses it sent. The error message does not give the last position read or the target either. The likely cases call for different fixes: the keyboard ignored the presses, the reader picked up some other cyan, or a stale frame made the loop oscillate. Log read (r,c) -> target (r,c), path on each try, and write the message as cursor at (0,2) '3', target (1,6) 'u' after 10 tries. Keep "cursor unreadable" separate from "cursor read but not moving".

  2. Typed characters and Shift are never checked. Only cursor moves are verified. A dropped A or LCLICK types a wrong name with no error, and the code assumes Shift is off, so a keyboard that opens auto-capitalised or in caps lock inverts the case. Check that the field changed after each A (for example, its text grew), or say plainly in the header that this is not covered.

  3. Check every character before the first press, and throw an internal error for a bad one. An unsupported character in the middle of the string throws after the first half is already typed. It is a caller bug, so it should not use SEND_ERROR_REPORT; keyboard_enter_code uses NO_ERROR_REPORT, and InternalProgramError would fit too.

  4. Replace the output parameters and sentinels with a returned value. find_system_keyboard_cursor(screen, int& row, int& col) returns a bool and writes two int&. The typer then casts them to uint8_t, and the test uses -1 with a separate keyboard bool to mean "no keyboard". Return std::optional<...> of a row/col struct, and give the test an optional expected cursor.

  5. system_keyboard_field_has_text() can misread a busy background as text. It treats stddev sum > 30 over the field as text, but the game shows through the dimmed field. A busy PLZA background could pass for text. The caller would then press B on an empty field and close the keyboard, which is the hazard the function's own comment warns about. Count bright (white) text pixels instead. Also, neither this function nor SystemKeyboardDetector/Watcher has a caller in this PR. Upstream will ask what uses them, so link the first caller (PLZA/BDSP name entry) or name it in the description.

  6. The typer decodes the action bits itself and hard-codes its timings. ((uint8_t)action & 0x80) repeats the existing is_wrap(action). & 0x7f copies the enum's encoding, so put that in a helper in CodeEntryTools.h. MOVE_HOLD/WRAP_RELEASE/... and the unnamed 100ms, 250ms for LCLICK don't use the CODEBOARD_ENTRY_* delays (wired/wireless, Switch 1/2). Either say why the feedback loop needs its own fixed values, or use the settings. Every screenshot is a Switch 1 capture, so the header should say Switch 2 is untested.

Korean: out of scope for the first upstream version

The Korean keyboard (~/BDSPLab/fails/plza-ko-keyboard.png) adds a candidate strip (< ... >) above the keys, which moves the whole grid down about 0.08 of the screen height. Its letter keys are Hangul jamo, so neither the detector boxes nor KEYBOARD_POSITIONS fit it. Typing a Latin name on it would also need the globe key to switch layouts first. The detector returning false is the safe way to fail. Keep it that way. Name the supported keyboards in the detector/typer header (Latin QWERTY/AZERTY; console-tested in EN/ES/DE/IT/FR/JA), and state that Korean, and probably Chinese (untested), are not supported. The caller's error should say so rather than time out without a reason.

API shape and diff hygiene

  • Exposing keyboard_get_path() is fine. It already had external linkage, and the declaration and its comment are accurate. In that header, use the full-path include style of its siblings ("NintendoSwitch/Programs/FastCodeEntry/NintendoSwitch_CodeEntryTools.h").
  • Upstream needs the ComputerPrograms_UnitTestRunner.cpp change. Upstream registers image tests in that file, as it does for add_tests_CheckOnlineDetector.
  • Handoff: this branch is one commit plus 13 merges of fork main, which is 39 commits ahead of upstream main. Cut the handoff as one commit on upstream/main. The PR diff applies cleanly there (git apply --check).
  • The description is out of date: it says this version "hasn't been run on the console yet", but the console runs above used identical typer code.
  • CRLF, no tabs, file headers, include guards and brace style are all fine.

Nits

  • #include <map> in the typer (it uses std::map directly).
  • Name the LCLICK timings like the other constants (SHIFT_HOLD/SHIFT_RELEASE).
  • Give the OK colour ratio {0.172, 0.276, 0.552} and the grey range 600, 690 a short "measured on ..." comment.

Not checked: CI and a local build; no compile was needed for these claims.

dayfine and others added 3 commits November 13, 2026 08:24
…ecked typing

Review of #62:
- The cursor reader covers the right column (backspace, Return, OK) and the
  bottom row; the typer steps back onto the character grid from there.
- The detector's key-panel check uses two strips (rows 0/1 and 2/3), so a
  cursor anywhere can only spoil one of them.
- system_keyboard_field_has_text() reads the backspace key (dark with text),
  not the field over the game's background.
- The typer checks every character before the first press (UserSetupError),
  logs the cursor read and the moves of each try, names the last cursor and
  the target in its error, and checks that each key press adds white pixels
  to the field.
- Presses use the code entry hold for Switch 1; the read-back waits stay fixed.
- read_system_keyboard_cursor() returns std::optional<SystemKeyboardKey>.
- Tests derive ImageUnitTest, add six Legends: Z-A frames (en/fr/de/it/ja/es)
  and list the screens still to capture, which are added once their file
  exists.
- Headers state the supported keyboards (Latin QWERTY/AZERTY), Korean and
  Chinese out of scope, Switch 2 untested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…sor frames

The cursor border pulses between bright cyan and a darker teal (0, 185, 210);
the reader only matched the bright phase, so a cursor on '/' read as nothing.
Widen the colour test (red < 130, green > 160, blue > 170, green - red > 60),
which still leaves out the OK button and ABC underline blue.

Add tests on Legends: Z-A console frames with the cursor on T, /, !, 0, -,
backspace, Return, OK, globe, the N-accent key and Space. AZERTY and Korean
stay planned: the layout follows the system language, not the game's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
…s (function length)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 13, 2026
Conflict: NintendoSwitch_SystemKeyboardDetector.{h,cpp} were added on both
sides (lab copy vs core #62). Kept core's version; typing moved to
Programs/FastCodeEntry/NintendoSwitch_SystemKeyboardTyping.h. SourceFiles.cmake:
dropped the lab entry in the core library list, kept core's main-list entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 13, 2026
…_keyboard

autostory/engine (core #62) moved type_on_system_keyboard out of
SystemKeyboardDetector.h into FastCodeEntry/NintendoSwitch_SystemKeyboardTyping.h.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 13, 2026
Conflicts: the system keyboard detector takes #62's version (its typing is
FastCodeEntry::type_on_system_keyboard, which NewGame now calls); the unit
test runner keeps both include sets; SourceFiles lists the RNG handoff once.
RouteIO includes Pokemon_RouteLog.h, which no header brings in any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine and others added 3 commits November 13, 2026 22:34
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
The SystemKeyboardDetector screenshots are on the fork's CommandLineTests
main, not upstream's, so the image tests could not load them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013VUubhHeYSMGSSDXZkKZss
dayfine added a commit that referenced this pull request Nov 16, 2026
dayfine added a commit that referenced this pull request Nov 16, 2026
…62's typer

- Launch: no blind A presses after the user select; wait up to 60 s for the
  language list (about 15 s of logos on the console, 2026-11-15).
- Language list: D-pad taps 600 ms apart.
- Look picker: tilt left (150 ms) until a highlight reads, since the first
  tilt lands on a different portrait each launch; A only once the target reads.
- Name: type_on_system_keyboard() (switch/system-keyboard), Simplified
  Chinese switched to letters first; + until Confirm Choices shows (Chinese
  takes one + to accept the text, one to submit).
- Drop the blind name typer (PokemonLZA_AutoStoryNameKeys) and its tests;
  new_game_problem() checks the name against the typer's key map.
- Revert this PR's CommandLineTests pin bump.
- Tests on the 2026-11-15 frames: language list, look highlight (10),
  keyboard counter (10), the system keyboard reader (9), Confirm Choices (12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AKYzhrMnPK61XHWEch4a6E
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

detector Screen detectors and readers game:shared Shared or game-neutral code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant