Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
24187dd
Switch system keyboard: detector, cursor reader and closed-loop typer
dayfine Nov 13, 2026
0053160
PLZA AutoStory 01.00: start the new game from HOME or the language list
dayfine Nov 16, 2026
ca54e11
Merge #62 (system keyboard typer, its upstream branch upstream/system…
dayfine Nov 16, 2026
ff2c7f5
PLZA AutoStory 01.00: intro as the console showed it; name typed with…
dayfine Nov 16, 2026
fdc0de0
CI (fork only): pin CommandLineTests 41d8566 — drop before upstream
dayfine Nov 16, 2026
55575c9
Merge remote-tracking branch 'origin/plza/slice5b' into plza/newgame
dayfine Nov 16, 2026
62265a8
Merge branch 'plza/slice5b' into plza/newgame
dayfine Nov 16, 2026
09d3ed0
PLZA new game: find Z-A's HOME tile by its title before launching
dayfine Nov 16, 2026
84925e6
CI (fork only): pin CommandLineTests 989adaf (HOME tile frames) — dro…
dayfine Nov 16, 2026
fa69cdd
Merge branch 'plza/slice5b' into plza/newgame
dayfine Nov 16, 2026
7636a21
CI (fork only): pin CommandLineTests 8b376ff (01.02 Italian choice me…
dayfine Nov 16, 2026
ec709fb
Merge branch 'plza/slice5b' into plza/newgame
dayfine Nov 16, 2026
f4cf196
CI (fork only): pin CommandLineTests 848e0b0 (01.02 Italian talk prom…
dayfine Nov 16, 2026
30bd778
Merge branch 'plza/slice5b' into plza/newgame
dayfine Nov 16, 2026
7d47855
PLZA AutoStory: one launch from HOME for every start and reload
dayfine Nov 16, 2026
7db5b1c
PLZA AutoStory launch: UserSetupError's header; the test expectation …
dayfine Nov 16, 2026
9dfa366
Merge branch 'plza/slice5b' into plza/newgame
dayfine Nov 16, 2026
c2ec1d3
PLZA AutoStory: rename the launch test's image list so the base class…
dayfine Nov 16, 2026
a9f6195
PLZA AutoStory: the run's autosave sightings decide saves and chain r…
dayfine Nov 16, 2026
62be72f
Merge branch 'plza/slice5b' into plza/newgame
dayfine Nov 16, 2026
589597b
PLZA AutoStory: a talk presses A only on its prompt; 03.02 split at i…
dayfine Nov 16, 2026
0f9f382
PLZA AutoStory: a talk has started on any scene, not only a text box
dayfine Nov 16, 2026
9145e7f
PLZA AutoStory: story battles start from story_battle_start; a talk's…
dayfine Nov 16, 2026
edcd305
PLZA AutoStory: 03.02b walks to Taunie after the plaza scene; a battl…
dayfine Nov 16, 2026
18e3073
PLZA AutoStory: scene machine's classify_open back under 50 lines
dayfine Nov 16, 2026
73a04a5
PLZA AutoStory: a battle shows by the player's Pokemon panel, not onl…
dayfine Nov 16, 2026
a7a72c3
PLZA AutoStory tests: chain and battle-start fixtures match the platf…
dayfine Nov 16, 2026
bb1e275
PLZA AutoStory: 03.02b map talk pages the Map & Missions card with A …
dayfine Nov 16, 2026
615a767
PLZA AutoStory: a scene step must follow what starts it
dayfine Nov 16, 2026
1cc2391
PLZA AutoStory: 03.04 ends on its autosave outside the Center
dayfine Nov 16, 2026
223fc84
PLZA AutoStory: B through a scene's text box, so a closing box cannot…
dayfine Nov 16, 2026
cf0b73b
Merge origin/plza/slice5b into plza/newgame
dayfine Nov 16, 2026
6162fe2
PLZA 03.06/03.07: split at the Wild Zones card autosave
dayfine Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/cpp-ci-serial-programs-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -247,9 +247,9 @@ jobs:
with:
# The fork's fixtures: upstream's plus the PLZA AutoStory frames (Confirm Choices, opening and arrival folders), dayfine/CommandLineTests PR #18,
# and the 01.01 banner and controls hint in English and Italian (branch plza/hint-banner-any-language),
# and the 01.02 choice menu and talk prompt in Italian (branch plza/new-game-intro). Fork-only pin: dropped at upstream handoff.
# and the 01.00 new game frames, HOME tiles (FireRed first, Z-A second too) and the 01.02 Italian choice menu and talk prompt (branch plza/new-game-intro), and the 03.02b battle frame with no lock-on and Map & Missions card (branch plza/battle-panel). Fork-only pin: dropped at upstream handoff.
repository: 'dayfine/CommandLineTests'
ref: '848e0b03a03d9cdca580f21f3389486478d52f4d'
ref: '0414d97e16f77d6405d7f456147007ae317b1e72'
path: 'CommandLineTests'

# The unit tests assume the downloadable resources they need are already downloaded
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@

#include "NintendoSwitch/Inference/NintendoSwitch_CheckOnlineDetector.h"
#include "NintendoSwitch/Inference/NintendoSwitch_FailedToConnectDetector.h"
#include "NintendoSwitch/Inference/NintendoSwitch_SystemKeyboardDetector.h"
#include "NintendoSwitch/Inference/NintendoSwitch_UpdatePopupDetector.h"
#include "NintendoSwitch/Programs/Typewriter/NintendoSwitch_Typewriter_Tests.h"
#include "NintendoSwitch/Controllers/SysbotBase/SysbotBase_Connection.h"
Expand All @@ -56,7 +57,7 @@



UnitTestDatabase make_UNIT_TESTS_ALL(){

Check warning on line 60 in SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.cpp

View workflow job for this annotation

GitHub Actions / code-health

Code health (advisory)

PokemonAutomation::ComputerPrograms::make_UNIT_TESTS_ALL(): function_lines=41 (advisory above 25, limit 50)
UnitTestDatabase ret;

add_tests_QtEventThreadPool(ret, GlobalThreadPools::computation_normal());
Expand All @@ -82,6 +83,7 @@
add_tests_ScreenMachine(ret);
NintendoSwitch::add_tests_CheckOnlineDetector(ret);
NintendoSwitch::add_tests_FailedToConnectDetector(ret);
NintendoSwitch::add_tests_SystemKeyboardDetector(ret);
NintendoSwitch::add_tests_UpdatePopupDetector(ret);
NintendoSwitch::Typewriter::add_tests_TypewriterTextMapping(ret);
NintendoSwitch::PokemonBDSP::add_tests_DialogDetector(ret);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,280 @@
/* System Keyboard Detector
*
* From: https://github.com/PokemonAutomation/
*
*/

#include <cmath>
#include <format>
#include <string>
#include <string_view>
#include <utility>
#include <vector>
#include "Common/Cpp/Filesystem/Filesystem.h"
#include "CommonFramework/GlobalAutoPaths.h"
#include "CommonFramework/ImageTypes/ImageRGB32.h"
#include "CommonFramework/ImageTools/ImageStats.h"
#include "CommonFramework/VideoPipeline/VideoOverlayScopes.h"
#include "CommonTools/Images/SolidColorTest.h"
#include "Tests/ImageUnitTest.h"
#include "NintendoSwitch_SystemKeyboardDetector.h"

namespace PokemonAutomation{
namespace NintendoSwitch{

namespace{

// Centres of the five rows of keys, and of the first column and the spacing
// between columns, as fractions of the screen.
constexpr double KEY_ROW_Y[] = {0.500, 0.593, 0.681, 0.770, 0.858};
constexpr double KEY_COL_X0 = 0.077;
constexpr double KEY_COL_STEP = 0.075;

// In the right column, Return spans rows 1-2 and OK rows 3-4. These are the
// gaps between those keys.
constexpr double RIGHT_COLUMN_RETURN_TOP = 0.548;
constexpr double RIGHT_COLUMN_OK_TOP = 0.725;

// The cursor border and everything else the reader looks at, from the top of the
// digit row to the bottom of the bottom row.
const ImageFloatBox KEYS_AREA(0.0, 0.43, 1.0, 0.49);

// The cursor border pulses between bright cyan (about 50, 240, 220) and a darker
// teal (about 0, 185, 210), measured on Legends: Z-A. The blue of the OK button
// and of the ABC underline (40, 90, 250) is not green enough to match.
bool is_cursor_pixel(uint32_t pixel){
const int red = (pixel >> 16) & 0xff;
const int green = (pixel >> 8) & 0xff;
const int blue = pixel & 0xff;
return red < 130 && green > 160 && blue > 170 && green - red > 60;
}

uint8_t nearest_row(double y){
uint8_t row = 0;
for (uint8_t r = 1; r < 5; r++){
if (std::abs(KEY_ROW_Y[r] - y) < std::abs(KEY_ROW_Y[row] - y)){
row = r;
}
}
return row;
}

// The key whose cursor border has its centre at (x, y), or nothing if that is
// off the keyboard.
std::optional<SystemKeyboardKey> key_at(double x, double y){
const long col = std::lround((x - KEY_COL_X0) / KEY_COL_STEP);
if (col < 0 || col > 11){
return std::nullopt;
}
if (col < 11){
return SystemKeyboardKey{nearest_row(y), (uint8_t)col};
}
if (y < RIGHT_COLUMN_RETURN_TOP){
return SystemKeyboardKey{0, 11};
}
if (y < RIGHT_COLUMN_OK_TOP){
return SystemKeyboardKey{1, 11};
}
return SystemKeyboardKey{3, 11};
}

} // namespace



SystemKeyboardDetector::SystemKeyboardDetector(Color color)
: m_color(color)
, m_key_panel_upper(0.15, 0.532, 0.65, 0.036)
, m_key_panel_lower(0.15, 0.715, 0.65, 0.032)
, m_bottom_bar(0.10, 0.935, 0.35, 0.030)
, m_ok_button(0.875, 0.760, 0.045, 0.020)
{}
void SystemKeyboardDetector::make_overlays(VideoOverlaySet& items) const{
items.add(m_color, m_key_panel_upper);
items.add(m_color, m_key_panel_lower);
items.add(m_color, m_bottom_bar);
items.add(m_color, m_ok_button);
}
bool SystemKeyboardDetector::detect(const ImageViewRGB32& screen){
// Each strip crosses white keys and the slightly brighter gaps between them,
// which gives a stddev sum of about 11 (measured on BDSP and Legends: Z-A).
const double max_panel_stddev = 20;
const bool panel =
is_white(extract_box_reference(screen, m_key_panel_upper), 500, max_panel_stddev) ||

Check warning on line 103 in SerialPrograms/Source/NintendoSwitch/Inference/NintendoSwitch_SystemKeyboardDetector.cpp

View workflow job for this annotation

GitHub Actions / code-health

Code health (advisory)

style: is_white(): literal argument 500 needs a /*param=*/ comment (Google "Function Arguments", user review on fork #590)
is_white(extract_box_reference(screen, m_key_panel_lower), 500, max_panel_stddev);

Check warning on line 104 in SerialPrograms/Source/NintendoSwitch/Inference/NintendoSwitch_SystemKeyboardDetector.cpp

View workflow job for this annotation

GitHub Actions / code-health

Code health (advisory)

style: is_white(): literal argument 500 needs a /*param=*/ comment (Google "Function Arguments", user review on fork #590)
if (!panel || !is_white(extract_box_reference(screen, m_bottom_bar))){
return false;
}
// OK is blue (40, 93, 255), or flat grey (221, 221, 221) while the field is
// empty (disabled). Measured on BDSP and Legends: Z-A.
const ImageStats ok = image_stats(extract_box_reference(screen, m_ok_button));
return is_solid(ok, {0.103, 0.240, 0.657}) || is_grey(ok, 600, 690);

Check warning on line 111 in SerialPrograms/Source/NintendoSwitch/Inference/NintendoSwitch_SystemKeyboardDetector.cpp

View workflow job for this annotation

GitHub Actions / code-health

Code health (advisory)

style: is_grey(): literal arguments 600, 690 need a /*param=*/ comment (Google "Function Arguments", user review on fork #590)
}


std::optional<SystemKeyboardKey> read_system_keyboard_cursor(const ImageViewRGB32& screen){
// The cursor is a cyan border around one key. Neighbouring keys overlap that border,
// so take the centroid of all cyan pixels and map it to the nearest key.
const ImageViewRGB32 keys = extract_box_reference(screen, KEYS_AREA);
double count = 0, sum_x = 0, sum_y = 0;
for (size_t y = 0; y < keys.height(); y++){
for (size_t x = 0; x < keys.width(); x++){
if (is_cursor_pixel(keys.pixel(x, y))){
count++;
sum_x += (double)x;
sum_y += (double)y;
}
}
}
// The border is about 0.18% of the screen; a few stray pixels are not a cursor.
if (count < 0.0005 * screen.width() * screen.height()){
return std::nullopt;
}
return key_at(
KEYS_AREA.x + sum_x / count / keys.width() * KEYS_AREA.width,
KEYS_AREA.y + sum_y / count / keys.height() * KEYS_AREA.height
);
}

bool system_keyboard_field_has_text(const ImageViewRGB32& screen){
// Left part of the backspace key, clear of its icon and of the cursor border.
// Light grey (221) with an empty field, dark grey (52) with text.
const ImageStats stats = image_stats(extract_box_reference(screen, ImageFloatBox(0.878, 0.470, 0.008, 0.060)));
return stats.average.sum() < 400;
}

size_t system_keyboard_text_pixels(const ImageViewRGB32& screen){
// The field, above its underline, which is white in some games.
const ImageViewRGB32 field = extract_box_reference(screen, ImageFloatBox(0.03, 0.17, 0.87, 0.105));
size_t count = 0;
for (size_t y = 0; y < field.height(); y++){
for (size_t x = 0; x < field.width(); x++){
const uint32_t pixel = field.pixel(x, y);
if (((pixel >> 16) & 0xff) > 190 && ((pixel >> 8) & 0xff) > 190 && (pixel & 0xff) > 190){
count++;
}
}
}
return count;
}



namespace{

// One screenshot and what each function should read from it.
class Test_SystemKeyboardDetector : public ImageUnitTest{
public:
// `cursor`: the key under the cursor, or nothing for a screen without the keyboard.
Test_SystemKeyboardDetector(const std::string& image, std::optional<SystemKeyboardKey> cursor, bool has_text)
: ImageUnitTest(std::format("NintendoSwitch::SystemKeyboardDetector - {}", image), {image})
, m_cursor(cursor)
, m_has_text(has_text)
{}

virtual UnitTestResult run_on_images(

Check warning on line 175 in SerialPrograms/Source/NintendoSwitch/Inference/NintendoSwitch_SystemKeyboardDetector.cpp

View workflow job for this annotation

GitHub Actions / code-health

Code health (advisory)

PokemonAutomation::NintendoSwitch::Test_SystemKeyboardDetector::run_on_images( Logger & logger , CancellableScope & scope , const std :: vector<ImageRGB32> & images) const: function_lines=29 (advisory above 25, limit 50)
Logger& logger, CancellableScope& scope,
const std::vector<ImageRGB32>& images
) const override{
const ImageRGB32& image = images[0];
const bool keyboard = SystemKeyboardDetector().detect(image);
if (keyboard != m_cursor.has_value()){
return std::format("detect() returned {}.", keyboard);
}
if (!keyboard){
return true;
}
const std::optional<SystemKeyboardKey> cursor = read_system_keyboard_cursor(image);
if (cursor != m_cursor){
return cursor
? std::format("Read the cursor at ({}, {}).", cursor->row, cursor->col)
: std::format("Read no cursor.");
}
const bool has_text = system_keyboard_field_has_text(image);
if (has_text != m_has_text){
return std::format("system_keyboard_field_has_text() returned {}.", has_text);
}
// An empty field has no white text pixels; one typed character has many.
const size_t text_pixels = system_keyboard_text_pixels(image);
if ((text_pixels >= 20) != m_has_text){
return std::format("system_keyboard_text_pixels() returned {}.", text_pixels);
}
return true;
}

private:
std::optional<SystemKeyboardKey> m_cursor;
bool m_has_text;
};

std::string test_image_path(std::string_view image){
return std::format("NintendoSwitch/SystemKeyboardDetector/{}", image);
}

// The cursor on keys other than "1".
void add_cursor_tests(UnitTestDatabase& database){
// Legends: Z-A, English and Spanish prompts. The
// border pulses between bright cyan and a darker teal; T, / and ! are dark.
const std::pair<std::string_view, SystemKeyboardKey> cursors[] = {
{"PLZA-Cursor-T.png", {1, 4}},
{"PLZA-Cursor-Slash.png", {1, 10}},
{"PLZA-Cursor-Exclamation.png", {3, 10}},
{"PLZA-Cursor-0.png", {0, 9}},
{"PLZA-Cursor-Minus.png", {0, 10}},
{"PLZA-Cursor-Backspace.png", {0, 11}},
{"PLZA-Cursor-Return.png", {1, 11}},
{"PLZA-Cursor-OK.png", {3, 11}},
{"PLZA-Cursor-Globe.png", {4, 0}},
{"PLZA-Cursor-NTilde.png", {4, 5}},
{"PLZA-Cursor-Space.png", {4, 8}},
};
for (const auto& [image, cursor] : cursors){
database.add<Test_SystemKeyboardDetector>(test_image_path(image), cursor, false);
}
}

// Tests whose screen is not captured yet.
void add_planned_tests(UnitTestDatabase& database){
const SystemKeyboardKey key_1{0, 0};
// Screens still to be captured: AZERTY needs a console whose system language
// uses it (the layout follows the system, not the game). Each test is added
// once its image is in the test repo.
struct Planned{
std::string_view image;
std::optional<SystemKeyboardKey> cursor;
bool has_text;
};
const Planned planned[] = {
{"AZERTY-Empty.png", key_1, false},
{"PLZA-Korean.png", std::nullopt, false},
};
for (const Planned& test : planned){
const std::string image = test_image_path(test.image);
if (Filesystem::exists(std::format("{}{}", UNIT_TEST_RESOURCE_PATH(), image))){
database.add<Test_SystemKeyboardDetector>(image, test.cursor, test.has_text);
}
}
}

} // namespace


void add_tests_SystemKeyboardDetector(UnitTestDatabase& database){
const SystemKeyboardKey key_1{0, 0};
// The OK button is blue with text in the field and grey without.
database.add<Test_SystemKeyboardDetector>(test_image_path("BDSP-PlayerName-Empty.png"), key_1, false);
database.add<Test_SystemKeyboardDetector>(test_image_path("BDSP-FriendName-Typed.png"), key_1, true);
for (std::string_view language : {"English", "French", "German", "Italian", "Japanese", "Spanish"}){
database.add<Test_SystemKeyboardDetector>(test_image_path(std::format("PLZA-{}-Empty.png", language)), key_1, false);
}
for (std::string_view image : {"BDSP-Dialog.png", "HomeMenu.png", "HomeMenu-OnlineLicensePopup.png", "UserSelect.png"}){
database.add<Test_SystemKeyboardDetector>(test_image_path(image), std::nullopt, false);
}
add_cursor_tests(database);
add_planned_tests(database);
}



} // namespace NintendoSwitch
} // namespace PokemonAutomation
Loading
Loading