Repository navigation
Command-line test runner: filter tests by name substring - #1895
Conversation
Add a "--test-substring <value>" argument to the command-line unit test runner (--command-line-test-mode): it only runs the tests whose name contains the given substring, using the same case-sensitive matching as the GUI's "Run All Tests Containing This Substring" mode. Without the argument (or with an empty value) all tests run, as before. - Main.cpp: parse the argument (error + exit 1 if it has no value). - CommandLineTests.h/.cpp: pass the substring through to the runner. - CommandLineUnitTestRunner::run(): add only the matching tests; log how many will run, and warn in orange if nothing matches.
The two runners (UnitTestRunner::program and CommandLineUnitTestRunner::run) each had their own copy of the "add the tests whose name contains this substring" loop (and the GUI's "Tests to Run" preview had a third copy), and each of their on_test_finished listeners re-implemented the same result classification into passed/failed/skipped/OOM buckets. - Add a file-local add_tests_matching(filter, add_test) helper and use it in the GUI's run-substring-match / run-all modes, the GUI's "Tests to Run" preview, and the command-line runner. - Add UnitTestResultCounters, shared by both runners: one place defines which bucket each result kind goes into (OOM is a failure and has its own counter) and returns the result's 1-based index within its bucket, which the GUI uses for its 1000-entry panel caps, exactly like the old fetch_add(1) + 1 sequences. The write-only m_total_failed goes away.
Per review: UnitTestResultCounters/m_results was redundant for the GUI, whose Stats already track the same passed/failed/skipped/OOM buckets, and its one unique use (the combined FAILED+OOM cap sequence) was just the old m_total_failed. Revert both runners to their original atomics and per-case counting, keeping only the shared add_tests_matching() helper from the previous commit.
pa-devbot
left a comment
There was a problem hiding this comment.
AI review: Reviewed the full diff (all 3 commits, net diff vs main) and verified it end-to-end on aarch64: pa-build + pa-test reproduce all four documented cases exactly (BlackBorder -> 1 test; Typewriter -> 44 passed; NoSuchTestXYZ -> warning, 0 tests, exit 0; missing value -> error, exit 1). CI is green on all five platforms and the branch merges cleanly into current main. Docs in CommandLineTests.h/ComputerPrograms_UnitTestRunner.h match the behavior.
The shared add_tests_matching() helper is a good cleanup: it preserves the original semantics in all three call sites (name-order iteration over the std::map database, the 1000-entry display cap in the GUI preview counts identically to the old loop, and test->name() is the same string as the old test.first since the map key is set from it at insertion). The counter dedup that got reverted mid-PR left no residue: the diff vs main touches only the intended regions, and both runners keep their original atomics.
| if (num_matching == 0){ | ||
| m_logger.log("No test name contains \"" + test_substring + "\". No tests will be run.", COLOR_ORANGE); | ||
| }else{ | ||
| m_logger.log("Running the " + tostr_u_commas(num_matching) + " tests whose name contains \"" + test_substring + "\""); |
There was a problem hiding this comment.
AI review: Nit: when exactly one test matches this reads "Running the 1 tests whose name contains ...". No need to block on it, but if you're touching this line anyway a "test(s)" or singular/plural branch would read better.
| }); | ||
| if (!test_substring.empty()){ | ||
| if (num_matching == 0){ | ||
| m_logger.log("No test name contains \"" + test_substring + "\". No tests will be run.", COLOR_ORANGE); |
There was a problem hiding this comment.
AI review: FYI (documented, so not a blocker): a substring that matches nothing exits 0, so a typo'd filter in CI would produce a green run with zero tests - the orange warning is the only signal. Worth knowing if anyone wires this into a pipeline; if that becomes a problem, a distinct exit code for "zero tests ran due to filter" would be the natural fix.
| } | ||
| } | ||
| if (test_substring_missing_value){ | ||
| global_logger_command_line().log("--test-substring requires a value", COLOR_RED); |
There was a problem hiding this comment.
AI review: FYI: this missing-value error fires even when --command-line-test-mode isn't passed, i.e. a stray --test-substring on a GUI launch also exits 1. That strict behavior seems right for a malformed flag, just noting the flag is parsed outside of test mode.
|
I had Gin's bot add this PR so that we could ask agents to run the test suite on just a subset of tests. |
Adds a
--test-substring <value>argument to the command-line unit test runner (--command-line-test-mode) so you can run only the tests whose name contains a given substring - the same case-sensitive matching as the GUI's "Run All Tests Containing This Substring" mode.Usage
./SerialPrograms --command-line-test-mode --test-substring BlackBorder # or, in this sandbox: pa-test Arduino-Source --test-substring BlackBorderBehavior
No test name contains "...". No tests will be run.) and the run exits 0.--test-substringwith no value: error on stdout + exit 1.Changes
SerialPrograms/Source/CommonFramework/Main.cpp: parse--test-substring <value>alongside the existing flags.SerialPrograms/Source/Tests/CommandLineTests.h/.cpp: pass the substring to the runner and echo it in the start banner.SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.h/.cpp:CommandLineUnitTestRunner::run()takes the substring and only adds matching tests.Companion docs PR: PokemonAutomation/CommandLineTests#48.
Testing (aarch64, pa-build + pa-test)
--test-substring BlackBorder-> 1 test ran, exit 0--test-substring Typewriter-> 44 tests, all passed, exit 0--test-substring NoSuchTestXYZ-> warning, 0 tests, exit 0--test-substring(no value) -> error, exit 1