Skip to content

Command-line test runner: filter tests by name substring - #1895

Merged
Mysticial merged 3 commits into
mainfrom
cli-test-substring-filter
Nov 12, 2026
Merged

Mysticial merged 3 commits into
mainfrom
cli-test-substring-filter

Conversation

@pa-devbot

Copy link
Copy Markdown
Collaborator

Adds a --test-substring <value> argument to the command-line unit test runner (--command-line-test-mode) so you can run only the tests whose name contains a given substring - the same case-sensitive matching as the GUI's "Run All Tests Containing This Substring" mode.

Usage

./SerialPrograms --command-line-test-mode --test-substring BlackBorder
# or, in this sandbox:
pa-test Arduino-Source --test-substring BlackBorder

Behavior

  • No argument (or empty value): runs every test, exactly as before.
  • Substring given: only matching tests run; the runner logs how many will run.
  • Substring matches nothing: a warning is logged (No test name contains "...". No tests will be run.) and the run exits 0.
  • --test-substring with no value: error on stdout + exit 1.

Changes

  • SerialPrograms/Source/CommonFramework/Main.cpp: parse --test-substring <value> alongside the existing flags.
  • SerialPrograms/Source/Tests/CommandLineTests.h/.cpp: pass the substring to the runner and echo it in the start banner.
  • SerialPrograms/Source/ComputerPrograms/ComputerPrograms_UnitTestRunner.h/.cpp: CommandLineUnitTestRunner::run() takes the substring and only adds matching tests.

Companion docs PR: PokemonAutomation/CommandLineTests#48.

Testing (aarch64, pa-build + pa-test)

  • --test-substring BlackBorder -> 1 test ran, exit 0
  • --test-substring Typewriter -> 44 tests, all passed, exit 0
  • --test-substring NoSuchTestXYZ -> warning, 0 tests, exit 0
  • --test-substring (no value) -> error, exit 1
  • full unfiltered suite: 20,520 passed, 0 failed, 5 skipped, exit 0

Add a "--test-substring <value>" argument to the command-line unit test
runner (--command-line-test-mode): it only runs the tests whose name contains
the given substring, using the same case-sensitive matching as the GUI's
"Run All Tests Containing This Substring" mode. Without the argument (or
with an empty value) all tests run, as before.

- Main.cpp: parse the argument (error + exit 1 if it has no value).
- CommandLineTests.h/.cpp: pass the substring through to the runner.
- CommandLineUnitTestRunner::run(): add only the matching tests; log how many
  will run, and warn in orange if nothing matches.
The two runners (UnitTestRunner::program and CommandLineUnitTestRunner::run)
each had their own copy of the "add the tests whose name contains this
substring" loop (and the GUI's "Tests to Run" preview had a third copy),
and each of their on_test_finished listeners re-implemented the same result
classification into passed/failed/skipped/OOM buckets.

- Add a file-local add_tests_matching(filter, add_test) helper and use it
  in the GUI's run-substring-match / run-all modes, the GUI's "Tests to
  Run" preview, and the command-line runner.
- Add UnitTestResultCounters, shared by both runners: one place defines
  which bucket each result kind goes into (OOM is a failure and has its own
  counter) and returns the result's 1-based index within its bucket, which
  the GUI uses for its 1000-entry panel caps, exactly like the old
  fetch_add(1) + 1 sequences. The write-only m_total_failed goes away.
Per review: UnitTestResultCounters/m_results was redundant for the GUI,
whose Stats already track the same passed/failed/skipped/OOM buckets, and
its one unique use (the combined FAILED+OOM cap sequence) was just the old
m_total_failed. Revert both runners to their original atomics and
per-case counting, keeping only the shared add_tests_matching() helper
from the previous commit.

@pa-devbot pa-devbot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: Reviewed the full diff (all 3 commits, net diff vs main) and verified it end-to-end on aarch64: pa-build + pa-test reproduce all four documented cases exactly (BlackBorder -> 1 test; Typewriter -> 44 passed; NoSuchTestXYZ -> warning, 0 tests, exit 0; missing value -> error, exit 1). CI is green on all five platforms and the branch merges cleanly into current main. Docs in CommandLineTests.h/ComputerPrograms_UnitTestRunner.h match the behavior.

The shared add_tests_matching() helper is a good cleanup: it preserves the original semantics in all three call sites (name-order iteration over the std::map database, the 1000-entry display cap in the GUI preview counts identically to the old loop, and test->name() is the same string as the old test.first since the map key is set from it at insertion). The counter dedup that got reverted mid-PR left no residue: the diff vs main touches only the intended regions, and both runners keep their original atomics.

if (num_matching == 0){
m_logger.log("No test name contains \"" + test_substring + "\". No tests will be run.", COLOR_ORANGE);
}else{
m_logger.log("Running the " + tostr_u_commas(num_matching) + " tests whose name contains \"" + test_substring + "\"");

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: Nit: when exactly one test matches this reads "Running the 1 tests whose name contains ...". No need to block on it, but if you're touching this line anyway a "test(s)" or singular/plural branch would read better.

});
if (!test_substring.empty()){
if (num_matching == 0){
m_logger.log("No test name contains \"" + test_substring + "\". No tests will be run.", COLOR_ORANGE);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: FYI (documented, so not a blocker): a substring that matches nothing exits 0, so a typo'd filter in CI would produce a green run with zero tests - the orange warning is the only signal. Worth knowing if anyone wires this into a pipeline; if that becomes a problem, a distinct exit code for "zero tests ran due to filter" would be the natural fix.

}
}
if (test_substring_missing_value){
global_logger_command_line().log("--test-substring requires a value", COLOR_RED);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: FYI: this missing-value error fires even when --command-line-test-mode isn't passed, i.e. a stray --test-substring on a GUI launch also exits 1. That strict behavior seems right for a malformed flag, just noting the flag is parsed outside of test mode.

@jw098

jw098 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

I had Gin's bot add this PR so that we could ask agents to run the test suite on just a subset of tests.
I reviewed the PR, and it looks fine to me.

@Mysticial
Mysticial merged commit 8958cda into main Nov 12, 2026
8 checks passed
@Mysticial
Mysticial deleted the cli-test-substring-filter branch November 12, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants