diff --git a/devtools/code_health/README.md b/devtools/code_health/README.md index 3c2afd3c87..2a9620025f 100644 --- a/devtools/code_health/README.md +++ b/devtools/code_health/README.md @@ -97,13 +97,14 @@ function or class exceed their limit. Checks are the metric names in the table a `file_lines`, `function_lines`, `nesting`, `parameters`, `public_methods`, `public_data`, `single_method_interface`, `header_symbols`. Six style findings take a tag too: `char_pointer`, `overload_prefix`, `out_param_order`, `string_concat`, `param_comment` and `test_only_api`, for a C API -or a fixed interface that needs them. Other style and layering +or a fixed interface that needs them; so do `learned_rule_test` and `special_case` (see +[Systematic fixes](#systematic-fixes)). Other style and layering findings have no bypass. - **Place it** on the function's or class's first line, or in the comment block directly above it (only `//` comment and `template` lines between). A `file_lines` tag goes in the first 20 lines of the file, and so does a `header_symbols` tag. A style tag (`char_pointer`, `overload_prefix`, - `out_param_order`, `string_concat`, `param_comment`, `test_only_api`) goes on the flagged line or in the comment block directly above it, and waives the finding whether it is an error or advisory. A tag that attaches to nothing fails. Style checks report only changed lines, so a style tag on a line the change left alone (a later PR in a stack) attaches if it covers a finding of the whole file. + `out_param_order`, `string_concat`, `param_comment`, `test_only_api`, `learned_rule_test`, `special_case`) goes on the flagged line or in the comment block directly above it, and waives the finding whether it is an error or advisory. A tag that attaches to nothing fails. Style checks report only changed lines, so a style tag on a line the change left alone (a later PR in a stack) attaches if it covers a finding of the whole file. - **Justify it:** the text after `--`, on the tag line itself, must be at least 40 characters and 6 words, and say why splitting would make the code worse. Unknown checks, a missing or short justification, or a malformed tag fail. @@ -173,6 +174,29 @@ non-test caller later in the stack (API added one PR before its caller), and ups `Common/ControllerStates` C-style headers (constants and `*_equals` used only by their own .cpp) are reported too. +## Systematic fixes + +Three blocking rules keep a console fix a rule with a test, not a patch at one story point (the +user, 2026-11-15: "make sure our solutions are systematic and avoid patching here and there"). +Like the style checks they look only at what a change adds. + +| Check | Level | Rule | +| --- | --- | --- | +| `learned_rule_test` (`learned_rules.py`) | error (tag) | A comment in non-test C++ that cites a console finding, `(console B, 2026-11-10)` or `(17.03, console B 2026-11-15)` ("console B" or "lab-B", then a date in the same clause), needs a test that cites the same date, and the checkpoint when the citation names one: a `*_Tests.cpp`/`.h` or a `ConsoleFailures/*.json` replay, in the same game's directory or in shared code. Reported on added lines as `learned rule without a test: (console B, )`, unless the base file cited that date already (a reworded comment). Also reported: a change to test files after which a citation anywhere in the tree has no covering test while it had one at the base (`learned rule lost its last test`). | +| `special_case` (`special_cases.py`) | error (tag) | In non-test C++ under `SerialPrograms/Source/**/AutoStory/`: a string literal that is a checkpoint id (`NN.NN[a-z]`) or the id of an event a `StoryData/*.plan.json` declares, used as a key: `==`, `!=`, `<=>`, `.find/count/contains/at/starts_with/ends_with/compare(...)`, `[...]`, or the first element of `{"id", ...}`. Not reported: plain id lists, ids passed to other functions, one-word event ids (`"bicycle"` is also an item) and ids that are also a `kind`/`action` name. Message: `special case by name: "" at ; move it into the plan or a general rule`. | +| Plan lint (`plan_lint.py`) | error | Every `StoryData/*.plan.json` (any game): a new or edited event whose `kind`/`action` is `legs` or whose `args` hold a `legs` list (FRLG `kind: legs`, PLZA `action: walk`) needs `not planned: ` in its `why`. Events unchanged since the base are grandfathered; each run prints the changed plans' legs counts per game. The reason in the `why` is the bypass; there is no tag in JSON. | + +Measured on 2026-11-15 (whole trees, every line counted as new): `learned_rule_test` 1 of 250 +citations uncovered on `frlg/moved-trainers` (`ScreenFades.h`, 2026-11-08) and 12 of 41 on +`plza/newgame` (PLZA 2026-11-10 notes: no PLZA test cites that day); `special_case` 1 +(`PokemonFRLG_AutoStoryRngLoop.cpp`, `step.name == "rival_takes_bulbasaur"`); plan lint 134 FRLG +legs events (FRLG's `legs_before_2026_11_15()` list) and 26 PLZA walks, none with a reason. On the +diffs of fork #599, #604 and #610 against their bases all three report nothing. The date match is +coarse: any test of the same game that names the same day covers a citation, so the rule catches +a learned rule from a day no test cites, not a rule whose own test is missing on a busy day. +Detector and reader measurements whose image tests live in `CommandLineTests` take a +`learned_rule_test` tag saying so. + ## Build and native C++ tests in one command The existing `.github/workflows/cpp-ci-serial-programs-base.yml` already runs diff --git a/devtools/code_health/__main__.py b/devtools/code_health/__main__.py index 6cb3349c0c..7cc318f2cb 100644 --- a/devtools/code_health/__main__.py +++ b/devtools/code_health/__main__.py @@ -5,10 +5,11 @@ import sys from pathlib import Path -from . import allow, exposure, header_api, layering, param_comments, style +from . import allow, exposure, header_api, layering, learned_rules, param_comments, plan_lint, special_cases, style from .classes import measure_classes +from .cpp_text import normalize from .findings import annotations -from .git_files import changed_files, default_base, read_revision +from .git_files import changed_files, changed_paths, default_base, read_revision from .metrics import measure from .policy import advisories, bypasses, evaluate @@ -17,23 +18,88 @@ def analyze(path, source): return measure(path, source) + measure_classes(path, source) + header_api.measure_header(path, source) -def style_findings(path, source, previous_source, users): +class Repository: + """Whole-tree facts some line checks need, each read once and only when asked for.""" + + def __init__(self, root): + self.root = root + self.users = exposure.git_users(root) + self._tests = None + self._event_ids = None + + @property + def tests(self): + if self._tests is None: + self._tests = learned_rules.tracked_tests(self.root) + return self._tests + + @property + def event_ids(self): + if self._event_ids is None: + self._event_ids, _ = special_cases.plan_names(special_cases.tracked_plans(self.root)) + return self._event_ids + + +def style_findings(path, source, previous_source, repo): """(style errors, style advisories) on lines new since `previous_source` (None: all lines).""" errors, notes = style.check(path, source, previous_source) errors += header_api.prefix_overloads(path, source, previous_source) # Advisory: both report many sites a reviewer may accept (pbf_* timings, API for a # later PR in a stack); see the README's Style table for the measured rates. notes += param_comments.check(path, source, previous_source) - notes += exposure.check(path, source, previous_source, users) + notes += exposure.check(path, source, previous_source, repo.users) + added = style.added_lines(normalize(source).split('\n'), + None if previous_source is None else normalize(previous_source).split('\n')) + if special_cases.autostory_code(path): + errors += special_cases.check(path, source, added, repo.event_ids) + if learned_rules.CONSOLE.search(source): + errors += learned_rules.check(path, source, previous_source, added, repo.tests) return errors, notes +def lost_tests(root, base_ref, paths, repo): + """Failures for console citations whose last covering test this change removed or edited.""" + changed = {current: original for _, current, original in paths + if learned_rules.is_test(current) or (original and learned_rules.is_test(original))} + if not changed: + return [], [] + base_tests = [t for t in repo.tests if t.path not in changed] + for original in filter(None, changed.values()): + try: + base_tests.append(learned_rules.test_cite(original, read_revision(root, base_ref, original))) + except subprocess.CalledProcessError: + continue + found = learned_rules.lost_coverage(learned_rules.code_citations(root), base_tests, repo.tests) + kept, waived = [], [] + for path in sorted({f.path for f in found}): + source = (root / path).read_text(encoding='utf-8-sig') + allowances, _ = allow.parse(path, source) + left, notices, _ = allow.waive(path, source, allowances, [f for f in found if f.path == path]) + kept += left + waived += notices + return kept, waived + + +def plan_findings(root, base_ref, paths): + """(failures for unexplained new legs events in changed plans, per-game legs counts).""" + failures, plans = [], [] + for status, current, original in paths: + if status == 'D' or not plan_lint.is_plan(current): + continue + text = (root / current).read_text(encoding='utf-8-sig') + previous = read_revision(root, base_ref, original) if original and plan_lint.is_plan(original) else None + failures += plan_lint.check(current, text, previous) + plans.append((current, text)) + return failures, plan_lint.counts(plans) + + def run(root, base_ref=None, github=False): if base_ref is None: base_ref, how = default_base(root) print(f'Base: {base_ref} ({how}).') changes = changed_files(root, base_ref) - users = exposure.git_users(root) + paths = changed_paths(root, base_ref) + repo = Repository(root) failures, notes, bypassed = [], [], [] advisory = 0 for current, original in changes: @@ -50,17 +116,25 @@ def run(root, base_ref=None, github=False): notices, unneeded = bypasses(metrics, allowed) bypassed.extend(notices) notes.extend(unneeded) - style_errors, style_notes = style_findings(current, source, previous_source, users) + style_errors, style_notes = style_findings(current, source, previous_source, repo) # Waive errors and advisories in one pass, so a tag on either counts as used. A tag on # a line this change left alone attaches to the whole file's findings. kept, waived, unattached = allow.waive( current, source, allowances, style_errors + style_notes, - whole_file=lambda: sum(style_findings(current, source, None, users), [])) + whole_file=lambda: sum(style_findings(current, source, None, repo), [])) failures.extend([f for f in kept if f in style_errors] + unattached) bypassed.extend(waived) notes.extend(f for f in kept if f not in style_errors) advisory += sum(m.check == 'file_lines' and 500 < m.value <= 1000 for m in metrics) + lost, lost_waived = lost_tests(root, base_ref, paths, repo) + failures.extend(lost) + bypassed.extend(lost_waived) + plan_failures, legs = plan_findings(root, base_ref, paths) + failures.extend(plan_failures) print(f'Scanned {len(changes)} changed C++ files against {base_ref}.') + for name, (total, with_reason) in sorted(legs.items()): + print(f'Plan lint: {name}: {total} legs events in the changed plans, {with_reason} with a ' + '"not planned:" reason; only new or edited ones must have one.') print(f'Advisory: {advisory} changed files exceed 500 lines; prefer functions <=25 lines.') for note in notes: print(f'Advisory: {note}') diff --git a/devtools/code_health/allow.py b/devtools/code_health/allow.py index badb4dcdec..a075e50843 100644 --- a/devtools/code_health/allow.py +++ b/devtools/code_health/allow.py @@ -4,7 +4,7 @@ that function's or class's listed metrics exceed their limit up to a hard ceiling. A `file_lines` or `header_symbols` tag goes in the first FILE_TAG_LINES lines. A STYLE_CHECKS tag (`char_pointer`, `overload_prefix`, `out_param_order`, `string_concat`, `param_comment`, -`test_only_api`) waives that style finding +`test_only_api`, `learned_rule_test`, `special_case`) waives that line finding on its own line or the line below its comment block. Every bypass is printed, quoting its justification, so reviewers see it; malformed, unknown or misplaced tags fail.""" import re @@ -15,7 +15,7 @@ # Style findings a tag can bypass; they attach like a function's tag, to the finding's line. STYLE_CHECKS = ('char_pointer', 'overload_prefix', 'out_param_order', 'string_concat', 'param_comment', - 'test_only_api') + 'test_only_api', 'learned_rule_test', 'special_case') # File-level metrics: the tag goes in the first FILE_TAG_LINES lines. FILE_CHECKS = ('file_lines',) + tuple(HEADER_LIMITS) CHECKS = (('file_lines', 'function_lines', 'nesting', 'parameters') + tuple(CLASS_LIMITS) diff --git a/devtools/code_health/cpp_text.py b/devtools/code_health/cpp_text.py new file mode 100644 index 0000000000..bfe22fcf59 --- /dev/null +++ b/devtools/code_health/cpp_text.py @@ -0,0 +1,65 @@ +"""Comments and string literals of C++ source, with their line numbers. + +Shares `style.strip_code`'s lexing (comments, raw strings, digit separators), so a `//` inside +a string or a quote inside a comment is read the same way by every check.""" +import re + +from .style import strip_code + +STRING = re.compile(r'"( *)"') + + +def normalize(source): + return source.replace('\r\n', '\n').replace('\r', '\n') + + +def comment_lines(source): + """{line: comment text on that line}, the `//`, `/*`, `*/` and leading `*` removed. + + A character is comment text when `strip_code` blanked it and it lies outside the quotes + that `strip_code` keeps around literals.""" + raw_lines = normalize(source).split('\n') + code_lines = strip_code(normalize(source)) + found = {} + for number, (raw, code) in enumerate(zip(raw_lines, code_lines), 1): + if not code.strip() and raw.lstrip().startswith('#'): + continue # strip_code blanks preprocessor lines whole; they hold no comments we read + text, quote = [], None + for raw_char, code_char in zip(raw, code): + if code_char in '"\'' and quote in (None, code_char): + quote = None if quote else code_char + elif code_char == ' ' and raw_char != ' ' and quote is None: + text.append(raw_char) + elif text and text[-1] != ' ': + text.append(' ') + cleaned = re.sub(r'^\s*(?:/[/*]+|\*+(?!/))\s*|\*+/\s*$', '', ''.join(text).strip()).strip() + if cleaned: + found[number] = cleaned + return found + + +def comment_blocks(source): + """[(first line, {line: text})] for runs of consecutive comment lines.""" + blocks, current, last = [], {}, None + for number, text in sorted(comment_lines(source).items()): + if last is not None and number != last + 1: + blocks.append(current) + current = {} + current[number] = text + last = number + if current: + blocks.append(current) + return blocks + + +def string_literals(source): + """[(line, start column, end column, value)] for one-line string literals. + + Columns index the normalized line; `end` is one past the closing quote. Escapes stay as + written, which is enough for the identifier-like values the checks compare.""" + raw_lines = normalize(source).split('\n') + found = [] + for number, (raw, code) in enumerate(zip(raw_lines, strip_code(normalize(source))), 1): + for match in STRING.finditer(code): + found.append((number, match.start(), match.end(), raw[match.start() + 1:match.end() - 1])) + return found diff --git a/devtools/code_health/git_files.py b/devtools/code_health/git_files.py index 89615e908e..45e9f658d6 100644 --- a/devtools/code_health/git_files.py +++ b/devtools/code_health/git_files.py @@ -31,6 +31,21 @@ def changed_files(root, base_ref): return changes +def changed_paths(root, base_ref): + """[(status letter, current path, base path or None)] for every changed tracked file. + + Deleted files keep their base path as `current`; added files have no base path.""" + fields = iter(git(root, 'diff', '--name-status', '-z', '--find-renames', base_ref, '--').decode().split('\0')) + changes = [] + for status in fields: + if not status: + continue + original = next(fields) + current = next(fields) if status[0] in 'RC' else original + changes.append((status[0], current, None if status[0] in 'AC' else original)) + return changes + + UPSTREAMS = ('upstream/main', 'origin/main') diff --git a/devtools/code_health/learned_rules.py b/devtools/code_health/learned_rules.py new file mode 100644 index 0000000000..5d315783a9 --- /dev/null +++ b/devtools/code_health/learned_rules.py @@ -0,0 +1,180 @@ +"""A rule learned on the console needs a test: `learned_rule_test`. + +A comment in non-test C++ that cites a console finding, "(console B, 2026-11-10)" or +"(17.03, console B 2026-11-15)", records a rule the code learned from a run. That rule is kept +only if a test fails when it breaks (the user, 2026-11-15: "make sure our solutions are +systematic and avoid patching here and there"). So each citation needs a test that cites the +same finding: a `*_Tests.cpp` / `*_Tests.h` file, or a `ConsoleFailures/*.json` replay, that +names the same date and, when the citation names a checkpoint id, that checkpoint too. The test +lives in the same game's directory (`VideoGames//`, `PokemonBDSP/`), or in shared code +outside those, which tests every game. + +Reported, on lines this change adds: a citation no test covers, unless the base file cited the +same date (and checkpoint) already, so rewording an old comment is not a new rule. And for the whole tree: a +citation that a test covered at the base and none covers now, i.e. the change deleted or edited +the last test that cited it. A citation is "console B" or "lab-B", then within the same clause +(no `(`, `)`, `;` or sentence end between) a date; the checkpoint is a `NN.NN[a-z]` id in that +clause or in the parenthesis that holds the citation.""" +import re +import subprocess +from dataclasses import dataclass +from pathlib import PurePosixPath + +from .cpp_text import comment_blocks, normalize +from .exposure import test_file +from .findings import Finding +from .git_files import EXTENSIONS, git, included + +CHECK = 'learned_rule_test' +CITE = 'user, 2026-11-15: rules learned on the console are kept by tests' +CONSOLE = re.compile(r'\b(?:console|lab)[ -]B\b', re.I) +DATE = re.compile(r'\b20\d\d-[01]\d-[0-3]\d\b') +CHECKPOINT = re.compile(r'(? closing else '' + checkpoint = CHECKPOINT.search(before + ' ' + clause[:date.start()]) + line = max(number for offset, number in starts if offset <= console.start()) + found.append(Citation(path, line, date.group(), checkpoint.group() if checkpoint else '')) + return found + + +@dataclass(frozen=True) +class TestCite: + path: str + dates: frozenset + checkpoints: frozenset + + +def test_cite(path, text): + return TestCite(path, frozenset(DATE.findall(text)), frozenset(CHECKPOINT.findall(text))) + + +def covers(test, citation): + if test.path == citation.path or citation.date not in test.dates: + return False + if citation.checkpoint and citation.checkpoint not in test.checkpoints: + return False + home = area(citation.path) + return not home or area(test.path) in ('', home) + + +def covered(citation, tests): + return any(covers(test, citation) for test in tests) + + +def tracked_tests(root, revision=None): + """TestCites for every test file at `revision` (None: the working tree) that names a date.""" + args = ['grep', '-l', '-I', '-E', '20[0-9]{2}-[01][0-9]-[0-3][0-9]', + *([revision] if revision else []), '--', 'Common', 'SerialPrograms'] + result = subprocess.run(['git', '-C', str(root), *args], capture_output=True, text=True) + if result.returncode not in (0, 1): + raise ValueError(f'git grep for test citations failed: {result.stderr.strip()}') + tests = [] + for line in result.stdout.splitlines(): + path = line.split(':', 1)[1] if revision else line + if not is_test(path): + continue + if revision: + text = git(root, 'show', f'{revision}:{path}').decode('utf-8', 'replace') + else: + text = (root / path).read_text(encoding='utf-8', errors='replace') + tests.append(test_cite(path, text)) + return tests + + +def code_citations(root, revision=None): + """Citations in every non-test first-party C++ file at `revision` (None: the working tree).""" + args = ['grep', '-l', '-I', '-i', '-E', r'(console|lab)[ -]B', + *([revision] if revision else []), '--', 'Common', 'SerialPrograms'] + result = subprocess.run(['git', '-C', str(root), *args], capture_output=True, text=True) + if result.returncode not in (0, 1): + raise ValueError(f'git grep for console citations failed: {result.stderr.strip()}') + found = [] + for line in result.stdout.splitlines(): + path = line.split(':', 1)[1] if revision else line + if not included(path) or is_test(path) or PurePosixPath(path).suffix not in EXTENSIONS: + continue + if revision: + source = git(root, 'show', f'{revision}:{path}').decode('utf-8', 'replace') + else: + source = (root / path).read_text(encoding='utf-8', errors='replace') + found.extend(citations(path, source)) + return found + + +def message(citation, why='learned rule without a test'): + return (f'{why}: {citation.path}:{citation.line} {citation.label()}; add a test or a ' + f'ConsoleFailures replay that cites it ({CITE})') + + +def already_cited(citation, before): + """Whether the base file cited this finding already: a reworded comment is no new rule.""" + return any(old.date == citation.date and citation.checkpoint in ('', old.checkpoint) for old in before) + + +def check(path, source, previous_source, added, tests): + """Findings for new citations on `added` lines of a non-test C++ file that no test covers. + + A citation of a date (and checkpoint) the base file already cited is not new.""" + if is_test(path): + return [] + before = citations(path, normalize(previous_source)) if previous_source is not None else [] + return [Finding(path, c.line, message(c), check=CHECK) + for c in citations(path, normalize(source)) + if c.line in added and not already_cited(c, before) and not covered(c, tests)] + + +def lost_coverage(current_code, base_tests, tests): + """Findings for citations a base test covered that no current test covers.""" + return [Finding(c.path, c.line, message(c, 'learned rule lost its last test'), check=CHECK) + for c in current_code if covered(c, base_tests) and not covered(c, tests)] diff --git a/devtools/code_health/plan_lint.py b/devtools/code_health/plan_lint.py new file mode 100644 index 0000000000..88956baa7e --- /dev/null +++ b/devtools/code_health/plan_lint.py @@ -0,0 +1,91 @@ +"""Walks are planned: a hand-written `legs` event says why the planner can't walk it. + +A walk in a story plan is a planned route to a target the planner walks; a fixed list of legs +("east 1, south 6, north 1") is a recording of one run that breaks when anything differs +(the user, 2026-11-15, 17.03's hand-written legs: "not ok"). FRLG's step lint +(`test_legs_need_a_reason` in `PokemonFRLG_AutoStoryStepLint_Tests.cpp`) enforces this for +FRLG at test time; this check applies it to every game's `StoryData/*.plan.json` at review. + +An event is a legs event when its `kind` or `action` is "legs" or its `args` hold a "legs" +list (FRLG `kind: legs`, PLZA `action: walk` with `args.legs`). A new legs event, or one this +change edits, must say "not planned: " in its `why`. Events already at the base and +left unchanged are grandfathered; their count per game is printed with each run.""" +import json +import re +from pathlib import PurePosixPath + +from .findings import Finding + +NOT_PLANNED = 'not planned: ' +CITE = 'user, 2026-11-15: walks are planned routes' +PLAN = re.compile(r'(?:^|/)StoryData/[^/]+\.plan\.json$') + + +def is_plan(path): + return PLAN.search(path) is not None + + +def events(text): + """{event id: event} of a plan's checkpoint, or None when the text is not a plan.""" + try: + plan = json.loads(text) + except ValueError: + return None + checkpoint = plan.get('checkpoint') if isinstance(plan, dict) else None + listed = checkpoint.get('events') if isinstance(checkpoint, dict) else None + if not isinstance(listed, list): + return None + return {e['id']: e for e in listed if isinstance(e, dict) and isinstance(e.get('id'), str)} + + +def is_legs(event): + args = event.get('args') + return ('legs' in (event.get('kind'), event.get('action')) + or (isinstance(args, dict) and isinstance(args.get('legs'), list))) + + +def reasoned(event): + why = event.get('why') + return isinstance(why, str) and NOT_PLANNED in why + + +def game(path, text): + try: + name = json.loads(text).get('game') + except (ValueError, AttributeError): + name = None + return name if isinstance(name, str) else PurePosixPath(path).name.split('-')[0] + + +def id_line(text, event_id): + pattern = re.compile(r'"id"\s*:\s*' + re.escape(json.dumps(event_id))) + for number, line in enumerate(text.split('\n'), 1): + if pattern.search(line): + return number + return 1 + + +def check(path, text, previous_text): + """Findings for new or changed legs events in plan `path` without a "not planned:" reason.""" + if not is_plan(path): + return [] + current = events(text) + if current is None: + return [Finding(path, 1, 'plan lint: not a plan with checkpoint.events; cannot check its walks')] + before = (events(previous_text) if previous_text is not None else None) or {} + return [Finding(path, id_line(text, event_id), f'hand-written legs: {event_id} in {path} ' + f'has no "{NOT_PLANNED}" in its why; make it a planned route to its ' + f'target, or say why the planner cannot walk it ({CITE})') + for event_id, event in current.items() + if is_legs(event) and before.get(event_id) != event and not reasoned(event)] + + +def counts(plans): + """{game: (legs events, legs events with a reason)} over `plans`, [(path, text)].""" + totals = {} + for path, text in plans: + for event in (events(text) or {}).values(): + if is_legs(event): + legs, with_reason = totals.get(game(path, text), (0, 0)) + totals[game(path, text)] = (legs + 1, with_reason + reasoned(event)) + return totals diff --git a/devtools/code_health/special_cases.py b/devtools/code_health/special_cases.py new file mode 100644 index 0000000000..ca9b19f7ab --- /dev/null +++ b/devtools/code_health/special_cases.py @@ -0,0 +1,99 @@ +"""No per-checkpoint or per-event special cases in AutoStory code: `special_case`. + +`if (checkpoint.id == "17.03")` or `if (event.name == "right_of_takashi")` patches one story +point instead of fixing the rule that failed there; the next checkpoint with the same shape +fails the same way (the user, 2026-11-15: "make sure our solutions are systematic and avoid +patching here and there"). What differs per checkpoint or event belongs in its plan +(`StoryData/*.plan.json`); what is common belongs in a general rule. + +Checked: non-test C++ under an `AutoStory/` directory of `SerialPrograms/Source`, on the lines +a change adds. Reported: a string literal that is a checkpoint id (`NN.NN`, `NN.NNb`) or the id +of an event some plan declares, used as a key: compared with `==` / `!=` / `<=>`, an argument +of `find`, `count`, `contains`, `at`, `starts_with`, `ends_with`, `compare` or `[...]`, or the +first element of a brace pair `{"id", ...}` (a map entry). Not reported: plain lists of ids +(a checkpoint table), ids passed to other functions, and literals that are also an event's +`kind` or `action` (those name a general handler, not one event), and one-word event ids such as "bicycle", +which are also item or species names (`item == "bicycle"` in FRLG's item use).""" +import json +import re +import subprocess +from pathlib import PurePosixPath + +from .cpp_text import normalize, string_literals +from .exposure import test_file +from .findings import Finding + +CHECK = 'special_case' +CITE = 'user, 2026-11-15: systematic fixes, no patching here and there' +CHECKPOINT_ID = re.compile(r'\d\d\.\d\d[a-z]?') +COMPARE_BEFORE = re.compile(r'(?:==|!=|<=>)\s*$') +COMPARE_AFTER = re.compile(r'^\s*(?:==|!=|<=>)') +KEY_CALL = re.compile(r'(?:\.|->)\s*(?:find|count|contains|at|starts_with|ends_with|compare)\s*\(\s*$') +SUBSCRIPT = re.compile(r'\[\s*$') +MAP_ENTRY_BEFORE = re.compile(r'(?:^|[{,(=]\s*)\{\s*$') +MAP_ENTRY_AFTER = re.compile(r'^\s*,') +PLANS = 'StoryData/' + + +def autostory_code(path): + parts = PurePosixPath(path).parts + return (path.startswith('SerialPrograms/Source/') and 'AutoStory' in parts[:-1] + and PLANS.rstrip('/') not in parts and not test_file(path)) + + +def plan_names(texts): + """(event ids, kind and action names) declared by the plan JSON `texts`.""" + events, handlers = set(), set() + for text in texts: + try: + plan = json.loads(text) + except ValueError: + continue + for event in plan_events(plan): + if isinstance(event.get('id'), str): + events.add(event['id']) + for field in ('kind', 'action'): + if isinstance(event.get(field), str): + handlers.add(event[field]) + # A one-word id ("bicycle", "heal") is also an item, species or menu name the code may + # compare for its own reasons; an id with an underscore names one story point. + return {e for e in events - handlers if '_' in e}, handlers + + +def plan_events(plan): + checkpoint = plan.get('checkpoint') if isinstance(plan, dict) else None + events = checkpoint.get('events') if isinstance(checkpoint, dict) else None + return [e for e in events if isinstance(e, dict)] if isinstance(events, list) else [] + + +def tracked_plans(root): + """Texts of every `StoryData/*.plan.json` in the working tree.""" + result = subprocess.run(['git', '-C', str(root), 'ls-files', '--', '*StoryData/*.plan.json'], + capture_output=True, text=True) + if result.returncode != 0: + raise ValueError(f'git ls-files for plans failed: {result.stderr.strip()}') + return [(root / path).read_text(encoding='utf-8', errors='replace') + for path in result.stdout.splitlines() if (root / path).is_file()] + + +def key_use(line, start, end): + """Whether the literal at [start, end) of `line` is compared or looked up.""" + before, after = line[:start], line[end:] + return bool(COMPARE_BEFORE.search(before) or COMPARE_AFTER.match(after) + or KEY_CALL.search(before) or (SUBSCRIPT.search(before) and after.lstrip().startswith(']')) + or (MAP_ENTRY_BEFORE.search(before) and MAP_ENTRY_AFTER.match(after))) + + +def check(path, source, added, event_ids): + """Findings for checkpoint or event ids used as keys on `added` lines of AutoStory code.""" + if not autostory_code(path): + return [] + lines = normalize(source).split('\n') + findings = [] + for line, start, end, value in string_literals(source): + if line not in added or not (CHECKPOINT_ID.fullmatch(value) or value in event_ids): + continue + if key_use(lines[line - 1], start, end): + findings.append(Finding(path, line, f'special case by name: "{value}" at {path}:{line}; ' + f'move it into the plan or a general rule ({CITE})', check=CHECK)) + return findings diff --git a/devtools/code_health/tests/test_systematic.py b/devtools/code_health/tests/test_systematic.py new file mode 100644 index 0000000000..adbb33e723 --- /dev/null +++ b/devtools/code_health/tests/test_systematic.py @@ -0,0 +1,264 @@ +"""Rules that keep fixes systematic: learned rules have tests, no special cases by name, planned walks.""" +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from devtools.code_health import learned_rules, plan_lint, special_cases +from devtools.code_health.cpp_text import comment_lines, string_literals + + +REPO = Path(__file__).resolve().parents[3] +ENV = {key: value for key, value in os.environ.items() if key != 'GITHUB_ACTIONS'} +FRLG = 'SerialPrograms/Source/VideoGames/PokemonFRLG/Programs/AutoStory/' +PLZA = 'SerialPrograms/Source/VideoGames/PokemonLZA/Programs/AutoStory/' +SHARED = 'SerialPrograms/Source/Pokemon/AutoStory/' + + +class CppTextTests(unittest.TestCase): + def test_comments_skip_string_contents_and_keep_line_numbers(self): + source = ('int a = 1; // first\r\n' + 'std::string s = "// not a comment";\r\n' + '/* block\r\n * second */\r\n' + 'char c = \'"\'; // after a quote char\r\n') + self.assertEqual({1: 'first', 3: 'block', 4: 'second', 5: 'after a quote char'}, + comment_lines(source)) + + def test_string_literals_with_columns(self): + found = string_literals('if (id == "17.03"){ f("a // b"); }\n') + self.assertEqual([(1, 10, 17, '17.03'), (1, 22, 30, 'a // b')], found) + + +class CitationTests(unittest.TestCase): + def cites(self, comment): + return [(c.line, c.date, c.checkpoint) for c in + learned_rules.citations(FRLG + 'X.cpp', comment)] + + def test_citation_shapes_seen_in_the_tree(self): + cases = [ + ('// measured (console B, 2026-11-12).\n', [(1, '2026-11-12', '')]), + ('// partner is out (03.08, console B 2026-11-14).\n', [(1, '2026-11-14', '03.08')]), + ('// Console B, FRLG 07.02 run 2, 2026-11-09: it\n', [(1, '2026-11-09', '07.02')]), + ('// a fade (BDSP\'s Oreburgh Gate, mean 46, lab-B\n// 2026-11-08). Check\n', + [(1, '2026-11-08', '')]), + ('// (11.01 ride_bike, console B, 2026-11-12 05:42: "not riding")\n', + [(1, '2026-11-12', '11.01')]), + ('// 12.02b, 2026-11-14: no console named\n', []), + ('// measured on console B. The camera turns; 2026-11-14\n', []), + ('const char* s = "console B, 2026-11-14";\n', []), + ] + for comment, expected in cases: + with self.subTest(comment=comment): + self.assertEqual(expected, self.cites(comment)) + + +def cite(path, text): + return learned_rules.test_cite(path, text) + + +class LearnedRuleTests(unittest.TestCase): + SOURCE = '// A press during a slide is lost (17.03, console B, 2026-11-15).\nvoid f();\n' + + def findings(self, tests, previous=None, added=(1,)): + return learned_rules.check(FRLG + 'Walk.h', self.SOURCE, previous, set(added), tests) + + def test_new_citation_needs_a_test_of_its_date_and_checkpoint(self): + found = self.findings([]) + self.assertEqual(1, len(found)) + self.assertIn('learned rule without a test: ' + FRLG + 'Walk.h:1 (17.03, console B, 2026-11-15)', + found[0].message) + self.assertEqual('learned_rule_test', found[0].check) + # Another day, or the same day at another checkpoint, is another finding. + self.assertEqual(1, len(self.findings([cite(FRLG + 'Walk_Tests.cpp', '2026-11-14, 17.03')]))) + self.assertEqual(1, len(self.findings([cite(FRLG + 'Walk_Tests.cpp', '2026-11-15, 17.02')]))) + + def test_a_test_or_replay_of_the_same_game_or_shared_code_covers_it(self): + replay = FRLG + 'ConsoleFailures/17.03-21-12-slide.json' + for test in (cite(FRLG + 'Walk_Tests.cpp', '// 17.03, console B, 2026-11-15'), + cite(replay, json.dumps({'about': 'lab-B 2026-11-15', 'checkpoint': '17.03'})), + cite(SHARED + 'Engine_Tests.cpp', '2026-11-15 17.03')): + with self.subTest(test=test.path): + self.assertEqual([], self.findings([test])) + other_game = cite(PLZA + 'Walk_Tests.cpp', '17.03 2026-11-15') + self.assertEqual(1, len(self.findings([other_game]))) + + def test_old_and_untouched_citations_are_not_reported(self): + reworded = '// The slide loses presses (17.03, console B, 2026-11-15).\n' + self.assertEqual([], self.findings([], previous=reworded)) + self.assertEqual([], self.findings([], added=(2,))) + self.assertEqual([], learned_rules.check(FRLG + 'Walk_Tests.cpp', self.SOURCE, None, {1}, [])) + + def test_deleting_the_last_test_of_a_citation_fails(self): + code = learned_rules.citations(FRLG + 'Walk.h', self.SOURCE) + test = cite(FRLG + 'Walk_Tests.cpp', '17.03 2026-11-15') + lost = learned_rules.lost_coverage(code, [test], []) + self.assertEqual(1, len(lost)) + self.assertIn('learned rule lost its last test', lost[0].message) + self.assertEqual([], learned_rules.lost_coverage(code, [test], [test])) + self.assertEqual([], learned_rules.lost_coverage(code, [], [])) + + +class SpecialCaseTests(unittest.TestCase): + EVENTS = {'right_of_takashi', 'rival_takes_bulbasaur'} + + def findings(self, line, path=FRLG + 'PokemonFRLG_AutoStoryWalk.cpp'): + return [f.message.split('"')[1] for f in special_cases.check(path, line + '\n', {1}, self.EVENTS)] + + def test_ids_used_as_keys_are_reported(self): + cases = [ + 'if (checkpoint.id == "17.03"){', + 'if ("12.02b" != row.id){', + 'return step.name == "rival_takes_bulbasaur";', + 'auto it = waits.find("right_of_takashi");', + 'const auto& w = waits["right_of_takashi"];', + 'const std::map WAITS{{"17.03", 2}, {"right_of_takashi", 1}};', + ] + expected = [['17.03'], ['12.02b'], ['rival_takes_bulbasaur'], ['right_of_takashi'], + ['right_of_takashi'], ['17.03', 'right_of_takashi']] + for line, names in zip(cases, expected): + with self.subTest(line=line): + self.assertEqual(names, self.findings(line)) + + def test_lists_calls_other_strings_tests_and_other_code_pass(self): + for line in ('"01.01", "01.02", "01.03",', + 'first = checkpoint_index(checkpoints, "01.02");', + 'if (item == "bicycle"){', + 'if (action == "taps"){', + 'log("17.03 done");'): + with self.subTest(line=line): + self.assertEqual([], self.findings(line)) + line = 'if (checkpoint.id == "17.03"){' + for path in (FRLG + 'PokemonFRLG_AutoStoryWalk_Tests.cpp', FRLG + 'StoryData/x.cpp', + 'SerialPrograms/Source/VideoGames/PokemonFRLG/Programs/PokemonFRLG_Rng.cpp'): + with self.subTest(path=path): + self.assertEqual([], self.findings(line, path)) + + def test_event_ids_come_from_plans_without_handler_names_or_one_word_ids(self): + plan = {'checkpoint': {'id': '17.03', 'events': [ + {'id': 'right_of_takashi', 'kind': 'route', 'action': 'run'}, + {'id': 'bicycle', 'kind': 'game_action', 'action': 'use_item'}, + {'id': 'use_item', 'kind': 'step'}, + ]}} + events, handlers = special_cases.plan_names([json.dumps(plan), 'not json']) + self.assertEqual({'right_of_takashi'}, events) + self.assertEqual({'route', 'run', 'game_action', 'use_item', 'step'}, handlers) + + +def plan(events, game='frlg'): + return json.dumps({'game': game, 'checkpoint': {'id': '17.03', 'events': events}}, indent=2) + + +class PlanLintTests(unittest.TestCase): + PATH = FRLG + 'StoryData/frlg-17.03.plan.json' + FRLG_LEGS = {'id': 'right_of_takashi', 'kind': 'legs', 'action': 'taps', + 'args': {'legs': [{'direction': 'east', 'tiles': 1}]}, 'why': 'east then north'} + PLZA_WALK = {'id': 'to_the_gate', 'kind': 'game_action', 'action': 'walk', + 'args': {'legs': [[0, 4500], [90, 10000]]}, 'why': 'up the road'} + ROUTE = {'id': 'below_giovanni', 'kind': 'route', 'action': 'run', 'args': {'target': {}}, 'why': 'planned'} + + def ids(self, text, previous=None, path=PATH): + return [f.message.split()[2] for f in plan_lint.check(path, text, previous)] + + def test_new_legs_events_need_a_not_planned_reason(self): + self.assertEqual(['right_of_takashi', 'to_the_gate'], + self.ids(plan([self.FRLG_LEGS, self.PLZA_WALK, self.ROUTE]))) + reasoned = dict(self.FRLG_LEGS, why='not planned: a tap beside a wall where a run overshoots') + self.assertEqual([], self.ids(plan([reasoned, self.ROUTE]))) + + def test_unchanged_legs_at_the_base_are_grandfathered_and_edits_are_not(self): + base = plan([self.FRLG_LEGS, self.ROUTE]) + self.assertEqual([], self.ids(plan([self.FRLG_LEGS, self.ROUTE, dict(self.ROUTE, id='b')]), base)) + edited = dict(self.FRLG_LEGS, args={'legs': [{'direction': 'east', 'tiles': 2}]}) + self.assertEqual(['right_of_takashi'], self.ids(plan([edited, self.ROUTE]), base)) + + def test_finding_line_other_files_and_counts(self): + text = plan([self.ROUTE, self.FRLG_LEGS]) + line = plan_lint.check(self.PATH, text, None)[0].line + self.assertIn('"id": "right_of_takashi"', text.split('\n')[line - 1]) + self.assertEqual([], self.ids(plan([self.FRLG_LEGS]), path=FRLG + 'StoryData/maps.json')) + self.assertEqual(['not'], self.ids('{"checkpoint": 3}')) + counts = plan_lint.counts([(self.PATH, plan([self.FRLG_LEGS, self.ROUTE])), + (PLZA + 'StoryData/plza-01.01.plan.json', plan([self.PLZA_WALK], 'plza'))]) + self.assertEqual({'frlg': (1, 0), 'plza': (1, 0)}, counts) + + +class SystematicCliTests(unittest.TestCase): + """The three rules through the command line, in a temporary Git repository.""" + + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + for args in (('init', '-q'), ('config', 'user.email', 'test@example.invalid'), + ('config', 'user.name', 'Test')): + self.git(*args) + + def git(self, *args): + return subprocess.check_output(['git', '-C', str(self.root), *args]) + + def write(self, path, text): + file = self.root / path + file.parent.mkdir(parents=True, exist_ok=True) + file.write_text(text) + self.git('add', '-A') + + def commit(self): + self.git('commit', '-qm', 'base revision') + + def cli(self): + return subprocess.run([sys.executable, '-m', 'devtools.code_health', '--root', str(self.root), + '--base-ref', 'HEAD'], cwd=REPO, capture_output=True, text=True, env=ENV) + + def test_learned_rule_without_a_test_fails_until_a_test_cites_it(self): + self.write(FRLG + 'Walk.h', '#pragma once\n') + self.commit() + self.write(FRLG + 'Walk.h', '#pragma once\n// Presses in a slide are lost (console B, 2026-11-15).\n') + result = self.cli() + self.assertEqual(1, result.returncode, result.stdout + result.stderr) + self.assertIn('learned rule without a test: ' + FRLG + 'Walk.h:2 (console B, 2026-11-15)', result.stderr) + self.write(FRLG + 'Walk_Tests.cpp', '// The slide (console B, 2026-11-15).\n') + result = self.cli() + self.assertEqual(0, result.returncode, result.stdout + result.stderr) + + def test_deleting_the_last_test_fails_and_a_tag_waives_it(self): + self.write(FRLG + 'Walk.h', '// Presses in a slide are lost (console B, 2026-11-15).\n') + self.write(FRLG + 'Walk_Tests.cpp', '// The slide (console B, 2026-11-15).\n') + self.commit() + self.git('rm', '-q', FRLG + 'Walk_Tests.cpp') + result = self.cli() + self.assertEqual(1, result.returncode, result.stdout + result.stderr) + self.assertIn('learned rule lost its last test: ' + FRLG + 'Walk.h:1', result.stderr) + self.write(FRLG + 'Walk.h', '// code-health: allow learned_rule_test -- the image test for this lives ' + 'in CommandLineTests, outside this repository\n' + '// Presses in a slide are lost (console B, 2026-11-15).\n') + result = self.cli() + self.assertEqual(0, result.returncode, result.stdout + result.stderr) + self.assertIn('learned_rule_test bypassed by the tag at line 1', result.stdout) + + def test_special_case_by_event_name_fails(self): + self.write(FRLG + 'StoryData/frlg-17.03.plan.json', + plan([{'id': 'right_of_takashi', 'kind': 'route', 'action': 'run', 'why': 'planned'}])) + self.write(FRLG + 'Walk.cpp', 'bool f(){\n return true;\n}\n') + self.commit() + self.write(FRLG + 'Walk.cpp', 'bool f(const Step& step){\n return step.name == "right_of_takashi";\n}\n') + result = self.cli() + self.assertEqual(1, result.returncode, result.stdout + result.stderr) + self.assertIn('special case by name: "right_of_takashi" at ' + FRLG + 'Walk.cpp:2', result.stderr) + + def test_plan_lint_blocks_new_legs_and_reports_counts(self): + legs = PlanLintTests.FRLG_LEGS + self.write(FRLG + 'StoryData/frlg-17.03.plan.json', plan([legs])) + self.commit() + self.write(FRLG + 'StoryData/frlg-17.03.plan.json', plan([legs, dict(legs, id='second')])) + result = self.cli() + self.assertEqual(1, result.returncode, result.stdout + result.stderr) + self.assertIn('hand-written legs: second in', result.stderr) + self.assertNotIn('hand-written legs: right_of_takashi', result.stderr) + self.assertIn('Plan lint: frlg: 2 legs events in the changed plans, 0 with', result.stdout) + + +if __name__ == '__main__': + unittest.main()