Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions devtools/code_health/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,13 +97,14 @@ function or class exceed their limit. Checks are the metric names in the table a
`file_lines`, `function_lines`, `nesting`, `parameters`, `public_methods`, `public_data`,
`single_method_interface`, `header_symbols`. Six style findings take a tag too: `char_pointer`,
`overload_prefix`, `out_param_order`, `string_concat`, `param_comment` and `test_only_api`, for a C API
or a fixed interface that needs them. Other style and layering
or a fixed interface that needs them; so do `learned_rule_test` and `special_case` (see
[Systematic fixes](#systematic-fixes)). Other style and layering
findings have no bypass.

- **Place it** on the function's or class's first line, or in the comment block directly above
it (only `//` comment and `template` lines between). A `file_lines` tag goes in the first 20
lines of the file, and so does a `header_symbols` tag. A style tag (`char_pointer`, `overload_prefix`,
`out_param_order`, `string_concat`, `param_comment`, `test_only_api`) goes on the flagged line or in the comment block directly above it, and waives the finding whether it is an error or advisory. A tag that attaches to nothing fails. Style checks report only changed lines, so a style tag on a line the change left alone (a later PR in a stack) attaches if it covers a finding of the whole file.
`out_param_order`, `string_concat`, `param_comment`, `test_only_api`, `learned_rule_test`, `special_case`) goes on the flagged line or in the comment block directly above it, and waives the finding whether it is an error or advisory. A tag that attaches to nothing fails. Style checks report only changed lines, so a style tag on a line the change left alone (a later PR in a stack) attaches if it covers a finding of the whole file.
- **Justify it:** the text after `--`, on the tag line itself, must be at least 40 characters and 6 words,
and say why splitting would make the code worse. Unknown checks, a missing or short
justification, or a malformed tag fail.
Expand Down Expand Up @@ -173,6 +174,29 @@ non-test caller later in the stack (API added one PR before its caller), and ups
`Common/ControllerStates` C-style headers (constants and `*_equals` used only by their own .cpp)
are reported too.

## Systematic fixes

Three blocking rules keep a console fix a rule with a test, not a patch at one story point (the
user, 2026-11-15: "make sure our solutions are systematic and avoid patching here and there").
Like the style checks they look only at what a change adds.

| Check | Level | Rule |
| --- | --- | --- |
| `learned_rule_test` (`learned_rules.py`) | error (tag) | A comment in non-test C++ that cites a console finding, `(console B, 2026-11-10)` or `(17.03, console B 2026-11-15)` ("console B" or "lab-B", then a date in the same clause), needs a test that cites the same date, and the checkpoint when the citation names one: a `*_Tests.cpp`/`.h` or a `ConsoleFailures/*.json` replay, in the same game's directory or in shared code. Reported on added lines as `learned rule without a test: <file:line> (console B, <date>)`, unless the base file cited that date already (a reworded comment). Also reported: a change to test files after which a citation anywhere in the tree has no covering test while it had one at the base (`learned rule lost its last test`). |
| `special_case` (`special_cases.py`) | error (tag) | In non-test C++ under `SerialPrograms/Source/**/AutoStory/`: a string literal that is a checkpoint id (`NN.NN[a-z]`) or the id of an event a `StoryData/*.plan.json` declares, used as a key: `==`, `!=`, `<=>`, `.find/count/contains/at/starts_with/ends_with/compare(...)`, `[...]`, or the first element of `{"id", ...}`. Not reported: plain id lists, ids passed to other functions, one-word event ids (`"bicycle"` is also an item) and ids that are also a `kind`/`action` name. Message: `special case by name: "<literal>" at <file:line>; move it into the plan or a general rule`. |
| Plan lint (`plan_lint.py`) | error | Every `StoryData/*.plan.json` (any game): a new or edited event whose `kind`/`action` is `legs` or whose `args` hold a `legs` list (FRLG `kind: legs`, PLZA `action: walk`) needs `not planned: <reason>` in its `why`. Events unchanged since the base are grandfathered; each run prints the changed plans' legs counts per game. The reason in the `why` is the bypass; there is no tag in JSON. |

Measured on 2026-11-15 (whole trees, every line counted as new): `learned_rule_test` 1 of 250
citations uncovered on `frlg/moved-trainers` (`ScreenFades.h`, 2026-11-08) and 12 of 41 on
`plza/newgame` (PLZA 2026-11-10 notes: no PLZA test cites that day); `special_case` 1
(`PokemonFRLG_AutoStoryRngLoop.cpp`, `step.name == "rival_takes_bulbasaur"`); plan lint 134 FRLG
legs events (FRLG's `legs_before_2026_11_15()` list) and 26 PLZA walks, none with a reason. On the
diffs of fork #599, #604 and #610 against their bases all three report nothing. The date match is
coarse: any test of the same game that names the same day covers a citation, so the rule catches
a learned rule from a day no test cites, not a rule whose own test is missing on a busy day.
Detector and reader measurements whose image tests live in `CommandLineTests` take a
`learned_rule_test` tag saying so.

## Build and native C++ tests in one command

The existing `.github/workflows/cpp-ci-serial-programs-base.yml` already runs
Expand Down
88 changes: 81 additions & 7 deletions devtools/code_health/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@
import sys
from pathlib import Path

from . import allow, exposure, header_api, layering, param_comments, style
from . import allow, exposure, header_api, layering, learned_rules, param_comments, plan_lint, special_cases, style
from .classes import measure_classes
from .cpp_text import normalize
from .findings import annotations
from .git_files import changed_files, default_base, read_revision
from .git_files import changed_files, changed_paths, default_base, read_revision
from .metrics import measure
from .policy import advisories, bypasses, evaluate

Expand All @@ -17,23 +18,88 @@ def analyze(path, source):
return measure(path, source) + measure_classes(path, source) + header_api.measure_header(path, source)


def style_findings(path, source, previous_source, users):
class Repository:
"""Whole-tree facts some line checks need, each read once and only when asked for."""

def __init__(self, root):
self.root = root
self.users = exposure.git_users(root)
self._tests = None
self._event_ids = None

@property
def tests(self):
if self._tests is None:
self._tests = learned_rules.tracked_tests(self.root)
return self._tests

@property
def event_ids(self):
if self._event_ids is None:
self._event_ids, _ = special_cases.plan_names(special_cases.tracked_plans(self.root))
return self._event_ids


def style_findings(path, source, previous_source, repo):
"""(style errors, style advisories) on lines new since `previous_source` (None: all lines)."""
errors, notes = style.check(path, source, previous_source)
errors += header_api.prefix_overloads(path, source, previous_source)
# Advisory: both report many sites a reviewer may accept (pbf_* timings, API for a
# later PR in a stack); see the README's Style table for the measured rates.
notes += param_comments.check(path, source, previous_source)
notes += exposure.check(path, source, previous_source, users)
notes += exposure.check(path, source, previous_source, repo.users)
added = style.added_lines(normalize(source).split('\n'),
None if previous_source is None else normalize(previous_source).split('\n'))
if special_cases.autostory_code(path):
errors += special_cases.check(path, source, added, repo.event_ids)
if learned_rules.CONSOLE.search(source):
errors += learned_rules.check(path, source, previous_source, added, repo.tests)
return errors, notes


def lost_tests(root, base_ref, paths, repo):
"""Failures for console citations whose last covering test this change removed or edited."""
changed = {current: original for _, current, original in paths
if learned_rules.is_test(current) or (original and learned_rules.is_test(original))}
if not changed:
return [], []
base_tests = [t for t in repo.tests if t.path not in changed]
for original in filter(None, changed.values()):
try:
base_tests.append(learned_rules.test_cite(original, read_revision(root, base_ref, original)))
except subprocess.CalledProcessError:
continue
found = learned_rules.lost_coverage(learned_rules.code_citations(root), base_tests, repo.tests)
kept, waived = [], []
for path in sorted({f.path for f in found}):
source = (root / path).read_text(encoding='utf-8-sig')
allowances, _ = allow.parse(path, source)
left, notices, _ = allow.waive(path, source, allowances, [f for f in found if f.path == path])
kept += left
waived += notices
return kept, waived


def plan_findings(root, base_ref, paths):
"""(failures for unexplained new legs events in changed plans, per-game legs counts)."""
failures, plans = [], []
for status, current, original in paths:
if status == 'D' or not plan_lint.is_plan(current):
continue
text = (root / current).read_text(encoding='utf-8-sig')
previous = read_revision(root, base_ref, original) if original and plan_lint.is_plan(original) else None
failures += plan_lint.check(current, text, previous)
plans.append((current, text))
return failures, plan_lint.counts(plans)


def run(root, base_ref=None, github=False):
if base_ref is None:
base_ref, how = default_base(root)
print(f'Base: {base_ref} ({how}).')
changes = changed_files(root, base_ref)
users = exposure.git_users(root)
paths = changed_paths(root, base_ref)
repo = Repository(root)
failures, notes, bypassed = [], [], []
advisory = 0
for current, original in changes:
Expand All @@ -50,17 +116,25 @@ def run(root, base_ref=None, github=False):
notices, unneeded = bypasses(metrics, allowed)
bypassed.extend(notices)
notes.extend(unneeded)
style_errors, style_notes = style_findings(current, source, previous_source, users)
style_errors, style_notes = style_findings(current, source, previous_source, repo)
# Waive errors and advisories in one pass, so a tag on either counts as used. A tag on
# a line this change left alone attaches to the whole file's findings.
kept, waived, unattached = allow.waive(
current, source, allowances, style_errors + style_notes,
whole_file=lambda: sum(style_findings(current, source, None, users), []))
whole_file=lambda: sum(style_findings(current, source, None, repo), []))
failures.extend([f for f in kept if f in style_errors] + unattached)
bypassed.extend(waived)
notes.extend(f for f in kept if f not in style_errors)
advisory += sum(m.check == 'file_lines' and 500 < m.value <= 1000 for m in metrics)
lost, lost_waived = lost_tests(root, base_ref, paths, repo)
failures.extend(lost)
bypassed.extend(lost_waived)
plan_failures, legs = plan_findings(root, base_ref, paths)
failures.extend(plan_failures)
print(f'Scanned {len(changes)} changed C++ files against {base_ref}.')
for name, (total, with_reason) in sorted(legs.items()):
print(f'Plan lint: {name}: {total} legs events in the changed plans, {with_reason} with a '
'"not planned:" reason; only new or edited ones must have one.')
print(f'Advisory: {advisory} changed files exceed 500 lines; prefer functions <=25 lines.')
for note in notes:
print(f'Advisory: {note}')
Expand Down
4 changes: 2 additions & 2 deletions devtools/code_health/allow.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
that function's or class's listed metrics exceed their limit up to a hard ceiling. A
`file_lines` or `header_symbols` tag goes in the first FILE_TAG_LINES lines. A STYLE_CHECKS tag
(`char_pointer`, `overload_prefix`, `out_param_order`, `string_concat`, `param_comment`,
`test_only_api`) waives that style finding
`test_only_api`, `learned_rule_test`, `special_case`) waives that line finding
on its own line or the line below its comment block. Every bypass is printed, quoting its
justification, so reviewers see it; malformed, unknown or misplaced tags fail."""
import re
Expand All @@ -15,7 +15,7 @@

# Style findings a tag can bypass; they attach like a function's tag, to the finding's line.
STYLE_CHECKS = ('char_pointer', 'overload_prefix', 'out_param_order', 'string_concat', 'param_comment',
'test_only_api')
'test_only_api', 'learned_rule_test', 'special_case')
# File-level metrics: the tag goes in the first FILE_TAG_LINES lines.
FILE_CHECKS = ('file_lines',) + tuple(HEADER_LIMITS)
CHECKS = (('file_lines', 'function_lines', 'nesting', 'parameters') + tuple(CLASS_LIMITS)
Expand Down
65 changes: 65 additions & 0 deletions devtools/code_health/cpp_text.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
"""Comments and string literals of C++ source, with their line numbers.

Shares `style.strip_code`'s lexing (comments, raw strings, digit separators), so a `//` inside
a string or a quote inside a comment is read the same way by every check."""
import re

from .style import strip_code

STRING = re.compile(r'"( *)"')


def normalize(source):
return source.replace('\r\n', '\n').replace('\r', '\n')


def comment_lines(source):
"""{line: comment text on that line}, the `//`, `/*`, `*/` and leading `*` removed.

A character is comment text when `strip_code` blanked it and it lies outside the quotes
that `strip_code` keeps around literals."""
raw_lines = normalize(source).split('\n')
code_lines = strip_code(normalize(source))
found = {}
for number, (raw, code) in enumerate(zip(raw_lines, code_lines), 1):
if not code.strip() and raw.lstrip().startswith('#'):
continue # strip_code blanks preprocessor lines whole; they hold no comments we read
text, quote = [], None
for raw_char, code_char in zip(raw, code):
if code_char in '"\'' and quote in (None, code_char):
quote = None if quote else code_char
elif code_char == ' ' and raw_char != ' ' and quote is None:
text.append(raw_char)
elif text and text[-1] != ' ':
text.append(' ')
cleaned = re.sub(r'^\s*(?:/[/*]+|\*+(?!/))\s*|\*+/\s*$', '', ''.join(text).strip()).strip()
if cleaned:
found[number] = cleaned
return found


def comment_blocks(source):
"""[(first line, {line: text})] for runs of consecutive comment lines."""
blocks, current, last = [], {}, None
for number, text in sorted(comment_lines(source).items()):
if last is not None and number != last + 1:
blocks.append(current)
current = {}
current[number] = text
last = number
if current:
blocks.append(current)
return blocks


def string_literals(source):
"""[(line, start column, end column, value)] for one-line string literals.

Columns index the normalized line; `end` is one past the closing quote. Escapes stay as
written, which is enough for the identifier-like values the checks compare."""
raw_lines = normalize(source).split('\n')
found = []
for number, (raw, code) in enumerate(zip(raw_lines, strip_code(normalize(source))), 1):
for match in STRING.finditer(code):
found.append((number, match.start(), match.end(), raw[match.start() + 1:match.end() - 1]))
return found
15 changes: 15 additions & 0 deletions devtools/code_health/git_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,21 @@ def changed_files(root, base_ref):
return changes


def changed_paths(root, base_ref):
"""[(status letter, current path, base path or None)] for every changed tracked file.

Deleted files keep their base path as `current`; added files have no base path."""
fields = iter(git(root, 'diff', '--name-status', '-z', '--find-renames', base_ref, '--').decode().split('\0'))
changes = []
for status in fields:
if not status:
continue
original = next(fields)
current = next(fields) if status[0] in 'RC' else original
changes.append((status[0], current, None if status[0] in 'AC' else original))
return changes


UPSTREAMS = ('upstream/main', 'origin/main')


Expand Down
Loading
Loading