[gapilongo]

FIX-2: decision policy v2.1. Catalogued unmapped noise, and authentication failures as a never-close class

Closes #342. Tier T3: this changes the decision policy and INV-03. Follows #340 and #341.

Scope change after review: the campaign split is deferred. #342 asked for a campaign check that tells one rule repeating apart from an attack. Three independent review rounds each found a way to launder a block through it:

1. through a sibling that FIX-1 would block;
2. one hop: a kill-chain step leaves the window, and the sibling it blocked looks clean;
3. two or more hops.

The lead decided to drop it from this PR. FIX-1's campaign check stays exactly as on main: a burst, related_threats or related_open_reviews blocks. No new correlation digest fields ship.

What changed

1. Noise catalogue (src/lg_sotf/domain/noise_families.py, version 2026-11-09.2).

   • It is reviewed and versioned, and keyed by vendor and detection rule id.
   • The rule id is read from the vendor's own event (raw.rule.id, digits only). It counts only when both hold:
     • the Wazuh normalizer built the alert: the ingestion service records CanonicalAlert.normalizer, never taken from the event, and model_view keeps it out of prompts;
     • the canonical rule name is that normalizer's "<id>: <description>".
   • Title, description and rule-name text are never matched.
   • A catalogued rule lifts only the unmapped class (closure.model_close_class). Every other check still applies: the tier, severity, both model tiers benign at 8/85, steering text, a recent true positive and FIX-1's campaign check.
   • A model close that used the catalogue names the family and the catalogue version in its reason.

   Family	Wazuh rule ids	Why it is routine (ruleset semantics)	Abuse it could hide	Abuse test
   linux_package_install	2901, 2902 (dpkg), 2932, 2933 (yum)	Package-manager log of installs and updates. Removals (2903, 2934) are left out: removing a security agent is T1562.001.	A malicious or backdoored package (T1546.016, T1195). Any local user can also forge a syslog line (logger -t yum), so a repeat can be a decoy.	yes
   windows_software_install	60612	Windows Installer's record of a completed install. The product name is never read.	A malicious MSI run through msiexec (T1218.007) or a remote-access tool (T1219). A per-user install needs no admin rights.	yes
   antivirus_status	62107, 62108, 62130	Defender scan and signature telemetry with no finding.	None known	n/a
   web_client_error	31101	A single 4xx answer. Attack signatures, repeated 4xx (31151, level 10 = medium) and 5xx are separate rules and are not catalogued.	Probing, or a failed exploit (T1595, T1190)	yes

   These entries were chosen from the ruleset's meaning, not from corpus labels. For example, 554 and 31120 are benign in parts of corpus_v1 but are left out.

2. Authentication failures are a never-close class whatever their ATT&CK id (review H1). Each of these counts as credential_access:

   • T1531;
   • a Wazuh rule in the groups authentication_failed, authentication_failures, invalid_login or login_denied;
   • rule ids 6250 and 44733.

   These checks only block, so they read the raw event whatever format built the alert.

3. State cap (cap_alert).

   • When it drops an oversized raw event, the cap keeps a shortened raw.rule (id, level, description, groups, mitre).
   • A Wazuh-built alert whose rule can't be read is the new never-close class vendor_unknown, which also blocks the catalogue lift.
   • So is one whose groups list was cut, past 20 in the kept subset or past 50 in a raw event kept whole: the list is marked groups_truncated.

4. INV-03 text and proofs, plus docs: threat-model.md (TM-WRK-2, with corrected residual risk), architecture.md, security.md, ai-system-card.md, changelog.d/FIX-2.md, and the closure, routing and catalogue docstrings.

   • CanonicalAlert.normalizer is new. Alerts stored before this release have none, so the catalogue does not apply to them (fail closed).
   • openapi.json and schema.gen.ts are unchanged against main.

Review fixes kept

Finding	Fix	Proof
H1: authentication failures outside credential access	T1531, the auth-failure groups, ids 6250 and 44733	test_noise_families.py::test_review_h1_*, covering 60122, 60130, 99919, 60196, 5302, 6250 and 44733 through the real Wazuh normalizer
L1: vendor from raw shape	normalizer recorded at ingestion; the catalogue requires wazuh	test_noise_families.py; test_ingest.py::test_a_pushed_event_cannot_claim_the_wazuh_normalizer
L2 / L3	Abuse and residual-risk text corrected: 60204 is frequency-based, and 31151 is level 10 = medium	catalogue entries, threat model
Dropped raw, more than 20 groups	raw.rule subset kept; vendor_unknown	test_an_oversized_raw_event_keeps_its_rule_through_the_state_cap, test_recheck_a_wazuh_alert_without_a_readable_rule_never_closes, test_final_a_cut_groups_list_never_closes
M1, L4, L5, the laundering findings	n/a: they concerned the deferred campaign split

Measured (HEAD)

Fake model. Eval harness plus the dataset gates, corpus_v1 replayed from its committed cassettes. No cassette changed.

Dataset (benign / malicious)	Tier off: benign / malicious closed	auto_full_audit: benign closed (95% Wilson CI)	auto_full_audit: malicious closed	Gate
corpus_v1 (312 / 208)	0 / 0	0/312 (0.0%; 0.0-1.2%)	0	pass
adversarial_v2 (24 / 71)	3 / 0	3/24 (12.5%; 4.3-31.0%)	0	pass
synthetic_v1 (21 / 35)	5 / 0	5/21 (23.8%; 10.6-45.1%)	0	pass
phishing_v1 (11 / 24)	3 / 0	3/11 (27.3%; 9.7-56.6%)	0	pass

DeepSeek, dev run 2 (100 Wazuh alerts, FIX-1 deployed), simulated with the real decide, the run's own correlation digests (FIX-1 campaign check) and the raw event from corpus_v1:

	Benign closed by the model	Malicious closed
Tier off	0/60	0/40
auto_full_audit	2/60 (3.3%; 0.9-11.4%): windows_software_install, antivirus_status	0/40

Blockers on the 27 benign alerts that reach the model-close check:

Blocker	FIX-1	v2.1
never_close_class	12	3
campaign_in_progress	20	20
fast_tier_not_benign	18	18
deep_tier_not_benign	18	18

The campaign check and the model tiers remain the main blockers. The scorecard corpus_v1 simulation and the fooled-model worst case were not rerun for this scope.

Threat note

• What can go wrong: an actor on a host picks which catalogued rule fires, often without privileges (logger -t yum, a per-user MSI), and both model tiers are persuaded it is routine.
• Mitigations:
  • the catalogue lifts only unmapped, and keys on the Wazuh normalizer's rule id, never on text;
  • authentication failures are never-close by vendor group, listed id and T1531;
  • an unreadable or cut rule is vendor_unknown;
  • every other condition still holds, including FIX-1's campaign check;
  • every such close is model_closed / ai_closed:audit_required and names its family and catalogue version;
  • abuse tests cover each abusable family.
• Residual risk:
  • both tiers come from one model family;
  • the rule id and groups are trusted as far as the vendor event;
  • Wazuh's frequency rules are no backstop for a slow attack (60204 needs 8 failures from one IP in 240 s);
  • FIX-1's campaign check still blocks most closes on busy hosts. That is the trade-off of deferring the split.
• Follow-ups:
  • a campaign split that holds under review: a design that carries the reason for a block, not hop-by-hop digests;
  • dev rerun of the 100 Wazuh alerts.

Merged with main (verdict cache, #344)

• INV-03 keeps both texts.
• Because authentication failures are now a never-close class, they get no verdict key:
  • the analyst-oracle replay on corpus_v1 closes 4 benign alerts instead of 9 (still 0 attacks);
  • the verdict-key replay closes 14 instead of more than 15.
• Both test floors are updated, with a comment.

Checks

• ruff, mypy (375 files), lint-imports (14 kept), changelog check.
• pytest tests/unit tests/invariants tests/deploy after the merge: 5137 passed.
• tests/integration eval, engine and ingestion after the merge: 130 passed, 1 skipped, 1 failed. The failure is the phishing_v1 cassette replay, which fails on main too ([BUG] phishing_v1 cassette replay is time-dependent: 4 items miss after some hours #346, fixed separately). The two oracle tests pass after the floor update.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JmdYVXkgpNHoygVJCrUeZc

[claude]

Review of #345 (FIX-2). I read noise_families.py, closure.py and correlation/base.py; I did not run the tests.

Overall: a careful, fail-closed design that matches the invariants it documents. The catalogue is keyed on vendor rule id (never title or description text), lifts only unmapped, and never lifts a mapped never-close class. Missing raw events, bad counts and incomplete related sets all block the close. Docs, OpenAPI, the TS schema and the changelog moved with the code, and the tests are thorough.

Suggestions

1. quiet_rule_repeat accepts a deep verdict of None (never analysed) as quiet. That matches the docs, but such an alert then vouches for a burst. Require an explicit benign verdict or a terminal status, or add a test that pins the choice.
2. quiet_rule_repeat checks fast-tier threat_assessment == benign without llm_used, whereas _fast_tier_benign requires llm_used is True for the current alert. A rules-only benign is weaker evidence, so the two checks should be consistent.
3. related_count is len(related), but the tally runs over related_alerts(...), which can return fewer rows. This fails closed, since repeats won't equal related, but a comment or test for a missing row would make it explicit.
4. Forged events (for example a dpkg.log line that fires 2902) are accepted by design, so the defence rests on both model tiers being benign at 8/85. A per-tenant opt-out of individual families would be worth considering.

Tests to add: a related alert with a missing deep verdict, a related alert with rules-only benign triage, and raw.rule.id as an int or bool in a related alert's raw event.

No security blockers found. I'd address 1 and 2 before merge.

[claude]

Review of PR #345 (FIX-2: noise catalogue and campaign split)

I read the diff for the source, the ingestion change, the invariants and the tests. I haven't run the test suite, and the docs and changelog files got only a skim.

Summary

The design is careful. A catalogued rule lifts only unmapped, and the rule id is trusted only when the ingestion service recorded normalizer == "wazuh". Unknown, missing or inconsistent counts all fail closed. The tests are strong. They cover forged pushes (test_a_pushed_event_cannot_claim_the_wazuh_normalizer), the escalation of malicious verdicts per family, incomplete related sets, and an end-to-end engine run. I found no blocking bugs. The points below are worth a look before merge.

Security and policy (INV-03)

1. Repeat flooding of a catalogued rule. Up to 9 quiet repeats of the same rule can stand behind a model-led close. For 31101 (web 4xx) and the 2932/2933 rules, anyone who can send a request or write a syslog line can create those repeats. Slow probing that stays under the 31151 threshold is T1595 or T1190, and each alert would be closed as noise. The PR documents this, but you could add a per-family max_repeats, or require repeats to be below a lower cap, for the families with abuse_known=True. At minimum, say explicitly that the 9-repeat cap is the accepted risk.
2. Stale sibling state. quiet_rule_repeat judges siblings from their stored triage/analysis digests, using the thresholds in force now. If a tenant tightens model_close_* thresholds, old siblings are re-evaluated, which is fine. A sibling in waiting_human that is a "candidate" has never been checked by a human, though, so a model-persuaded benign verdict vouches for the next alert. The M1 test covers steering text. Please also consider requiring the sibling to be closed for the abuse_known families.
3. T1531 -> credential_access is broader than the PR title suggests. It will stop model-led closes for the Windows logon-failure rules (60122-60130) and for anything else Wazuh maps to T1531. That is the safe direction, and I'd mention it in the changelog as a behaviour change.
4. model_view coverage. It is applied in triage and analysis, but I only saw those two call sites in the diff. Please confirm that no other prompt builder (correlation digests, response planning, the llm_normalize path, the chat/copilot paths) passes the raw alert mapping to a model. If one does, normalizer is visible to it. It is not exploitable, because the value comes from the platform, but it breaks the stated "models never see it" claim.

Correctness and edge cases

• Field length. normalizer: Field(max_length=32) is not enforced by model_copy(update=...) in _finish. A normalizer name longer than 32 characters (e.g. a future plugin) would be stored, then fail validation when the alert is re-read. Add a unit test over all registered normalizer names, or validate in _finish.
• Old digests. _campaign_in_progress returns False for related_count == 0 when related_complete is absent (a pre-FIX-2 digest). That matches the documented behaviour. It does rely on related_count == 0 being trustworthy for old digests, which hid blind-audit alerts rather than flagging them. Re-running correlation on in-flight alerts would remove the doubt.
• _related limit check. complete = len(related) < self.max_related is correct only if the repo query returns at most limit rows. Please add a test that pins the exact boundary (max_related - 1 complete, max_related incomplete).
• Invariant check. The new chain case in check_routing_policy covers related_rule_repeats: 0. A second case with related_complete: False and all repeats counted would pin the incompleteness rule at install time too.

Performance

Each correlation run evaluates model_close_blockers once per related alert, up to max_correlations (default 10, 20 at most). That is cheap and in-memory. related_document builds a merged dict per alert, which is fine at this size.

Style and maintainability

• A few docstring lines in closure.py (the campaign bullet) run past the repo's line length. Please run the linter or formatter.
• NEVER_CLOSE_RULE_IDS has two entries (Asterisk 6250, FortiAuthenticator 44733) with no stated source or test. Add a one-line justification or a test for each, like the catalogue entries have.
• CATALOGUE_VERSION is a manual string. A test that fails when CATALOGUE changes without a version bump (e.g. a hash snapshot) would enforce the "bump on every change" rule.

Tests

Coverage is good. Gaps I'd close:

• a 31101 flood of exactly 9 repeats (closes) and 10 (blocks);
• an alert whose raw was dropped by the size cap, as an end-to-end case through routing;
• the max_length check on normalizer mentioned above.

Nice work on the fail-closed discipline and on documenting the abuse each family could hide.

[claude]

Review of FIX-2 (decision policy v2.1). I read the PR description, the docs, closure.py, noise_families.py and the graph.py size-cap change. I did not run the tests. The diff I had was truncated, so I only skimmed routing.py, ingestion/service.py, models/alert.py and the tests. There is no CLAUDE.md in the repo.

What looks good:

• The catalogue lifts only the unmapped class. Mapped never-close classes are never lifted and every other blocker still applies.
• It is keyed on the vendor rule id (digits only), and only when normalizer == wazuh and the rule name starts with ': '. Titles and descriptions are never matched, which answers title camouflage.
• It fails closed. A Wazuh alert with no readable rule, or a cut groups list, becomes vendor_unknown. cap_alert keeps the raw.rule subset, so inflating an event cannot hide its groups.
• Each catalogue entry records its rationale, the abuse it could hide, and an abuse-test flag. Deferring the campaign split is a defensible call and the PR is open about it.

Issues and suggestions:

1. Non-list groups skips the auth-failure check (noise_families.py, vendor_never_close_class). If rule.groups is present but not a list (for example a comma-separated string), the group check is skipped and nothing marks it unknown. For a never-close gate, treat a present non-list groups as VENDOR_UNKNOWN, with a test. Or confirm the Wazuh normalizer always emits a list.
2. Inconsistent group limits. vendor_never_close_class reads groups[:MAX_RULE_GROUPS] (50), while cap_alert keeps 20 for the oversized-raw subset and MAX_LIST_ITEMS otherwise. If an alert with more than 50 groups does not pass through cap_alert, an authentication_failed group past index 50 is silently ignored, with no groups_truncated flag. Return VENDOR_UNKNOWN when len(groups) > MAX_RULE_GROUPS instead of slicing.
3. T1531 (Account Access Removal) now maps to credential_access. This also blocks genuine account-removal alerts, which is probably right, but the changelog should say so.
4. Residual risk, already acknowledged in the PR: a local user can choose which catalogued rule fires (logger -t yum, a per-user MSI install). As a follow-up, consider a per-entity or per-rule rate cap, or mandatory audit sampling, on catalogue-lifted closes. 31101 and the install families are the likeliest decoy targets.
5. Docs: the architecture.md rewrap has uneven line lengths, and the TM-WRK-2 cell in threat-model.md is now very long, so consider a subsection. The comment 'and likely dpkg lines through syslog' in noise_families.py is speculative, so verify it or drop it.
6. In graph.py, RAW_RULE_SUBSET_ITEMS sits apart from the other RAW_* constants. Group them.
7. Tests to add: non-list groups; more than 50 groups; an int or whitespace-padded rule id; a leading-zero id such as 02901 (must not match); a normalizer of wazuh with a mismatched rule-name prefix; a cap_alert case where the raw event is kept whole but MAX_LIST_ITEMS cuts the groups; and a regression test that a pre-release alert with no normalizer stays unmapped and blocked.

Security and performance: I found no injection path in the lookup, which is an exact match on a digit-validated id, and the per-alert cost is negligible. Items 1 and 2 are the only places I found where a never-close gate can fail open, so I would fix those before merge. Otherwise this is a careful, well-documented T3 change.