[FIX-0] Verdict cache and suppression suggestions keyed by behaviour; a threat revokes them

changelog.d/AI-E1.9.md

@@ -0,0 +1,7 @@
+### Changed
+
+- The attack-emulation corpus declares its authorised vulnerability scanner,
+  as a tenant would (`_meta.tenant_settings`, `triage.known_scanners`).
+- Evaluation reports record, per item, what the routing decision read
+  (`decision_inputs`: the confidence, the triage, correlation and analysis
+  scores and the verdict).

changelog.d/fatigue-1-measurement.md

@@ -0,0 +1,28 @@
+### Added
+
+- `lg-sotf eval --oracle propose|accept`: a measurement mode in which two
+  simulated analysts, taking turns, decide every escalation at once from the dataset's
+  labels (`false_positive` for benign, `true_positive` for malicious;
+  response actions are rejected), through the same decision path as the API,
+  with items in event-time order. It shows how much the verdict cache, the
+  tenant's entity history and suppression rules could close with perfect,
+  instant analysts. `accept` also enables the suppression rules the
+  decisions suggest. Each item's status still records what automation did
+  before the oracle acted.
+- Every evaluation report and scorecard now leads with two numbers: the
+  benign closure rate (with n and a 95% interval) and malicious auto-close
+  (with n and an exact one-sided 95% upper bound). A table attributes each
+  automatic close to rules, the model, history, the verdict cache, a
+  suppression rule or a completed response, and every malicious alert closed
+  by any of them is listed by id.
+- Datasets can declare the tenant's context (`_meta.tenant_context`): IT
+  admin accounts and groups, helpdesk accounts, automation accounts and
+  hosts, jump hosts, patch windows and the authorised scanner. The
+  attack-emulation corpus declares its estate's. Nothing applies it yet;
+  reports show how many benign alerts it covers ("closable with context")
+  and how many attacks it covers too.
+- A loader for the public AIT Alert Data Set (2.29 million Wazuh alerts, CC
+  BY 4.0): `python -m tests.fixtures.aitads fetch DIR` and `build DIR --out
+  FILE` sample `aitads_v1` (5,000 alerts per scenario, stratified by attack
+  phase, seeded). Labels come from the dataset's attack-phase times; see
+  `docs/evaluation.md#ait-ads` for what they can and cannot say.

changelog.d/fix-verdict-cache-fingerprint.md

@@ -0,0 +1,44 @@
+### Security
+
+- The verdict cache could close a malicious alert at the gate because it
+  matched alerts by fingerprint alone (rule, hosts, users, IPs, hashes,
+  domains). On one host, a ransomware `PSEXESVC` service matched an Edge
+  update service an analyst had closed as benign, and a web shell's `ps`
+  running as uid 33 matched monitoring's `ps` running as root. The cache now
+  uses a verdict key: the fingerprint plus the process image, command line,
+  service name and image path, user/uid, parent process, the objects an
+  attacker creates (scheduled tasks, registry keys, target files, Sysmon WMI
+  subscriptions) and the request context (URLs, user agents, O365 operations
+  and parameters, queries, process-access rights), compared in full: nothing
+  is masked or truncated.
+- Alerts with no process, command, service or created object, alerts in a
+  never-close class (credential access, log clearing, tampering, ransomware,
+  persistence through named objects) and alerts normalised by a model never
+  close from the cache and get no suggested rule; request context alone (a
+  user agent, a URL, an O365 operation) is never enough. Model-normalised
+  alerts also skip suppression rules.
+- A cached verdict now closes an alert only after two different analysts
+  decided it benign within 7 days, at the alert's severity or higher. One
+  analyst's decisions, including a bulk close, count once. It is never used
+  for critical alerts, nor within 7 days of a confirmed threat on that key or
+  on any alert with the same fingerprint.
+- Suppression rules suggested from false-positive decisions (one decision, or
+  a batch of escalations) match the alert's verdict key instead of its
+  fingerprint, so enabling one no longer suppresses a different service or
+  command on the same host. With the analyst oracle on corpus_v1, enabling
+  every suggestion closed 4 attacks before this change and none after.
+- When a threat is confirmed on an alert (an analyst `true_positive`, a
+  blind-audit malicious label, a planned response or a malicious close),
+  every enabled suppression rule that would close that alert is disabled and
+  recorded as `suppression.revoked` in the audit log. A failure is logged as
+  an error and counted in `lgsotf_revocation_failures_total`.
+- Migration 0016 disables every enabled suggested rule that matches a
+  fingerprint alone (audited `suppression.revoked`, reason
+  `fingerprint_suggestion`), and the gate ignores such rules even if they are
+  enabled again. Replace them with rules on `verdict_key`.
+- A benign decision on a grouped escalation now closes an attached alert only
+  when it has the same verdict key as the reviewed alert; otherwise the
+  attached alert gets its own escalation, as before.
+- Cached verdicts written before this change are no longer read, so repeats
+  are investigated until two analysts have decided them again. They expire
+  within 7 days; `DELETE /tenant/verdict-cache` removes them now.

docs/ai-system-card.md

@@ -174,7 +174,7 @@ dangerous case.
 
 | Decision | Who makes it | The model's part |
 |---|---|---|
-| Close an alert as benign | The routing policy (`lg_sotf.domain.routing`). Never for high or critical alerts (INV-03): those close only by an analyst decision, an admin-enabled suppression rule, or an analyst-confirmed verdict (not for critical). Low and medium alerts close only when rules corroborate the model | Its assessment is one input; a benign verdict alone never closes |
+| Close an alert as benign | The routing policy (`lg_sotf.domain.routing`). Never for high or critical alerts (INV-03): those close only by an analyst decision, an admin-enabled suppression rule, or an analyst-confirmed verdict (not for critical). Low and medium alerts close on a benign verdict only with approved benign evidence an attacker cannot write (the detection rule's own words, the EICAR hash, a test source, or a scanner the tenant declared as the source of the activity, never for a never-close class), or as a model-led close when the tenant's autonomy tier is `auto_full_audit` (FIX-1) | Its assessment is one input; a benign verdict alone never closes |
 | Escalate to an analyst | The routing policy. Uncertain cases, model refusals (`model_refused`), alerts from the fallback normaliser, and failed responses always go to a person | Its uncertainty raises escalation |
 | Plan a response | The routing policy, from the threat score | It recommends actions; the planner validates every target |
 | Run a response action | The tenant's **autonomy policy** per risk level (`auto`, `approval` or `disabled`; default `approval` for low, medium and high, `disabled` for critical), then the **guardrails** (protected assets, target checks, hourly caps, blast radius). Containment on model-only evidence always waits for a person, even under `auto` (INV-11) | None beyond the recommendation |
@@ -202,7 +202,7 @@ with `lg-sotf verify-invariants` ([security.md](security.md#11-safety-invariants
 |---|---|---|---|---|
 | INV-01 | Every tenant table has forced row-level security, and the runtime role cannot bypass it | enforced | `rls_catalog`, `runtime_role`, `rls_probe` | TM-DB-1, TM-DB-2, TM-SCH-1, TM-API-3 |
 | INV-02 | No response action runs without an approval or an autonomy rule that allows it; the kill switch wins | enforced | `kill_switch` | TM-WRK-3, TM-RSP-4, TM-RSP-5, TM-MCP-6 |
-| INV-03 | An AI verdict alone never closes a high or critical alert or a never-close class; a low or medium alert closes on the models' word only at the tenant's autonomy tier | enforced | `routing_policy` | TM-WRK-1, TM-WRK-2 |
+| INV-03 | An AI verdict alone never closes a high or critical alert or a never-close class; a low or medium alert closes on the models' word only at the tenant's autonomy tier | enforced | `routing_policy` | TM-WRK-1, TM-WRK-2, TM-WRK-10 |
 | INV-04 | Every LLM call records its prompt hash and model id | enforced | none | TM-WRK-8 |
 | INV-05 | Every LLM and tool step of a run is in the trace store | enforced | none | TM-WRK-8, TM-EVT-6 |
 | INV-06 | The audit log is append-only for the runtime role | enforced | `audit_append_only` | TM-DB-3, TM-API-9, TM-SCH-2 |
@@ -357,16 +357,19 @@ split since AI-E6.3 ([evaluation.md](evaluation.md#adversarial-corpus-v2)).
   * **The gate.** The items are in the gated `adversarial` split, 0 of 3
     closed.
 * **Activity words in rule names** (AI-E6.4,
-  [#305](https://github.com/gapilongo/SOC/issues/305), in review as #314 on
+  [#305](https://github.com/gapilongo/SOC/issues/305), merged as #314 on
   2026-11-08). `scheduled`, `update`, `backup` and `scan` in a rule name
-  usually describe the detected activity, not benign intent, yet they still
-  count as benign evidence. AI-E6.4 counts them only with a trusted
-  qualifier, such as a maintenance window or a known scanner, and adds
-  adversarial items for them. Until it merges, the gap is open.
+  usually describe the detected activity, not benign intent. AI-E6.4 counts
+  them only with a trusted qualifier (an intent word in the rule name, a
+  test-environment source or, for `scan`, a known scanner) and gates the
+  `activity-camouflage` family.
 * **No benign auto-close on the attack-emulation corpus** (section 7.1). The
   routing policy and the eval replay, not the model, held all 312 benign
   alerts. The proposed changes are hypotheses in
-  [RUN.md](scorecards/v2/deepseek-v4/RUN.md), not made.
+  [RUN.md](scorecards/v2/deepseek-v4/RUN.md), measured offline in
+  [CALIBRATION.md](scorecards/v2/deepseek-v4/CALIBRATION.md); decision policy
+  v2 (FIX-1) replaced the routing they tuned.
+  A real-model re-run (v2.1) is still to do.
 * **Models follow injected text.** Up to 44% of the time in one family with
   a 7B local model. The policy contains the effect; the model is not robust.
 * **Steering through the verdict.** The injection-followed metric checks
@@ -467,8 +470,8 @@ change a customer's environment.
 * On the attack-emulation corpus automation closed no benign alert (0 of
   312), so the measured analyst-time saving is zero (section 7.1).
 * Rule-name words that describe the detected activity (`scheduled`,
-  `update`, `backup`, `scan`) still count as benign evidence until AI-E6.4
-  (#305) merges. Title camouflage (#299) is fixed and gated (AI-E6.3).
+  `update`, `backup`, `scan`) count as benign evidence only with a trusted
+  qualifier (AI-E6.4). Title camouflage (#299) is fixed and gated (AI-E6.3).
 * Models follow injected text a substantial share of the time; only the
   policy contains it.
 * Shadow mode (running a candidate configuration side by side without side

docs/api.md

@@ -366,8 +366,10 @@ window.
 * `POST /escalations/bulk/propose-suppression` `{"escalation_ids", "name"?}`
   is all or nothing: 404 when an escalation is not found, 422
   `not_confirmed_benign` (`details` lists them) when one has no
-  `false_positive` or `benign_positive` decision. Otherwise one disabled rule
-  matching the alerts' fingerprints and sources, up to the most severe of
+  `false_positive` or `benign_positive` decision, 422 `no_verdict_key` when an
+  alert has no verdict key (no process, command, service or named object, a
+  never-close class, or a model-normalised event). Otherwise one disabled rule
+  matching the alerts' verdict keys and sources, up to the most severe of
   their severities: 201 `{suppression, created: true, escalation_ids}`, or
   200 with the unexpired rule that already has the same match.
 * `GET /escalations/{id}/response-preview` computes the plan a

docs/architecture.md

@@ -220,8 +220,11 @@ Graph (compiled once per worker process with `AsyncPostgresSaver`,
 
 ```
 START → gate → triage → correlation → analysis → decide
-gate:        suppression rules / verdict cache (same or lower severity than the
-             analyst decided; never for critical or LLM-normalised alerts) →
+gate:        suppression rules / verdict cache (same verdict key, i.e. same
+             fingerprint and behaviour, decided benign by 2 different
+             analysts at the same or a higher severity in 7 days, no threat
+             since; never for critical, never-close-class or LLM-normalised
+             alerts, which also skip suppression rules) →
              close (benign) | triage
 triage:      rules + fast-tier LLM → close (obvious FP) | correlation
 correlation: indicator sightings + optional LLM → analysis