Create observability dashboards and alert thresholds

[labgadget015-dotcom]

Roadmap item 8/10 · Workstream: Reliability · Risk: Medium · Approval required: No · Target milestone: M2 — Governed intake

Objective

Give one place to see whether the pipeline is actually working — not merely reachable — with alert thresholds that page on real breakage and never report missing data as healthy.

Non-goals

• A new monitoring vendor
• Business/product analytics

Acceptance criteria

• Every probe reports ok / fail / unknown; unknown is never rendered green
• Dashboard panels: executions by status per workflow, intake-to-DRC latency, audit-event failure rate, n8n execution budget burn vs plan, canary verdicts
• Documented thresholds with rationale, e.g. fresh failure streak → page; stale open incident → warning plus periodic reminder; execution budget > 80% → warn, > 95% → page
• Monitors have their own heartbeat below the monitor interval and are checked from outside the system they monitor
• Alerts link to the correlation ID and the run/execution that fired them

Dependencies

• Implement correlation IDs, idempotency, and structured audit events #319
• Complete n8n incident remediation and write the post-incident review #320

Risk

Medium — bad thresholds cause alert fatigue or false greens; monitoring must not consume the pipeline's own quota.

Test evidence required

• Forced-failure test for each alert (fires, links correctly, clears on recovery)
• Blind-path test: remove the data source and confirm the panel shows UNKNOWN, not green
• One week of alert volume reviewed

Definition of done

• Dashboards and thresholds documented in docs/
• Every alert has a runbook link
• No green state derivable from absent data
• Evidence URL and Correlation ID recorded on the project item