Skip to content

Prometheus exporter + Grafana dashboards as code #2

Description

@rubinder

Problem

The platform measures a great deal and exports none of it. make monitor persists every metric to ops.monitor_results and routes alerts to ops.alert_log (src/ops/runner.py, src/ops/alerts.py), with robust median/MAD baselines and history (ADR-0007). But the only way to see any of it is to run a make target and read stdout, or query the Iceberg table by hand.

That is the gap between "this platform has checks" and "this platform is operated". A monitoring story that ends at a terminal transcript is the thing the README's own framing — data quality as an engineered property, not a dashboard someone occasionally checks — argues against.

Proposed change

A thin export layer over data that is already persisted. No new measurement.

  • src/ops/exporter.py: read the latest run from ops.monitor_results and ops.alert_log and render Prometheus text-format metrics. Batch jobs write a .prom file for the node_exporter textfile collector — no Pushgateway, no always-on process, consistent with the repo's offline-by-default posture.
  • Metric surface, labelled by table / monitor / kind:
    • adp_monitor_value — the observed metric
    • adp_monitor_baseline_median — the trailing median it was judged against
    • adp_monitor_status — 0 ok / 1 warn / 2 breach
    • adp_arrival_lag_days, adp_arrival_status
    • adp_agent_findings_total{severity=...}
    • adp_promotions_blocked_total
    • adp_run_timestamp_seconds
  • dashboards/ as code: one Grafana JSON dashboard + Prometheus alert rules, provisioned by a docker-compose.yml that brings up Prometheus + Grafana only (no Kafka, no ClickHouse).

Constraints

  • make all, make monitor, make agent and make test must keep working with nothing running. The exporter writes a file; it must never require a scrape target to exist. Docker is strictly opt-in, and uv sync && uv run make all from a clean clone stays true.
  • Cardinality stays bounded: labels are table/monitor/kind, never row values or IDs.
  • The dashboard is committed as JSON and loaded by provisioning config, not clicked together in the UI.

Acceptance

  • make monitor emits a valid Prometheus text-format file; a test parses it and asserts one series per monitor with the right labels.
  • docker compose up brings up Prometheus + Grafana with the dashboard already provisioned, and no other service.
  • An alert rule fires on a breach, demonstrated by make drift-demo (README transcript, with the sequence stated).
  • Every make target still runs with docker stopped, and CI does not gain a service dependency.
  • An ADR records why textfile-collector over Pushgateway.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P4Roadmap position 4enhancementNew feature or requestobservabilityMetrics, dashboards, alerting

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions