Skip to content

feat(api): add optional Prometheus metrics - #526

Merged
jsenecal merged 2 commits into
mainfrom
feat/519-prometheus-metrics
Nov 21, 2026
Merged

jsenecal merged 2 commits into
mainfrom
feat/519-prometheus-metrics

Conversation

@jsenecal

@jsenecal jsenecal commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #519.

Problem

Operators cannot see whether queries are failing, on which device, or why, without reading logs. #355 asks for metrics; #519 scopes the port from hyperglass-ng.

Change

  • Config: metrics.enable (default false, because the endpoint is unauthenticated) and metrics.path (default /metrics). Not exported to the UI.
  • hyperglass/api/metrics.py (new dependency prometheus-client>=0.21.0; uv lock adds only prometheus-client 0.26.0 and changes no other locked version):
    • hyperglass_queries_total{device,query_type,status}
    • hyperglass_query_errors_total{device,query_type,reason}: reason is the snake_case exception class (device_timeout, auth_error, query_location_not_found, parsing_error, …), invalid_request for 4xx HTTPExceptions (request validation), service_unavailable for the 503 from the state recovery hook, else internal_error
    • hyperglass_query_duration_seconds{device,query_type}: device execution time on cache misses, success only
    • hyperglass_query_cache_total{result}: hit / miss (main has no force field, so the fork's bypass is dropped)
    • hyperglass_build_info{version}
    • HTTP metrics from Litestar's Prometheus middleware, labeled by route template (/api/devices/{id}); /api/health and the metrics path are excluded.
  • Labels come only from validated device/directive ids and exception class names. Anything failing before Query validation is labeled unknown, so request input cannot add label values.
  • Instrumentation hooks the existing flow: labels are attached in query() after validation; failures are recorded from the existing exception handlers for routes marked with opt={"query_metrics": True}. The execution deadline from Transfer transport reliability and state recovery #396 already raises DeviceTimeout, so it lands as device_timeout with no new error types. Recording is wrapped so a metrics failure can never fail a query, and each request's failure is counted once (an exception handler that fails during a Redis outage is called again by Litestar; found while testing the image).
  • Metrics endpoint skips the app-wide recover_state hook (reuses skip_state_recovery from feat(api): add a health check endpoint and healthcheck command #524), so it answers during a Redis outage.
  • Multiprocess: hyperglass start sets PROMETHEUS_MULTIPROC_DIR to a new tempfile.mkdtemp() directory (under TMPDIR, /tmp in the image, which UID 10001 can write; /etc/hyperglass is not used) and removes it at exit, or creates and empties *.db in a directory the operator set. This runs before uvicorn imports the app, which is when prometheus_client picks its mode.
  • create_app(): app construction moves into a function so tests can build an app with metrics enabled; module-level app is unchanged.
  • Docs: metrics section on configuration/config/monitoring (metric/label tables, reasons, PromQL examples, multiple workers), restricting the endpoint at NGINX (sample config does not forward /metrics), Caddy (sample forwards everything) and Compose (published on all addresses by default), plus config.mdx, complete-config.mdx and .samples/sample_config.yaml.

Tests

New hyperglass/util/tests/test_metrics_endpoint.py: endpoint content, disabled by default (404), configurable path, skips state recovery, not in schema; success/duration/cache hit+miss; device failures with reason (device_timeout, auth_error, internal_error); execution deadline → device_timeout; unknown location counted as unknown and attacker-chosen absent from /metrics; malformed body (400) → invalid_request; state unavailable (503) → service_unavailable; non-query routes not counted; route-template grouping; health/metrics excluded from HTTP metrics; failing metric does not fail the query; failure counted once when the handler fails (fails with 2.0 == 1 without the guard); prepare_metrics_dir temp/configured/stale cleanup; start() prepares the dir only when enabled.

$ pytest hyperglass --ignore hyperglass/plugins/external -q -p no:cacheprovider
4193 passed, 4 skipped, 347 warnings in 82.27s (0:01:22)
$ ruff check .
All checks passed!

Image built locally and run as UID 10001 with 8 workers (the default), fake_output: true, metrics.enable: true, next to Valkey. /tmp/hyperglass-metrics-* was created owned by hyperglass and held one .db file per worker. After 24 concurrent queries and one unknown location:

hyperglass_query_cache_total{result="miss"} 24.0
hyperglass_queries_total{device="example_device",query_type="__hyperglass_juniper_bgp_route__",status="success"} 24.0
hyperglass_queries_total{device="unknown",query_type="unknown",status="error"} 1.0
hyperglass_query_duration_seconds_count{device="example_device",query_type="__hyperglass_juniper_bgp_route__"} 24.0
hyperglass_query_errors_total{device="unknown",query_type="unknown",reason="query_location_not_found"} 1.0
hyperglass_http_requests_total{app_name="hyperglass",method="POST",path="/api/query",status_code="201"} 24.0

With Valkey stopped, /metrics answered 200 in 15 ms and a query failure was counted once as internal_error (on the rebuilt image with the fix). docker stop removed the temp directory.

Changelog

### Added
- Optional Prometheus metrics (`metrics.enable`, off by default): query outcomes and failure reasons per device and query type, device response time, cache hits and misses, build info, and HTTP request metrics by route, aggregated across workers. The endpoint is unauthenticated; restrict it at your reverse proxy or firewall (#519).

Notes

@github-actions github-actions Bot added docs Improve documentation backend Python backend: API, models, state, execution labels Nov 20, 2026
@coveralls

coveralls commented Nov 20, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 85.699% (+0.3%) from 85.355% — feat/519-prometheus-metrics into main

@jsenecal
jsenecal force-pushed the feat/519-prometheus-metrics branch from 48219ff to e42addb Compare November 20, 2026 21:07
@jsenecal
jsenecal changed the base branch from feat/519-health-check to build/527-rye-to-uv November 20, 2026 21:07
@github-actions github-actions Bot added the container Docker image and compose label Nov 20, 2026
@jsenecal jsenecal closed this Nov 20, 2026
@jsenecal jsenecal reopened this Nov 20, 2026
@jsenecal
jsenecal force-pushed the feat/519-prometheus-metrics branch from e42addb to 821a05b Compare November 20, 2026 21:55
@jsenecal
jsenecal marked this pull request as ready for review November 21, 2026 04:29
Operators could not see whether queries were failing, on which device,
or why, without reading logs. Expose Prometheus metrics behind
`metrics.enable`, off by default because the endpoint is unauthenticated.

Query metrics count outcomes and failure reasons per device and directive,
time device execution on cache misses, and count cache hits and misses.
Failures are recorded from the existing exception handlers, so the
overall execution deadline (DeviceTimeout), state-unavailable 503s and
request validation errors are all attributed without new error types.
Label values come only from validated device and directive ids and
exception class names; failures before validation are labeled `unknown`,
so request input cannot inflate cardinality. Recording is best-effort and
can never fail a query. HTTP metrics come from Litestar's Prometheus
middleware, labeled by route template.

uvicorn workers are separate processes, so `hyperglass start` points
PROMETHEUS_MULTIPROC_DIR at a fresh directory under the system temporary
directory (writable by the image's non-root user, unlike the app path's
parents) and removes it on exit, or empties a configured one. The metrics
endpoint skips the state recovery hook so it still answers during a Redis
outage.

App construction moves into create_app() so tests can build an app with
metrics enabled; the module-level `app` is unchanged.

Closes #519.
During a Redis outage default_handler fails while reading params, and
Litestar then calls an exception handler again for the same request. Both
calls recorded the failure, so one failed query was counted twice. Mark
the request once its failure is recorded.

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 0c379c6d.

I read the metrics module, the route and error handler changes, and the start-up path for the multiprocess directory. Metrics are off by default, the labels come only from validated ids and exception class names, and metrics is not sent to the UI. CI gate passes, every commit is verified and the prometheus-client licence is Apache-2.0.

This PR and #566 both change hyperglass/api/routes.py, and this PR and #538/#543 both change hyperglass/api/error_handlers.py. The second one to merge needs to add the bypass cache result and recheck the error reason for a Redis outage.

Nits:

  • hyperglass/api/metrics.py:88: BUILD_INFO.labels(__version__).set(1) runs at import and is not covered by _never_raise. If PROMETHEUS_MULTIPROC_DIR points to a missing or unwritable directory while metrics.enable is false, the import can raise and the app does not start. Setting it from create_app() when metrics are enabled would avoid that.
  • The body has a Changelog entry but CHANGELOG.md is not edited. Please add it under [Unreleased], here or in the release PR.

@WilhelmZA WilhelmZA added the reviewing Issue is being reviewed by maintainers label Nov 21, 2026
@WilhelmZA WilhelmZA added the dependencies Pull requests that update a dependency file label Nov 21, 2026
@jsenecal
jsenecal merged commit 849ab0e into main Nov 21, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Python backend: API, models, state, execution container Docker image and compose dependencies Pull requests that update a dependency file docs Improve documentation reviewing Issue is being reviewed by maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Port /api/health, Prometheus metrics and the healthcheck CLI from hyperglass-ng

3 participants