Repository navigation
feat(api): add optional Prometheus metrics - #526
Merged
Merged
Conversation
jsenecal
force-pushed
the
feat/519-prometheus-metrics
branch
from
November 20, 2026 21:07
48219ff to
e42addb
Compare
jsenecal
changed the base branch from
feat/519-health-check
to
build/527-rye-to-uv
November 20, 2026 21:07
This was referenced Nov 20, 2026
jsenecal
force-pushed
the
feat/519-prometheus-metrics
branch
from
November 20, 2026 21:55
e42addb to
821a05b
Compare
jsenecal
marked this pull request as ready for review
November 21, 2026 04:29
Operators could not see whether queries were failing, on which device, or why, without reading logs. Expose Prometheus metrics behind `metrics.enable`, off by default because the endpoint is unauthenticated. Query metrics count outcomes and failure reasons per device and directive, time device execution on cache misses, and count cache hits and misses. Failures are recorded from the existing exception handlers, so the overall execution deadline (DeviceTimeout), state-unavailable 503s and request validation errors are all attributed without new error types. Label values come only from validated device and directive ids and exception class names; failures before validation are labeled `unknown`, so request input cannot inflate cardinality. Recording is best-effort and can never fail a query. HTTP metrics come from Litestar's Prometheus middleware, labeled by route template. uvicorn workers are separate processes, so `hyperglass start` points PROMETHEUS_MULTIPROC_DIR at a fresh directory under the system temporary directory (writable by the image's non-root user, unlike the app path's parents) and removes it on exit, or empties a configured one. The metrics endpoint skips the state recovery hook so it still answers during a Redis outage. App construction moves into create_app() so tests can build an app with metrics enabled; the module-level `app` is unchanged. Closes #519.
During a Redis outage default_handler fails while reading params, and Litestar then calls an exception handler again for the same request. Both calls recorded the failure, so one failed query was counted twice. Mark the request once its failure is recorded.
jsenecal
force-pushed
the
feat/519-prometheus-metrics
branch
from
November 21, 2026 04:33
821a05b to
0c379c6
Compare
This was referenced Nov 21, 2026
WilhelmZA
approved these changes
Nov 21, 2026
WilhelmZA
left a comment
Collaborator
There was a problem hiding this comment.
Approving at 0c379c6d.
I read the metrics module, the route and error handler changes, and the start-up path for the multiprocess directory. Metrics are off by default, the labels come only from validated ids and exception class names, and metrics is not sent to the UI. CI gate passes, every commit is verified and the prometheus-client licence is Apache-2.0.
This PR and #566 both change hyperglass/api/routes.py, and this PR and #538/#543 both change hyperglass/api/error_handlers.py. The second one to merge needs to add the bypass cache result and recheck the error reason for a Redis outage.
Nits:
hyperglass/api/metrics.py:88:BUILD_INFO.labels(__version__).set(1)runs at import and is not covered by_never_raise. IfPROMETHEUS_MULTIPROC_DIRpoints to a missing or unwritable directory whilemetrics.enableis false, the import can raise and the app does not start. Setting it fromcreate_app()when metrics are enabled would avoid that.- The body has a Changelog entry but
CHANGELOG.mdis not edited. Please add it under[Unreleased], here or in the release PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #519.
Problem
Operators cannot see whether queries are failing, on which device, or why, without reading logs. #355 asks for metrics; #519 scopes the port from hyperglass-ng.
Change
metrics.enable(defaultfalse, because the endpoint is unauthenticated) andmetrics.path(default/metrics). Not exported to the UI.hyperglass/api/metrics.py(new dependencyprometheus-client>=0.21.0;uv lockadds onlyprometheus-client0.26.0 and changes no other locked version):hyperglass_queries_total{device,query_type,status}hyperglass_query_errors_total{device,query_type,reason}:reasonis the snake_case exception class (device_timeout,auth_error,query_location_not_found,parsing_error, …),invalid_requestfor 4xxHTTPExceptions (request validation),service_unavailablefor the 503 from the state recovery hook, elseinternal_errorhyperglass_query_duration_seconds{device,query_type}: device execution time on cache misses, success onlyhyperglass_query_cache_total{result}:hit/miss(mainhas noforcefield, so the fork'sbypassis dropped)hyperglass_build_info{version}/api/devices/{id});/api/healthand the metrics path are excluded.Queryvalidation is labeledunknown, so request input cannot add label values.query()after validation; failures are recorded from the existing exception handlers for routes marked withopt={"query_metrics": True}. The execution deadline from Transfer transport reliability and state recovery #396 already raisesDeviceTimeout, so it lands asdevice_timeoutwith no new error types. Recording is wrapped so a metrics failure can never fail a query, and each request's failure is counted once (an exception handler that fails during a Redis outage is called again by Litestar; found while testing the image).recover_statehook (reusesskip_state_recoveryfrom feat(api): add a health check endpoint and healthcheck command #524), so it answers during a Redis outage.hyperglass startsetsPROMETHEUS_MULTIPROC_DIRto a newtempfile.mkdtemp()directory (underTMPDIR,/tmpin the image, which UID 10001 can write;/etc/hyperglassis not used) and removes it at exit, or creates and empties*.dbin a directory the operator set. This runs before uvicorn imports the app, which is whenprometheus_clientpicks its mode.create_app(): app construction moves into a function so tests can build an app with metrics enabled; module-levelappis unchanged.configuration/config/monitoring(metric/label tables, reasons, PromQL examples, multiple workers), restricting the endpoint at NGINX (sample config does not forward/metrics), Caddy (sample forwards everything) and Compose (published on all addresses by default), plusconfig.mdx,complete-config.mdxand.samples/sample_config.yaml.Tests
New
hyperglass/util/tests/test_metrics_endpoint.py: endpoint content, disabled by default (404), configurable path, skips state recovery, not in schema; success/duration/cache hit+miss; device failures with reason (device_timeout,auth_error,internal_error); execution deadline →device_timeout; unknown location counted asunknownandattacker-chosenabsent from/metrics; malformed body (400) →invalid_request; state unavailable (503) →service_unavailable; non-query routes not counted; route-template grouping; health/metrics excluded from HTTP metrics; failing metric does not fail the query; failure counted once when the handler fails (fails with2.0 == 1without the guard);prepare_metrics_dirtemp/configured/stale cleanup;start()prepares the dir only when enabled.Image built locally and run as UID 10001 with 8 workers (the default),
fake_output: true,metrics.enable: true, next to Valkey./tmp/hyperglass-metrics-*was created owned byhyperglassand held one.dbfile per worker. After 24 concurrent queries and one unknown location:With Valkey stopped,
/metricsanswered 200 in 15 ms and a query failure was counted once asinternal_error(on the rebuilt image with the fix).docker stopremoved the temp directory.Changelog
Notes
main(Request validation errors return HTTP 200 #490 is closed), so HTTP metrics record them correctly; the fork's caveat no longer applies./api/queryreturns a bare 500 after ~15 s becauserecover_statelets the RedisConnectionErrorthrough (onlyStateErrorbecomes 503) anddefault_handlerthen readsparamsfrom Redis.GET /api/devices/<unknown id>also returns 500 (Devices.__getitem__raisesIndexError), not 404. Filed as Requests during a Redis outage get a bare 500 instead of 503 #529 and GET /api/devices/<id> returns 500 for an unknown device #530.requests_in_progressgauge useslivesum, so a worker killed mid-request leaves its value until restart; the temp dir is recreated on eachhyperglass start.