Repository navigation
feat(api): add a health check endpoint and healthcheck command - #524
Conversation
Operators and container runtimes had no reliable way to tell whether hyperglass can serve queries. The UI root keeps answering 200 through a Redis outage, and the Compose file worked around that by fetching /api/info, which only covers part of the picture and has no IPv6 or custom-host support. GET /api/health checks Redis with a dedicated short-timeout connection in a worker thread (the shared pool has no timeouts, so an unreachable Redis could hang the check), then that configuration state is loaded or can be recovered, then that the UI is built (or disabled). It answers 200 or 503 and logs failure details instead of returning them. The handler replaces the app-wide state recovery hook, which reads Redis through the shared pool, and runs recovery itself after the bounded probe. `hyperglass healthcheck` requests the endpoint on the configured listen address, maps 0.0.0.0 and :: to the loopback address of the same family, and bypasses HTTP proxies. The image gets a HEALTHCHECK using it, with a 10 minute start period because the UI is built on first boot before the server listens. The Compose healthcheck is removed in favour of the image's, so an older image without the command shows no health status rather than permanently unhealthy. Part of #519.
WilhelmZA
left a comment
There was a problem hiding this comment.
Reviewed at 60f49ddc.
The endpoint, the command and the tests read well: the response discloses only ok, error or disabled per check, the handler never reaches a device, and Part of #519 is right. Two points have to change before approval:
GET /api/healthis unauthenticated and opens a new Redis connection on every request. See the comment onhyperglass/api/routes.py.- The default
compose.yamlloses its health status, because it still pins an image that has noHEALTHCHECK. See the comment oncompose.yaml.
#528 also changes Dockerfile, in the build stage only, so the two do not conflict. #526 carries this commit and needs a rebase after this merges.
/api/health is unauthenticated, and every request opened a new Redis connection and used two jobs on the shared thread pool. A client that floods it could open connections to Redis faster than they close. - Each worker now runs the checks at most once per second. Requests that arrive during a run wait for it, and requests in the next second reuse its result. A request that disconnects does not cancel the shared run. - The Redis probe reuses one client with the short timeouts, so repeated probes share a connection. compose.yaml keeps its /api/info health check. It pins 3.0.0-rc.2, which has no HEALTHCHECK, so removing it would leave the default deployment without a health status until it pins an image that has one.
WilhelmZA
left a comment
There was a problem hiding this comment.
Reviewed at b36846a0.
Both points from the earlier review are resolved. /api/health now keeps one Redis client per DSN and runs the checks at most once per second per worker, with concurrent requests sharing one probe, and the tests cover it. compose.yaml keeps its /api/info check until it pins an image that has the HEALTHCHECK. CI gate passes, both commits are verified, and I found no code defect in the current diff.
One thing to change before approval:
- The PR description still describes the first commit. The compose.yaml bullet says the
wget /api/infohealthcheck is removed, and the Changelog block has a### Changedentry sayingcompose.yamlno longer defines its own healthcheck.b36846a0restored that healthcheck, so that entry would put a change that did not happen into the release notes. Please update the bullet and drop the### Changedentry.
Nits:
- One optional docs suggestion inline.
WilhelmZA
left a comment
There was a problem hiding this comment.
Approving at 2e1348c3.
Both points from the last review are resolved: the description now says compose.yaml keeps its /api/info check and the ### Changed changelog entry is gone, and the docs sentence is in monitoring.mdx. The only change since b36846a0 is that one sentence; the rest of the diff is unchanged. CI gate passes and all three commits are verified.
One thing to watch at merge, not a defect in this PR: #535 also changes hyperglass/cli/main.py and adds test_commands_cover_cli in hyperglass/cli/tests/test_help.py, which requires the registered commands to equal a fixed COMMANDS tuple. That tuple has no ("healthcheck",) entry. The two branches merge without a text conflict, so whichever merges second leaves main failing that test unless it adds the entry. If #535 merges first, please update this branch, add the entry and let CI run again before merging.
|
One addition to the merge note in my approval at |
Typer 0.29 ships its own CliRunner, which takes only charset and env, so CliRunner(mix_stderr=False) in test_healthcheck.py raised TypeError at collection once #535 lifted the Typer hold, and main's CI broke. The healthcheck command prints only to stdout and the tests check exit codes, so a plain CliRunner() is enough. That collection error also hid test_help.py::test_commands_cover_cli, which fails because COMMANDS lacked healthcheck (#524); it is added.
Part of #519 (health check half; Prometheus metrics follow in a stacked PR).
Problem
There is no reliable way to tell whether hyperglass can serve queries. The UI root keeps answering 200 through a Redis outage, the image has no
HEALTHCHECK, andcompose.yamlworks around that bywget-ing/api/info, which only covers part of the picture.Change
GET /api/health(not in the OpenAPI schema). Checks, in order:redis: ping over a dedicated client with 3 s connect/read timeouts, in a worker thread. The shared pool has no timeouts, so an unreachable Redis could otherwise hang the check. The client is kept per DSN, so repeated probes reuse its connection.state:ensure_ready(), i.e. configuration is in Redis or can be recovered (same recovery other requests trigger). Skipped (reportederror) when Redis is down.ui:static/ui/index.htmlexists, ordisabledwhenHYPERGLASS_DISABLE_UIis set.Each worker runs the checks at most once per second: requests that arrive during a run wait for it, requests in the next second reuse its result, and the run is shielded so a disconnecting client doesn't cancel it for the others. A flood therefore costs one probe and two thread-pool jobs per second per worker.
Answers
200/503with{"status", "checks"}. Failure details are logged, not returned. The handler replaces the app-widerecover_statehook (newskip_state_recoveryinmiddleware.py), because that hook reads Redis through the unbounded shared pool before the handler runs; recovery happens inside the handler after the bounded probe instead.hyperglass healthcheckrequests the endpoint onHYPERGLASS_HOST/HYPERGLASS_PORT, maps0.0.0.0→127.0.0.1and::→::1, brackets IPv6, never usesHTTP_PROXY, exits 1 when unhealthy or unreachable.--timeout(default 8 s, under the 10 s Docker timeout).Dockerfile
HEALTHCHECK --interval=30s --timeout=10s --start-period=10m --retries=3running that command as the image user (UID 10001). The 10 min start period covers the first-boot UI build.compose.yaml: keeps its
wget /api/infohealthcheck, with a comment to remove it once the file pins an image that has theHEALTHCHECK.compose.yamlonmainstill pins3.0.0-rc.2, which has noHEALTHCHECK, nohyperglass healthcheckand no/api/health, so removing it now would leave the default deployment with no health status. Until then the Compose check overrides the image's, which the Docker guide says.Docs: new
configuration/config/monitoringpage (nav "Health & Monitoring") and a "Health check" section in the Docker guide; the monitoring page points to it for the Compose override.Tests
New
hyperglass/util/tests/test_health_endpoint.py(ok; Redis unreachable; app-wide state hook not run when Redis is unreachable; lost state recovered; unrecoverable state; UI not built; UI disabled; not in schema; probe uses short timeouts; probe fails fast against TEST-NET-1; one Redis client reused across probes; result reused within the interval; checks again after it; concurrent requests share one probe; a disconnecting request doesn't cancel the shared probe) andhyperglass/cli/tests/test_healthcheck.py(healthy, unhealthy,0.0.0.0,::1and::over IPv6, ignoreshttp_proxy, unreachable). Mutation-checked: removing thebefore_requestoverride, the proxy bypass, client reuse, result reuse or the shield makes the corresponding tests fail.Image built locally (
docker build) and run as UID 10001 next tovalkey/valkey:8-alpine:Health went
starting→healthyafter the UI build. Afterdocker stopof Valkey, three checks returned{"status":"error","checks":{"redis":"error","state":"error","ui":"ok"}}and the container wentunhealthy; after starting Valkey again it went back tohealthy.Changelog
Notes
main: thestatecheck and therecover_stateoverride are new here, becausemainrecovers lost state per request.test_health_recovers_lost_statecovers it.routes.py).create_app()refactor here: these tests use the module-levelapp. The refactor is in the metrics PR, which needs an app built from different params.