Skip to content

feat(api): add a health check endpoint and healthcheck command - #524

Merged
jsenecal merged 3 commits into
mainfrom
feat/519-health-check
Nov 21, 2026
Merged

jsenecal merged 3 commits into
mainfrom
feat/519-health-check

Conversation

@jsenecal

@jsenecal jsenecal commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #519 (health check half; Prometheus metrics follow in a stacked PR).

Problem

There is no reliable way to tell whether hyperglass can serve queries. The UI root keeps answering 200 through a Redis outage, the image has no HEALTHCHECK, and compose.yaml works around that by wget-ing /api/info, which only covers part of the picture.

Change

  • GET /api/health (not in the OpenAPI schema). Checks, in order:

    • redis: ping over a dedicated client with 3 s connect/read timeouts, in a worker thread. The shared pool has no timeouts, so an unreachable Redis could otherwise hang the check. The client is kept per DSN, so repeated probes reuse its connection.
    • state: ensure_ready(), i.e. configuration is in Redis or can be recovered (same recovery other requests trigger). Skipped (reported error) when Redis is down.
    • ui: static/ui/index.html exists, or disabled when HYPERGLASS_DISABLE_UI is set.

    Each worker runs the checks at most once per second: requests that arrive during a run wait for it, requests in the next second reuse its result, and the run is shielded so a disconnecting client doesn't cancel it for the others. A flood therefore costs one probe and two thread-pool jobs per second per worker.

    Answers 200 / 503 with {"status", "checks"}. Failure details are logged, not returned. The handler replaces the app-wide recover_state hook (new skip_state_recovery in middleware.py), because that hook reads Redis through the unbounded shared pool before the handler runs; recovery happens inside the handler after the bounded probe instead.

  • hyperglass healthcheck requests the endpoint on HYPERGLASS_HOST/HYPERGLASS_PORT, maps 0.0.0.0 → 127.0.0.1 and :: → ::1, brackets IPv6, never uses HTTP_PROXY, exits 1 when unhealthy or unreachable. --timeout (default 8 s, under the 10 s Docker timeout).

  • Dockerfile HEALTHCHECK --interval=30s --timeout=10s --start-period=10m --retries=3 running that command as the image user (UID 10001). The 10 min start period covers the first-boot UI build.

  • compose.yaml: keeps its wget /api/info healthcheck, with a comment to remove it once the file pins an image that has the HEALTHCHECK. compose.yaml on main still pins 3.0.0-rc.2, which has no HEALTHCHECK, no hyperglass healthcheck and no /api/health, so removing it now would leave the default deployment with no health status. Until then the Compose check overrides the image's, which the Docker guide says.

  • Docs: new configuration/config/monitoring page (nav "Health & Monitoring") and a "Health check" section in the Docker guide; the monitoring page points to it for the Compose override.

Tests

New hyperglass/util/tests/test_health_endpoint.py (ok; Redis unreachable; app-wide state hook not run when Redis is unreachable; lost state recovered; unrecoverable state; UI not built; UI disabled; not in schema; probe uses short timeouts; probe fails fast against TEST-NET-1; one Redis client reused across probes; result reused within the interval; checks again after it; concurrent requests share one probe; a disconnecting request doesn't cancel the shared probe) and hyperglass/cli/tests/test_healthcheck.py (healthy, unhealthy, 0.0.0.0, ::1 and :: over IPv6, ignores http_proxy, unreachable). Mutation-checked: removing the before_request override, the proxy bypass, client reuse, result reuse or the shield makes the corresponding tests fail.

$ pytest hyperglass --ignore hyperglass/plugins/external -q -p no:cacheprovider
4168 passed, 4 skipped, 185 warnings in 79.92s (0:01:19)
$ ruff check .
All checks passed!

Image built locally (docker build) and run as UID 10001 next to valkey/valkey:8-alpine:

$ docker inspect --format '{{json .Config.Healthcheck}} user={{.Config.User}}' hc519-hg
{"Test":["CMD","python3","-m","hyperglass.console","healthcheck"],"Interval":30000000000,"Timeout":10000000000,"StartPeriod":600000000000,"Retries":3} user=10001:10001
$ docker exec hc519-hg python3 -m hyperglass.console healthcheck
hyperglass is healthy
$ curl -s -w ' %{http_code}\n' http://127.0.0.1:18001/api/health
{"status":"ok","checks":{"redis":"ok","state":"ok","ui":"ok"}} 200

Health went starting → healthy after the UI build. After docker stop of Valkey, three checks returned {"status":"error","checks":{"redis":"error","state":"error","ui":"ok"}} and the container went unhealthy; after starting Valkey again it went back to healthy.

Changelog

### Added
- `GET /api/health` reports whether Redis is reachable, configuration is loaded and the UI is built (`200`/`503`), and `hyperglass healthcheck` queries it on the configured IPv4 or IPv6 listen address. The container image now has a Docker `HEALTHCHECK` using it (#519).

Notes

  • Ported from hyperglass-ng's health check, adapted to main: the state check and the recover_state override are new here, because main recovers lost state per request.
  • Valkey restored its data from its RDB snapshot on restart in the container test, so that run did not exercise state recovery; test_health_recovers_lost_state covers it.
  • The 3 s Redis probe timeout is an internal bound, not a user-visible setting (commented in routes.py).
  • No create_app() refactor here: these tests use the module-level app. The refactor is in the metrics PR, which needs an app built from different params.

Operators and container runtimes had no reliable way to tell whether
hyperglass can serve queries. The UI root keeps answering 200 through a
Redis outage, and the Compose file worked around that by fetching
/api/info, which only covers part of the picture and has no IPv6 or
custom-host support.

GET /api/health checks Redis with a dedicated short-timeout connection in
a worker thread (the shared pool has no timeouts, so an unreachable Redis
could hang the check), then that configuration state is loaded or can be
recovered, then that the UI is built (or disabled). It answers 200 or 503
and logs failure details instead of returning them. The handler replaces
the app-wide state recovery hook, which reads Redis through the shared
pool, and runs recovery itself after the bounded probe.

`hyperglass healthcheck` requests the endpoint on the configured listen
address, maps 0.0.0.0 and :: to the loopback address of the same family,
and bypasses HTTP proxies. The image gets a HEALTHCHECK using it, with a
10 minute start period because the UI is built on first boot before the
server listens. The Compose healthcheck is removed in favour of the
image's, so an older image without the command shows no health status
rather than permanently unhealthy.

Part of #519.
@github-actions github-actions Bot added docs Improve documentation backend Python backend: API, models, state, execution container Docker image and compose labels Nov 20, 2026
@coveralls

coveralls commented Nov 20, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 85.355% (+0.2%) from 85.11% — feat/519-health-check into main

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 60f49ddc.

The endpoint, the command and the tests read well: the response discloses only ok, error or disabled per check, the handler never reaches a device, and Part of #519 is right. Two points have to change before approval:

  1. GET /api/health is unauthenticated and opens a new Redis connection on every request. See the comment on hyperglass/api/routes.py.
  2. The default compose.yaml loses its health status, because it still pins an image that has no HEALTHCHECK. See the comment on compose.yaml.

#528 also changes Dockerfile, in the build stage only, so the two do not conflict. #526 carries this commit and needs a rebase after this merges.

Comment thread hyperglass/api/routes.py Resolved
Comment thread compose.yaml Resolved
@WilhelmZA WilhelmZA added the reviewing Issue is being reviewed by maintainers label Nov 20, 2026
/api/health is unauthenticated, and every request opened a new Redis
connection and used two jobs on the shared thread pool. A client that
floods it could open connections to Redis faster than they close.

- Each worker now runs the checks at most once per second. Requests that
  arrive during a run wait for it, and requests in the next second reuse
  its result. A request that disconnects does not cancel the shared run.
- The Redis probe reuses one client with the short timeouts, so repeated
  probes share a connection.

compose.yaml keeps its /api/info health check. It pins 3.0.0-rc.2, which
has no HEALTHCHECK, so removing it would leave the default deployment
without a health status until it pins an image that has one.

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b36846a0.

Both points from the earlier review are resolved. /api/health now keeps one Redis client per DSN and runs the checks at most once per second per worker, with concurrent requests sharing one probe, and the tests cover it. compose.yaml keeps its /api/info check until it pins an image that has the HEALTHCHECK. CI gate passes, both commits are verified, and I found no code defect in the current diff.

One thing to change before approval:

  • The PR description still describes the first commit. The compose.yaml bullet says the wget /api/info healthcheck is removed, and the Changelog block has a ### Changed entry saying compose.yaml no longer defines its own healthcheck. b36846a0 restored that healthcheck, so that entry would put a change that did not happen into the release notes. Please update the bullet and drop the ### Changed entry.

Nits:

  • One optional docs suggestion inline.

@jsenecal
jsenecal requested a review from WilhelmZA November 21, 2026 01:54

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 2e1348c3.

Both points from the last review are resolved: the description now says compose.yaml keeps its /api/info check and the ### Changed changelog entry is gone, and the docs sentence is in monitoring.mdx. The only change since b36846a0 is that one sentence; the rest of the diff is unchanged. CI gate passes and all three commits are verified.

One thing to watch at merge, not a defect in this PR: #535 also changes hyperglass/cli/main.py and adds test_commands_cover_cli in hyperglass/cli/tests/test_help.py, which requires the registered commands to equal a fixed COMMANDS tuple. That tuple has no ("healthcheck",) entry. The two branches merge without a text conflict, so whichever merges second leaves main failing that test unless it adds the entry. If #535 merges first, please update this branch, add the entry and let CI run again before merging.

@WilhelmZA

Copy link
Copy Markdown
Collaborator

One addition to the merge note in my approval at 2e1348c3. If #535 merges first, this branch needs a second change besides the ("healthcheck",) entry in COMMANDS: hyperglass/cli/tests/test_healthcheck.py line 18 creates CliRunner(mix_stderr=False), and the Typer version that #535 locks no longer accepts mix_stderr, so the module fails at import. CliRunner() is the form #535 uses in the other CLI tests.

@jsenecal
jsenecal merged commit 8398ce8 into main Nov 21, 2026
16 checks passed
jsenecal added a commit that referenced this pull request Nov 21, 2026
Typer 0.29 ships its own CliRunner, which takes only charset and env, so CliRunner(mix_stderr=False) in test_healthcheck.py raised TypeError at collection once #535 lifted the Typer hold, and main's CI broke. The healthcheck command prints only to stdout and the tests check exit codes, so a plain CliRunner() is enough. That collection error also hid test_help.py::test_commands_cover_cli, which fails because COMMANDS lacked healthcheck (#524); it is added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Python backend: API, models, state, execution container Docker image and compose docs Improve documentation reviewing Issue is being reviewed by maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants