Repository navigation
Conversation
A Redis connection error or timeout, raised by the state hook or later in a request, reached default_handler, which read params from the same Redis, raised again and left Litestar to send a plain-text 500. default_handler now answers those errors with 503 and the fixed message used when state is missing, and falls back to that message when params cannot be read.
This was referenced Nov 21, 2026
WilhelmZA
reviewed
Nov 21, 2026
WilhelmZA
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed at 8d71babe.
The handler change is correct. redis-py ConnectionError and TimeoutError now reach unavailable_handler and get a JSON 503 in the shape the UI already reads, and the 500 path no longer depends on Redis to build its message. CI gate passes and the commit is verified.
Before approval:
Closes #529closes the issue on merge, but #529 also lists the roughly 15 second hang from the Redis pool having no socket timeouts, and says that could be part of this fix or a follow-up. Please either change the keyword toPart of #529, or note on #529 that #532 now carries the socket timeout item so nothing is lost when it closes.
#543 is stacked on this branch and #526 also changes hyperglass/api/error_handlers.py. After both merge, a Redis outage on POST /api/query returns through unavailable_handler before record_query_error, so it is not counted in the query error metric. Whichever merges second should settle that.
Nits:
- The body has a Changelog entry but
CHANGELOG.mdis not edited. Please add it under[Unreleased], here or in the release PR.
Collaborator
Author
|
Thanks.
|
This was referenced Nov 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
While Redis is unreachable, API requests get Litestar's plain-text 500 instead of a 503 with a JSON body.
recover_state, the app-widebefore_requesthook, turns onlyStateErrorinto a 503. The redis-pyConnectionErrorraised byuse_state().ensure_ready()(fromis_populated, before any rebuild is attempted) passes through.default_handler, theExceptionhandler, which readsstate.params.messages.generalfrom the same Redis. The handler raises in turn, and Litestar sends its own plain 500 with no JSON body.The same happens when Redis goes away after the hook passed, for example at the query cache read in
POST /api/query.Change
hyperglass/api/error_handlers.py:default_handlersends redis-pyConnectionErrorandTimeoutErrorto a newunavailable_handler, which answers 503 with{"output": ..., "level": "danger", "keywords": []}, the shapehttp_handleralready gives the missing-state 503 and the shape the UI reads. The subclassesBusyLoadingError,AuthenticationErrorandMaxConnectionsErrorare covered too. Other Redis errors, such asResponseError, are bugs and stay 500. The Redis error text names the Redis host, so it is logged, not returned.default_handlergets its message fromgeneral_message(), which returnsparams.messages.generaland falls back to the fixed message when params cannot be read, so the handler never fails in turn.hyperglass/state/store.py: the fixed messageensure_readyraises ("Hyperglass state is unavailable"), whichrecover_statereturns with its 503, becomes the constantSTATE_UNAVAILABLE, with a comment saying why it can't come fromparams.messages. Both new code paths use it, so there is still one fallback message.hyperglass/api/middleware.py: docstring only.recover_stateis not changed. Exceptions raised in abefore_requesthook go through the app's exception handlers, so one mapping indefault_handlercovers the hook and route handlers. The alternative, catching Redis errors inrecover_statetoo, would put the same mapping in two places.The other handlers (
http_handler,app_handler,validation_handler) don't read state, so they needed no change.Redis errors during a request
These also get a 503. Redis going away mid-request is the same temporary condition the hook reports a moment earlier, and the next request either succeeds or gets the hook's 503. A 500 would read as a bug in hyperglass. The handler can't tell whether the error came from the hook or a route, so both get the same answer. If Redis fails after a device query has run, while the result is being cached, that result is lost and the client gets a 503. Retrying is the right response.
Tests
New
hyperglass/exceptions/tests/test_redis_outage_response.py. Redis goes down by disconnecting the test's fakeredis server (FakeServer.connected = False), so every command raises redis-py'sConnectionError:POST /api/query,GET /api/devicesandGET /api/infowith Redis down return 503 with the JSON body above, and no Redis error text.RedisManager.get_mapraisingConnectionError, with an IPv4 and an IPv6 target): 503 with the same body.default_handlerdirectly:ConnectionErrorandTimeoutErrorgive 503; with Redis down, an unrelated error and a RedisResponseErrorboth still give a 500 with the fallback message instead of raising; with Redis up, it still usesmessages.general.Without the fix (
origin/main'serror_handlers.py, new tests kept):The before-request cases fail with
redis.exceptions.ConnectionError: FakeRedis is emulating a connection error.raised out ofdefault_handler. The mid-request cases return 500. Removing only the 503 dispatch fails 7 tests. Removing only the params fallback fails the 2answers_without_paramstests.With the fix:
Changelog
Notes
/api/healthalready uses its own short-timeout client and skipsrecover_state, and is unchanged.error_handlers.py.git merge-treeof this branch withfeat/519-prometheus-metricsmerges without conflicts. Whichever merges second should decide whetherunavailable_handleralso callsrecord_query_error. As it stands, Redis-outage errors return fromdefault_handlerbefore reaching feat(api): add optional Prometheus metrics #526's call.StateErrorraised byHyperglassState._requiredgoes toapp_handler. That handler answers 400, becauseStateError's level iswarning, and returns the internal message, such as'devices' ('hyperglass.state.devices') does not exist in Redis store. A 503 with the fixed message would be more accurate. That is a separate issue.Closes #529