Skip to content

fix(api): answer 503 with a JSON body while Redis is unreachable - #538

Open
jsenecal wants to merge 1 commit into
mainfrom
fix/529-redis-outage-503
Open

jsenecal wants to merge 1 commit into
mainfrom
fix/529-redis-outage-503

Conversation

@jsenecal

Copy link
Copy Markdown
Collaborator

Problem

While Redis is unreachable, API requests get Litestar's plain-text 500 instead of a 503 with a JSON body.

  1. recover_state, the app-wide before_request hook, turns only StateError into a 503. The redis-py ConnectionError raised by use_state().ensure_ready() (from is_populated, before any rebuild is attempted) passes through.
  2. It lands in default_handler, the Exception handler, which reads state.params.messages.general from the same Redis. The handler raises in turn, and Litestar sends its own plain 500 with no JSON body.

The same happens when Redis goes away after the hook passed, for example at the query cache read in POST /api/query.

Change

  • hyperglass/api/error_handlers.py:
    • default_handler sends redis-py ConnectionError and TimeoutError to a new unavailable_handler, which answers 503 with {"output": ..., "level": "danger", "keywords": []}, the shape http_handler already gives the missing-state 503 and the shape the UI reads. The subclasses BusyLoadingError, AuthenticationError and MaxConnectionsError are covered too. Other Redis errors, such as ResponseError, are bugs and stay 500. The Redis error text names the Redis host, so it is logged, not returned.
    • default_handler gets its message from general_message(), which returns params.messages.general and falls back to the fixed message when params cannot be read, so the handler never fails in turn.
  • hyperglass/state/store.py: the fixed message ensure_ready raises ("Hyperglass state is unavailable"), which recover_state returns with its 503, becomes the constant STATE_UNAVAILABLE, with a comment saying why it can't come from params.messages. Both new code paths use it, so there is still one fallback message.
  • hyperglass/api/middleware.py: docstring only. recover_state is not changed. Exceptions raised in a before_request hook go through the app's exception handlers, so one mapping in default_handler covers the hook and route handlers. The alternative, catching Redis errors in recover_state too, would put the same mapping in two places.

The other handlers (http_handler, app_handler, validation_handler) don't read state, so they needed no change.

Redis errors during a request

These also get a 503. Redis going away mid-request is the same temporary condition the hook reports a moment earlier, and the next request either succeeds or gets the hook's 503. A 500 would read as a bug in hyperglass. The handler can't tell whether the error came from the hook or a route, so both get the same answer. If Redis fails after a device query has run, while the result is being cached, that result is lost and the client gets a 503. Retrying is the right response.

Tests

New hyperglass/exceptions/tests/test_redis_outage_response.py. Redis goes down by disconnecting the test's fakeredis server (FakeServer.connected = False), so every command raises redis-py's ConnectionError:

  • POST /api/query, GET /api/devices and GET /api/info with Redis down return 503 with the JSON body above, and no Redis error text.
  • Redis lost after the hook passed, at the query cache read (RedisManager.get_map raising ConnectionError, with an IPv4 and an IPv6 target): 503 with the same body.
  • default_handler directly: ConnectionError and TimeoutError give 503; with Redis down, an unrelated error and a Redis ResponseError both still give a 500 with the fallback message instead of raising; with Redis up, it still uses messages.general.

Without the fix (origin/main's error_handlers.py, new tests kept):

FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_redis_down_before_request_returns_503[query]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_redis_down_before_request_returns_503[devices]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_redis_down_before_request_returns_503[info]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_redis_down_during_request_returns_503[192.0.2.0/24]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_redis_down_during_request_returns_503[2001:db8::/32]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_default_handler_answers_redis_outage_with_503[connection]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_default_handler_answers_redis_outage_with_503[timeout]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_default_handler_answers_without_params[unrelated]
FAILED hyperglass/exceptions/tests/test_redis_outage_response.py::test_default_handler_answers_without_params[redis-response-error]
9 failed, 1 passed in 0.82s

The before-request cases fail with redis.exceptions.ConnectionError: FakeRedis is emulating a connection error. raised out of default_handler. The mid-request cases return 500. Removing only the 503 dispatch fails 7 tests. Removing only the params fallback fails the 2 answers_without_params tests.

With the fix:

$ .venv/bin/pytest hyperglass --ignore hyperglass/plugins/external -q -p no:cacheprovider
4178 passed, 4 skipped in 89.83s (0:01:29)
$ .venv/bin/ruff check .
All checks passed!

Changelog

Notes

  • Not changed here: the shared Redis connection pool has no socket timeouts, so a request to an unreachable Redis host can still hang about 15 s before it gets the 503. That is Unauthenticated requests on any route can exhaust the shared thread pool #532. /api/health already uses its own short-timeout client and skips recover_state, and is unchanged.
  • feat(api): add optional Prometheus metrics #526 (Prometheus metrics) also edits error_handlers.py. git merge-tree of this branch with feat/519-prometheus-metrics merges without conflicts. Whichever merges second should decide whether unavailable_handler also calls record_query_error. As it stands, Redis-outage errors return from default_handler before reaching feat(api): add optional Prometheus metrics #526's call.
  • Not changed here, and found by reading the code rather than by a test: if state goes missing from Redis mid-request while Redis is still up, the StateError raised by HyperglassState._required goes to app_handler. That handler answers 400, because StateError's level is warning, and returns the internal message, such as 'devices' ('hyperglass.state.devices') does not exist in Redis store. A 503 with the fixed message would be more accurate. That is a separate issue.

Closes #529

A Redis connection error or timeout, raised by the state hook or later in a
request, reached default_handler, which read params from the same Redis,
raised again and left Litestar to send a plain-text 500. default_handler
now answers those errors with 503 and the fixed message used when state
is missing, and falls back to that message when params cannot be read.
@jsenecal jsenecal added backend Python backend: API, models, state, execution bug Confirmed bug or unexpected behavior labels Nov 21, 2026
@jsenecal jsenecal self-assigned this Nov 21, 2026
@jsenecal
jsenecal requested a review from WilhelmZA November 21, 2026 04:46
@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 85.435% (+0.08%) from 85.355% — fix/529-redis-outage-503 into main

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 8d71babe.

The handler change is correct. redis-py ConnectionError and TimeoutError now reach unavailable_handler and get a JSON 503 in the shape the UI already reads, and the 500 path no longer depends on Redis to build its message. CI gate passes and the commit is verified.

Before approval:

  • Closes #529 closes the issue on merge, but #529 also lists the roughly 15 second hang from the Redis pool having no socket timeouts, and says that could be part of this fix or a follow-up. Please either change the keyword to Part of #529, or note on #529 that #532 now carries the socket timeout item so nothing is lost when it closes.

#543 is stacked on this branch and #526 also changes hyperglass/api/error_handlers.py. After both merge, a Redis outage on POST /api/query returns through unavailable_handler before record_query_error, so it is not counted in the query error metric. Whichever merges second should settle that.

Nits:

  • The body has a Changelog entry but CHANGELOG.md is not edited. Please add it under [Unreleased], here or in the release PR.

@WilhelmZA WilhelmZA added the reviewing Issue is being reviewed by maintainers label Nov 21, 2026
@jsenecal

Copy link
Copy Markdown
Collaborator Author

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Python backend: API, models, state, execution bug Confirmed bug or unexpected behavior reviewing Issue is being reviewed by maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Requests during a Redis outage get a bare 500 instead of 503

3 participants