Skip to content

fix(api): stop returning internal error messages to clients - #543

Open
jsenecal wants to merge 2 commits into
fix/529-redis-outage-503from
fix/539-private-errors
Open

jsenecal wants to merge 2 commits into
fix/529-redis-outage-503from
fix/539-private-errors

Conversation

@jsenecal

Copy link
Copy Markdown
Collaborator

Stacked on #538 (#529); review that first. This PR is the last commit.

Problem

app_handler handles every HyperglassError and returns the exception's own message with a status taken from its level. That is right for PublicHyperglassError, whose messages come from params.messages. For PrivateHyperglassError, which is meant for internal errors, it is wrong twice:

  • The message is internal. It can name Redis keys, configuration paths, device attributes or parser details.
  • The status is wrong. Private errors default to the warning level, which STATUS_CODE_MAP maps to 400, so the client is told it sent a bad request when the server failed.

One reachable case: when state goes missing from a running Redis mid-request, HyperglassState._required re-raises StateError and the client gets a 400 with 'params' ('hyperglass.state.params') does not exist in Redis store.

Change

hyperglass/api/error_handlers.py, app_handler only:

The check is "is it public", not "is it private", so an error class nobody has classified is internal by default.

Audit of private errors raised during a request

Error Where it's raised in a request Decision
InputValidationError Directive rules and input plugin validate Never reaches the handler. Query.__init__ converts it to the public InputInvalid, which uses messages.invalid_input. Unchanged.
ParsingError Built-in output plugins Now 500 with messages.general. The texts are hard-coded developer strings ("Error parsing response data", "Failed to parse FRR traceroute output: {exc}"), not operator-configured messages, and some include parser exception text. A parsing failure is a server-side failure, not a bad request.
ConfigError Construct.format (directive uses a device attribute that isn't defined), _shell.py (custom shell command quoting) Now 500 with messages.general. These describe the operator's configuration (directive, attribute and device names) and the client can't fix them.
StateError HyperglassState._required, RedisManager.get(..., raise_if_none=True) 503 with STATE_UNAVAILABLE, as above.
ExternalError BaseExternal (webhooks, which run as a background task after the response) No change in practice. If one reached the handler it would now get the general 500.
UnsupportedDevice, ConfigInvalid, ConfigMissing, ConfigLoaderMissing, PluginError, UnsupportedError, DependencyError Startup, config loading and plugin registration only Not raised during a request.
Plain HyperglassError routes.py when a query returns None Internal: 500 with messages.general, which is what that line means to send. See Notes.

Device and driver failures (AuthError, DeviceTimeout, RestError, ScrapeError, ResponseEmpty, DeviceCommandError, QueryLocationNotFound, QueryTypeNotFound, InputInvalid, InputNotAllowed) are all public errors with operator-configured messages, so what users see for them doesn't change. No private error was found that a route raises on purpose for the client to read, so none was converted to a public error.

UI

No change needed. useLGQuery reads the JSON body whatever the HTTP status, and Result shows any body whose level isn't success as an alert. danger maps to the error alert, covered by the existing "shows a danger response at the error level" test in individual.test.tsx. A parsing or configuration failure now shows the configured general message in a red alert instead of the internal text in an orange one.

Tests

New hyperglass/exceptions/tests/test_private_error_response.py. Errors are raised where POST /api/query runs the device query, with an IPv4 and an IPv6 target:

  • ParsingError, ConfigError (danger level, as Construct raises it), ExternalError and a plain HyperglassError each return 500 with {"output": messages.general, "level": "danger", "keywords": []}, and the internal text is absent from the response.
  • params deleted from Redis mid-request, then read as a route reads it: 503 with the STATE_UNAVAILABLE body, and no Redis key text.
  • Public errors unchanged: InputInvalid (warning) answers 400 and DeviceTimeout (danger) answers 500, each with its own message, level and keywords.
  • app_handler called directly with Redis down: a private error still answers 500 with the fallback message.

#538's state and client fixtures moved from test_redis_outage_response.py to hyperglass/exceptions/tests/conftest.py so both files can use them (importing fixtures from a test module trips Ruff F811). They aren't otherwise changed.

Changed assertion: test_arcos.py::test_malformed_output_never_returns_device_text asserted that a ParsingError reached the client as "Error parsing ArcOS response data". It now asserts a 500 with messages.general. Its point, that no device text leaks, still holds.

Without the fix (#538's error_handlers.py, new tests kept):

FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[parsing-192.0.2.0/24]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[parsing-2001:db8::/32]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[config-192.0.2.0/24]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[config-2001:db8::/32]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[external-192.0.2.0/24]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[external-2001:db8::/32]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[base-192.0.2.0/24]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_in_route_returns_500[base-2001:db8::/32]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_state_missing_during_request_returns_503[192.0.2.0/24]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_state_missing_during_request_returns_503[2001:db8::/32]
FAILED hyperglass/exceptions/tests/test_private_error_response.py::test_private_error_answers_without_params
11 failed, 2 passed in 1.08s

Every failure is the status: assert 400 == 500 or assert 400 == 503. With test_arcos.py included, 21 fail (its 10 updated cases too). Removing only the StateError dispatch fails the 2 mid-request state tests. Removing only the public check fails the other 19.

With the fix:

$ .venv/bin/pytest hyperglass --ignore hyperglass/plugins/external -q -p no:cacheprovider
4191 passed, 4 skipped in 86.71s (0:01:26)
$ .venv/bin/ruff check .
All checks passed!

Changelog

  • Internal errors raised while answering a query, such as an output parsing failure or a configuration problem found at query time, are answered with 500 and the configured general error message. Before, they were answered with 400 and their internal message. State missing from Redis during a request is answered with 503 (Internal errors are returned to clients as 400 with their internal message #539).

Notes

  • feat(api): add optional Prometheus metrics #526 (Prometheus metrics) also edits error_handlers.py. git merge-tree of this branch with feat/519-prometheus-metrics merges without conflicts. Once both are in, internal errors reach feat(api): add optional Prometheus metrics #526's record_query_error call in default_handler and return before the one in app_handler, so each is counted once. A StateError goes to unavailable_handler and isn't recorded, the same open question fix(api): answer 503 with a JSON body while Redis is unreachable #538 notes for Redis outages.
  • Every StateError answers 503, including the one use_state() raises for an attribute that doesn't exist on HyperglassState. That is a programming error, not an outage, but it can't be reached with a correct call and recover_state already answers every StateError with 503.
  • Found while auditing, not fixed here:
    • routes.py raises HyperglassError(message=..., alert="danger") when a query returns None. HyperglassError.__init__ has no alert argument, so this raises TypeError and reaches default_handler. The client gets the 500 with messages.general the line was written to send, so the outcome is the same, but the log says "Error" with a TypeError. Fix: level="danger".
    • ConfigError(..., level="danger") in execution/drivers/_construct.py doesn't set the level. PrivateHyperglassError.__init__ takes level as a format keyword, so the error stays at warning and was answered 400. With this PR the level no longer affects the response.
    • ParsingError('Error from device: "{}"', err) in bgp_route_juniper.py and PluginError("...", repr(cls)) in plugins/_manager.py pass a second positional argument, which PrivateHyperglassError.__init__ doesn't accept, so they raise TypeError instead. In the Juniper plugin, reached when a device answers with an xnm:error, the plugin's own except Exception turns that TypeError into ParsingError("Error parsing response data"), so the device's error text was never shown. It is now a 500 with messages.general.
    • HyperglassError.keywords deduplicates through a set, so keyword order varies between reads. The new public-error test compares them sorted.

Closes #539

app_handler returned every HyperglassError's own message with a status taken
from its level. Private errors default to the warning level, so an internal
failure answered 400 with text such as Redis key names or config paths.

Only public errors, whose messages come from params.messages, are now returned
as raised. Any other hyperglass error is logged and answered like an
unexpected error: 500 with messages.general. A StateError during a request is
the temporary condition recover_state reports, so it gets the same 503.

Closes #539
@jsenecal jsenecal added backend Python backend: API, models, state, execution bug Confirmed bug or unexpected behavior security Hardening and security-relevant changes labels Nov 21, 2026
@jsenecal jsenecal self-assigned this Nov 21, 2026
@jsenecal
jsenecal requested a review from WilhelmZA November 21, 2026 05:22
@coveralls

coveralls commented Nov 21, 2026 •

Copy link
Copy Markdown

Coverage Status

Coverage is 85.505% — fix/539-private-errors into fix/529-redis-outage-503. No base build found for fix/529-redis-outage-503.

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 6378882d.

I read the diff against fix/529-redis-outage-503. app_handler now returns only PublicHyperglassError messages as raised, sends StateError to the 503 handler and sends every other HyperglassError to the generic 500. Validation answers are unchanged because InputValidationError still becomes the public InputInvalid. CI gate passes and the commit is verified.

Before approval:

  • This changes HTTP statuses for API clients. A parsing error or a query-time configuration error returned 400 and now returns 500. State lost during a request returned 400 and now returns 503. It needs a CHANGELOG.md entry under [Unreleased] marked **API change:**, like the entries for #412 and #426, so clients that branch on 400 are warned.
  • The base is not main, so Closes #539 does not take effect on this merge, and the #538 body only closes #529. Please retarget this PR to main after #538 merges, or close #539 by hand once the fix is on main.

#526 also changes hyperglass/api/error_handlers.py and adds record_query_error calls in the same handlers.

@WilhelmZA WilhelmZA added the reviewing Issue is being reviewed by maintainers label Nov 21, 2026
@jsenecal

Copy link
Copy Markdown
Collaborator Author

Thanks.

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 6281df18.

The new commit adds the CHANGELOG.md entry with the **API change:** note, and it matches app_handler at this head: a state error returns 503, other internal errors return 500 with messages.general, and public errors are unchanged. CI gate passes and both commits are verified.

Closes #539 takes effect only after this PR is retargeted to main, as described in the comment above, so #538 has to merge first.

Nits:

  • CHANGELOG.md: the entry says these internal errors returned HTTP 400. That holds for the default warning level and for error. An internal error raised with level danger already returned 500 through STATUS_CODE_MAP, and only its body changes. "usually returned HTTP 400" would be exact.

@jsenecal

Copy link
Copy Markdown
Collaborator Author

Thanks. Agreed that "usually returned HTTP 400" is exact. To keep your approval on this head, I'll apply that wording when the Unreleased entries are consolidated in the release PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Python backend: API, models, state, execution bug Confirmed bug or unexpected behavior reviewing Issue is being reviewed by maintainers security Hardening and security-relevant changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants