Repository navigation
Conversation
app_handler returned every HyperglassError's own message with a status taken from its level. Private errors default to the warning level, so an internal failure answered 400 with text such as Redis key names or config paths. Only public errors, whose messages come from params.messages, are now returned as raised. Any other hyperglass error is logged and answered like an unexpected error: 500 with messages.general. A StateError during a request is the temporary condition recover_state reports, so it gets the same 503. Closes #539
WilhelmZA
left a comment
There was a problem hiding this comment.
Reviewed at 6378882d.
I read the diff against fix/529-redis-outage-503. app_handler now returns only PublicHyperglassError messages as raised, sends StateError to the 503 handler and sends every other HyperglassError to the generic 500. Validation answers are unchanged because InputValidationError still becomes the public InputInvalid. CI gate passes and the commit is verified.
Before approval:
- This changes HTTP statuses for API clients. A parsing error or a query-time configuration error returned 400 and now returns 500. State lost during a request returned 400 and now returns 503. It needs a
CHANGELOG.mdentry under[Unreleased]marked**API change:**, like the entries for #412 and #426, so clients that branch on 400 are warned. - The base is not
main, soCloses #539does not take effect on this merge, and the #538 body only closes #529. Please retarget this PR tomainafter #538 merges, or close #539 by hand once the fix is onmain.
#526 also changes hyperglass/api/error_handlers.py and adds record_query_error calls in the same handlers.
|
Thanks.
|
WilhelmZA
left a comment
There was a problem hiding this comment.
Approving at 6281df18.
The new commit adds the CHANGELOG.md entry with the **API change:** note, and it matches app_handler at this head: a state error returns 503, other internal errors return 500 with messages.general, and public errors are unchanged. CI gate passes and both commits are verified.
Closes #539 takes effect only after this PR is retargeted to main, as described in the comment above, so #538 has to merge first.
Nits:
CHANGELOG.md: the entry says these internal errors returned HTTP 400. That holds for the defaultwarninglevel and forerror. An internal error raised with leveldangeralready returned 500 throughSTATUS_CODE_MAP, and only its body changes. "usually returned HTTP 400" would be exact.
|
Thanks. Agreed that "usually returned HTTP 400" is exact. To keep your approval on this head, I'll apply that wording when the Unreleased entries are consolidated in the release PR. |
Stacked on #538 (#529); review that first. This PR is the last commit.
Problem
app_handlerhandles everyHyperglassErrorand returns the exception's own message with a status taken from its level. That is right forPublicHyperglassError, whose messages come fromparams.messages. ForPrivateHyperglassError, which is meant for internal errors, it is wrong twice:warninglevel, whichSTATUS_CODE_MAPmaps to 400, so the client is told it sent a bad request when the server failed.One reachable case: when state goes missing from a running Redis mid-request,
HyperglassState._requiredre-raisesStateErrorand the client gets a 400 with'params' ('hyperglass.state.params') does not exist in Redis store.Change
hyperglass/api/error_handlers.py,app_handleronly:PublicHyperglassError: unchanged. Same status, message, level and keywords.StateError: goes to fix(api): answer 503 with a JSON body while Redis is unreachable #538'sunavailable_handler, so 503 with the fixedSTATE_UNAVAILABLEmessage, the same answerrecover_stategives a moment earlier in the request.HyperglassErrorthat is neither public nor private: goes todefault_handler, which logs the real message and answers 500 withmessages.general, read through fix(api): answer 503 with a JSON body while Redis is unreachable #538'sgeneral_message()so the handler still answers when params can't be read.The check is "is it public", not "is it private", so an error class nobody has classified is internal by default.
Audit of private errors raised during a request
InputValidationErrorvalidateQuery.__init__converts it to the publicInputInvalid, which usesmessages.invalid_input. Unchanged.ParsingErrormessages.general. The texts are hard-coded developer strings ("Error parsing response data", "Failed to parse FRR traceroute output: {exc}"), not operator-configured messages, and some include parser exception text. A parsing failure is a server-side failure, not a bad request.ConfigErrorConstruct.format(directive uses a device attribute that isn't defined),_shell.py(custom shell command quoting)messages.general. These describe the operator's configuration (directive, attribute and device names) and the client can't fix them.StateErrorHyperglassState._required,RedisManager.get(..., raise_if_none=True)STATE_UNAVAILABLE, as above.ExternalErrorBaseExternal(webhooks, which run as a background task after the response)UnsupportedDevice,ConfigInvalid,ConfigMissing,ConfigLoaderMissing,PluginError,UnsupportedError,DependencyErrorHyperglassErrorroutes.pywhen a query returnsNonemessages.general, which is what that line means to send. See Notes.Device and driver failures (
AuthError,DeviceTimeout,RestError,ScrapeError,ResponseEmpty,DeviceCommandError,QueryLocationNotFound,QueryTypeNotFound,InputInvalid,InputNotAllowed) are all public errors with operator-configured messages, so what users see for them doesn't change. No private error was found that a route raises on purpose for the client to read, so none was converted to a public error.UI
No change needed.
useLGQueryreads the JSON body whatever the HTTP status, andResultshows any body whoselevelisn'tsuccessas an alert.dangermaps to theerroralert, covered by the existing "shows a danger response at the error level" test inindividual.test.tsx. A parsing or configuration failure now shows the configured general message in a red alert instead of the internal text in an orange one.Tests
New
hyperglass/exceptions/tests/test_private_error_response.py. Errors are raised wherePOST /api/queryruns the device query, with an IPv4 and an IPv6 target:ParsingError,ConfigError(danger level, asConstructraises it),ExternalErrorand a plainHyperglassErroreach return 500 with{"output": messages.general, "level": "danger", "keywords": []}, and the internal text is absent from the response.paramsdeleted from Redis mid-request, then read as a route reads it: 503 with theSTATE_UNAVAILABLEbody, and no Redis key text.InputInvalid(warning) answers 400 andDeviceTimeout(danger) answers 500, each with its own message, level and keywords.app_handlercalled directly with Redis down: a private error still answers 500 with the fallback message.#538's
stateandclientfixtures moved fromtest_redis_outage_response.pytohyperglass/exceptions/tests/conftest.pyso both files can use them (importing fixtures from a test module trips Ruff F811). They aren't otherwise changed.Changed assertion:
test_arcos.py::test_malformed_output_never_returns_device_textasserted that aParsingErrorreached the client as"Error parsing ArcOS response data". It now asserts a 500 withmessages.general. Its point, that no device text leaks, still holds.Without the fix (#538's
error_handlers.py, new tests kept):Every failure is the status:
assert 400 == 500orassert 400 == 503. Withtest_arcos.pyincluded, 21 fail (its 10 updated cases too). Removing only theStateErrordispatch fails the 2 mid-request state tests. Removing only the public check fails the other 19.With the fix:
Changelog
Notes
error_handlers.py.git merge-treeof this branch withfeat/519-prometheus-metricsmerges without conflicts. Once both are in, internal errors reach feat(api): add optional Prometheus metrics #526'srecord_query_errorcall indefault_handlerand return before the one inapp_handler, so each is counted once. AStateErrorgoes tounavailable_handlerand isn't recorded, the same open question fix(api): answer 503 with a JSON body while Redis is unreachable #538 notes for Redis outages.StateErroranswers 503, including the oneuse_state()raises for an attribute that doesn't exist onHyperglassState. That is a programming error, not an outage, but it can't be reached with a correct call andrecover_statealready answers everyStateErrorwith 503.routes.pyraisesHyperglassError(message=..., alert="danger")when a query returnsNone.HyperglassError.__init__has noalertargument, so this raisesTypeErrorand reachesdefault_handler. The client gets the 500 withmessages.generalthe line was written to send, so the outcome is the same, but the log says "Error" with aTypeError. Fix:level="danger".ConfigError(..., level="danger")inexecution/drivers/_construct.pydoesn't set the level.PrivateHyperglassError.__init__takeslevelas a format keyword, so the error stays atwarningand was answered 400. With this PR the level no longer affects the response.ParsingError('Error from device: "{}"', err)inbgp_route_juniper.pyandPluginError("...", repr(cls))inplugins/_manager.pypass a second positional argument, whichPrivateHyperglassError.__init__doesn't accept, so they raiseTypeErrorinstead. In the Juniper plugin, reached when a device answers with anxnm:error, the plugin's ownexcept Exceptionturns thatTypeErrorintoParsingError("Error parsing response data"), so the device's error text was never shown. It is now a 500 withmessages.general.HyperglassError.keywordsdeduplicates through aset, so keyword order varies between reads. The new public-error test compares them sorted.Closes #539