Repository navigation
feat(api): let a forced query bypass the cache, at most once per interval - #566
Conversation
WilhelmZA
left a comment
There was a problem hiding this comment.
Reviewed at 129df4f9.
I read hyperglass/api/refresh.py, the route change and the model change. The bound holds: the SET NX lease, the Lua compare-and-set for the cooldown and the guarded cooldown write inside the result transaction allow one forced run per cache key per min_interval plus run time, for any number of clients or workers. With refresh.enable false, force is ignored and unforced requests make no extra Redis call. force is a boolean, stays out of digest() and never reaches command templating. CI gate passes and all five commits are verified.
Before approval:
- #552 ends with an open question for the maintainers on the semantics and defaults, and
Closes #552closes it on merge. That question needs an answer on the issue first. This PR also deviates from the issue in two places the body explains (a TTL key in place ofexecuted_at, and<key>.refreshin place of<key>:refresh), so the answer should cover those. hyperglass/api/refresh.py: theLEASE_MARGINcomment andcaching.mdxsay only one forced run happens at a time. The lease covers the coroutine only._execute_querycancels the await atrequest_timeout - 1, but the netmiko work runs inrun_in_executorand its thread keeps the device session open after the cancellation, with a connect timeout ofrequest_timeout * 1.25. Withrequest_timeout: 90andmin_interval: 30, a device that hangs during a forced run can still hold a session at 119 seconds when the next forced request takes the lease and opens a second one. The start rate stays bounded and unforced misses onmainbehave the same way. Please either reword both places to "at most one forced run starts per interval", or set a longer cooldown after a device timeout.
#526 also changes hyperglass/api/routes.py. Whichever merges second needs to count forced runs as bypass in the cache metric.
Nits:
CHANGELOG.md:forceis now a declared boolean, so a value that cannot be read as a boolean gets HTTP 400 even with refresh disabled.caching.mdxsays so and the changelog entry does not.
|
Thanks.
|
WilhelmZA
left a comment
There was a problem hiding this comment.
Approving at e22d4ce5.
The new commit changes only comments, docs and CHANGELOG.md. The docstring, the LEASE_MARGIN comment, caching.mdx, the complete-config comment and the changelog now say that at most one forced run starts per interval and that a session which outlives the timeout can overlap the next run. That matches the code. The changelog also states that a force value that cannot be read as a boolean gets HTTP 400. The semantics and defaults are confirmed on #552. CI gate passes and all six commits are verified.
#526 also changes hyperglass/api/routes.py. Whichever merges second counts forced runs as bypass in the cache metric.
Nits:
caching.mdx: the "at most one starts per interval" statement has one more exception. If a worker dies during a forced run, no cooldown is written and the lease expires afterrequest_timeout + 5seconds. When that is less thanmin_interval(for examplerequest_timeout: 10withmin_interval: 30), the next forced run can start early. The defaults are not affected. A sentence next to the eviction note would cover it.
|
Thanks, and thanks for settling #552. Agreed on the dead-worker exception. If |
|
Rebased onto
The dead-worker note for |
e22d4ce to
b190d05
Compare
|
Also folded in the dead-worker note you suggested, since this head needs a fresh review anyway. It is in |
WilhelmZA
left a comment
There was a problem hiding this comment.
Reviewed at e089190f.
The code is good. refresh.py is unchanged from the head I approved. In query() every path records one cache result: hit for a cached answer, bypass only for a forced run that won the lease, miss otherwise. Success is recorded after the response is built, so a Redis failure in refresh_in() no longer counts as both success and error, and that order does not change what #526 shipped. The metrics helpers cannot raise, so no lease is left without a cooldown. The dead-worker note in caching.mdx matches the lease TTL in the code. All nine commits are verified, and CI gate passed on the run merged into main at f89ab22a.
Before approval:
- #564 has merged and
CHANGELOG.mdnow conflicts withmain. Please rebase and I will approve that head onceCI gatepasses on it.
Nits:
hyperglass/api/routes.py: the duration is observed before the result pipeline andrefresh_in(), which can still fail. A query can then add a duration sample and also count as an error, while the docs table describes the histogram as successful queries. The pipeline case already behaves this way onmain. Movingrecord_query_durationnext torecord_query_successwould make the two agree.
…rval POST /api/query accepts force: true. With cache.refresh.enable (off by default) a forced query for a cached result runs on the device again and replaces the entry, once the result is cache.refresh.min_interval seconds old (default 30, below cache.timeout). A per-key Redis lease allows one forced run at a time across workers and becomes a cooldown of min_interval when the run ends, successful or not. Requests that are refused get the cached result. Every response carries refresh_in.
Every run of a query now sets <cache key>.refresh to a cooldown that expires after cache.refresh.min_interval, in the transaction that writes the result, and a forced query takes that key with SET NX. That replaces the executed_at field, so no worker clocks are compared. The lease outlives the device deadline (request_timeout - 1) by five seconds, and a run whose lease expired still starts a cooldown. A Redis error while starting the cooldown is logged instead of replacing the query's result or error. During another request's forced run, refresh_in is min_interval rather than the lease's remaining time plus min_interval.
A run that wrote its result set the refresh key to the cooldown unconditionally. When the cached entry expired during a forced run, an unforced run that finished first replaced the forced run's lease, so a second forced run could start once that cooldown expired. The write now starts the cooldown only over no key, a cooldown or the request's own lease. Params are read once for the refresh settings.
UIParameters typed cache as the full Cache model, so validating the projected parameters re-added refresh with its defaults, whatever the configuration said. A UICache model with only timeout and show_text keeps it out.
The forced-refresh lease covers the request, not the Netmiko session, which runs in an executor thread that keeps going after the request timeout. A device that hangs during one forced run can still hold its session when the next forced run starts. Reword the docs, comments and changelog to say at most one forced run starts per interval, and note in the changelog that a non-boolean force is rejected with HTTP 400.
A forced run skips a cached result and runs on the device, but the query cache metric counted it as a miss. Count it as `result="bypass"`, so miss plus bypass is the number of device runs and a miss still means there was no cached result. A forced query answered from the cache, during a cooldown or while another request's forced run holds the lease, stays a hit: the client gets the cached result and the device is not queried. A forced query with no cached result is a miss, and with refresh disabled, where force is ignored, a forced query counts as it did before. A forced run's outcome, failure reason and duration are counted like any other run's; the duration metric's help text now says it covers bypasses.
The rebase onto the metrics left record_query_success before refresh.refresh_in(), which reads Redis while the response is built. If that read failed, the query was counted as both a success and an error. Record the success last, after everything that can fail. Also say that only queries reaching the cache lookup are counted in the cache metric, and assert the exact status of a failed forced run.
The device run's duration was observed before the cache write and refresh_in(), both of which use Redis and can still fail. Such a query added a duration sample and was also counted as an error, while the monitoring docs describe the histogram as successful queries only. Measure the device run where it was, but record its duration next to the success, after everything that can fail. record_query_duration now takes None for a query answered from the cache, which keeps the route under the complexity limit.
|
Rebased onto
|
e089190 to
07b87b7
Compare
WilhelmZA
left a comment
There was a problem hiding this comment.
Approving at 07b87b79.
The nine earlier commits are the patches reviewed at e089190f; the range diff shows only CHANGELOG.md context from #564. The new commit moves record_query_duration next to record_query_success, after the cache write and refresh_in(). The elapsed time starts as None, is set only around the device run, and the helper records nothing for None, so a cache hit adds no sample and a query that fails after the device run adds neither a success nor a duration. The test now covers both a failed cache write and a failed refresh_in(). The branch contains current main, CI gate passes on it and all ten commits are verified.
Base:
main. Piece F1 of #550, the server side of jsenecal#6's refresh. Backend only; the UI is F2 (#553).Problem
Within
cache.timeoutevery request for the same query gets the cached result, and each hit renews the entry's TTL, so there is no way to get fresh output from the device until the entry expires. The fork added aforceflag with a cooldown enforced only in the browser, so any client could sendforce: trueon every request and turn the cache into a pass-through.A cooldown checked only against the entry's
timestampis not enough either:query()writes the entry only after the device answers, so N forced requests that arrive during one run all see the old entry and all reach the device. Errors are not cached, so a device that keeps failing could also be forced back to back.Change
The semantics below are the proposal from #550 Q1 / #552, pending maintainer confirmation.
Config, off by default (
models/config/cache.py):Not sent to the UI.
params.frontend()projects onlycache.{show_text,timeout}, butUIParameters(models/ui.py) typedcacheas the fullCachemodel, so validation re-addedrefreshwith its defaults whatever the configuration said. A newUICachemodel with onlytimeoutandshow_text(whattypes/config.ts_Cachedeclares) keeps it out, and the UI test fixture is unchanged frommain. F2 can addenableif it needs it.Query.force: bool = False. It is not in__repr__, sodigest()and the cache key are the same with or without it. Withrefresh.enable: false,forceis ignored, as today.api/refresh.py(new),ForcedRefresh, used fromquery(). With refresh enabled, one Redis key per query,hyperglass.state.query.<digest>.refresh, carries both the lease and the cooldown:cooldownwithEX min_interval, in the sameMULTIpipeline that writes the result. That write is a Lua script that sets the cooldown only over no key, a cooldown or the request's own lease, never over another request's forced run. Otherwise an unforced run that finished during a forced run (the entry having expired meanwhile) would replace that run's lease, and a second forced run could start once the cooldown expired.SET <key> <token> NX EX request_timeout + 5. If the key exists (a cooldown, or another forced run), the request gets the cached entry at once rather than waiting: nothing holds a connection open polling Redis, and the next request after the run gets the new entry anyway. The lease outlives the request deadline (request_timeout - 1,_execute_query) by 6 s, so it can't expire while a late request is still writing its result; a lease left by a worker that died expires after that. The lease covers the request, not the device session: a Netmiko session runs in an executor thread that keeps going after the deadline cancels the await (each command has its own read timeout ofrequest_timeout, counted from after the connect), so a device that hangs can still hold one forced run's session when the next forced run starts after the cooldown. Unforced misses onmainbehave the same way.cooldownthat expires aftermin_interval. It also sets the cooldown if the lease has expired meanwhile, and leaves alone a lease or cooldown that another request set since. So a failing device can't be forced back to back, and the gap between the device answering and the new entry being written is covered. A Redis error here is logged; it doesn't replace the device's error or discard a good result.DELand the new fields go in oneMULTIpipeline, so no field of the old result survives (the fork's overwrite keptenrichment_generation, which would also let an old background enrichment pass its generation check). On failure the old entry is kept.maxmemory(any policy butnoevictioncan, since the key has an expiry), a second forced run can start.caching.mdxsays so. A worker that dies mid-run leaves its lease until it expires (request_timeout + 5); meanwhile forced queries get the cached result andrefresh_inreportsmin_interval. Also incaching.mdx.executed_athash field; a TTL'd key does the same job without clock comparison, and lets oneSET NXcheck both the interval and a run in progress.Every successful
/api/queryresponse getsrefresh_in: seconds until a forced request would run (0= now), ornullwhen refresh is disabled. During a cooldown it is the cooldown's remaining TTL; while another request's forced run holds the lease it ismin_interval(that run starts a cooldown of that length when it ends; an earlier forced request just gets the cached result and an updated value).GETandPTTLare read in one transaction. Error responses don't carry it, so after a failed forced run F2 learns the cooldown from the next response. F2 can show the cooldown without comparing clocks with the server. Added toQueryResponsefor the OpenAPI schema.routes.py: 18 lines added, 9 removed;_state.paramsis read once for the refresh settings (the device call is wrapped inwith refresh.run():, re-indenting it).query()'s complexity stays at the Ruff limit of 10.external/tests/test_reverse_dns.py: itsSimpleNamespacestand-in forQuerygetsforce=False, sincequery()now reads that attribute.Docs:
caching.mdx(new "Forced refresh" section),api-docs.mdx(the body field andrefresh_in),complete-config.mdxtable and example,.samples/sample_config.yaml(and its copy incomplete-config.mdx). CHANGELOG under Added.Bound: with refresh enabled, a forced run of one query key starts on the device at most once per
min_intervalplus the request time, whatever the number of clients or workers. This bounds the start rate, not session overlap (see the lease note above).Behaviour to confirm with the semantics:
forceis now a declaredboolfield (it was accepted and ignored throughextra="allow"). A value Pydantic can't read as a boolean, such as"sometimes", now gets HTTP 400 even with refresh disabled.true/false,1/0and the usual string forms still work.Why these defaults
/api/dns,/api/aspath/enrich).min_interval30 s, strictly belowcache.timeout, and not the fork's 600 scache.timeout. An entry only lives pasttimeoutwhile hits keep renewing it, so withtimeout: 120and an interval of 120 a user's refresh would be refused until the entry had expired anyway. With 30/120, a result can be forced from 30 s until it expires. A shorttimeoutalone still validates while refresh is off.Serve cached, not 429
Inside the interval or while the lease is held, the request gets the cached result with
cached: trueandrefresh_in, not 429. The caller asked for that result and gets a valid one; it just isn't newer. A 429 would make every client handle an error for a normal button press, and the UI would have nothing to show.refresh_incarries whatRetry-Afterwould.No per-client limit
Not added.
forcebounds itself per cache key: across all clients, at most one forced run of a key starts per interval, and a per-client quota wouldn't lower that.For a free-text target, what a single client can do with
force(one extra run per key per 30 s) is less than what it can already do without it: every new target is an uncached run, with no limit at all. That is #532's scope (unauthenticated floods on every route), and a per-client limit there would coverforcetoo.For a directive with a select list of targets, or no target, that argument doesn't hold: there the cache was the bound, and
forceraises it from one run percache.timeoutto one permin_interval(4x with the defaults) for any anonymous client. That is the bound an operator opts into withcache.refresh.enable, andcaching.mdxnow says so. If maintainers want a per-client limit anyway,auxiliary.py's_RATEscript andquota_identity()would apply as for the lookups.Tests
hyperglass/util/tests/test_query_refresh.py(new, 25 tests) drivesPOST /api/querythrough Litestar's test client on fakeredis, with_execute_queryreplaced by a counting stub. It covers:digest()unchanged byforce;force: "sometimes"gives 400 without reaching the device.cached: true).refresh_in == 30 - 10.refresh_in == 30.request_timeout.forceignored andrefresh_in: nullwhen refresh is disabled.min_intervalequal to or abovetimeoutrejected, and 0 rejected; with refresh enabled in the configuration, the UI parameters (init_ui_params(...).export_json(by_alias=True)) carry onlycache: {timeout, showText}. TypingUIParameters.cacheasCacheagain fails that test (refresh: {enable: False, minInterval: 30}reappears).Without the change, the new tests can't run: the fixture's
cache.refreshconfig is rejected byParams(extra="forbid"), so most error at setup. The mutation runs below are the useful check. Each changes one part ofapi/refresh.py, keeping the tests (warning counts trimmed):M4 (no cooldown when the run raises) is caught only by the expired-lease test: with a live lease, the stuck lease blocks the retry anyway, until it expires.
The lease and cooldown logic was also run against a real Redis (the local
hyperglass-rediscontainer): the lease TTL was 95 s withrequest_timeout90, a second forced request was refused withrefresh_in30, and after deleting the lease mid-run and failing the run, the key heldcooldownwith a 30 s TTL. The reviewer's overlap case (min_interval 2: forced run F holds the lease, the entry expires, an unforced run writes first) was replayed on real Redis: F's token survived the unforced write, a second forced request was refused, and F's end turned the key into a 2 s cooldown.With the change:
pytest hyperglass --ignore hyperglass/plugins/externalgives4195 passed, 4 skipped;ruff check .givesAll checks passed!. CI's fixture step run locally (python .tests/ui-fixture/generate.pythengit diff --exit-code -- hyperglass/ui/__tests__/fixtures/config.json) leaves the fixture unchanged, and it has no diff againstorigin/main. Black and isort are clean on the new files and changed lines.Changelog
cache.refresh.enable: true,POST /api/queryaccepts"force": true, which runs a cached query on the device again and replaces its cache entry. The server answers a forced query from the cache when the query ran less thancache.refresh.min_interval(default 30 seconds) ago, or while another forced request for the same query is in progress, so at most one forced run of each query starts per interval. Successful responses includerefresh_in, the seconds until a forced query would run.forcemust be a boolean: a value that cannot be read as one is rejected with HTTP 400, even when forced refresh is disabled (Allow a query to bypass the cache, at most once per interval per query #552).Notes
git merge-treeagainstorigin/feat/519-prometheus-metricsmerges cleanly. A forced run goes through the miss branch, so feat(api): add optional Prometheus metrics #526 would count it asmiss. Whichever of the two merges second should add abypassvalue torecord_query_cache()for forced runs (refresh.forcedis the flag).fix/529-redis-outage-503): merges cleanly. The lease calls raiseRedisErrorlike the other Redis calls inquery(), so they get whatever fix(api): answer 503 with a JSON body while Redis is unreachable #538 does for those (not tested here).<key>:refresh; this uses<key>.refresh, becauseRedisManager.key()joins on dots. It lives underhyperglass.state.*, sostate.clear()on restart removes it with the entries.SETin the existing pipeline. With refresh disabled, nothing changes for unforced requests.main.refresh_inis snake_case, as Allow a query to bypass the cache, at most once per interval per query #552 specifies. The other response keys are single words, so there is no precedent either way;/api/infokeys are camelCase.retry: 3(Failed looking-glass queries are retried three times, reaching the device four times #546) re-sends a failed request. F2's one-shot flag means retries go out unforced; the cooldown bounds them anyway.Closes #552