Skip to content

feat(api): let a forced query bypass the cache, at most once per interval - #566

Merged
jsenecal merged 10 commits into
mainfrom
feat/port-f1-force-refresh
Nov 21, 2026
Merged

jsenecal merged 10 commits into
mainfrom
feat/port-f1-force-refresh

Conversation

@jsenecal

@jsenecal jsenecal commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Base: main. Piece F1 of #550, the server side of jsenecal#6's refresh. Backend only; the UI is F2 (#553).

Problem

Within cache.timeout every request for the same query gets the cached result, and each hit renews the entry's TTL, so there is no way to get fresh output from the device until the entry expires. The fork added a force flag with a cooldown enforced only in the browser, so any client could send force: true on every request and turn the cache into a pass-through.

A cooldown checked only against the entry's timestamp is not enough either: query() writes the entry only after the device answers, so N forced requests that arrive during one run all see the old entry and all reach the device. Errors are not cached, so a device that keeps failing could also be forced back to back.

Change

The semantics below are the proposal from #550 Q1 / #552, pending maintainer confirmation.

  • Config, off by default (models/config/cache.py):

    cache:
      refresh:
        enable: false
        min_interval: 30   # > 0, and < cache.timeout when enable is true

    Not sent to the UI. params.frontend() projects only cache.{show_text,timeout}, but UIParameters (models/ui.py) typed cache as the full Cache model, so validation re-added refresh with its defaults whatever the configuration said. A new UICache model with only timeout and show_text (what types/config.ts _Cache declares) keeps it out, and the UI test fixture is unchanged from main. F2 can add enable if it needs it.

  • Query.force: bool = False. It is not in __repr__, so digest() and the cache key are the same with or without it. With refresh.enable: false, force is ignored, as today.

  • api/refresh.py (new), ForcedRefresh, used from query(). With refresh enabled, one Redis key per query, hyperglass.state.query.<digest>.refresh, carries both the lease and the cooldown:

    • Every run of the query, forced or not, sets it to cooldown with EX min_interval, in the same MULTI pipeline that writes the result. That write is a Lua script that sets the cooldown only over no key, a cooldown or the request's own lease, never over another request's forced run. Otherwise an unforced run that finished during a forced run (the entry having expired meanwhile) would replace that run's lease, and a second forced run could start once the cooldown expired.
    • A forced request for a cached entry takes it with SET <key> <token> NX EX request_timeout + 5. If the key exists (a cooldown, or another forced run), the request gets the cached entry at once rather than waiting: nothing holds a connection open polling Redis, and the next request after the run gets the new entry anyway. The lease outlives the request deadline (request_timeout - 1, _execute_query) by 6 s, so it can't expire while a late request is still writing its result; a lease left by a worker that died expires after that. The lease covers the request, not the device session: a Netmiko session runs in an executor thread that keeps going after the deadline cancels the await (each command has its own read timeout of request_timeout, counted from after the connect), so a device that hangs can still hold one forced run's session when the next forced run starts after the cooldown. Unforced misses on main behave the same way.
    • The winner runs the query. When the device answers or fails, a Lua compare-and-set turns its lease into a cooldown that expires after min_interval. It also sets the cooldown if the lease has expired meanwhile, and leaves alone a lease or cooldown that another request set since. So a failing device can't be forced back to back, and the gap between the device answering and the new entry being written is covered. A Redis error here is logged; it doesn't replace the device's error or discard a good result.
    • On success the entry is replaced: DEL and the new fields go in one MULTI pipeline, so no field of the old result survives (the fork's overwrite kept enrichment_generation, which would also let an old background enrichment pass its generation check). On failure the old entry is kept.
    • A forced request with no cached entry is an ordinary cache miss.
    • "At most one forced run starts per interval" holds as long as Redis keeps the lease key. If Redis evicts it under maxmemory (any policy but noeviction can, since the key has an expiry), a second forced run can start. caching.mdx says so. A worker that dies mid-run leaves its lease until it expires (request_timeout + 5); meanwhile forced queries get the cached result and refresh_in reports min_interval. Also in caching.mdx.
    • Only Redis TTLs are compared, never worker clocks. Allow a query to bypass the cache, at most once per interval per query #552 proposed an executed_at hash field; a TTL'd key does the same job without clock comparison, and lets one SET NX check both the interval and a run in progress.
  • Every successful /api/query response gets refresh_in: seconds until a forced request would run (0 = now), or null when refresh is disabled. During a cooldown it is the cooldown's remaining TTL; while another request's forced run holds the lease it is min_interval (that run starts a cooldown of that length when it ends; an earlier forced request just gets the cached result and an updated value). GET and PTTL are read in one transaction. Error responses don't carry it, so after a failed forced run F2 learns the cooldown from the next response. F2 can show the cooldown without comparing clocks with the server. Added to QueryResponse for the OpenAPI schema.

  • routes.py: 18 lines added, 9 removed; _state.params is read once for the refresh settings (the device call is wrapped in with refresh.run():, re-indenting it). query()'s complexity stays at the Ruff limit of 10.

  • external/tests/test_reverse_dns.py: its SimpleNamespace stand-in for Query gets force=False, since query() now reads that attribute.

  • Docs: caching.mdx (new "Forced refresh" section), api-docs.mdx (the body field and refresh_in), complete-config.mdx table and example, .samples/sample_config.yaml (and its copy in complete-config.mdx). CHANGELOG under Added.

Bound: with refresh enabled, a forced run of one query key starts on the device at most once per min_interval plus the request time, whatever the number of clients or workers. This bounds the start rate, not session overlap (see the lease note above).

Behaviour to confirm with the semantics:

  • A failed forced run returns the device's error (like any failed query), even though a valid cached entry exists. The entry is kept and the next request gets it. The alternative, answering a failed refresh with the old entry, would hide the failure from the user who asked for fresh output.
  • force is now a declared bool field (it was accepted and ignored through extra="allow"). A value Pydantic can't read as a boolean, such as "sometimes", now gets HTTP 400 even with refresh disabled. true/false, 1/0 and the usual string forms still work.

Why these defaults

  • Off by default, like every other server-side feature unauthenticated clients can drive (/api/dns, /api/aspath/enrich).
  • min_interval 30 s, strictly below cache.timeout, and not the fork's 600 s cache.timeout. An entry only lives past timeout while hits keep renewing it, so with timeout: 120 and an interval of 120 a user's refresh would be refused until the entry had expired anyway. With 30/120, a result can be forced from 30 s until it expires. A short timeout alone still validates while refresh is off.

Serve cached, not 429

Inside the interval or while the lease is held, the request gets the cached result with cached: true and refresh_in, not 429. The caller asked for that result and gets a valid one; it just isn't newer. A 429 would make every client handle an error for a normal button press, and the UI would have nothing to show. refresh_in carries what Retry-After would.

No per-client limit

Not added. force bounds itself per cache key: across all clients, at most one forced run of a key starts per interval, and a per-client quota wouldn't lower that.

For a free-text target, what a single client can do with force (one extra run per key per 30 s) is less than what it can already do without it: every new target is an uncached run, with no limit at all. That is #532's scope (unauthenticated floods on every route), and a per-client limit there would cover force too.

For a directive with a select list of targets, or no target, that argument doesn't hold: there the cache was the bound, and force raises it from one run per cache.timeout to one per min_interval (4x with the defaults) for any anonymous client. That is the bound an operator opts into with cache.refresh.enable, and caching.mdx now says so. If maintainers want a per-client limit anyway, auxiliary.py's _RATE script and quota_identity() would apply as for the lookups.

Tests

hyperglass/util/tests/test_query_refresh.py (new, 25 tests) drives POST /api/query through Litestar's test client on fakeredis, with _execute_query replaced by a counting stub. It covers:

  • digest() unchanged by force; force: "sometimes" gives 400 without reaching the device.
  • Unforced requests still served from the cache (one device run, cached: true).
  • A forced request after the cooldown re-runs and replaces the entry, removing stale fields, for an IPv4 and an IPv6 target.
  • Inside the interval it serves the cached entry with refresh_in == 30 - 10.
  • A forced request with no entry runs once.
  • An unforced request followed at once by a forced one: one device run, the forced one served cached.
  • A lease held by another run serves cached with refresh_in == 30.
  • 5 concurrent forced requests (threads, device delay 0.5 s) give exactly one run and four cached answers.
  • A failed forced run keeps the old entry, and a retry falls in the cooldown.
  • A forced request right after a forced run is served cached; once the cooldown expires, a forced request runs again.
  • A lease that expires mid-run, before the device fails, still leaves a cooldown of at most 30 s.
  • The lease TTL during a run is above request_timeout.
  • An entry that expires during a forced run is rewritten with the new output.
  • An unforced run that writes while another request's lease is in the key leaves the lease alone.
  • A forced run whose lease was replaced by another request's mid-run leaves that lease alone when it ends.
  • A Redis error while starting the cooldown still returns the new output, and the result's write still turns the request's lease into a cooldown.
  • force ignored and refresh_in: null when refresh is disabled.
  • Config: refresh off by default; min_interval equal to or above timeout rejected, and 0 rejected; with refresh enabled in the configuration, the UI parameters (init_ui_params(...).export_json(by_alias=True)) carry only cache: {timeout, showText}. Typing UIParameters.cache as Cache again fails that test (refresh: {enable: False, minInterval: 30} reappears).

Without the change, the new tests can't run: the fixture's cache.refresh config is rejected by Params (extra="forbid"), so most error at setup. The mutation runs below are the useful check. Each changes one part of api/refresh.py, keeping the tests (warning counts trimmed):

== M1 lease always granted
FAILED hyperglass/util/tests/test_query_refresh.py::test_forced_query_within_interval_serves_cached
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_held_by_a_run_serves_cached
FAILED hyperglass/util/tests/test_query_refresh.py::test_concurrent_forced_queries_run_once
FAILED hyperglass/util/tests/test_query_refresh.py::test_failed_forced_run_keeps_entry_and_cools_down
FAILED hyperglass/util/tests/test_query_refresh.py::test_forced_query_runs_again_after_the_cooldown
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_that_expires_mid_run_still_cools_down
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_outlives_the_device_deadline
FAILED hyperglass/util/tests/test_query_refresh.py::test_forced_query_right_after_a_run_serves_cached
8 failed, 17 passed
== M2 runs do not start a cooldown (no interval)
FAILED hyperglass/util/tests/test_query_refresh.py::test_unforced_queries_still_use_the_cache
FAILED hyperglass/util/tests/test_query_refresh.py::test_cooldown_failure_keeps_the_result
FAILED hyperglass/util/tests/test_query_refresh.py::test_forced_query_right_after_a_run_serves_cached
3 failed, 22 passed
== M3 lease deleted after a forced run instead of cooling down
FAILED hyperglass/util/tests/test_query_refresh.py::test_failed_forced_run_keeps_entry_and_cools_down
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_that_expires_mid_run_still_cools_down
2 failed, 23 passed
== M4 cooldown only on success (no finally)
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_that_expires_mid_run_still_cools_down
1 failed, 24 passed
== M5 expired lease gets no cooldown
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_that_expires_mid_run_still_cools_down
1 failed, 24 passed
== M6 no lease margin
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_outlives_the_device_deadline
1 failed, 24 passed
== M7 refresh_in during a run = lease TTL + interval
FAILED hyperglass/util/tests/test_query_refresh.py::test_lease_held_by_a_run_serves_cached
FAILED hyperglass/util/tests/test_query_refresh.py::test_unforced_run_keeps_another_requests_lease
2 failed, 23 passed
== M8 cooldown RedisError not caught
FAILED hyperglass/util/tests/test_query_refresh.py::test_cooldown_failure_keeps_the_result
1 failed, 24 passed
== M9 replaced entry not deleted
FAILED hyperglass/util/tests/test_query_refresh.py::test_forced_query_reruns_and_replaces_the_entry
1 failed, 24 passed
== M10 enable ignored
FAILED hyperglass/util/tests/test_query_refresh.py::test_force_is_ignored_when_refresh_is_disabled
1 failed, 24 passed
== M11 _COOL_DOWN sets the cooldown unconditionally
FAILED hyperglass/util/tests/test_query_refresh.py::test_forced_run_keeps_a_lease_taken_after_its_own
1 failed, 24 passed
== M12 record() sets the cooldown unconditionally
FAILED hyperglass/util/tests/test_query_refresh.py::test_unforced_run_keeps_another_requests_lease
FAILED hyperglass/util/tests/test_query_refresh.py::test_forced_run_keeps_a_lease_taken_after_its_own
2 failed, 23 passed
== M13 record() ignores this request's own lease
FAILED hyperglass/util/tests/test_query_refresh.py::test_cooldown_failure_keeps_the_result
1 failed, 24 passed

M4 (no cooldown when the run raises) is caught only by the expired-lease test: with a live lease, the stuck lease blocks the retry anyway, until it expires.

The lease and cooldown logic was also run against a real Redis (the local hyperglass-redis container): the lease TTL was 95 s with request_timeout 90, a second forced request was refused with refresh_in 30, and after deleting the lease mid-run and failing the run, the key held cooldown with a 30 s TTL. The reviewer's overlap case (min_interval 2: forced run F holds the lease, the entry expires, an unforced run writes first) was replayed on real Redis: F's token survived the unforced write, a second forced request was refused, and F's end turned the key into a 2 s cooldown.

With the change: pytest hyperglass --ignore hyperglass/plugins/external gives 4195 passed, 4 skipped; ruff check . gives All checks passed!. CI's fixture step run locally (python .tests/ui-fixture/generate.py then git diff --exit-code -- hyperglass/ui/__tests__/fixtures/config.json) leaves the fixture unchanged, and it has no diff against origin/main. Black and isort are clean on the new files and changed lines.

Changelog

  • Forced refresh, off by default. With cache.refresh.enable: true, POST /api/query accepts "force": true, which runs a cached query on the device again and replaces its cache entry. The server answers a forced query from the cache when the query ran less than cache.refresh.min_interval (default 30 seconds) ago, or while another forced request for the same query is in progress, so at most one forced run of each query starts per interval. Successful responses include refresh_in, the seconds until a forced query would run. force must be a boolean: a value that cannot be read as one is rejected with HTTP 400, even when forced refresh is disabled (Allow a query to bypass the cache, at most once per interval per query #552).

Notes

Closes #552

@jsenecal jsenecal added backend Python backend: API, models, state, execution feature New feature or request labels Nov 21, 2026
@jsenecal jsenecal self-assigned this Nov 21, 2026
@jsenecal
jsenecal requested a review from WilhelmZA November 21, 2026 06:11
@github-actions github-actions Bot added the docs Improve documentation label Nov 21, 2026
@coveralls

coveralls commented Nov 21, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 86.521% (+0.3%) from 86.241% — feat/port-f1-force-refresh into main

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 129df4f9.

I read hyperglass/api/refresh.py, the route change and the model change. The bound holds: the SET NX lease, the Lua compare-and-set for the cooldown and the guarded cooldown write inside the result transaction allow one forced run per cache key per min_interval plus run time, for any number of clients or workers. With refresh.enable false, force is ignored and unforced requests make no extra Redis call. force is a boolean, stays out of digest() and never reaches command templating. CI gate passes and all five commits are verified.

Before approval:

  • #552 ends with an open question for the maintainers on the semantics and defaults, and Closes #552 closes it on merge. That question needs an answer on the issue first. This PR also deviates from the issue in two places the body explains (a TTL key in place of executed_at, and <key>.refresh in place of <key>:refresh), so the answer should cover those.
  • hyperglass/api/refresh.py: the LEASE_MARGIN comment and caching.mdx say only one forced run happens at a time. The lease covers the coroutine only. _execute_query cancels the await at request_timeout - 1, but the netmiko work runs in run_in_executor and its thread keeps the device session open after the cancellation, with a connect timeout of request_timeout * 1.25. With request_timeout: 90 and min_interval: 30, a device that hangs during a forced run can still hold a session at 119 seconds when the next forced request takes the lease and opens a second one. The start rate stays bounded and unforced misses on main behave the same way. Please either reword both places to "at most one forced run starts per interval", or set a longer cooldown after a device timeout.

#526 also changes hyperglass/api/routes.py. Whichever merges second needs to count forced runs as bypass in the cache metric.

Nits:

  • CHANGELOG.md: force is now a declared boolean, so a value that cannot be read as a boolean gets HTTP 400 even with refresh disabled. caching.mdx says so and the changelog entry does not.

@WilhelmZA WilhelmZA added the reviewing Issue is being reviewed by maintainers label Nov 21, 2026
@jsenecal

Copy link
Copy Markdown
Collaborator Author

Thanks.

  • Lease wording: confirmed. The lease and the deadline cover the coroutine only, and the Netmiko executor thread keeps running after request_timeout - 1 cancels the await. A hung session can therefore overlap the next forced run once the cooldown ends. One correction to the numbers: timeout=request_timeout * 1.25 is not the connect timeout in netmiko 4.6. Connect uses conn_timeout (default 10). The thread stays alive because each command has its own read_timeout=request_timeout, plus the connect-retry budget. I took the rewording option. The refresh.py docstring, the LEASE_MARGIN comment, caching.mdx, the complete-config comment, CHANGELOG and the PR body now say at most one forced run starts per interval, and that a session outliving the timeout can overlap the next one, as unforced misses already can on main (e22d4ce).
  • CHANGELOG: the entry now says that a force value that cannot be read as a boolean gets HTTP 400, even with refresh disabled. caching.mdx now says the same, since pydantic accepts "yes", 1 and "on", and rejects null.
  • Allow a query to bypass the cache, at most once per interval per query #552 question: agreed, this shouldn't merge until the semantics and defaults are settled on Allow a query to bypass the cache, at most once per interval per query #552. The answer there should cover the TTL key used in place of executed_at, and .refresh in place of :refresh.
  • feat(api): add optional Prometheus metrics #526 overlap: noted. Whichever merges second counts forced runs as bypass in the cache metric.

WilhelmZA
WilhelmZA previously approved these changes Nov 21, 2026

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at e22d4ce5.

The new commit changes only comments, docs and CHANGELOG.md. The docstring, the LEASE_MARGIN comment, caching.mdx, the complete-config comment and the changelog now say that at most one forced run starts per interval and that a session which outlives the timeout can overlap the next run. That matches the code. The changelog also states that a force value that cannot be read as a boolean gets HTTP 400. The semantics and defaults are confirmed on #552. CI gate passes and all six commits are verified.

#526 also changes hyperglass/api/routes.py. Whichever merges second counts forced runs as bypass in the cache metric.

Nits:

  • caching.mdx: the "at most one starts per interval" statement has one more exception. If a worker dies during a forced run, no cooldown is written and the lease expires after request_timeout + 5 seconds. When that is less than min_interval (for example request_timeout: 10 with min_interval: 30), the next forced run can start early. The defaults are not affected. A sentence next to the eviction note would cover it.

@jsenecal

Copy link
Copy Markdown
Collaborator Author

Thanks, and thanks for settling #552. Agreed on the dead-worker exception. If request_timeout + 5 is below min_interval, the lease left by a dead worker expires before a cooldown would have. To keep your approval on this head, I'll add that sentence next to the eviction note in caching.mdx in a follow-up docs commit after merge.

@jsenecal

Copy link
Copy Markdown
Collaborator Author

Rebased onto main after #526 merged; this needs another look, sorry for dismissing your approval. The range-diff shows commits 2–6 unchanged. Commit 1 differs only in import context. Two commits are new:

  • 03a67a2, forced runs in the cache metric: a forced run that executes on the device counts as result="bypass", including when it fails. A forced query answered from the cache, during a cooldown or while another forced run holds the lease, counts as hit, because the device isn't queried. With nothing cached it's a miss. With refresh disabled, force changes nothing. So miss + bypass = device runs, and the hit-ratio example keeps its meaning. monitoring.mdx has a new "Cache Results" section. TestForcedRefresh covers each case.
  • b190d05: the rebase left record_query_success before refresh.refresh_in(), which reads Redis while the response is built. A Redis failure there counted the query as both a success and an error. The success is now recorded last, with a regression test. The docs also say that only queries reaching the cache lookup are counted in hyperglass_query_cache_total.

The dead-worker note for caching.mdx is still a follow-up after merge, as agreed. CI fails until #571 merges, because main broke when #535 and #524 met.

@jsenecal

Copy link
Copy Markdown
Collaborator Author

Also folded in the dead-worker note you suggested, since this head needs a fresh review anyway. It is in caching.mdx, next to the eviction note (latest commit): a worker that dies mid-run writes no cooldown, and its lease expires after request_timeout + 5. If that is below min_interval (e.g. 10 and 30), the next forced run can start early. The defaults are not affected (95 > 30).

@jsenecal jsenecal closed this Nov 21, 2026
@jsenecal jsenecal reopened this Nov 21, 2026

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at e089190f.

The code is good. refresh.py is unchanged from the head I approved. In query() every path records one cache result: hit for a cached answer, bypass only for a forced run that won the lease, miss otherwise. Success is recorded after the response is built, so a Redis failure in refresh_in() no longer counts as both success and error, and that order does not change what #526 shipped. The metrics helpers cannot raise, so no lease is left without a cooldown. The dead-worker note in caching.mdx matches the lease TTL in the code. All nine commits are verified, and CI gate passed on the run merged into main at f89ab22a.

Before approval:

  • #564 has merged and CHANGELOG.md now conflicts with main. Please rebase and I will approve that head once CI gate passes on it.

Nits:

  • hyperglass/api/routes.py: the duration is observed before the result pipeline and refresh_in(), which can still fail. A query can then add a duration sample and also count as an error, while the docs table describes the histogram as successful queries. The pipeline case already behaves this way on main. Moving record_query_duration next to record_query_success would make the two agree.

…rval

POST /api/query accepts force: true. With cache.refresh.enable (off by
default) a forced query for a cached result runs on the device again and
replaces the entry, once the result is cache.refresh.min_interval seconds
old (default 30, below cache.timeout). A per-key Redis lease allows one
forced run at a time across workers and becomes a cooldown of
min_interval when the run ends, successful or not. Requests that are
refused get the cached result. Every response carries refresh_in.
Every run of a query now sets <cache key>.refresh to a cooldown that
expires after cache.refresh.min_interval, in the transaction that writes
the result, and a forced query takes that key with SET NX. That replaces
the executed_at field, so no worker clocks are compared.

The lease outlives the device deadline (request_timeout - 1) by five
seconds, and a run whose lease expired still starts a cooldown. A Redis
error while starting the cooldown is logged instead of replacing the
query's result or error. During another request's forced run,
refresh_in is min_interval rather than the lease's remaining time plus
min_interval.
A run that wrote its result set the refresh key to the cooldown
unconditionally. When the cached entry expired during a forced run, an
unforced run that finished first replaced the forced run's lease, so a
second forced run could start once that cooldown expired. The write now
starts the cooldown only over no key, a cooldown or the request's own
lease. Params are read once for the refresh settings.
UIParameters typed cache as the full Cache model, so validating the
projected parameters re-added refresh with its defaults, whatever the
configuration said. A UICache model with only timeout and show_text
keeps it out.
The forced-refresh lease covers the request, not the Netmiko session, which runs in an executor thread that keeps going after the request timeout. A device that hangs during one forced run can still hold its session when the next forced run starts. Reword the docs, comments and changelog to say at most one forced run starts per interval, and note in the changelog that a non-boolean force is rejected with HTTP 400.
A forced run skips a cached result and runs on the device, but the query
cache metric counted it as a miss. Count it as `result="bypass"`, so
miss plus bypass is the number of device runs and a miss still means
there was no cached result.

A forced query answered from the cache, during a cooldown or while
another request's forced run holds the lease, stays a hit: the client
gets the cached result and the device is not queried. A forced query
with no cached result is a miss, and with refresh disabled, where force
is ignored, a forced query counts as it did before.

A forced run's outcome, failure reason and duration are counted like any
other run's; the duration metric's help text now says it covers bypasses.
The rebase onto the metrics left record_query_success before
refresh.refresh_in(), which reads Redis while the response is built. If
that read failed, the query was counted as both a success and an error.
Record the success last, after everything that can fail.

Also say that only queries reaching the cache lookup are counted in the
cache metric, and assert the exact status of a failed forced run.
The device run's duration was observed before the cache write and
refresh_in(), both of which use Redis and can still fail. Such a query
added a duration sample and was also counted as an error, while the
monitoring docs describe the histogram as successful queries only.

Measure the device run where it was, but record its duration next to
the success, after everything that can fail. record_query_duration now
takes None for a query answered from the cache, which keeps the route
under the complexity limit.
@jsenecal

Copy link
Copy Markdown
Collaborator Author

Rebased onto main after #564 and #574 merged (07b87b7).

  • Rebase: the range-diff shows only CHANGELOG context changes. [Unreleased] now reads Added (forced refresh), Changed (feat(ui): show the cache time in the viewer's locale and time zone #564, untouched), Fixed.
  • Duration nit: record_query_duration now runs next to record_query_success, after the cache write and refresh_in(). The elapsed time is still measured around the device run only, so the histogram holds successful queries, as the docs say. A cache hit records nothing.
  • Test: test_failure_after_device_run_counted_as_error_only[cache_write|refresh_in] makes either step fail. It checks that the query counts as an error, with no success and no duration sample. Both cases fail on the old order.
  • Backend: 4260 passed, and ruff is clean.

@jsenecal
jsenecal force-pushed the feat/port-f1-force-refresh branch from e089190 to 07b87b7 Compare November 21, 2026 16:11
@jsenecal
jsenecal requested a review from WilhelmZA November 21, 2026 16:11

@WilhelmZA WilhelmZA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 07b87b79.

The nine earlier commits are the patches reviewed at e089190f; the range diff shows only CHANGELOG.md context from #564. The new commit moves record_query_duration next to record_query_success, after the cache write and refresh_in(). The elapsed time starts as None, is set only around the device run, and the helper records nothing for None, so a cache hit adds no sample and a query that fails after the device run adds neither a success nor a duration. The test now covers both a failed cache write and a failed refresh_in(). The branch contains current main, CI gate passes on it and all ten commits are verified.

@jsenecal
jsenecal merged commit 0a10a9c into main Nov 21, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Python backend: API, models, state, execution docs Improve documentation feature New feature or request reviewing Issue is being reviewed by maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow a query to bypass the cache, at most once per interval per query

3 participants