Skip to content

fix: apply one rule to every distance threshold, and let 2 match everything - #838

Open
vishal-bala wants to merge 19 commits into
mainfrom
fix/semantic-cache-uniform-distance-threshold
Open

vishal-bala wants to merge 19 commits into
mainfrom
fix/semantic-cache-uniform-distance-threshold

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

A distance threshold tells a semantic lookup how close a stored vector must be to count as a match. RedisVL takes one in six places: SemanticCache, SemanticMessageHistory and LangCacheSemanticCache each have a default for every lookup and an override for a single call, routes in SemanticRouter each have their own, and VectorRangeQuery and MultiVectorQuery's Vector take one directly.

cache = SemanticCache(name="llmcache", distance_threshold=0.1)  # default for every lookup
cache.check(prompt="...", distance_threshold=0.05)              # override for this lookup only

Each place validated its threshold its own way, and most let through values that break lookups without an error. A bool passes as a number, because Python's True is the integer 1. A string from a config file passes in some places and fails in others. NaN is the worst case: every comparison with NaN is false, so a NaN threshold makes every lookup match nothing. Before this PR, SemanticCache.check(distance_threshold=float("nan")), SemanticMessageHistory(distance_threshold="0.3") and router.update_route_thresholds({"greeting": 3}) were all accepted. This PR gives every site the same rule.

Changes

One rule for COSINE thresholds

A threshold must now be a real number from 0 to 2 inclusive, the range of the COSINE distance these indexes use. An int, a float or a numpy scalar qualifies. A bool, a string, a Decimal or NaN does not. validate_cosine_distance_threshold in redisvl/utils/utils.py applies the rule and returns the value as a plain float.

SemanticCache, SemanticMessageHistory and LangCacheSemanticCache all call it, for the default and for an override. LangCacheSemanticCache checks against 0 to 2 on its "redis" scale and 0 to 1 on its "normalized" scale. The constructors check before they load the default HuggingFace vectorizer, so a bad threshold fails without a model download. SemanticMessageHistory.get_relevant checks the threshold before its top_k == 0 early return.

The rule raises TypeError: distance_threshold must be a real number, got <type> for a value of the wrong type. It raises ValueError: Distance must be between 0 and 2, got <value> for a value out of range or NaN, which is the message SemanticCache.set_threshold already used.

The same gaps closed at the other sites

The other three sites keep their own range and error type, because their callers already rely on them. They now refuse the same bad values.

  • SemanticRouter routes keep their (0, 2] range and pydantic ValidationError. Route now validates on assignment as well as construction, and update_route_thresholds checks every value before it applies any, so one bad value leaves every threshold unchanged.
  • VectorRangeQuery.set_distance_threshold now refuses a bool and NaN. It has no upper bound, because it serves L2 and IP indexes as well as COSINE.
  • Vector.max_distance now refuses a bool and a string before pydantic coerces them to floats, and validates on assignment. The value is written into the query string.

A threshold of 2 matches everything

A COSINE distance of 2 is the distance between two exactly opposite vectors, so a threshold of 2 should match every vector. Float rounding breaks that at the very edge. Measured on Redis 8.10.3, a FLOAT32 vector with 1536 dimensions reports a distance of 2.00000023842 from its own negation, and FLOAT16 reports up to 2.0006. A radius of 2, or FILTER @distance <= 2.0, drops that vector, and a radius of 3 keeps it. Before this PR a caller could pass 3 to the cache or history override to match everything. With 2 now the maximum, 2 has to cover that case.

cosine_distance_bound returns the threshold unchanged below 2, and 3 at 2. Three comparisons use it:

  • SemanticCache's client-side check of each candidate.
  • SemanticMessageHistory's range radius.
  • SemanticRouter's range radius, and its FILTER for the min and avg aggregation methods.

A summed router distance keeps its threshold unchanged, because a sum of several reference distances can genuinely exceed 2.

Tests

  • tests/unit/test_utils.py tests the rule once. It accepts 0, 2, np.float32(0.5) and 1 against a maximum of 1, each returned as a float. It refuses 2 + 1e-9, -1e-9, NaN, 1 + 1e-9 against a maximum of 1, True and "0.5". It also tests cosine_distance_bound.
  • tests/unit/test_semantic_cache_threshold.py and test_semantic_history_threshold.py check that each entry point applies the rule. They cover the constructor (before the vectorizer loads), the setter and each override, plus the widened bound at 2. test_route_schema.py, test_semantic_router_queries.py, test_langcache_semantic_cache.py, test_query_types.py and test_aggregation_types.py gained a case or two each.

Notes

Numeric strings and Decimal are refused rather than converted. The parameters are typed float, and converting strings would also let "nan" and "inf" through float(). Most sites already refused strings for an override. A caller that passes a threshold straight from an environment variable or a config file now has to convert it with float().

The router's rule still differs from the others in three ways, all from before this PR. It excludes 0, it accepts Decimal (pydantic's strict float allows it), and it raises ValidationError rather than TypeError. Changing any of them would break existing router configurations, so this PR only closes the paths that skipped validation.

This PR should merge before the change that deprecates the implicit default vectorizer (branch feat/deprecate-implicit-vectorizer), because both edit SemanticCache.__init__.

Release Notes

Distance thresholds on SemanticCache, SemanticMessageHistory, LangCacheSemanticCache, SemanticRouter routes, VectorRangeQuery and MultiVectorQuery vectors now raise an error for a boolean, a string or NaN (and, outside routes, a Decimal), and the cache, history and LangCache thresholds also for anything outside 0 to 2 (0 to 1 on LangCache's "normalized" scale), so convert a threshold read from config with float(). A COSINE threshold of 2 now matches every vector, including an exactly opposite one that rounding reports just above 2.

AI Assistance

  • No AI assistance was used for this PR.
  • This PR is AI-assisted. I have reviewed and understood everything in it, and I take responsibility for it.

Claude Code (claude-opus-5-5): implementation, tests, pre-PR review and this description.

Reviewer Brief

Start here: redisvl/utils/utils.py, validate_cosine_distance_threshold and cosine_distance_bound. The cache, history and LangCache call the first for every threshold, and the cache, history and router call the second to decide what a threshold of 2 means.

Where it could fail

  • redisvl/extensions/router/semantic.py, _distance_threshold_filter: widening the bound is only correct for an aggregation whose result can't exceed one reference's distance. Check that min and avg get it and sum doesn't.
  • redisvl/extensions/router/schema.py: Route now validates every field on assignment, not only the threshold. Code outside the library that sets route.references = [] or an empty name now raises. The library's own delete path edits the list in place for that reason.
  • SemanticMessageHistory.get_relevant now raises for a bad threshold even with top_k=0, which used to return [].

Not verified: the rounding overshoot was measured on FLAT indexes over HASH storage with RESP2. HNSW, JSON storage, RESP3 and BFLOAT16 weren't measured; the bound of 3 leaves a margin of about 1 over the largest overshoot seen. The live LangCache API (mocked here), and cluster mode, which this change doesn't touch.

Beyond CI: all 12 new or changed tests outside test_utils.py fail against main's library code. The validator's tests in test_utils.py, which can't be collected against main, each catch a mutation of the inclusive bound, the bool check or the float conversion. On Redis 8.10.3, VECTOR_RANGE 2 and FILTER @distance <= 2.0 drop a vector reported at 2.00000023842, and 3.0 keeps it.

Reviewed e434aff4, fixes in a80b6b4f and 7f6d9854 · Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs, Redis behaviour


Note

Medium Risk
Changes input validation and lookup filtering across cache, history, and router paths; callers passing booleans, strings, or overrides above 2 will see new errors, but behavior for valid thresholds is tightened rather than broadened except at threshold 2.

Overview
Introduces shared cosine distance threshold validation (validate_cosine_distance_threshold) and rounding-aware bounds (cosine_distance_bound) in redisvl/utils/utils.py, then wires them through semantic cache, LangCache, message history, router, and query types so defaults and per-call overrides follow the same rules.

Validation: Thresholds must be real numbers in [0, 2] for Redis COSINE (or [0, 1] where normalized). bool, NaN, out-of-range values, and (for several paths) numeric strings now raise TypeError or ValueError before Redis or vectorizers run. SemanticCache drops its private _validate_distance_threshold in favor of the shared helper; LangCache validates before converting to similarity; SemanticMessageHistory and SemanticCache constructor/setters/check paths align; router Route thresholds get validate_assignment, and update_route_thresholds validates all values before applying any.

Threshold = 2: When the max threshold is 2 (“match everything”), comparisons and VECTOR_RANGE radii use cosine_distance_bound so Redis-reported distances slightly above 2 (float rounding) are still included; the router widens aggregate FILTER bounds for min/avg but not for sum.

Query layer: Vector.max_distance and set_distance_threshold reject booleans and non-real numbers; assignment-time validation is enabled where values are interpolated into queries.

Tests: New/extended unit tests cover invalid thresholds, bound widening at 2, router filters, and assignment validation.

Reviewed by Cursor Bugbot for commit 7f6d985. Bugbot is set up for automated code reviews on this repo. Configure here.

check() and acheck() searched with a VectorRangeQuery, so Redis
discarded every candidate beyond distance_threshold and a miss carried
no information about how close it came. They now run a KNN VectorQuery
for num_results candidates and keep those at or within the threshold in
Python, inclusive as the range radius was.

On the FLAT index the cache always uses, both queries are exhaustive,
so hit and miss decisions match. Measured on Redis 8.4.7 they agreed on
3,000 of 3,000 samples per configuration and on 200 of 200 queries
within 2e-6 of the threshold. The one theoretical difference is that
the client compares the distance as Redis returns it, rounded to 12
significant digits, so a distance within about 1e-12 of the threshold
can now fall on the other side of it.

Unfiltered lookups take 2-3% longer. With a tag filter matching 10% of
entries they take 13% less time at 1,000 entries and 36% less at
10,000, because KNN compares vectors only within the filtered subset
where the range query scanned the whole index. The nearest candidate's
distance is now in hand on every lookup, which the planned
OpenTelemetry instrumentation reports.

A per-call distance_threshold is still validated as the range query
validated it: TypeError for a non-number, ValueError for a negative
value. An id-only row (the Redis 8.8+ expiry race) still reaches
_process_cache_results and is skipped there rather than raising. Query
construction and the threshold filter are shared by the sync and async
paths.
The cache-wide threshold (__init__, set_threshold) accepted 0 to 2,
while a per-call override (check, acheck) accepted any non-negative
number, including values above 2, booleans and NaN, which turned every
lookup into a silent miss.

Both paths now share _validate_distance_threshold: a real number from
0 to 2 inclusive, refusing bool (TypeError), strings (TypeError) and
NaN (ValueError), with set_threshold's existing ValueError message.
Cover np.float32 and the 0/2 boundaries through check and acheck as
well as set_threshold, and list the new TypeError on __init__.
…olds

Apply SemanticCache's distance-threshold rule to SemanticMessageHistory: the
constructor, set_distance_threshold and the get_relevant override now accept a
real number from 0 to 2 inclusive and refuse strings, bools, NaN and values
outside the COSINE range. Move the rule into
redisvl.utils.utils.validate_cosine_distance_threshold so that both classes
share it.
Refuse a bad distance threshold before the constructor loads the default
vectorizer, and before get_relevant returns early for top_k=0, so that every
path refuses the same values.
Route.distance_threshold was checked only at construction, so
update_route_thresholds and direct assignment accepted NaN, bools,
strings and out-of-range numbers, which then reached the aggregation
FILTER. Route now validates on assignment, and update_route_thresholds
checks every value before applying any.
…iform-distance-threshold

# Conflicts:
#	redisvl/extensions/cache/llm/semantic.py
…rapper reads

main's check and acheck now run inside _telemetry.operation, which reads
cache.name, and resolve refresh_ttl_on_hit.
@vishal-bala vishal-bala added the auto:patch Increment the patch version when merged label Nov 19, 2026
@vishal-bala
vishal-bala marked this pull request as ready for review November 19, 2026 15:11
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T15:13:03.090566Z b8808d0 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vishal-bala
vishal-bala added this pull request to stack #858 November 19, 2026 15:16
Drop the 0 and 2 accept cases, which the integration tests already run
end to end, and the False case, which reaches the same bool branch as
True. Reword the validator docstring's NaN sentence.
…ic-cache-uniform-distance-threshold

# Conflicts:
#	redisvl/extensions/cache/llm/semantic.py
Apply the cosine threshold rule to LangCacheSemanticCache overrides (0-2
on the redis scale, 0-1 normalized), refuse bool and NaN in
VectorRangeQuery.set_distance_threshold, and refuse bool and strings in
Vector.max_distance before pydantic coerces them.

SemanticCache now clamps a reported distance to 2 before comparing it,
as norm_cosine_distance does, because float32 rounding can put an exactly
opposite vector just above 2 and past the largest allowed threshold.

Trim router and history tests that pass without the fix.
SemanticCache now refuses a bad threshold before it builds the default
vectorizer, as SemanticMessageHistory does. Vector validates max_distance
on assignment as well as construction, since it is interpolated into the
query. Name the real-number rule in VectorRangeQuery's TypeError and list
the new errors in LangCache's check and acheck.

Test the shared rule once in test_utils.py and keep one wiring check per
entry point; move the router cases into test_route_schema.py.
Redis can report an exactly opposite vector a little above 2 (measured on
8.10.3: 2.00000024 for FLOAT32). cosine_distance_bound widens a threshold
of 2 past that, and SemanticCache's client-side compare, the message
history's range radius, and the router's range radius and min/avg FILTER
all use it. A summed router distance keeps its threshold unchanged.
@vishal-bala vishal-bala changed the title fix(cache): apply one distance-threshold rule to SemanticCache fix: apply one rule to every distance threshold, and let 2 match everything Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:patch Increment the patch version when merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant