Repository navigation
fix(mcp): allow locked exists on vector fields with index_missing - #845
Purv-Kabaria wants to merge 2 commits into
Conversation
A locked exists filter on a vector field was rejected at startup even when the field declares index_missing. Since redis#771, Redis accepts INDEXMISSING on vector fields and answers ismissing() on them, so vector fields now go through the same index_missing check as every other field type. Closes redis#788
|
Hi! Following up on the locked exists filter fix for #788. The PR includes the regression test, the 598 passing MCP tests, and the live Redis 8.10.3 check. Bugbot is green and there are no requested changes so far. Happy to clarify anything in the Reviewer Brief or adjust the approach. Thanks for taking a look when you can! |
|
|
||
|
|
||
| @pytest.mark.parametrize("field_name", ["category", "embedding"]) | ||
| def test_validate_profile_against_schema_accepts_locked_exists_with_index_missing( |
There was a problem hiding this comment.
This test lets the profile through on the grounds that Redis answers ismissing() on the vector field. Nothing in tests/integration checks that, though: ismissing doesn't appear there at all. If FT.CREATE ever stopped carrying INDEXMISSING for a vector field, this validator would still pass the profile, and every call would fail as a retryable backend error. That's the failure mode the validator exists to prevent.
Could you add one integration test? It would create a vector field with index_missing: True, load one document with an embedding and one without, and assert that (-ismissing(@embedding)) returns only the first, both as a filter and as a KNN prefilter. I checked that behaviour by hand on 8.2, 8.4 and 8.10, for FLAT, HNSW and SVS-VAMANA, on HASH and JSON, so the test should pass on every CI version.
There was a problem hiding this comment.
@Purv-Kabaria once this change is in, I can merge the PR 👍
There was a problem hiding this comment.
Fixed in 9620555. Added integration coverage using RedisVL's own SearchIndex.create: one document with an embedding and one without, asserting that (-ismissing(@embedding)) selects only the first, both as a filter and as a KNN prefilter. The matrix covers HASH/JSON with FLAT, HNSW and SVS-VAMANA. All 12 cases and the 598 MCP unit tests pass against redis:8.10.3; formatting and mypy pass. The Reviewer Brief is updated. Thanks for spelling out the backend failure this needs to catch.
chrikrah
left a comment
There was a problem hiding this comment.
@Purv-Kabaria this is ready to go in. I built the index with RedisVL's own SearchIndex.create, the path #771 changed, on redis:8.2.11, older than your 8.10.3. The locked filter works there too.
# head 8f8c0fe, uv sync --frozen --extra mcp, Python 3.12
# redis:8.2.11 container, SearchIndex.create, one doc with an embedding and one without
index_missing=True: profile accepted; '(-ismissing(@embedding))' -> 1 rows
index_missing=False: profile rejected (ValueError)
# not run: Redis 7.4 or a cluster
The probe is ten lines. Want it as an integration test?
vishal-bala
left a comment
There was a problem hiding this comment.
Per my last review, these changes are ready except for one last request - once it's addressed, we can merge this. See my comment here: #845 (comment)
Signed-off-by: Purv Kabaria <purvkabaria@gmail.com>
|
Thanks for checking the |
Motivation
A custom tool profile whose
lock.filterusesexistson a vector field fails at startup, even when the field is declared withindex_missing: true. Since #771, RedisVL emitsINDEXMISSINGon vector fields correctly, and Redis answersismissing(@field)on them, so a locked "only documents that have an embedding" filter is a query that works. Closes #788.Changes
_validate_locked_exists_fieldsinredisvl/mcp/tools/profiles.pyno longer rejects vector fields outright. A vector field now goes through the sameindex_missingcheck as every other field type:existsis accepted when the field declaresindex_missing, and a field without it still fails at startup with the existing "not indexed with INDEXMISSING" error. The unusedvector_field_namesparameter is removed.The new unit tests cover a tag field and a vector field, each with and without
index_missing.Release Notes
MCP custom tool profiles can lock an
existsfilter on a vector field declared withindex_missing.Reviewer Brief
Start here:
redisvl/mcp/tools/profiles.py,_validate_locked_exists_fields(): vector fields now use the sameindex_missingguard as other fields.tests/integration/test_mcp/test_locked_exists_vector.pyexercises the realSearchIndex.createpath with a document with an embedding and one without.Not verified: cluster mode; the full integration suite; live Redis versions other than 8.10.3 in this review.
Beyond CI: 598 MCP unit tests and all 12 new integration cases pass against redis:8.10.3, covering filter and KNN queries on HASH/JSON with FLAT, HNSW and SVS-VAMANA; mypy passes for 125 source files and isort/black pass.
Reviewed
9620555? Codex ? lenses: correctness, security, design, tests, docs, Redis behaviour; independent challenger passNote
Low Risk
Narrows startup validation for MCP custom tool profiles only; no auth or query-path behavior change beyond accepting previously-rejected valid configs.
Overview
MCP profile startup validation no longer blocks
existslocked filters on vector fields when the field hasindex_missing._validate_locked_exists_fieldsdrops the hard vector-field rejection and applies the same INDEXMISSING rule as for tag/text fields, so profiles like “only docs with an embedding” can register when the schema supportsismissing(@embedding).Unit tests parametrize tag and vector fields with and without
index_missing. New Redis integration tests assert(-ismissing(@embedding))works as a filter and as a KNN pre-filter across vector algorithms and hash/json storage.Reviewed by Cursor Bugbot for commit 9620555. Bugbot is set up for automated code reviews on this repo. Configure here.