Skip to content

test(cluster): seed aclear multipage test with one amset - #800

Merged
vishal-bala merged 1 commit into
mainfrom
fix/cluster-aclear-test-seed-with-amset
Nov 19, 2026
Merged

vishal-bala merged 1 commit into
mainfrom
fix/cluster-aclear-test-seed-with-amset

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

On the first run of the nightly cluster job (#796), test_embeddings_cache_aclear_multipage_cluster failed on the redis-py 8.x leg with AttributeError: 'NoneType' object has no attribute 'writelines'. It passed on the 6.x and 7.x legs, and it passes locally on 8.1.0. The fault is in redis-py 8.x, not RedisVL, but the test only reaches it because of how it seeds its data.

Root cause

The test seeded its 600 cache entries with asyncio.gather over 600 aset() calls on one async RedisCluster. Each aset is a single-key pipeline, so up to about 200 pipelines are in flight on each of the three primaries.

redis-py 8.0 lowered the async cluster's default max_connections per node from 2**31 to 100. When connecting is slow, as on the CI runner, more than 100 pipelines hold a connection at once and ClusterNode.acquire_connection raises MaxConnectionsError. That error subclasses ConnectionError, so the 8.1.0 pipeline treats it as retryable and calls cluster_client.aclose(). That closes the shared client and disconnects connections other coroutines are part-way through using. On Python 3.11, asyncio.wait_for yields between the connection's is_connected check and its write. A connection in the middle of its CLIENT SETINFO handshake therefore writes after disconnect() has set _writer = None, which is the CI traceback. On Python 3.12 and later the same race surfaces as ConnectionError or TimeoutError instead.

A reproduction with redis-py alone (300 concurrent single-key pipelines, max_connections=4, 20 rounds, local six-node cluster):

Python redis-py Result
3.11.9 8.1.0 55 of the CI AttributeError, plus ConnectionError and MaxConnectionsError
3.14.0 8.1.0 MaxConnectionsError, ConnectionError, TimeoutError
3.11.9 7.4.1 MaxConnectionsError only
3.11.9 6.4.0 MaxConnectionsError only

redis/redis-py#4448, merged but not yet released, stops the pipeline closing the client and makes MaxConnectionsError fail immediately. Once it ships, the old seeding would fail with MaxConnectionsError instead, so the test needs changing either way.

Changes

The test now seeds with one amset() call, which is a single pipeline using at most one connection per node. The comment saying amset writes nothing on an async cluster client was out of date, because #697 fixed that, so it's replaced. The now-unused import asyncio is removed.

Notes

This PR goes between #795 and #796 in the stack: #794 → #795 → this PR → #796. It doesn't depend on #794 or #795. It's stacked because #796's nightly job needs it to pass.

Tested locally:

  • On this branch, 11 of the 12 tests in test_redis_cluster_support.py pass against a local six-node Redis 8.4 cluster (Python 3.14, redis-py 8.1.0).
  • The remaining test, test_semantic_router_cluster_client, errored only because that venv lacks sentence-transformers.
  • With redis-py 8.1.0 on both Python 3.11 and 3.14, single-amset seeding writes all 600 keys even with max_connections=4.

Not tested:

  • The edited test on Python 3.11.
  • Anything on redis-py 8.0.x.

Cluster tests don't run on pull-request CI, so #796's nightly job is the first place this runs in CI.

Reviewer Brief

Start here: tests/integration/test_redis_cluster_support.py, the seeding step in test_embeddings_cache_aclear_multipage_cluster. It's the only behavioural change: 600 entries now go in through one amset() pipeline, not 600 concurrent aset() calls.

Not verified: the edited test hasn't run on Python 3.11, where the nightly redis-py 8.x leg failed, or against redis-py 6.x, 7.x or 8.0.x. requires_cluster tests don't run on pull-request CI, so #796's nightly job is the first CI run.

Beyond CI: on a local six-node Redis 8.4 cluster (Python 3.14, redis-py 8.1.0), the edited test and its sync twin pass. Single-amset seeding wrote all 600 keys with max_connections=4 on both Python 3.11 and 3.14. A redis-py-only script that runs concurrent single-key pipelines on one client reproduced the CI AttributeError 55 times in 20 rounds on Python 3.11 with redis-py 8.1.0. The old test itself never failed locally.

Since the last brief: the stack was rebased and main merged in; #800's own change is unchanged, now as 1a85721b.

Reviewed 1a85721b (originally 8e694b3d) · Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs, Redis behaviour


Note

Low Risk
Integration test-only change with no production code paths affected.

Overview
Updates test_embeddings_cache_aclear_multipage_cluster so multipage aclear coverage is seeded with a single amset batch instead of hundreds of concurrent aset calls on one async cluster client.

The new comment documents that redis-py 8’s per-node connection cap can trip pipeline retries and tear down the shared client under the old pattern; the obsolete note that async amset was broken on cluster is removed. Unused asyncio import is dropped.

Reviewed by Cursor Bugbot for commit aeb6008. Bugbot is set up for automated code reviews on this repo. Configure here.

vishal-bala added a commit that referenced this pull request Oct 30, 2026
Co-Authored-By: Claude <noreply@anthropic.com>
@vishal-bala
vishal-bala added this pull request to stack #801 October 30, 2026 12:39
@vishal-bala vishal-bala added the auto:tests Add or improve existing tests label Oct 30, 2026
@vishal-bala
vishal-bala marked this pull request as ready for review November 9, 2026 14:00
@vishal-bala
vishal-bala force-pushed the fix/cluster-aclear-test-seed-with-amset branch from 49e8570 to 1a85721 Compare November 11, 2026 09:29
@raphaeldelio
raphaeldelio self-requested a review November 19, 2026 10:39

@raphaeldelio raphaeldelio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Base automatically changed from fix/cluster-ft-info-resp3-async-init to main November 19, 2026 13:03
The test seeded 600 keys with 600 concurrent aset() calls, each a
single-key pipeline. On redis-py 8.x, whose async cluster defaults to
100 connections per node, that raised MaxConnectionsError on slow
connects; the 8.1.0 pipeline retry then closed the shared client and
tore down other coroutines' connections mid-handshake, failing with
"'NoneType' object has no attribute 'writelines'" on Python 3.11.

Seed with a single amset() instead. The comment that amset writes
nothing on an async cluster was stale: #697 fixed that.
@vishal-bala
vishal-bala force-pushed the fix/cluster-aclear-test-seed-with-amset branch from c7d58d3 to aeb6008 Compare November 19, 2026 13:03
@vishal-bala
vishal-bala merged commit 4a31b8d into main Nov 19, 2026
105 of 106 checks passed
@vishal-bala
vishal-bala deleted the fix/cluster-aclear-test-seed-with-amset branch November 19, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:tests Add or improve existing tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants