Repository navigation
ci: run the Redis Cluster tests nightly - #796
Merged
Merged
Conversation
vishal-bala
changed the base branch from
fix/cluster-ft-info-resp3-async-init
to
fix/cluster-aclear-test-seed-with-amset
October 30, 2026 12:39
vishal-bala
added a commit
that referenced
this pull request
Nov 5, 2026
…cluster clients The cluster branch of SearchIndex._info() and AsyncSearchIndex._info() sends a raw FT.INFO to one node, so redis-py's search parser never sees the reply. On RESP3, the redis-py 8 default, that reply is a map, and make_dict() raised KeyError: 1. Accept either shape. The async cluster client discovers its nodes on its first command, so on a fresh client get_random_node() raised IndexError and get_default_node() returned None. That broke create(), and delete() then failed with "'NoneType' object has no attribute 'connection_kwargs'". Initialise the client in _get_client(), and in _info() for callers that bypass it.
_info() sent FT.INFO to get_random_node(), which picks from every node in the cluster, replicas included. FT.CREATE goes to the default node, a primary, and Redis replicates asynchronously, so a replica can answer "no such index" straight after create(). The nightly cluster job hit this on redis-py 8.1.0, with FT.INFO landing on the replica at port 7005. Ask the default node instead, the node FT.CREATE, FT.DROPINDEX and FT.SEARCH already use. Async listall() had the same random-node read and now does the same; sync listall() already reached the default node through redis-py's keyless routing.
The previous commit said sync listall() already reached the default node
through redis-py's keyless routing. That holds only on redis-py 6.x. On 7.x
and 8.x, FT._LIST has no routing entry, so a keyless call on a cluster
raises RedisClusterException ("No way to dispatch this command to Redis
Cluster. Missing key"). Target the default node explicitly, as the async
twin and _info() do.
The test seeded 600 keys with 600 concurrent aset() calls, each a single-key pipeline. On redis-py 8.x, whose async cluster defaults to 100 connections per node, that raised MaxConnectionsError on slow connects; the 8.1.0 pipeline retry then closed the shared client and tore down other coroutines' connections mid-handshake, failing with "'NoneType' object has no attribute 'writelines'" on Python 3.11. Seed with a single amset() instead. The comment that amset writes nothing on an async cluster was stale: #697 fixed that.
Add a cluster-tests job to the test workflow that runs every requires_cluster test on the nightly schedule and on manual dispatch, across redis-py 6.x, 7.x and 8.x. None of these tests has run in CI before. The cluster nodes announce host.docker.internal, which Linux Docker Engine does not resolve. Map it to the host gateway inside each node, and to loopback on the runner, so the nodes can gossip and the client can follow MOVED redirects. The job runs serially because the cluster publishes fixed ports. Co-Authored-By: Claude <noreply@anthropic.com>
Two problems would have failed every leg of the new cluster job: - redis_cluster_container exported COMPOSE_PROJECT_NAME. Under -m requires_cluster, a cluster test runs before anything resolves redis_url, which then looked up the standalone redis service in the cluster's compose project and raised ContainerIsNotRunning. Every command in the fixture already passes -p, so stop exporting it. - redis-cluster-setup lacked the host.docker.internal mapping, but redis-cli --cluster create reconnects to the nodes at their announced address, so cluster creation could not finish on Linux. Also correct a workflow comment, and update AGENTS.md and the pre-PR review references, which said cluster tests never run in CI. Co-Authored-By: Claude <noreply@anthropic.com>
vishal-bala
force-pushed
the
ci/nightly-cluster-tests
branch
from
November 11, 2026 09:29
da1a85e to
5a27c94
Compare
vishal-bala
marked this pull request as ready for review
November 11, 2026 10:53
Index commands go to redis-py's default node, which redis-py re-points only after a keyed command is redirected or a connection fails. Until then, writes fail with ReadOnlyError and reads come from the demoted node. Say so, and how to recover.
Co-Authored-By: Claude <noreply@anthropic.com>
…r-aclear-test-seed-with-amset Co-Authored-By: Claude <noreply@anthropic.com>
…tly-cluster-tests Resolve references/tests.md by taking main's rewritten list and keeping this branch's change to the cluster bullet: requires_cluster tests are skipped in pull-request CI and run nightly. Co-Authored-By: Claude <noreply@anthropic.com>
raphaeldelio
self-requested a review
November 19, 2026 10:40
The test seeded 600 keys with 600 concurrent aset() calls, each a single-key pipeline. On redis-py 8.x, whose async cluster defaults to 100 connections per node, that raised MaxConnectionsError on slow connects; the 8.1.0 pipeline retry then closed the shared client and tore down other coroutines' connections mid-handshake, failing with "'NoneType' object has no attribute 'writelines'" on Python 3.11. Seed with a single amset() instead. The comment that amset writes nothing on an async cluster was stale: #697 fixed that.
…-tests Co-Authored-By: Claude <noreply@anthropic.com> # Conflicts: # docs/concepts/search-and-indexing.md
Base automatically changed from
fix/cluster-aclear-test-seed-with-amset
to
main
November 19, 2026 13:26
vishal-bala
added a commit
that referenced
this pull request
Nov 19, 2026
Rebasing onto main after #796 merged left uv.lock with a tomli dependency that matched two package entries, so uv could not parse the lockfile and every job that installs dependencies failed. Regenerate it from main's lock with pytest-cov added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
No CI job passes
--run-cluster-tests, so every test markedrequires_clusterhas been skipped on every run. Cluster-only regressions, and cluster tests that no longer test what they claim, have gone unnoticed as a result. This PR runs that suite nightly so the cluster code path gets regular coverage without slowing down pull requests.Changes
Nightly cluster job
A new
cluster-testsjob in.github/workflows/test.ymlrunspytest -n 0 --run-cluster-tests -m requires_cluster tests/on the nightly schedule and onworkflow_dispatch, and is skipped for pull requests and pushes. It runs on Python 3.11 across redis-py 6.x, 7.x and 8.x, because cluster replies differ by redis-py version: 8.x defaults to RESP3. The install step mirrors thetestjob's redis-py pinning. The run is serial becausetests/cluster-compose.ymlpublishes fixed ports 7001–7006, which per-worker clusters under xdist would fight over.Resolving
host.docker.internalon LinuxEvery cluster node announces
host.docker.internal. That is the address the nodes gossip to each other, the oneredis-cli --cluster createreconnects to while building the cluster, and the one the client is handed inMOVEDredirects. So it has to resolve inside every container and on the host. Docker Desktop provides the name inside containers, but Linux Docker Engine, which GitHub's ubuntu runners use, does not. Every container intests/cluster-compose.ymlnow maps it tohost-gateway, and the job adds127.0.0.1 host.docker.internalto the runner's/etc/hosts.Cluster fixture no longer exports
COMPOSE_PROJECT_NAMEredis_cluster_containerintests/conftest.pyused to setCOMPOSE_PROJECT_NAMEto the cluster's project. The standaloneredis_containerfixture's testcontainers commands read that variable. So once a cluster test had run,redis_urllooked for the standaloneredisservice in the cluster's project and raisedContainerIsNotRunning. Under-m requires_clustera cluster test runs first, so the new job would have hit this on every leg. Every command in the cluster fixture already passes-p, so it now leaves the variable alone. This also stops the standalone container from leaking after a cluster run.host.docker.internal, and that CI runs the suite nightly.Notes
This PR is the top of a stack, #794 → #795 → #800 → this PR, because the new job would fail on
maintoday. #794 fixes three cluster tests that silently used a single-node client, #795 fixesSearchIndex.info()andAsyncSearchIndex.info()on cluster clients, and #800 stops a cluster test from overrunning redis-py 8's per-node connection cap. The three fixes are independent; each branch merges the one below it only so that the stack is linear.The job lives in
test.ymlrather than in a workflow of its own, so the nightly schedule stays in one place. The cost is that an on-demand run dispatches all oftest.yml, including the service tests that use the provider API secrets.Reviewer Brief
Start here:
tests/cluster-compose.yml. The job only works if every container, and the runner, resolves the address the nodes announce, and this file is where most of that happens.Where it could fail
tests/conftest.py:84: the cluster fixture no longer exportsCOMPOSE_PROJECT_NAME. Check that nothing else relied on that variable pointing at the cluster project after the fixture had run..github/workflows/test.yml, thecluster-testsinstall step: it copies thetestjob's redis-pycaseblock, so a pin changed in only one copy leaves the nightly testing stale versions without failing.Not verified: local runs against this compose file on macOS with the
/etc/hostsentry (local runs used a stand-in topology announcing127.0.0.1); Docker Desktop on Windows; more than one green run on Linux, though two of the failures this job first exposed were timing-dependent.Beyond CI: a
workflow_dispatchrun onc220e6e3(run 41057698156) formed the cluster on an ubuntu runner and passed all 22 cluster tests on redis-py 6.4.0, 7.4.1 and 8.1.0, with every other job in the workflow green. Before the conftest change, the job's test order madetest_sync_client_validationerror withContainerIsNotRunninglocally; after it, that test passes.Since the last brief: merged the updated #795 and the new #800 below this PR in the stack; this PR's own diff is unchanged. The first dispatch, on
a55375f0, failed three tests on redis-py 8.x, all fixed in those two PRs.Reviewed
236b38de, fixes ina55375f0· Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs, Redis behaviourNote
Low Risk
Changes are limited to CI, test compose, and fixture isolation; no production library API or runtime behavior is modified.
Overview
Adds a nightly / manual-only
cluster-testsjob to.github/workflows/test.ymlthat is the sole CI path passing--run-cluster-testsforrequires_clustertests, matrixed across redis-py 6.x–8.x and run serially (-n 0) because the compose stack uses fixed ports.Linux cluster harness fixes:
tests/cluster-compose.ymlmapshost.docker.internalviaextra_hostson every node and the setup container; the job appends127.0.0.1 host.docker.internalon the runner.redis_cluster_containerintests/conftest.pystops exportingCOMPOSE_PROJECT_NAMEso cluster runs no longer break the standaloneredis_container/redis_urlfixture.Docs and pre-PR review references (
CONTRIBUTING.md,AGENTS.md, correctness/tests lenses) now state that cluster tests skip on pull-request CI and run only on the nightly schedule.Reviewed by Cursor Bugbot for commit 4c52b42. Bugbot is set up for automated code reviews on this repo. Configure here.