Skip to content

ci: run the Redis Cluster tests nightly - #796

Merged
vishal-bala merged 13 commits into
mainfrom
ci/nightly-cluster-tests
Nov 19, 2026
Merged

vishal-bala merged 13 commits into
mainfrom
ci/nightly-cluster-tests

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

No CI job passes --run-cluster-tests, so every test marked requires_cluster has been skipped on every run. Cluster-only regressions, and cluster tests that no longer test what they claim, have gone unnoticed as a result. This PR runs that suite nightly so the cluster code path gets regular coverage without slowing down pull requests.

Changes

Nightly cluster job

A new cluster-tests job in .github/workflows/test.yml runs pytest -n 0 --run-cluster-tests -m requires_cluster tests/ on the nightly schedule and on workflow_dispatch, and is skipped for pull requests and pushes. It runs on Python 3.11 across redis-py 6.x, 7.x and 8.x, because cluster replies differ by redis-py version: 8.x defaults to RESP3. The install step mirrors the test job's redis-py pinning. The run is serial because tests/cluster-compose.yml publishes fixed ports 7001–7006, which per-worker clusters under xdist would fight over.

Resolving host.docker.internal on Linux

Every cluster node announces host.docker.internal. That is the address the nodes gossip to each other, the one redis-cli --cluster create reconnects to while building the cluster, and the one the client is handed in MOVED redirects. So it has to resolve inside every container and on the host. Docker Desktop provides the name inside containers, but Linux Docker Engine, which GitHub's ubuntu runners use, does not. Every container in tests/cluster-compose.yml now maps it to host-gateway, and the job adds 127.0.0.1 host.docker.internal to the runner's /etc/hosts.

Cluster fixture no longer exports COMPOSE_PROJECT_NAME

redis_cluster_container in tests/conftest.py used to set COMPOSE_PROJECT_NAME to the cluster's project. The standalone redis_container fixture's testcontainers commands read that variable. So once a cluster test had run, redis_url looked for the standalone redis service in the cluster's project and raised ContainerIsNotRunning. Under -m requires_cluster a cluster test runs first, so the new job would have hit this on every leg. Every command in the cluster fixture already passes -p, so it now leaves the variable alone. This also stops the standalone container from leaking after a cluster run.

  • CONTRIBUTING.md now says the test host must resolve host.docker.internal, and that CI runs the suite nightly.
  • AGENTS.md and the pre-PR review references no longer say cluster tests never run in CI; they now say pull-request CI.

Notes

This PR is the top of a stack, #794 → #795 → #800 → this PR, because the new job would fail on main today. #794 fixes three cluster tests that silently used a single-node client, #795 fixes SearchIndex.info() and AsyncSearchIndex.info() on cluster clients, and #800 stops a cluster test from overrunning redis-py 8's per-node connection cap. The three fixes are independent; each branch merges the one below it only so that the stack is linear.

The job lives in test.yml rather than in a workflow of its own, so the nightly schedule stays in one place. The cost is that an on-demand run dispatches all of test.yml, including the service tests that use the provider API secrets.

Reviewer Brief

Start here: tests/cluster-compose.yml. The job only works if every container, and the runner, resolves the address the nodes announce, and this file is where most of that happens.

Where it could fail

  • tests/conftest.py:84: the cluster fixture no longer exports COMPOSE_PROJECT_NAME. Check that nothing else relied on that variable pointing at the cluster project after the fixture had run.
  • .github/workflows/test.yml, the cluster-tests install step: it copies the test job's redis-py case block, so a pin changed in only one copy leaves the nightly testing stale versions without failing.

Not verified: local runs against this compose file on macOS with the /etc/hosts entry (local runs used a stand-in topology announcing 127.0.0.1); Docker Desktop on Windows; more than one green run on Linux, though two of the failures this job first exposed were timing-dependent.

Beyond CI: a workflow_dispatch run on c220e6e3 (run 41057698156) formed the cluster on an ubuntu runner and passed all 22 cluster tests on redis-py 6.4.0, 7.4.1 and 8.1.0, with every other job in the workflow green. Before the conftest change, the job's test order made test_sync_client_validation error with ContainerIsNotRunning locally; after it, that test passes.

Since the last brief: merged the updated #795 and the new #800 below this PR in the stack; this PR's own diff is unchanged. The first dispatch, on a55375f0, failed three tests on redis-py 8.x, all fixed in those two PRs.

Reviewed 236b38de, fixes in a55375f0 · Claude Code (claude-opus-5-5) · lenses: correctness, security, design, tests, docs, Redis behaviour


Note

Low Risk
Changes are limited to CI, test compose, and fixture isolation; no production library API or runtime behavior is modified.

Overview
Adds a nightly / manual-only cluster-tests job to .github/workflows/test.yml that is the sole CI path passing --run-cluster-tests for requires_cluster tests, matrixed across redis-py 6.x–8.x and run serially (-n 0) because the compose stack uses fixed ports.

Linux cluster harness fixes: tests/cluster-compose.yml maps host.docker.internal via extra_hosts on every node and the setup container; the job appends 127.0.0.1 host.docker.internal on the runner. redis_cluster_container in tests/conftest.py stops exporting COMPOSE_PROJECT_NAME so cluster runs no longer break the standalone redis_container / redis_url fixture.

Docs and pre-PR review references (CONTRIBUTING.md, AGENTS.md, correctness/tests lenses) now state that cluster tests skip on pull-request CI and run only on the nightly schedule.

Reviewed by Cursor Bugbot for commit 4c52b42. Bugbot is set up for automated code reviews on this repo. Configure here.

@vishal-bala
vishal-bala added this pull request to stack #797 October 30, 2026 10:57
@vishal-bala vishal-bala added the auto:ci Updates to CI/CD workflows and processes label Oct 30, 2026
@vishal-bala
vishal-bala removed this pull request from stack #797 October 30, 2026 12:39
@vishal-bala
vishal-bala changed the base branch from fix/cluster-ft-info-resp3-async-init to fix/cluster-aclear-test-seed-with-amset October 30, 2026 12:39
@vishal-bala
vishal-bala added this pull request to stack #801 October 30, 2026 12:39
vishal-bala and others added 7 commits November 11, 2026 11:29
…cluster clients

The cluster branch of SearchIndex._info() and AsyncSearchIndex._info() sends
a raw FT.INFO to one node, so redis-py's search parser never sees the reply.
On RESP3, the redis-py 8 default, that reply is a map, and make_dict() raised
KeyError: 1. Accept either shape.

The async cluster client discovers its nodes on its first command, so on a
fresh client get_random_node() raised IndexError and get_default_node()
returned None. That broke create(), and delete() then failed with
"'NoneType' object has no attribute 'connection_kwargs'". Initialise the
client in _get_client(), and in _info() for callers that bypass it.
_info() sent FT.INFO to get_random_node(), which picks from every node in
the cluster, replicas included. FT.CREATE goes to the default node, a
primary, and Redis replicates asynchronously, so a replica can answer
"no such index" straight after create(). The nightly cluster job hit this
on redis-py 8.1.0, with FT.INFO landing on the replica at port 7005.

Ask the default node instead, the node FT.CREATE, FT.DROPINDEX and
FT.SEARCH already use. Async listall() had the same random-node read and
now does the same; sync listall() already reached the default node through
redis-py's keyless routing.
The previous commit said sync listall() already reached the default node
through redis-py's keyless routing. That holds only on redis-py 6.x. On 7.x
and 8.x, FT._LIST has no routing entry, so a keyless call on a cluster
raises RedisClusterException ("No way to dispatch this command to Redis
Cluster. Missing key"). Target the default node explicitly, as the async
twin and _info() do.
The test seeded 600 keys with 600 concurrent aset() calls, each a
single-key pipeline. On redis-py 8.x, whose async cluster defaults to
100 connections per node, that raised MaxConnectionsError on slow
connects; the 8.1.0 pipeline retry then closed the shared client and
tore down other coroutines' connections mid-handshake, failing with
"'NoneType' object has no attribute 'writelines'" on Python 3.11.

Seed with a single amset() instead. The comment that amset writes
nothing on an async cluster was stale: #697 fixed that.
Add a cluster-tests job to the test workflow that runs every
requires_cluster test on the nightly schedule and on manual dispatch,
across redis-py 6.x, 7.x and 8.x. None of these tests has run in CI
before.

The cluster nodes announce host.docker.internal, which Linux Docker
Engine does not resolve. Map it to the host gateway inside each node,
and to loopback on the runner, so the nodes can gossip and the client
can follow MOVED redirects. The job runs serially because the cluster
publishes fixed ports.

Co-Authored-By: Claude <noreply@anthropic.com>
Two problems would have failed every leg of the new cluster job:

- redis_cluster_container exported COMPOSE_PROJECT_NAME. Under
  -m requires_cluster, a cluster test runs before anything resolves
  redis_url, which then looked up the standalone redis service in the
  cluster's compose project and raised ContainerIsNotRunning. Every
  command in the fixture already passes -p, so stop exporting it.
- redis-cluster-setup lacked the host.docker.internal mapping, but
  redis-cli --cluster create reconnects to the nodes at their announced
  address, so cluster creation could not finish on Linux.

Also correct a workflow comment, and update AGENTS.md and the pre-PR
review references, which said cluster tests never run in CI.

Co-Authored-By: Claude <noreply@anthropic.com>
@vishal-bala
vishal-bala force-pushed the ci/nightly-cluster-tests branch from da1a85e to 5a27c94 Compare November 11, 2026 09:29
@vishal-bala
vishal-bala marked this pull request as ready for review November 11, 2026 10:53
vishal-bala and others added 4 commits November 12, 2026 13:55
Index commands go to redis-py's default node, which redis-py re-points
only after a keyed command is redirected or a connection fails. Until
then, writes fail with ReadOnlyError and reads come from the demoted
node. Say so, and how to recover.
Co-Authored-By: Claude <noreply@anthropic.com>
…r-aclear-test-seed-with-amset

Co-Authored-By: Claude <noreply@anthropic.com>
…tly-cluster-tests

Resolve references/tests.md by taking main's rewritten list and keeping
this branch's change to the cluster bullet: requires_cluster tests are
skipped in pull-request CI and run nightly.

Co-Authored-By: Claude <noreply@anthropic.com>
@raphaeldelio
raphaeldelio self-requested a review November 19, 2026 10:40

@raphaeldelio raphaeldelio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

The test seeded 600 keys with 600 concurrent aset() calls, each a
single-key pipeline. On redis-py 8.x, whose async cluster defaults to
100 connections per node, that raised MaxConnectionsError on slow
connects; the 8.1.0 pipeline retry then closed the shared client and
tore down other coroutines' connections mid-handshake, failing with
"'NoneType' object has no attribute 'writelines'" on Python 3.11.

Seed with a single amset() instead. The comment that amset writes
nothing on an async cluster was stale: #697 fixed that.
…-tests

Co-Authored-By: Claude <noreply@anthropic.com>

# Conflicts:
#	docs/concepts/search-and-indexing.md
vishal-bala added a commit that referenced this pull request Nov 19, 2026
#796's branch had moved on (main merged up the stack, dropping Python 3.10),
and the PR's merge ref produced a uv.lock that uv could not parse. uv.lock
is regenerated from #796's lock with pytest-cov added, not merged by text.
Base automatically changed from fix/cluster-aclear-test-seed-with-amset to main November 19, 2026 13:26
@vishal-bala
vishal-bala merged commit e7f3dc7 into main Nov 19, 2026
107 of 108 checks passed
@vishal-bala
vishal-bala deleted the ci/nightly-cluster-tests branch November 19, 2026 14:13
vishal-bala added a commit that referenced this pull request Nov 19, 2026
Rebasing onto main after #796 merged left uv.lock with a tomli dependency
that matched two package entries, so uv could not parse the lockfile and
every job that installs dependencies failed. Regenerate it from main's lock
with pytest-cov added.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:ci Updates to CI/CD workflows and processes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants