Skip to content

ci: measure test coverage and report it to Codecov - #826

Merged
vishal-bala merged 7 commits into
mainfrom
ci/code-coverage
Nov 19, 2026
Merged

vishal-bala merged 7 commits into
mainfrom
ci/code-coverage

Conversation

@vishal-bala

@vishal-bala vishal-bala commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

RedisVL has not measured test coverage since #262 removed the Codecov upload, so nobody can see which modules the suite leaves untested. This PR measures coverage on every run of test.yml and reports it to Codecov, which is what redis-py and other Redis projects use. The suite runs in three segments under different conditions (the PR matrix without API keys, the full suite with API keys on same-repo PRs, and the cluster suite that #796 adds to the nightly run), so each segment is uploaded under its own Codecov flag rather than as one number.

Changes

Collecting coverage in each segment

Every job that runs tests now writes its own coverage data file and uploads it as an artifact, even when tests fail:

Segment Job Runs on Data file
core test matrix every run .coverage.core-py<version>-redis-py-<version>-<image>
service service-tests (make test-all, Python 3.11) every run except fork and Dependabot PRs .coverage.service
cluster cluster-tests (-m requires_cluster) nightly schedule and manual dispatch .coverage.cluster-redis-py-<version>

The service-tests job passes --cov through a new optional pytest-args input on .github/actions/run-service-tests; test-fork-pr.yml does not pass it and runs exactly as before.

pyproject.toml gains [tool.coverage.*] settings: branch coverage of redisvl, paths stored relative to the checkout so data from different runners combines, and patch = ["subprocess"]. The last one matters because the CLI tests run python -m redisvl.cli.runner in a child process; without it the CLI is measured as untested.

Combining and reporting

A new coverage job runs after the three test jobs whatever their outcome (!cancelled()), so a skipped job just leaves its segment out. For each segment it combines that segment's data files, writes a Cobertura XML report and uploads it to Codecov under the segment's flag. It then combines all segments into the job summary, as a per-segment totals table plus a per-file table, and uploads an HTML report as an artifact. Uploading once per segment, instead of once per leg, means Codecov never comments before all the matrix legs have reported.

Per-segment data is kept in coverage-segments/ rather than beside .coverage, because patch = ["subprocess"] turns on coverage's parallel mode, and coverage report would otherwise absorb any .coverage.* file next to its data file.

codecov.yml makes both statuses informational and carries service and cluster forward on runs that skip them, from the nearest ancestor commit with a report for that flag.

  • pytest-cov>=7,<8 joins the dev group. Local make test is unchanged unless ARGS passes --cov.
  • CONTRIBUTING.md gains a "Coverage in CI" section and a working local command; AGENTS.md gains one line under Testing Notes.

Notes

The cluster segment instruments the cluster-tests job that #796 added to the nightly run; with #796 merged, this PR is based on main.

The service flag measures the full suite plus the API-key tests, because the service job runs make test-all. On a fork PR, which skips that job, Codecov carries service forward, so the headline number can hide a drop in code that service covered; the core row in the Codecov comment shows the PR's own change. Narrowing the service job to the API-key tests would remove the masking but also stop that job testing the rest of the suite, and not carrying the flag forward would make every fork PR show a false drop.

Dependabot PRs receive no Codecov token and Codecov refuses tokenless uploads from branches in this repository, so they are not reported to Codecov. The Coverage job summary still shows their totals.

codecov/codecov-action is pinned by tag (@v7), matching how every other third-party action in this repository is pinned.

Coverage is reported, not enforced: neither status is required, and a failed Codecov upload never fails CI (fail_ci_if_error: false). A target can be chosen once there is some history.

Reviewer Brief

Start here: .github/workflows/test.yml, the coverage job's "Combine coverage" step. It decides which segments reach Codecov and what the job summary shows.

Where it could fail

  • The coverage job's needs: [test, service-tests, cluster-tests] with if: ${{ !cancelled() }}: on every PR and push cluster-tests is skipped, and core must still upload; check the expression and the "not run" branch.
  • The three "Upload coverage data" steps: the data files are dotfiles, so they upload only because of include-hidden-files: true, and a re-run job replaces its earlier artifact only because of overwrite: true.
  • codecov.yml: cluster is carried forward through push-only commits on main, which upload core and service but no cluster; if Codecov does not carry it that far, PRs show no cluster flag.

Not verified: no CI run of this workflow yet, so the 3.14 and 3.15 legs (which use coverage's sys.monitoring core), the integration suite and the cluster suite have not run under coverage, and the slowdown per leg is unknown; how far Codecov carries a flag forward; tokenless uploads from fork PRs; actionlint was not run.

Beyond CI: on Python 3.13 the unit suite gives the same result with and without --cov (1,993 passed, 1 skipped), with wall time going from 51.5 s to 55.2 s; redisvl/cli/runner.py reads 0% without subprocess patching and 86% with it, and pytest-cov writes exactly the named data file both under xdist and with -n 0; the "Combine coverage" script, extracted from the workflow and run on local data files, produced the expected summary and outputs both with all three segments and with core alone; codecov.yml passes Codecov's validator (before the cluster flag was added); Codecov's API shows the run on 48d6eb32 processed with core at 78.73% and service at 79.41% across 121 files, all under redisvl/, which is every module in the package that has code; the core combine reports skipped 2, which in coverage 7.14 counts only data files identical to one already combined.

Since the last brief: after #796 merged, the branch was rebased onto main, which changed every commit SHA but not this PR's diff. c812cb84 lets a re-run test or coverage job replace its earlier coverage artifacts. 312bb98e switches to source_pkgs, because with source the XML named redisvl/version.py and redisvl/__init__.py as bare version.py and __init__.py, and Codecov dropped both. b29dc524 regenerates uv.lock, which the rebase had left unparseable.

Reviewed a5e4293b (correctness, security, design, tests, docs, Redis behaviour) and 2d5be637 (correctness, tests, docs), fixes in 7b93f6f3 and 4b781ccf · Claude Code (claude-opus-5-5)


Note

Low Risk
CI and dependency-only changes; coverage is informational and Codecov upload failures do not block merges.

Overview
CI re-enables coverage measurement across the test workflow: matrix core legs, service (make test-all), and nightly cluster runs each write a distinct .coverage.* file, upload it as an artifact, and a new coverage job merges segments per flag (core, service, cluster), publishes a workflow summary and HTML artifact, and uploads Cobertura XML to Codecov with fail_ci_if_error: false.

Tooling: pytest-cov joins the dev group; [tool.coverage.*] in pyproject.toml targets redisvl with branch coverage, repo-relative paths, subprocess patching for CLI tests, and report exclusions. New codecov.yml marks statuses informational and carry-forwards service and cluster when those jobs are skipped.

Workflow plumbing: .github/actions/run-service-tests accepts optional pytest-args (wired as --cov --cov-report= from test.yml); CONTRIBUTING.md and AGENTS.md document local and CI coverage behavior.

Reviewed by Cursor Bugbot for commit b29dc52. Bugbot is set up for automated code reviews on this repo. Configure here.

@vishal-bala
vishal-bala added this pull request to stack #801 November 5, 2026 12:35
@vishal-bala
vishal-bala marked this pull request as ready for review November 11, 2026 10:53

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4b756be. Configure here.

Comment thread .github/workflows/test.yml Resolved
@vishal-bala vishal-bala added the auto:ci Updates to CI/CD workflows and processes label Nov 18, 2026

@raphaeldelio raphaeldelio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The November 11 CI run shows that Codecov queued the core and service reports, but I couldn’t access the processed report. Can you confirm that Codecov processed both uploads and you verified the coverage maps to the expected files?

Base automatically changed from ci/nightly-cluster-tests to main November 19, 2026 14:13
RedisVL has not measured coverage since #262 removed the Codecov upload.
Every test.yml run now collects it and uploads each CI segment under its
own Codecov flag:

- core: every leg of the PR test matrix, combined
- service: the API-key suite, carried forward on fork and Dependabot PRs,
  which skip it

A new Coverage job combines the data, writes per-segment totals and a
per-file table to the job summary, uploads an HTML report and sends each
segment to Codecov. Statuses are informational and a failed upload never
fails CI.

Coverage follows subprocesses (patch = ["subprocess"]), so the CLI tests
that spawn `python -m redisvl.cli.runner` count. pytest-cov joins the dev
group; local `make test` is unchanged unless ARGS passes --cov.
…ately

The service flag measures the full suite plus the API-key tests, not the
API-key tests alone, so on fork PRs the carried-forward value can hide a
drop in non-API code; point readers at the core row. Dependabot PRs get no
Codecov token and are not reported. Note in codecov.yml that only service
is carried forward, and where a new segment must be added.
The cluster-tests job now measures coverage and uploads its data file, and
the coverage job combines it as a third segment under the Codecov flag
`cluster`. The job runs only on the schedule and on manual dispatch, so the
flag is carried forward on every other run; only the nightly run on main
measures every segment.
…ks a report

A workflow_dispatch run also runs the cluster and service jobs, so it
measures every segment, not just the nightly run. Codecov carries a flag
forward from the nearest ancestor commit with a report for it, not from the
latest report anywhere.
Artifacts are scoped to the workflow run, and upload-artifact refuses a
name that already exists unless overwrite is set, so re-running a failed
test or coverage job failed at its upload step. Set overwrite on all four
uploads. Prefix the data artifacts with coverage-data- so the coverage job's
download no longer also matches coverage-html from an earlier attempt.
With source = ["redisvl"], coverage writes XML paths relative to that
directory, so the root-level redisvl/version.py and redisvl/__init__.py
appeared as bare version.py and __init__.py. Codecov dropped both from the
report for 4b756be (119 of 121 modules with code), since
redisvl/cli/version.py and every package __init__.py share those names.
source_pkgs measures the same package, unexecuted files included, but
leaves the XML paths repo-relative.
@vishal-bala

Copy link
Copy Markdown
Collaborator Author

@raphaeldelio Yes, Codecov processed both uploads. On the latest run (48d6eb32), core is at 79% and service at 79%.

The file mapping had one gap, now fixed. Two small root files, redisvl/version.py and redisvl/__init__.py, were missing from the earlier report because their names clash with other files in the package. 3843cc5a writes every path in full (redisvl/...), and the latest report has all 121 modules that contain code, all under redisvl/.

Rebasing onto main after #796 merged left uv.lock with a tomli dependency
that matched two package entries, so uv could not parse the lockfile and
every job that installs dependencies failed. Regenerate it from main's lock
with pytest-cov added.
@vishal-bala
vishal-bala merged commit 4ce72d3 into main Nov 19, 2026
60 checks passed
@vishal-bala
vishal-bala deleted the ci/code-coverage branch November 19, 2026 14:52
vishal-bala added a commit that referenced this pull request Nov 19, 2026
Resolve the run-service-tests action and test.yml conflicts with the
coverage change (#826): keep both the save-cache and pytest-args inputs,
and save the HF cache after the coverage-enabled test run. Bring the new
coverage and cluster-tests jobs onto the same caching rules: coverage
does not cache, cluster-tests restores the 3.11 uv and v2 HF caches
without saving, and both pin ubuntu-24.04.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto:ci Updates to CI/CD workflows and processes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants