Repository navigation
ci: measure test coverage and report it to Codecov - #826
Conversation
7b67a74 to
4b756be
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4b756be. Configure here.
raphaeldelio
left a comment
There was a problem hiding this comment.
The November 11 CI run shows that Codecov queued the core and service reports, but I couldn’t access the processed report. Can you confirm that Codecov processed both uploads and you verified the coverage maps to the expected files?
RedisVL has not measured coverage since #262 removed the Codecov upload. Every test.yml run now collects it and uploads each CI segment under its own Codecov flag: - core: every leg of the PR test matrix, combined - service: the API-key suite, carried forward on fork and Dependabot PRs, which skip it A new Coverage job combines the data, writes per-segment totals and a per-file table to the job summary, uploads an HTML report and sends each segment to Codecov. Statuses are informational and a failed upload never fails CI. Coverage follows subprocesses (patch = ["subprocess"]), so the CLI tests that spawn `python -m redisvl.cli.runner` count. pytest-cov joins the dev group; local `make test` is unchanged unless ARGS passes --cov.
…ately The service flag measures the full suite plus the API-key tests, not the API-key tests alone, so on fork PRs the carried-forward value can hide a drop in non-API code; point readers at the core row. Dependabot PRs get no Codecov token and are not reported. Note in codecov.yml that only service is carried forward, and where a new segment must be added.
The cluster-tests job now measures coverage and uploads its data file, and the coverage job combines it as a third segment under the Codecov flag `cluster`. The job runs only on the schedule and on manual dispatch, so the flag is carried forward on every other run; only the nightly run on main measures every segment.
…ks a report A workflow_dispatch run also runs the cluster and service jobs, so it measures every segment, not just the nightly run. Codecov carries a flag forward from the nearest ancestor commit with a report for it, not from the latest report anywhere.
Artifacts are scoped to the workflow run, and upload-artifact refuses a name that already exists unless overwrite is set, so re-running a failed test or coverage job failed at its upload step. Set overwrite on all four uploads. Prefix the data artifacts with coverage-data- so the coverage job's download no longer also matches coverage-html from an earlier attempt.
With source = ["redisvl"], coverage writes XML paths relative to that directory, so the root-level redisvl/version.py and redisvl/__init__.py appeared as bare version.py and __init__.py. Codecov dropped both from the report for 4b756be (119 of 121 modules with code), since redisvl/cli/version.py and every package __init__.py share those names. source_pkgs measures the same package, unexecuted files included, but leaves the XML paths repo-relative.
48d6eb3 to
312bb98
Compare
|
@raphaeldelio Yes, Codecov processed both uploads. On the latest run ( The file mapping had one gap, now fixed. Two small root files, |
Rebasing onto main after #796 merged left uv.lock with a tomli dependency that matched two package entries, so uv could not parse the lockfile and every job that installs dependencies failed. Regenerate it from main's lock with pytest-cov added.
Resolve the run-service-tests action and test.yml conflicts with the coverage change (#826): keep both the save-cache and pytest-args inputs, and save the HF cache after the coverage-enabled test run. Bring the new coverage and cluster-tests jobs onto the same caching rules: coverage does not cache, cluster-tests restores the 3.11 uv and v2 HF caches without saving, and both pin ubuntu-24.04.

Motivation
RedisVL has not measured test coverage since #262 removed the Codecov upload, so nobody can see which modules the suite leaves untested. This PR measures coverage on every run of
test.ymland reports it to Codecov, which is what redis-py and other Redis projects use. The suite runs in three segments under different conditions (the PR matrix without API keys, the full suite with API keys on same-repo PRs, and the cluster suite that #796 adds to the nightly run), so each segment is uploaded under its own Codecov flag rather than as one number.Changes
Collecting coverage in each segment
Every job that runs tests now writes its own coverage data file and uploads it as an artifact, even when tests fail:
coretestmatrix.coverage.core-py<version>-redis-py-<version>-<image>serviceservice-tests(make test-all, Python 3.11).coverage.serviceclustercluster-tests(-m requires_cluster).coverage.cluster-redis-py-<version>The
service-testsjob passes--covthrough a new optionalpytest-argsinput on.github/actions/run-service-tests;test-fork-pr.ymldoes not pass it and runs exactly as before.pyproject.tomlgains[tool.coverage.*]settings: branch coverage ofredisvl, paths stored relative to the checkout so data from different runners combines, andpatch = ["subprocess"]. The last one matters because the CLI tests runpython -m redisvl.cli.runnerin a child process; without it the CLI is measured as untested.Combining and reporting
A new
coveragejob runs after the three test jobs whatever their outcome (!cancelled()), so a skipped job just leaves its segment out. For each segment it combines that segment's data files, writes a Cobertura XML report and uploads it to Codecov under the segment's flag. It then combines all segments into the job summary, as a per-segment totals table plus a per-file table, and uploads an HTML report as an artifact. Uploading once per segment, instead of once per leg, means Codecov never comments before all the matrix legs have reported.Per-segment data is kept in
coverage-segments/rather than beside.coverage, becausepatch = ["subprocess"]turns on coverage's parallel mode, andcoverage reportwould otherwise absorb any.coverage.*file next to its data file.codecov.ymlmakes both statuses informational and carriesserviceandclusterforward on runs that skip them, from the nearest ancestor commit with a report for that flag.pytest-cov>=7,<8joins thedevgroup. Localmake testis unchanged unlessARGSpasses--cov.CONTRIBUTING.mdgains a "Coverage in CI" section and a working local command;AGENTS.mdgains one line under Testing Notes.Notes
The
clustersegment instruments thecluster-testsjob that #796 added to the nightly run; with #796 merged, this PR is based onmain.The
serviceflag measures the full suite plus the API-key tests, because the service job runsmake test-all. On a fork PR, which skips that job, Codecov carriesserviceforward, so the headline number can hide a drop in code thatservicecovered; thecorerow in the Codecov comment shows the PR's own change. Narrowing the service job to the API-key tests would remove the masking but also stop that job testing the rest of the suite, and not carrying the flag forward would make every fork PR show a false drop.Dependabot PRs receive no Codecov token and Codecov refuses tokenless uploads from branches in this repository, so they are not reported to Codecov. The
Coveragejob summary still shows their totals.codecov/codecov-actionis pinned by tag (@v7), matching how every other third-party action in this repository is pinned.Coverage is reported, not enforced: neither status is required, and a failed Codecov upload never fails CI (
fail_ci_if_error: false). A target can be chosen once there is some history.Reviewer Brief
Start here:
.github/workflows/test.yml, thecoveragejob's "Combine coverage" step. It decides which segments reach Codecov and what the job summary shows.Where it could fail
coveragejob'sneeds: [test, service-tests, cluster-tests]withif: ${{ !cancelled() }}: on every PR and pushcluster-testsis skipped, andcoremust still upload; check the expression and the "not run" branch.include-hidden-files: true, and a re-run job replaces its earlier artifact only because ofoverwrite: true.codecov.yml:clusteris carried forward through push-only commits onmain, which uploadcoreandservicebut nocluster; if Codecov does not carry it that far, PRs show noclusterflag.Not verified: no CI run of this workflow yet, so the 3.14 and 3.15 legs (which use coverage's
sys.monitoringcore), the integration suite and the cluster suite have not run under coverage, and the slowdown per leg is unknown; how far Codecov carries a flag forward; tokenless uploads from fork PRs; actionlint was not run.Beyond CI: on Python 3.13 the unit suite gives the same result with and without
--cov(1,993 passed, 1 skipped), with wall time going from 51.5 s to 55.2 s;redisvl/cli/runner.pyreads 0% without subprocess patching and 86% with it, and pytest-cov writes exactly the named data file both under xdist and with-n 0; the "Combine coverage" script, extracted from the workflow and run on local data files, produced the expected summary and outputs both with all three segments and withcorealone;codecov.ymlpasses Codecov's validator (before theclusterflag was added); Codecov's API shows the run on48d6eb32processed withcoreat 78.73% andserviceat 79.41% across 121 files, all underredisvl/, which is every module in the package that has code; thecorecombine reportsskipped 2, which in coverage 7.14 counts only data files identical to one already combined.Since the last brief: after #796 merged, the branch was rebased onto
main, which changed every commit SHA but not this PR's diff.c812cb84lets a re-run test or coverage job replace its earlier coverage artifacts.312bb98eswitches tosource_pkgs, because withsourcethe XML namedredisvl/version.pyandredisvl/__init__.pyas bareversion.pyand__init__.py, and Codecov dropped both.b29dc524regeneratesuv.lock, which the rebase had left unparseable.Reviewed
a5e4293b(correctness, security, design, tests, docs, Redis behaviour) and2d5be637(correctness, tests, docs), fixes in7b93f6f3and4b781ccf· Claude Code (claude-opus-5-5)Note
Low Risk
CI and dependency-only changes; coverage is informational and Codecov upload failures do not block merges.
Overview
CI re-enables coverage measurement across the test workflow: matrix core legs, service (
make test-all), and nightly cluster runs each write a distinct.coverage.*file, upload it as an artifact, and a newcoveragejob merges segments per flag (core,service,cluster), publishes a workflow summary and HTML artifact, and uploads Cobertura XML to Codecov withfail_ci_if_error: false.Tooling:
pytest-covjoins the dev group;[tool.coverage.*]inpyproject.tomltargetsredisvlwith branch coverage, repo-relative paths, subprocess patching for CLI tests, and report exclusions. Newcodecov.ymlmarks statuses informational and carry-forwardsserviceandclusterwhen those jobs are skipped.Workflow plumbing:
.github/actions/run-service-testsaccepts optionalpytest-args(wired as--cov --cov-report=fromtest.yml);CONTRIBUTING.mdandAGENTS.mddocument local and CI coverage behavior.Reviewed by Cursor Bugbot for commit b29dc52. Bugbot is set up for automated code reviews on this repo. Configure here.