Skip to content

Add Codecov Test Analytics (JUnit upload + failed test reporting) #70

Description

@itsniper

Summary

Wire up Codecov Test Analytics so failed tests (and flaky tests) surface in Codecov PR comments and the Tests tab, separate from our existing coverage upload.

We already upload coverage (coverage.lcov via codecov/codecov-action@v5 + CODECOV_TOKEN). Test Analytics needs a second upload: JUnit XML test results with report-type: test_results.

What we get

  1. Failed test reporting on PRs — failed tests + stack traces in the Codecov PR comment
  2. Flaky test callouts — free for public/OSS repos
  3. Tests tab — per-test duration, failure rates, slow tests (results retained ~60 days)

How it works

swift test → JUnit XML → Codecov (report-type: test_results) → PR comment + Tests tab

Codecov only supports JUnit XML for test results today.

Implementation sketch

1. Emit JUnit from swift test

Today (.github/workflows/ci.yml):

- name: Test
  run: swift test --enable-code-coverage

Add xunit output, e.g.:

swift test --enable-code-coverage --xunit-output junit.xml

(--xunit-output is built into SwiftPM; filename should match *junit.xml or be passed explicitly to the uploader.)

2. Upload test results (even when tests fail)

Use the current codecov/codecov-action@v5 with report-type: test_results. The dedicated codecov/test-results-action is deprecated in favor of this.

Critical: if the Test step fails, later steps are skipped by default — so failed-test reporting never gets the file. Upload with if: ${{ !cancelled() }} (and ideally do the same for coverage export/upload so failures still report coverage when possible).

Sketch:

- name: Upload test results to Codecov
  if: ${{ !cancelled() }}
  uses: codecov/codecov-action@v5
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    report-type: test_results
    files: junit.xml
    fail_ci_if_error: false

Keep the existing coverage upload as a separate step (no report-type, or coverage files only).

3. Verify

  • CI log shows successful test-results upload
  • Codecov PR comment acknowledges test results received
  • Intentionally fail a test once and confirm failed tests + stack traces appear in the PR comment
  • Tests tab populates on the default branch after a few runs

Notes / caveats

  • Two independent uploads: coverage (lcov) vs test results (junit) — both use the same token
  • XCTest xunit detail can be thinner than some frameworks; still enough for names + pass/fail analytics. Swift Testing often has richer <failure> messages if we mix runners later
  • Not a CI gate — analytics only; the job still fails when tests fail
  • No change needed to codecov.yml for basic setup (flags optional if we ever matrix test envs)

References

Acceptance criteria

  • CI produces a JUnit XML report from the macOS test job
  • Test results are uploaded to Codecov with report-type: test_results
  • Upload runs when the workflow is not cancelled, including after test failures
  • Existing coverage upload continues to work
  • Codecov PR comments show test results / failed tests when applicable

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions