Skip to content

test: move the suites to xUnit 4 on Microsoft.Testing.Platform - #105

Merged
Reefact merged 2 commits into
mainfrom
claude/awesome-wozniak-xhdjsh
Oct 31, 2026
Merged

Reefact merged 2 commits into
mainfrom
claude/awesome-wozniak-xhdjsh

Conversation

@Reefact

@Reefact Reefact commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Summary

Moves every test project from xUnit 3.2.2 to xUnit 4.0.2 and runs the suites on Microsoft.Testing.Platform (MTP), the platform xUnit 4 is built for. Coverage moves to coverlet's MTP extension, so the report Sonar reads is the same as before.

Type of change

  • Tests
  • Build / CI / tooling

Changes

  • global.json opts dotnet test into MTP ("test": { "runner": "Microsoft.Testing.Platform" }). Without it, the .NET 10 SDK refuses to run xUnit 4 through VSTest. ci.yml and release.yml run unchanged.
  • The four test projects reference xunit.v3 4.0.2. Microsoft.NET.Test.Sdk and xunit.runner.visualstudio are dropped because they only served VSTest.
  • Coverage moves from coverlet.collector to coverlet.MTP 10.1.0. coverage.runsettings is replaced by tests/testconfig.json, linked into every test project by Directory.Build.targets. It is still one committed file, so a local run and the analysis produce the same report. The file is authoritative for coverlet (no defaults are injected), so it names the measured assemblies explicitly.
  • sonar.yml turns coverage on with --coverlet. coverlet.MTP adds a timestamp to each report's name, so sonar.cs.opencover.reportsPaths now matches coverage.opencover.*.xml.
  • DocumentationSampleCompiler keeps the test runner out of what doc samples compile against: Microsoft.Testing.* replaces testhost and the VSTest assemblies, which are no longer loaded.
  • The first commit is an intermediate step (xUnit 4 kept on VSTest through xunit.v3.mtp-off). The second commit supersedes it.

Testing

  • dotnet build -c Release: clean from deleted bin/obj, 0 warnings, 0 errors
  • dotnet test -c Release: 972 tests passed, the same count as under xUnit 3.2.2 (613 + 192 + 109 + 58)
  • tests/PackageSmokeTest/run.sh: not run. Packaging, the analyzers and the public entry points are untouched.

Also run:

  • the exact sonar.yml test command. It writes four OpenCover reports, all matched by the new reportsPaths glob. They list the same modules with the same sequence and branch counts as the VSTest + coverlet.collector baseline.
  • tools/style/lint-layout.sh: nothing to report.

Not verifiable locally: SonarCloud's import of the new reports. Check that the coverage on this PR's analysis does not drop.

IDE note: without xunit.runner.visualstudio, Test Explorer needs MTP support. Recent Visual Studio 2022, Rider and C# Dev Kit have it.

Public API

  • No change to the public surface

Documentation

  • No documentation change required

Generated by Claude Code

xunit.v3 3.2.2 -> 4.0.2 and xunit.runner.visualstudio 3.1.5 -> 4.0.1 in
the four test projects.

xUnit 4 enables Microsoft.Testing.Platform by default, and the .NET 10 SDK
then refuses the VSTest `dotnet test` that ci.yml, release.yml and
sonar.yml run ("Testing with VSTest target is no longer supported").
The mtp-off flavour is the same xUnit 4 without that switch, so the
workflows and coverlet.collector, which feeds Sonar's coverage, keep
working unchanged.

Verified: 972 tests discovered and passing, the same count as on 3.2.2
(613 + 192 + 109 + 58), and the Sonar command still writes one
non-empty coverage.opencover.xml per test project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019k6GsUrFe92DSgN3opCj6g
The previous commit kept xUnit 4 on VSTest through xunit.v3.mtp-off.
This one takes the platform xUnit 4 is built for.

- global.json opts `dotnet test` into Microsoft.Testing.Platform, so
  ci.yml and release.yml run unchanged (`dotnet test -c Release`).
- Test projects reference xunit.v3 4.0.2 and drop Microsoft.NET.Test.Sdk
  and xunit.runner.visualstudio, which only served VSTest.
- Coverage moves from coverlet.collector to coverlet.MTP 10.1.0. Its
  settings live in tests/testconfig.json, linked into every test project
  by Directory.Build.targets, replacing coverage.runsettings: still one
  committed file, so a local run and the analysis produce the same
  report. The file is authoritative for coverlet, so it names the
  measured assemblies instead of relying on injected defaults.
- sonar.yml switches on coverage with --coverlet; coverlet.MTP stamps
  each report with a timestamp, so reportsPaths matches
  coverage.opencover.*.xml.
- DocumentationSampleCompiler keeps the runner out of what samples
  compile against: Microsoft.Testing.* replaces testhost and the VSTest
  assemblies, which are no longer loaded.

Verified from clean bin/obj: 972 tests pass with the ci.yml and the
sonar.yml commands, and the four OpenCover reports cover the same
modules with the same sequence and branch counts as under VSTest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019k6GsUrFe92DSgN3opCj6g
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T14:55:11.395121Z 3941d13 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Reefact commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner Author

note: SonarCloud rejects SONAR_TOKEN at login; not caused by this PR

The SonarQube Cloud analysis job stops at dotnet-sonarscanner begin, before the build and test steps this PR changes:

WARNING: Authentication with the server has failed.
Pre-processing failed. Exit code: 1

SONAR_TOKEN reaches the job (masked as ***), and SonarCloud rejects it. The last successful Sonar run is on main on 2026-08-12, and there has been no run on main since. A re-run of the job failed the same way. The secret has most likely expired or been revoked.

The recent Dependabot PRs also fail Sonar, but differently: there the token arrives empty (sonar.token= is invalid), because Dependabot runs without repository secrets.

Fixing it needs a new token in SonarCloud (My Account → Security) stored as the SONAR_TOKEN repository secret, then a re-run of this job. Nothing in this PR can fix it.

Until then, SonarCloud's import of the coverage under MTP is unverified. Locally, the sonar.yml test command produces four coverage.opencover.*.xml reports with the same modules and the same sequence and branch counts as under VSTest.


Generated by Claude Code

Reefact commented Oct 31, 2026

Copy link
Copy Markdown
Owner Author

question (blocking): Verify Sonar coverage import after authentication is restored

[P2] The Sonar workflow run failed in Begin analysis with Authentication with the server has failed. Consequently, the updated --coverlet invocation and sonar.cs.opencover.reportsPaths have not executed in CI, so we cannot yet verify that SonarQube Cloud actually imports the four OpenCover reports and preserves the expected coverage. The regular CI (including both SDK matrices and package smoke tests) passed.

Could we restore the Sonar authentication and rerun this analysis, then verify that four reports were imported and the reported coverage did not regress? This is a verification blocker, not a confirmed defect in the MTP migration.

Reefact commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner Author

ack: stays blocking until Sonar analyses this PR; token rotation needed

Answers question (blocking): Verify Sonar coverage import after authentication is restored.

Restoring authentication needs a new SonarCloud token stored in the SONAR_TOKEN repository secret. That is a maintainer action, and nothing in this branch can do it. Once the secret is replaced and the SonarQube Cloud analysis job re-run, I will check, from the job log and the SonarCloud analysis:

  • four coverage.opencover.*.xml files imported (the scanner logs each report it parses);
  • coverage on SonarCloud unchanged from main. The local baseline per report is 286/291, 297/350, 83/433 and 83/83 sequence points, identical under VSTest and MTP.

If either does not hold, I will fix it on this branch.

Meanwhile, the regular CI confirms the tests themselves run under MTP: build (SDK 10.0.100) reports total: 972, succeeded: 972 from dotnet test -c Release --no-build, which is the same count as under xUnit 3.2.2.


Generated by Claude Code

Reefact commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner Author

note: Sonar imports all four MTP reports; coverage identical to main

Answers question (blocking): Verify Sonar coverage import after authentication is restored. No code change was needed.

SONAR_TOKEN has been rotated. The cause was SonarCloud's inactivity rule: a token unused for 60 days is removed, and the last analysis was on 2026-08-12. The re-run is green.

From the job log:

  • Test with coverage: total: 972, failed: 0.
  • Sensor C# Tests Coverage Report Import reads four reports, one Parsing the OpenCover report …/coverage.opencover.<timestamp>.xml line per test project, then logs 13 files, 13 main files, 13 main files with coverage.
  • QUALITY GATE STATUS: PASSED.

From the SonarCloud measures API, main compared with this PR:

Metric main PR 105
coverage 97.8 % 97.8 %
line coverage 99.8 % (1 / 433 uncovered) 99.8 % (1 / 433 uncovered)
branch coverage 94.8 % (15 / 286 uncovered) 94.8 % (15 / 286 uncovered)

Every number is identical, so moving from VSTest + coverlet.collector to MTP + coverlet.MTP changes nothing in what Sonar measures.


Generated by Claude Code

Reefact commented Oct 31, 2026

Copy link
Copy Markdown
Owner Author

reviewed: Sonar coverage import verified after the MTP migration

The successful Sonar rerun confirms that all 972 tests pass, all four timestamped OpenCover reports are parsed, all 13 main files have coverage, and the Quality Gate passes. The reported coverage matches main (97.8% overall, 99.8% lines, 94.8% branches), as documented in the follow-up. The previous verification blocker is resolved, the pull request head remains 3941d13, and no findings remain on my side.

@Reefact
Reefact merged commit 8376d76 into main Oct 31, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants