What happened
PR #202 added 205 lines of tests in src/commands/new/command.test.ts covering all three plugin types, validation, error cleanup, and interactive prompting. The CI checks job runs yarn backstage-cli repo test --coverage and the tests job runs yarn test (which expands to backstage-cli package test --coverage), so coverage data IS generated. However, SonarCloud's quality gate comment reported "0.0% Coverage on New Code" — and the quality gate still passed, providing false confidence about test coverage.
Investigation found three gaps:
.sonarcloud.properties does not set sonar.javascript.lcov.reportPaths — SonarCloud has no idea where to find coverage files.- No
sonarsource/sonarcloud-github-action step exists in any CI workflow — SonarCloud is running in Automatic Analysis mode, which performs its own checkout and cannot access CI-generated coverage artifacts.
- Coverage output from
backstage-cli package test --coverage (written to coverage/lcov.info) is never uploaded as a CI artifact.
This affects all PRs, not just #202 — coverage has likely been invisible to SonarCloud for the entire project history.
What could go better
Coverage visibility is a foundational quality signal. Without it, SonarCloud's quality gate is checking only duplication and static analysis — its coverage assessment is meaningless. This gap means:
- Neither the review agent nor humans can cross-reference SonarCloud coverage data to assess test adequacy.
- The quality gate provides false assurance ("passed" despite 0% coverage).
- Coverage trends over time are not tracked, making regressions invisible.
Confidence: High. The .sonarcloud.properties file was directly inspected and contains only a CPD exclusion. The CI workflows were read and confirmed to have no SonarCloud scanner step. The --coverage flag in test scripts confirms coverage data is generated but never consumed.
This is a systemic repo-level issue, not specific to PR #202. It likely predates the fullsend integration.
Proposed change
Two changes needed in redhat-developer/rhdh-cli:
-
Add coverage report path to .sonarcloud.properties:
sonar.javascript.lcov.reportPaths=coverage/lcov.info
-
Switch from Automatic Analysis to CI-based analysis by adding a SonarCloud scanner step to .github/workflows/pr.yaml after the test step (or uploading coverage as an artifact and running the scanner in a dependent job). Automatic Analysis cannot access CI-generated coverage files — CI-based analysis is required.
Alternatively, if staying with Automatic Analysis is preferred, the project should document that SonarCloud coverage data is not available and consider whether the quality gate should enforce a coverage threshold once CI-based analysis is enabled.
Validation criteria
After the fix: (1) SonarCloud reports non-zero coverage on PRs that add or modify tested code. (2) The sonar.javascript.lcov.reportPaths property is set in .sonarcloud.properties. (3) A SonarCloud scanner step exists in CI, or coverage is uploaded as an artifact consumed by the scanner. Verify on the next 2-3 PRs that include test changes.
Generated by retro agent from #202
What happened
PR #202 added 205 lines of tests in
src/commands/new/command.test.tscovering all three plugin types, validation, error cleanup, and interactive prompting. The CIchecksjob runsyarn backstage-cli repo test --coverageand thetestsjob runsyarn test(which expands tobackstage-cli package test --coverage), so coverage data IS generated. However, SonarCloud's quality gate comment reported "0.0% Coverage on New Code" — and the quality gate still passed, providing false confidence about test coverage.Investigation found three gaps:
.sonarcloud.propertiesdoes not setsonar.javascript.lcov.reportPaths— SonarCloud has no idea where to find coverage files.sonarsource/sonarcloud-github-actionstep exists in any CI workflow — SonarCloud is running in Automatic Analysis mode, which performs its own checkout and cannot access CI-generated coverage artifacts.backstage-cli package test --coverage(written tocoverage/lcov.info) is never uploaded as a CI artifact.This affects all PRs, not just #202 — coverage has likely been invisible to SonarCloud for the entire project history.
What could go better
Coverage visibility is a foundational quality signal. Without it, SonarCloud's quality gate is checking only duplication and static analysis — its coverage assessment is meaningless. This gap means:
Confidence: High. The
.sonarcloud.propertiesfile was directly inspected and contains only a CPD exclusion. The CI workflows were read and confirmed to have no SonarCloud scanner step. The--coverageflag in test scripts confirms coverage data is generated but never consumed.This is a systemic repo-level issue, not specific to PR #202. It likely predates the fullsend integration.
Proposed change
Two changes needed in
redhat-developer/rhdh-cli:Add coverage report path to
.sonarcloud.properties:Switch from Automatic Analysis to CI-based analysis by adding a SonarCloud scanner step to
.github/workflows/pr.yamlafter the test step (or uploading coverage as an artifact and running the scanner in a dependent job). Automatic Analysis cannot access CI-generated coverage files — CI-based analysis is required.Alternatively, if staying with Automatic Analysis is preferred, the project should document that SonarCloud coverage data is not available and consider whether the quality gate should enforce a coverage threshold once CI-based analysis is enabled.
Validation criteria
After the fix: (1) SonarCloud reports non-zero coverage on PRs that add or modify tested code. (2) The
sonar.javascript.lcov.reportPathsproperty is set in.sonarcloud.properties. (3) A SonarCloud scanner step exists in CI, or coverage is uploaded as an artifact consumed by the scanner. Verify on the next 2-3 PRs that include test changes.Generated by retro agent from #202