Repository navigation
ci: mutation testing with gremlins (just mutation, Test Action job) - #334
Conversation
Vulnetix config planManager
0 to create, 1 to update, 0 to delete, 0 unchanged |
0x73746F66
left a comment
0x73746F66
left a comment
There was a problem hiding this comment.
Reuse review: the CI job reuses the same private-module fetch step and pinned actions as the existing test jobs, and runs exactly the just mutation default so local and CI agree. gremlins is pinned (v0.6.0) like golangci-lint. No changes needed.
0x73746F66
left a comment
0x73746F66
left a comment
There was a problem hiding this comment.
Reuse review: the CI job mirrors the existing test job (same checkout/setup-go actions already recorded for test.yml in actions.lock, same private-module fetch step) and the just recipe pins gremlins the way golangci-lint is pinned, so no new mechanism is introduced. The 'Lockfile matches the workflows' failure was HTTP 403 rate limit on PACKAGES_TOKEN during 'gh extension install', not a lockfile mismatch; it will re-run on the rebased head.
Adds a pinned gremlins v0.6.0 run as a justfile recipe (any package, default internal/purl at 80% efficacy) and as a "Mutation testing" job in the Test Action workflow. The job sets up the private module fetch first, since gremlins runs go mod download itself. The threshold sits under the measured baseline (84.62% on internal/purl today; 100% once the #225 tests land), so a test change that stops catching a mutant fails the job. Closes #224
8af5789 to
5cb417e
Compare
Vulnetix config planManager
0 to create, 1 to update, 0 to delete, 0 unchanged |
Closes #224
What
justfile:mutation pkg="./internal/purl" efficacy="80"runsgo run github.com/go-gremlins/gremlins/cmd/gremlins@v0.6.0 unleash --threshold-efficacy <efficacy> --timeout-coefficient 10 <pkg>. Pinned like golangci-lint, so an upgrade is a deliberate commit.just mutation ./internal/versions 0surveys another package without gating..github/workflows/test.yml: aMutation testingjob (needs: [lint], same runner labels) that checks out, sets up Go, configures the private fetch of vdb-sca-match/vdb-manifest exactly like thetestandworktreejobs (gremlins runsgo mod downloaditself), then runs the same command. It uses onlyactions/checkout@v7andactions/setup-go@v7, already inactions.lockfor this workflow, so the lockfile is unchanged.Threshold
Measured locally with gremlins v0.6.0 on
internal/purl: 33 killed, 6 lived, efficacy 84.62% on main; 100% with the tests in #323 (#225). 80% passes today and fails if a test change stops catching mutants.--timeout-coefficient 10keeps a loaded self-hosted runner from turning slow tests into timeouts.Verified
just --dry-run mutationrenders the command.gremlins unleash --threshold-efficacy 80 --timeout-coefficient 10 ./internal/purlexits 0 at 84.62%.go build ./...unaffected.