Skip to content

fix(mutation): the Messaging gate filters out the S3 differential run, so the nightly finishes (#562) - #577

Draft
omercelikdev wants to merge 1 commit into
mainfrom
agent/562-messaging-mutation-differential
Draft

omercelikdev wants to merge 1 commit into
mainfrom
agent/562-messaging-mutation-differential

Conversation

@omercelikdev

Copy link
Copy Markdown
Collaborator

What and why

#562: the nightly mutation (Goldpath.Messaging) job hit its 300-minute timeout-minutes with no score. Run 42789231201 was cancelled, so the break-70 gate gave no verdict on a critical-path package.

Cause (the issue's evidence; I re-read the config to confirm it):

Fix: one line. This is the issue's preferred fix.

- "test-case-filter": "FullyQualifiedName!~Goldpath.Tests.Integration.|FullyQualifiedName~Goldpath.Tests.Integration.OutboxAtomicityTests.",
+ "test-case-filter": "(FullyQualifiedName!~Goldpath.Tests.Integration.|FullyQualifiedName~Goldpath.Tests.Integration.OutboxAtomicityTests.)&Category!=Differential",
  • The differential run is an engine-parity comparison, not part of the unit-level killer set.
  • It still runs on every PR: ci.yml's conformance step runs dotnet test tests/Goldpath.Messaging.Conformance/... with no filter. That file is untouched.
  • Unchanged: the break threshold (70), the timeout (300), the concurrency cap (2), and the differential test and its allow-list.

Ledgers:

  • stryker/README.md: the Messaging/Conformance borrowed-suite row now says why the differential run is filtered out, and where it still runs.
  • CHANGELOG [Unreleased] → Fixed.
  • No open thread is deferred by this change.

Reproduce

git fetch origin agent/562-messaging-mutation-differential && git checkout agent/562-messaging-mutation-differential
dotnet tool restore
GOLDPATH_MUTATION_CONCURRENCY=2 bash scripts/mutation-gate.sh Goldpath.Messaging      # the nightly's concurrency
#   expect: "Analyzing 3 test project(s)", an initial test run of about 4 min (not ~8), MUTATION GATE GREEN at >= 70

# RED (the fault back): restore the old filter, and the initial run doubles because the differential facts rejoin
git show origin/main:stryker/Goldpath.Messaging.json > stryker/Goldpath.Messaging.json
GOLDPATH_MUTATION_CONCURRENCY=2 bash scripts/mutation-gate.sh Goldpath.Messaging
#   initial test run ~7.8 min, as in the cancelled nightly (the full run does not finish in 300 min on the CI profile)
git checkout stryker/Goldpath.Messaging.json

The authoritative proof is the issue's second one: a workflow_dispatch of nightly (or the next scheduled night) with mutation (Goldpath.Messaging) green inside 300 minutes. I did not dispatch it, because it runs 31 jobs; the owner or the verifier can.

Evidence so far (local sandbox: 4 vCPU / 15 GB, concurrency 2, the CI shape; on c9d23ae + this change)

nightly 2026-11-09 (3968ab5) nightly 2026-11-10 (feb2b3e, cancelled) this branch
tests found 155 216 208
mutants created / to test 619 / 348 765 / 441 798 / 461
initial test run 3.6 min 7.8 min 4.2 min (00:32:05 → 00:36:17 UTC)
result 87.09 %, 179m 54s cancelled at 300 min running, see below

Reading the figures:

The full local run is still running as I open this. I will post its score and duration here when it ends. If the score falls below 70, I will stop and hand it to the owner, as the issue says, rather than lower anything.

Pre-PR self-check

  • Acceptance criterion 1 is open: mutation (Goldpath.Messaging) completes inside 300 min with a score of at least 70. The local full run is in progress; the nightly is the authoritative proof.
  • Acceptance criterion 2: the S3 differential facts still run on every PR. ci.yml's conformance step is unchanged and unfiltered.
  • Acceptance criterion 3: the preferred fix (exclude Category=Differential) is the whole change. Timeout, concurrency and break are untouched.
  • Ledgers: CHANGELOG [Unreleased] and stryker/README.md. No GA-gate status change; the RFC is not touched.
  • XML docs: n/a, no code.
  • Honesty: the PR claims the filter and the measured initial run. The score and duration are stated as pending, not as claimed.
  • Wired: the nightly matrix runs scripts/mutation-gate.sh Goldpath.Messaging, and that script reads this config (the local run above shows "Analyzing 3 test project(s)").
  • Contracts: n/a.
  • Bench / long test: no new test. The mutation run is the existing nightly job.
  • Deep self-review. Only compatibility and test quality apply:
    • the VSTest filter's precedence is explicit, with parentheses around the OR and & for the category;
    • the OutboxAtomicityTests borrowing is preserved;
    • no other config names the conformance project (grep -l Conformance stryker/*.json → only Messaging).

Gates run

Gate Result
docs-freshness.sh, ledger-check.sh, schema-honesty.sh, skills-parity.sh, template-pins.sh rc 0 ×5
dotnet build n/a: no source changed
Mutation gate (Messaging, full) running, to be posted here

What I could not verify

  • The nightly on the CI profile: needs a dispatch.
  • The final score and wall-clock time: pending, as above.

Closes #562


Generated by Claude Code

…, so the nightly finishes (#562)

Since #526 the S3 differential run (BusDifferentialTests, Category=Differential: every
conformance fact again on both engines, on real containers) sat in the test set of
stryker/Goldpath.Messaging.json, so Stryker ran it against every mutant. Nightly
42789231201 was cancelled at the job's 300-minute timeout with no score (216 tests found,
441 mutants to test, a 7.8-minute initial run, against 155 / 348 / 3.6 the night before).

The config's test-case-filter now also excludes Category=Differential. The differential
run compares engine outcomes; it is not the unit-level killer set, and ci.yml's conformance
step still runs it on every pull request. Threshold, timeout and concurrency unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AFdNd6h1NzwmonTGq2k4V
@omercelikdev omercelikdev added the agent-pr PR opened by the overnight agent label Nov 12, 2026
@github-actions github-actions Bot added the tier:C ADR-0014 Tier C: decisions, releases, gates, policy, agent layer; the owner merges label Nov 12, 2026
@omercelikdev
omercelikdev marked this pull request as draft November 12, 2026 02:52
@omercelikdev

omercelikdev commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status at the run's hard stop (06:00 Istanbul): I have converted this PR to a draft.

Done:

Remains: the score and the wall-clock time.

  • Mutant testing started at 00:40:23 UTC. At 02:52 UTC, after about 132 min, it had produced no verdict and was still running. This sandbox goes away when the session ends, so it may never report.
  • So the gate's verdict needs a nightly run: a workflow_dispatch of nightly, or the next scheduled night.
  • Watch the duration. 461 mutants is 32 % more than on 2026-11-09, which took 180 min, and a linear estimate of about 240 min leaves little margin under 300. If the nightly comes in near or over 300 even with the differential run excluded, this is an owner decision, as the issue says. The fix must not raise the timeout or lower the break.

CI on the head dc205c8:

  • merge-policy: red by design. This is a Tier C change (a gate), so it needs owner-approved (ADR-0014).
  • oracle: one fact failed, CampaignOracleTests.Thousands_release_paced_across_leader_handovers_and_a_live_throttle_lifts_the_ceiling (a TaskCanceledException in WaitForStateAsync). It is a timing-bound Campaign test on Oracle. This PR changes only a Stryker config and two documents, and no code that test runs. I re-ran the failed jobs once, to tell a one-off from a real failure.
  • gates, integration (which includes the unfiltered conformance and differential suite), kit, review and CodeQL: green.

Generated by Claude Code

@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Verifier verdict (run 2026-11-12)

Verdict: no blocking findings. Not agent-verified yet, because acceptance criterion 1 (the Messaging gate finishes inside 300 min with a score of at least 70) still needs the nightly. This is a draft that says it is waiting for that evidence, so I have not added changes-requested. Labels are unchanged (agent-pr, tier:C). Tier C: the owner merges.

  • Head verified: dc205c889ea0a40b95b4d0a15463ee380d117ab7
  • Base: main @ c9d23ae5020cbd4a87540dd58e9c20253764d52b (the merge base, with no drift)
  • Worked in my own worktree. Sandbox: 4 vCPU / 15 GB, Stryker 5.0.0 from the tool manifest, --concurrency 2 (the nightly's setting).

Steps 1–6

# Step Result Evidence
1 Scope ✓ The diff is 3 files: the one-line test-case-filter change in stryker/Goldpath.Messaging.json, the Messaging/Conformance row in stryker/README.md, and CHANGELOG [Unreleased] → ### Fixed. Unchanged: thresholds (85/75/70), mutate (no key, so the full package), ignore-methods, nightly.yml (timeout-minutes: 300, GOLDPATH_MUTATION_CONCURRENCY: "2") and the differential test and its allow-list. No dependency, pin or workflow change. AC2 ✓: ci.yml:101 still runs dotnet test tests/Goldpath.Messaging.Conformance/... --nologo with no filter. AC3 ✓: this is the preferred fix and the whole change. AC1 open: no full run and no nightly yet.
2 Rules ✓ The repo's own rule, in stryker/README.md, is that a borrowed suite or an exclusion needs a row there. It has one, and it says where the excluded tests still run. CHANGELOG entry ✓. No public API, XML docs or contract is touched. No open-threads row is needed: this is a permanent scoping decision, not deferred work (the same pattern as the MultiTenancy row's filtered integration suite). No T-number is used.
3 Confidentiality ✓ Case-insensitive grep of the diff for the whole list (BTK with word boundaries): no hit. The PR body and commit message are clean too.
4 Reproduce ✓ (mechanical); full run not run Gates: docs-freshness, ledger-check, schema-honesty, skills-parity, template-pins all rc 0. For the filter and Stryker proofs, see the probes below. I did not run the full Messaging mutation run: the author's run gave no verdict after 132 min, and it could not finish before the 08:45 deadline.
5 CI Required checks green, except merge-policy (red by design on Tier C) Required checks (merge-policy, gates, integration, kit, golden manifests (PR subset)): gates, integration (which runs the unfiltered conformance and differential suite), kit and GM subset are green. oracle (not required) is red. Its first run failed on CampaignOracleTests.Thousands_release_paced_…, and the same test fails on main at the base c9d23ae (job 130581794294) and at 440f4a0 (job 130504779979). The re-run (job 130938885267) failed on a different test, NotificationOracleTests…, with ORA-12537: TNS:connection closed in InitializeAsync, which is an infra failure. This PR touches no code either test runs. CodeQL green.
6 Charter n.a.

Deep review (each changed line, with the callers it reaches)

Area Result
a Security n.a.: test-selection config and docs only. No input, endpoint, secret or dependency.
b Correctness ✓ The filter's precedence is load-bearing, and the PR gets it right. Without the parentheses, VSTest reads A|B&C as A|(B&C) and all 24 differential tests stay in (probe P2). Category!=Differential keeps tests that have no Category trait (P3). The OutboxAtomicityTests borrowing is preserved: Goldpath.IntegrationTests lists 2 tests under both filters.
c Concurrency n.a.
d Data/migrations n.a.
e Performance ✓ This is the point of the change. On the same --mutate subset, the initial test run took 4m53s with the new config against 8m00s with main's, and the whole run 11m35s against 19m33s (P4).
f Compatibility ✓ scripts/mutation-gate.sh still sees 3 test projects ("Analyzing 3 test project(s)"), so the scope check holds. No other config names the conformance project.
g Test quality / score ✓ on the measured subset; ⚠ unmeasured on the full package. On the same subset, both configs score 77.27 % (20 tested, 2 NoCoverage), so the differential added no kill there (P4). A seam mutant that the differential would see (the correlation header dropped on publish, Filters.cs:52) is killed by the plain MassTransitBusConformance.S2_2 under the new filter (P5). Mutants that only the differential could kill are now not counted. No full score with the differential in was ever produced (the 2026-11-10 night was cancelled), so no measured score falls: the killer set goes back to the pre-#526 one, which scored 87.09 % on 2026-11-09.
h Code quality ✓
i Operability ✓ Timeout and concurrency unchanged. The PR flags the margin honestly: 461 mutants to test, against 348 on the 180-min night.
j Honesty ✓ with one NIT (N1). The PR states the score and duration as pending, not as claimed. The 208 tests found matches what I measured independently (P1, P4).

Edge-case probes (mine, nothing pushed)

# Probe Result
P1 dotnet vstest --ListTests --TestCaseFilter:<config filter> on each of the 3 test projects Conformance: 107 under main's filter → 83 under the new one. The 24 removed are exactly BusDifferentialTests (23 S3_… theory rows + Every_conformance_fact_is_in_the_differential); nothing else drops. Messaging.Tests: 123 → 123. IntegrationTests: 2 → 2. Total 208, which matches Stryker's "Number of tests found: 208".
P2 The same filter without parentheses 107 listed, all 24 differential tests still in. The parentheses are required, and they are present.
P3 Category=Differential / Category!=Differential alone 24 / 83: trait-less tests are kept.
P4 Stryker 5.0.0, --concurrency 2 --mutate "**/GoldpathMessageNameAttribute.cs", new config vs main's config New: "Number of tests found: 208", initial run 4m53s, 798 created / 20 tested, 77.27 %, 11m35s. Main's: "232", initial run 8m00s, 20 tested, 77.27 %, 19m33s. Stryker honours the filter, and the score on this subset is unchanged.
P5 Planted mutant Filters.cs:52 (the X-Correlation-Id header not set on publish), run under the new filter MassTransitBusConformance.S2_2_… goes RED (Assert.Equal() Failure: Strings differ). Restored: GREEN.

NIT (optional, does not block)

  1. stryker/README.md:41: "an engine-parity check, not a mutant killer". This overstates a little. Goldpath.Messaging contains the oracle engine (MassTransitMessagingEngine, Filters). A mutant that changes the oracle's recorded outcome (headers, attempts, error-queue or outbox/inbox rows) without failing a plain fact could make the differential diverge. ga-gate-status G5 Part 3/5 itself records a planted fault that only the differential sees, though on the Wolverine side. Suggested wording: "its kills are not counted: the score is measured on the plain conformance facts, as it was before test(messaging): S3 differential run — every conformance fact on both engines, outcomes compared against a reviewed allow-list (G5 Part 3/5, #332) #526". The rationale stays true either way.

What I could not run

  • The full Goldpath.Messaging mutation run: the final score and the wall-clock time against the 300-min limit. AC1 needs the nightly: a workflow_dispatch of nightly or the next scheduled night, with mutation (Goldpath.Messaging) green. Watch the duration: 461 mutants to test, against 348 on the 180-min night.

Owner: once the nightly shows mutation (Goldpath.Messaging) green inside 300 min with a score of at least 70, this is safe to merge, and the owner merges it (Tier C). Summary: the nightly Messaging mutation gate stops running the S3 differential suite against every mutant, so it can finish and give a verdict again. The differential still runs on every PR in ci.yml.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr PR opened by the overnight agent tier:C ADR-0014 Tier C: decisions, releases, gates, policy, agent layer; the owner merges

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI red on goldpath@main: nightly mutation (Goldpath.Messaging) hits the 300-min timeout since the S3 differential run joined its test set

2 participants