P41: measured test parallelism — dissolve HeavyCompilation, drop per-call GC, shard corpus Theories, caps 2/4 (#2179, #2180) by antonsynd · Pull Request #2252 · antonsynd/sharpy

antonsynd · 2026-10-05T21:47:28Z

P41: test-suite throughput (#2179, #2180). This changes tests, test infrastructure, CI, docs and scripts only; no product code changed.

Summary

Thread caps measured, not assumed. maxParallelThreads is now Compiler.Tests 2 and Stdlib.Tests 4. HeapHardLimit stays at 5 GB.
HeavyCompilation and Sequential dissolved. HeavyCompilation's [CollectionDefinition] lived in Sharpy.TestInfrastructure, which xUnit v2 ignores, so the attribute chained 173 classes into one serial collection. The only remaining run-alone collections are 8 audited ones, each guarding a process-global hazard.
Per-call forced gen-2 GC removed from the three IntegrationTestBase execute arms. It was ≈52% of each call. The arms now share one execute path.
Single-class corpus Theories hash-sharded. FileBased fixtures run as Compiler K=8 and Stdlib K=4, LetEmissionInvariance as K=8. Each set of shards lives in a namespace that keeps the old class name, so ~FileBasedIntegrationTests, DisplayName~ and the CI shard filters are unchanged. LetEmission's per-cell corpus rediscovery is cached: 466 s → 57 s.
New guards, each mutation-tested:
- process-global writes and the serial-collection roster, including a check that no collection attribute is inert;
- fixture-shard totality, which also refuses a skipped or re-categorised shard;
- a single execute path;
- guard citations that resolve a sharded group.
Docs and wrapper: the suite figure is now "~30 min", and the dotnet-serialized lock timeout is 4500 s.

Measured

Whole-solution gate Category!=Benchmark:

passed / failed / skipped sha wall

Before 57873 / 0 / 4 116c6e2 Compiler.Tests alone 5325 s

After 57937 / 0 / 4 a1504a4 1925 s for the whole solution

Test name × outcome compared against the control, in every project: nothing is missing, and the additions are the new guard rows only.
Order-independence: three full runs and one perturbed-order run (MaxParallelThreads=3, ParallelAlgorithm=aggressive) gave identical results.
Compiler.Tests at N=1/2/4/8: 2042 / 1641 / 1981 / 4707 s, peak tree RSS 7.1–8.2 GB. N≥4 loses to oversubscription by classes that are already internally parallel.
Stdlib.Tests at N=1/2/4/8: 142 / 33 / 26 / 23 s, under 1 GB at every N. At N=8 one fixed-window timer test went red once; that class now runs alone (stdlib: threading.Timer is a .NET thread-pool timer, not a threading.Thread — isinstance(t, Thread) is False, t.join() is SPY0203, the callback waits for a free pool thread #2251).

For CI review

Please check the per-shard wall times of the 7 Compiler shards in this run. The shard-balance comment in .github/workflows/dotnet10.yml will be re-balanced from them in a follow-up.

Filed from the round

concurrency: process-wide mutable state is not safe under concurrent in-process use — two compilations in one process gave 5 vs 1 diagnostics; module-level random shares one System.Random; crash bundles check-then-create #2250 concurrency: process-wide mutable state is not safe under concurrent in-process use.
stdlib: threading.Timer is a .NET thread-pool timer, not a threading.Thread — isinstance(t, Thread) is False, t.join() is SPY0203, the callback waits for a free pool thread #2251 threading.Timer is a pool timer, not a Thread.

Refs #2179 #2180 (both stay open for the cells and the #2180 in-process ruling).

🤖 Generated with Claude Code

https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…ss (refs #2179) P41 Phase 1 Task 1. The memory harness gains the axis the plan measures on: - --threads N appends the xUnit RunSettings override `-- xUnit.MaxParallelThreads=N` to the run. Route taken: the RunSettings override (no MSBuild fallback needed) - it is honoured over xunit.runner.json, which pins 1. - --out DIR (default .claude/tmp/memory-harness) so matrix cells do not clobber each other; --timing PATH exports SHARPY_TEST_TIMING (absolutized) for IntegrationTestBase's breakdown; --trx adds a trx logger at DIR/tests.trx. - One `ps -ww -A -o pid=,ppid=,rss=,args=` snapshot per sample, folded by awk into tree RSS (dotnet child + all descendants) and the largest testhost's own RSS. The testhost is matched by `exec --runtimeconfig <...>.Tests.runtimeconfig.json`: its comm is plain `dotnet`, like every spawned test program's, and args without -ww truncates `testhost.dll` away. - Machine used memory (vm_stat active+wired+compressor) and swap used at lock acquisition, swap again at exit; counts parsed from vstest's multi-line "Total tests:" block (zero lines are omitted, read as 0) or the one-line "Passed! - Failed:" form; "?" when neither is present. - Final line: threads=N peak_rss_mb peak_testhost_mb wall_s passed failed skipped base_used_mb swap_mb=A->B. --threshold and the exit-code conventions are unchanged. Override evidence @ 116c6e2 (worktree build, measured), filter FullyQualifiedName~Sharpy.Compiler.Tests.Lexer|FullyQualifiedName~Sharpy.Compiler.Tests.Parser, 2785 tests in 59 classes, trx start/end intervals swept for concurrency: - --threads 2: max_concurrent=2, 2696 cross-class overlapping pairs, e.g. PropertyDecoratorMatrixTests.SteerSpellingRuns [01:41:06.894758 .. 01:41:07.884882] and WrapperNodeUnwrapConformanceTests.ImportDispatchSites_AreWrapperAware [01:41:07.032700 .. 01:41:07.404927] (372 ms overlap). Summary: threads=2 peak_rss_mb=1105 peak_testhost_mb=800 wall_s=9 passed=2785 failed=0 skipped=0 - --threads 1 (control): max_concurrent=1, 0 cross-class overlapping pairs. Summary: threads=1 peak_rss_mb=1096 peak_testhost_mb=806 wall_s=10 passed=2785 failed=0 skipped=0 - The overlap sweep itself discriminates on the P0 control trx files: Stdlib (runner json pins 1) max_concurrent=1 / 0 pairs; Cli, Core, Lsp (no runner json) max_concurrent=14 each. Parser checks: summary awk -> 1234/2/3 on the one-line form, 441/0/1 (Cli) and 4162/0/0 (Stdlib) on the control logs, "? ? ?" on an empty log; the tree awk on a synthetic tree using the measured args spellings counts the whole tree and matches only the testhost (not vstest.console, not SharpyTestAssembly). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…efs #2180) P41 Phase 1 Task 2. Behind SHARPY_TEST_TIMING=<path> (read once into a static), every call of the three execute arms - CompileAndExecute ("single"), CompileAndExecuteProject ("project"), CompileAndExecuteEntryFile ("entry-file"; the latter two through the shared EmitAndRunProjectAssembly) - records six components plus its own total wall: pipeline (CompilerApi/ProjectCompiler compile; includes C# generation, which the compiler API exposes no seam for), emit (Roslyn parse + compile to the in-memory assembly), write+copy (temp source / assembly / runtimeconfig writes + Core, closure and native copies), process (start to exit), cleanup (temp dir deletes) and forced-gc. Calls are queued in a ConcurrentQueue; at ProcessExit the testhost appends pid-tagged rows per arm x component (n, median, p90, sum ms), the arm's total, and a coverage row comparing sum of components with sum of totals. Unset: CallTiming.Begin returns null, every hook is a `timing?.` no-op and Start() reads no clock; the arms' statements, order and ExecutionResult are untouched. Components are timed independently, not as laps, so a dropped recording surfaces as unattributed time in the coverage row rather than being folded into its neighbour. Instrument check @ 116c6e2 (worktree build, measured; default threads = runner json 1): (a) switch OFF, FullyQualifiedName~Sharpy.Compiler.Tests.Integration.FileBasedIntegrationTests: 3569 passed / 0 failed / 0 skipped = the P0 control's 3569 FileBasedIntegrationTests rows in Compiler.trx; the (fixture, isMultiFile, outcome) multiset is identical. Switch ON, the same 3569 and the 152 EntryFilePathMultiFileTests rows are identical to the control too. (b) switch ON, FileBasedIntegrationTests|EntryFilePathMultiFileTests (3721 passed / 0 / 0, 3718 calls), report written; per arm (ms): single n=3418 pipeline med 5.43 p90 10.44 | emit med 0.98 p90 2.48 (n=2401) write+copy med 1.87 p90 2.08 | process med 48.23 p90 51.10 (n=2400) cleanup med 1.20 p90 1.48 | forced-gc med 63.18 p90 67.69 total med 121.75 p90 137.05; coverage 363663.7 / 363842.1 = 0.9995 project n=150 pipeline med 30.73 | emit 1.47 | write+copy 1.64 | process 48.10 | cleanup 0.84 | forced-gc 62.74 | total med 148.71; coverage 0.9989 entry-file n=150 pipeline med 17.86 | emit 1.51 | write+copy 1.85 | process 48.42 | cleanup 0.85 | forced-gc 65.09 | total med 135.57; coverage 0.9996 (c) mutation - both `timing?.Add(Component.Process, ...)` lines commented out (worktree copy), rebuilt, filter EntryFilePathMultiFileTests|BacktickEscapedForwarderTests| ClrInteropIntegrationTests (173 passed): broken -> 5% check FAILS in all three arms (entry-file ratio 0.7396, project 0.8622, single 0.6650; within_5pct=no; process n=0); restored by cp from the saved copy, rebuilt, same filter -> passes in all three (0.9995, 0.9974, 0.9986; within_5pct=yes; process n=126/2/17). dotnet format whitespace --verify-no-changes clean with a repo-relative --include (the absolute-path spelling was shown vacuous by a positive control: a mis-indented copy exits 0 with the absolute path and 2 with the relative one). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…ms (refs #2180) P41 Phase 3 Task 1. CompileAndExecuteCore and EmitAndRunProjectAssembly (shared by CompileAndExecuteProject and CompileAndExecuteEntryFile) each carried their own copy of the deploy-and-spawn block; both now call one private static helper, RunEmittedProgram, so a per-call cost cured in one copy (#2180) or a behaviour fixed in one cannot stay in the other. Diff of the two blocks before the fold (every line not listed was byte-identical): - assembly / runtimeconfig file stem: "SharpyTestAssembly" vs "SharpyTestProject" -> preserved by the assemblyName parameter, passed as compilation.AssemblyName (the stems were already the compilations' own names, so the values are unchanged). - GeneratedCSharp on the three result shapes: generatedCSharp vs generatedCSharpReport -> preserved by the generatedCSharp parameter. - RawDiagnostics on success: `rawDiagnostics` vs `rawDiagnostics ?? new()` -> deliberate unification on `?? new()`; the single arm's list is never null, so no value changes. - spawn comment wording (the single arm's cited ".NET 10 on Linux x64"); kept that one for now, Task 3 replaces it. Identical in both, carried over unchanged: no environment overrides, WorkingDirectory = temp dir, `dotnet exec "<dll>"`, 30 s default timeout + Kill(entireProcessTree) + TimedOut result, non-zero exit -> "Process exited with code N: <stderr>", async stdout/stderr capture with the second WaitForExit, Sharpy.Core + CopyRuntimeClosure(Core + additional paths) + additional reference copies, the runtimeconfig literal (checked byte-equal), best-effort recursive delete in finally, and the timing hooks. Not in either block (left in the arms): the C# emit, the #line-mapped SPY0908 path (WithGeneratedCodeDiagnostics/RenderCompilationErrors), and the exception ladders - the helper catches nothing, so the single arm keeps its TargetInvocation/InvalidOperation/FileNotFound/TypeLoad/Exception mapping and the project arms their generic catch. Guard: Infrastructure/IntegrationTestBaseExecutePathTests counts, on the syntax tree of IntegrationTestBase.cs, exactly one `new ProcessStartInfo` and exactly one CopyRuntimeClosure(...) call; a third fact shows the counter sees 2 + 2 in an inline source that also mentions both in a comment and a string (positive control; a missing file throws). Mutation: the pre-fold IntegrationTestBase.cs (both blocks inlined) cp'd over the fold -> red (2 failed: Expected 1, Actual 2 for both counts; control 1 passed); restored by cp -> green (3 passed). Count identity @ b9d542c + this change (worktree build, measured; timing switch ON), against the P0 control trx by (testName with path args normalized, outcome) multiset per class: - FileBasedIntegrationTests: 3569/0/0, IDENTICAL. - EntryFilePathMultiFileTests, ClrInteropIntegrationTests, ProjectCompilationTests and every class calling CompileAndExecuteProject/EntryFile (ModuleMemberQualificationMatrixTests, CompileSideCorpusBoundaryTests, HarnessGeneratedCodeDiagnosticTests, BacktickEscapedForwarderTests, CrossModuleParityTests via DualModeIntegrationTestBase, ImportSourceRootParityTests, ExceptTypeResolutionMatrixTests, ModuleMemberImportAccessMatrixTests, OverloadApplicabilityMatrixTests, OverloadModifierResolutionMatrixTests) + Float32ArithmeticMatrixTests: 14 classes, 837/0/0, IDENTICAL. - Sharpy.Stdlib.Tests FileBasedIntegrationTests: 241/0/0, IDENTICAL. Native asset: stdlib_sqlite3_basic (Stdlib FileBased; e_sqlite3 via the copied closure) passes. GetAdditionalReferenceAssemblyPaths(): every Stdlib FileBased fixture (StdlibPaths) and Float32ArithmeticMatrixTests (47 passed) go through it. Timing coverage through the helper stays >= 0.95 in every arm of every run (FileBased single 0.9995 / project 0.9994; rest single 0.9982 / project 0.9996 / entry-file 0.9995; Stdlib single 0.9990 / project 0.9973). dotnet format whitespace --verify-no-changes (repo-relative --include) clean on both files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…refs #2179) xUnit v2 reads [CollectionDefinition] only from the test assembly itself. HeavyCompilation's definition has lived in Sharpy.TestInfrastructure since #679, so its DisableParallelization never took effect; only the cap of 1 kept those classes serial. Every collection below is therefore defined in its members' own project. HeavyCompilation's definition, its other members, and Sequential are unchanged here (Phase 4). | Global | Classes | Collection (defined in) | |---|---|---| | current directory | ModuleResolverTests (Directory.SetCurrentDirectory) | ProcessCwd, runs alone (Compiler.Tests) | | current directory + env | OsModuleTestsModuleTests (os.chdir, os.putenv; partial twin) | ProcessCwd, runs alone (Stdlib.Tests) | | Console.Out | ReplSessionTests (ReplSession swaps Console.Out on a Task.Run thread, outside ConsoleLock) | ConsoleCapture, runs alone (Compiler.Tests, new) | | Console.Error + logging root level | LoggingModuleTestsModuleTests, LoggingCompleteTestsModuleTests (CapturedStderr; twins) | ConsoleCapture, runs alone (Stdlib.Tests, existing) | | Console.Out | Print_Tests | ConsoleCapture, runs alone (Core.Tests, new) | | random's shared System.Random | Random{,Additional,Additional2}TestsModuleTests, CpythonBisectTestsModuleTests (twins) | PrngState, mutual only (Stdlib.Tests) | | memory (CsCheck Sample uses one thread per core) | 44 compile-heavy CsCheck classes, from HeavyCompilation | PropertySerial, runs alone (Compiler.Tests) | | TZ env + TZ cache | StrftimeHostTimeZoneTests | HostTimeZone (unchanged) | | console under a lock | ProjectCompilationHelper (TestHelpers.ConsoleLock), CliTestHarness (own lock) | none needed | | .claude/tmp reports | 9 Compiler + 4 Lsp writers, one distinct file each | none needed | | fixture bin/ under src | FileBased project arm (Sharpy.Test.*), MultiFileDeterminismTests (Test.*), one writer per file | none needed | | fixed temp names, ports | GUID or ephemeral everywhere (ClrInteropIntegrationTests' fixed /tmp names are distinct per test) | none needed | | culture | AnalysisLatencyLogTests sets CurrentCulture (per-thread) | none needed | Spy-generated classes get their [Collection] from hand-written partial twins in Spy/SpyTestCollections.cs. The file is outside generated/ because the regenerator removes orphan generated/*.cs files. check_spy_tests_staleness.sh: 119 files up to date. Print_Tests is not a measured collision. Nothing else in Core.Tests writes to the console today. It joins a run-alone collection because the alternative is a guard exemption resting on an invariant that nothing checks. Counts @ 116c6e2 + this change (measured; control at 116c6e2 in parentheses): Core.Tests 3989/0/0 (3989/0/0), Stdlib.Tests 4162/0/0 (4162/0/0). Compiler.Tests filtered (ModuleResolverTests | ReplSessionTests | ArithmeticPropertyTests | the guard of the next commit): 73/0/0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…2179) ProcessGlobalStateConformanceTests (Compiler.Tests/Conformance, CI shard 6). It does a Roslyn syntax walk over the five test projects and Sharpy.TestInfrastructure. Spy/generated is included; bin/obj and the uncompiled Integration/TestFixtures are excluded. A mutation site is a match against a literal roster: - invoked: SetCurrentDirectory, Chdir, SetOut, SetError, SetIn, CapturedOutput, CapturedStderr, SetEnvironmentVariable, Putenv, ClearCachedData - constructed: CapturedOutput, CapturedStderr, ReplSession, CompileServer - assigned: Environment.CurrentDirectory, CultureInfo.DefaultThreadCurrent{,UI}Culture Strings and comments never match. Each site is attributed to its outermost type. That type's [Collection] is merged across its partial declarations in the same project (the spy twins). The site passes only when the collection's [CollectionDefinition] is in the SAME project with DisableParallelization = true. That is the source-level form of the inert-HeavyCompilation finding. Sites in TestInfrastructure can only be exempt. There are 5 literal exemptions with reasons (CliTestHarness and ProjectCompilationHelper redirect under their locks; FrontEndParityTests only probes ReplSession), and a stale exemption fails. Positive control: 10 literal known sites must be found and isolated. Synthetic controls run the same classifier on in-memory sources: - one row per roster entry (17); - strings, comments, reads and CurrentCulture produce no site; - a class with no collection is red; - a definition in another project is red; - DisableParallelization=false is red; - a twin in the same namespace isolates, one in the wrong namespace does not; - a nested type is attributed to the outermost; - a site in TestInfrastructure is always red. The guard reads source at test time, so each mutation ran on the built binary without a rebuild. Each was restored by cp from a backup and checked with cmp: (1) [Collection("ProcessCwd")] removed from ModuleResolverTests broken -> red (2 failed, 25 passed), restored -> green (27 passed) (2) [Collection("ProcessCwd")] removed from the os partial twin in Spy/SpyTestCollections.cs broken -> red (2 failed, 25 passed), restored -> green (27 passed) (3) Compiler's ProcessCwd [CollectionDefinition] moved into Sharpy.TestInfrastructure/TestCollections.cs broken -> red (2 failed, 25 passed: "no [CollectionDefinition("ProcessCwd")] in src/Sharpy.Compiler.Tests"), restored -> green (27 passed) Guard: 27/0/0 (17 roster rows + 10 facts). Compiler.Tests FullyQualifiedName~Conformance&Category!=Benchmark&Category!=GapDiscovery: 17363/0/0 (measured, with the guard in the tree). dotnet format whitespace sharpy.sln --verify-no-changes: clean. Positive control: three trailing spaces in ProcessGlobalCollections.cs gave exit 2 naming the file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…, Analysis, Helpers, Parser, Stress and Stdlib (refs #2179) 16 classes: Project 8, Fuzz 3, Analysis/Helpers/Parser/Stress 1 each, and Stdlib.Tests' FileBasedIntegrationTests (the only Stdlib member). HeavyCompilation's DisableParallelization never took effect: its [CollectionDefinition] has lived in Sharpy.TestInfrastructure since #679, and xUnit v2 reads definitions only from the test assembly. The member classes therefore formed one ordinary named collection that the cap of 1 kept serial. Dropping the attribute gives each class its own default collection. Audit of every member before removal (Phase 2 terms; grep/read): - no mutable static field; - no static collection mutated at run time; the only shared statics are thread-safe Lazy<T>; - no ClearAll/ClearCache/Reset of a shared cache; every temp directory is GUID-named; - no timing assertion (Stopwatch values are only logged); - no process-global mutation: ProcessGlobalStateConformanceTests is green at 27/0/0. Compiler static state is guarded separately by StaticStateConformanceTests (#1036). Verification: the group's namespace filter, with Category!=Benchmark as in the P0 control, run at `-- xUnit.MaxParallelThreads=4` and `=1`. Each run is compared with the P0 control trx (.claude/tmp/p41-control/*.trx @ 116c6e2) by the multiset of test name x outcome. Walls are trx start-to-finish. Every call still forces a blocking GC until P3.3 lands, so the N=4 speedup is understated, and these classes are only one slice of their namespace. Compiler.Tests (Analysis|Fuzz|Helpers|Parser|Project|Stress namespaces), measured at this tree: N=4: 3428/0/0, wall 49 s; N=1: 3428/0/0, wall 80 s. Both IDENTICAL to P0 by name x outcome. Stdlib.Tests (whole project; its one member is the fixture Theory): N=4: 4162/0/0, wall 86 s; N=1: 4162/0/0, wall 57 s. Both IDENTICAL to P0. N=4 is slower here because the fixture Theory is still one class (xUnit never parallelises inside a class) and the extra threads only add contention; the fixture partition is P4.2. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…onformance, and dissolve Sequential (refs #2179) 38 HeavyCompilation classes (CodeGen 20, Conformance 18). Sequential is dissolved: 15 classes (14 CodeGen and Semantic/Validation/NamingConventionValidatorTests) plus its definition file. HeavyCompilation's DisableParallelization never took effect: its [CollectionDefinition] has lived in Sharpy.TestInfrastructure since #679, and xUnit v2 reads definitions only from the test assembly. The member classes therefore formed one ordinary named collection that the cap of 1 kept serial. Dropping the attribute gives each class its own default collection. Audit of every member before removal (Phase 2 terms; grep/read): - no mutable static field; - no static collection mutated at run time; the only shared statics are thread-safe Lazy<T>; - no ClearAll/ClearCache/Reset of a shared cache; every temp directory is GUID-named; - no timing assertion (Stopwatch values are only logged); - no process-global mutation: ProcessGlobalStateConformanceTests is green at 27/0/0. Compiler static state is guarded separately by StaticStateConformanceTests (#1036). Verification: the group's namespace filter, with Category!=Benchmark as in the P0 control, run at `-- xUnit.MaxParallelThreads=4` and `=1`. Each run is compared with the P0 control trx (.claude/tmp/p41-control/*.trx @ 116c6e2) by the multiset of test name x outcome. Walls are trx start-to-finish. Every call still forces a blocking GC until P3.3 lands, so the N=4 speedup is understated, and these classes are only one slice of their namespace. Sequential (DisableParallelization = true, defined in this assembly, so it did run last and alone) is obsolete. It came from 0e6429c (2025-11-10), whose reason was that the tests shared NameMangler state that added a numeric suffix. That state is gone: - StaticStateConformanceTests (#1036) bans mutable statics in the compiler assembly by reflection, and its allowlist holds only init-once tables, [ThreadStatic] fields and per-instance vending caches; - NameMangler has only readonly lookup tables; - grep of the 15 member files finds no mutable static and no process-global use. Compiler.Tests (CodeGen|Conformance namespaces), measured at this tree: N=4: 18129/0/0, wall 742 s; N=1: 18129/0/0, wall 820 s. Against P0 (18102 passed) the only difference at both N is +27 passed: ProcessGlobalStateConformanceTests, which landed after the control (0aecaa1). No other test is added, removed or changed in outcome. NamingConventionValidatorTests lives in the Semantic namespace, so it was verified by the Semantic filter at N=4 and N=1 (IDENTICAL to P0). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…#2179) 69 classes under Compiler.Tests/Semantic. HeavyCompilation's DisableParallelization never took effect: its [CollectionDefinition] has lived in Sharpy.TestInfrastructure since #679, and xUnit v2 reads definitions only from the test assembly. The member classes therefore formed one ordinary named collection that the cap of 1 kept serial. Dropping the attribute gives each class its own default collection. Audit of every member before removal (Phase 2 terms; grep/read): - no mutable static field; - no static collection mutated at run time; the only shared statics are thread-safe Lazy<T>; - no ClearAll/ClearCache/Reset of a shared cache; every temp directory is GUID-named; - no timing assertion (Stopwatch values are only logged); - no process-global mutation: ProcessGlobalStateConformanceTests is green at 27/0/0. Compiler static state is guarded separately by StaticStateConformanceTests (#1036). Verification: the group's namespace filter, with Category!=Benchmark as in the P0 control, run at `-- xUnit.MaxParallelThreads=4` and `=1`. Each run is compared with the P0 control trx (.claude/tmp/p41-control/*.trx @ 116c6e2) by the multiset of test name x outcome. Walls are trx start-to-finish. Every call still forces a blocking GC until P3.3 lands, so the N=4 speedup is understated, and these classes are only one slice of their namespace. Compiler.Tests (Semantic namespace), measured at this tree: N=4: 11196 passed / 1 not executed, wall 646 s; N=1: same counts, wall 801 s. Both IDENTICAL to P0 by name x outcome. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…refs #2179) 49 classes under Compiler.Tests/Integration, including FileBasedIntegrationTests (the fixture Theory, still one class until P4.2) and EntryFilePathMultiFileTests. HeavyCompilation's DisableParallelization never took effect: its [CollectionDefinition] has lived in Sharpy.TestInfrastructure since #679, and xUnit v2 reads definitions only from the test assembly. The member classes therefore formed one ordinary named collection that the cap of 1 kept serial. Dropping the attribute gives each class its own default collection. Audit of every member before removal (Phase 2 terms; grep/read): - no mutable static field; - no static collection mutated at run time; the only shared statics are thread-safe Lazy<T>; - no ClearAll/ClearCache/Reset of a shared cache; every temp directory is GUID-named; - no timing assertion (Stopwatch values are only logged); - no process-global mutation: ProcessGlobalStateConformanceTests is green at 27/0/0. Compiler static state is guarded separately by StaticStateConformanceTests (#1036). Verification: the group's namespace filter, with Category!=Benchmark as in the P0 control, run at `-- xUnit.MaxParallelThreads=4` and `=1`. Each run is compared with the P0 control trx (.claude/tmp/p41-control/*.trx @ 116c6e2) by the multiset of test name x outcome. Walls are trx start-to-finish. Every call still forces a blocking GC until P3.3 lands, so the N=4 speedup is understated, and these classes are only one slice of their namespace. Fixture bin/ dirs: the project arm writes <fixture>/bin/Debug/net10.0/Sharpy.Test.* (one writer) and MultiFileDeterminismTests writes Test.* (one writer, already uncollected). The entry-file arm emits nothing there (CompileFile writes only with an OutputAssemblyPath), so no two classes write the same file. Compiler.Tests (Integration namespace), measured at this tree: N=4: 8738/0/0, wall 907 s; N=1: 8738/0/0, wall 985 s. Both IDENTICAL to P0 by name x outcome. The fixture Theory dominates and runs serially inside its one class, so most of this namespace cannot use a second thread until P4.2. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…t definition (refs #2179) MetamorphicCorpusSweepTests was the last [Collection("HeavyCompilation")] class. With it gone, HeavyCompilationCollection is removed from Sharpy.TestInfrastructure/TestCollections.cs. The file stays as a comment saying no xUnit collection may be defined there: xUnit v2 reads [CollectionDefinition] only from the test assembly, so a definition in the shared library is inert, which is what HeavyCompilation's was from #679 to now. ProcessGlobalStateConformanceTests' doc comment is updated to the past tense. The other Properties classes were already in PropertySerial (bb02f61) or uncollected. HeavyCompilation's DisableParallelization never took effect: its [CollectionDefinition] has lived in Sharpy.TestInfrastructure since #679, and xUnit v2 reads definitions only from the test assembly. The member classes therefore formed one ordinary named collection that the cap of 1 kept serial. Dropping the attribute gives each class its own default collection. Audit of every member before removal (Phase 2 terms; grep/read): - no mutable static field; - no static collection mutated at run time; the only shared statics are thread-safe Lazy<T>; - no ClearAll/ClearCache/Reset of a shared cache; every temp directory is GUID-named; - no timing assertion (Stopwatch values are only logged); - no process-global mutation: ProcessGlobalStateConformanceTests is green at 27/0/0. Compiler static state is guarded separately by StaticStateConformanceTests (#1036). Verification: the group's namespace filter, with Category!=Benchmark as in the P0 control, run at `-- xUnit.MaxParallelThreads=4` and `=1`. Each run is compared with the P0 control trx (.claude/tmp/p41-control/*.trx @ 116c6e2) by the multiset of test name x outcome. Walls are trx start-to-finish. Every call still forces a blocking GC until P3.3 lands, so the N=4 speedup is understated, and these classes are only one slice of their namespace. Compiler.Tests (Properties namespace), measured at this tree: N=4: 3065 passed / 1 failed / 2 not executed, wall 860 s. The one red is ILCompilesPropertyTests.GeneratedCSharp_CompilesToValidIL, attributed as "ILCompiles fresh seed 8sLgyTw-IOM3 -> open #2139, reproduced at N=1 by lead" (CsCheck_Seed replay on the same build, .claude/tmp/lead-ilseed-n1.log): member access on the None literal emits `null.Res` (CS0023). It is not a concurrency failure and not this change. Against P0 that test's outcome is the ONLY difference (Passed -> Failed); every other name x outcome is equal to P0. N=1: 3066 passed / 2 not executed, wall 833 s. IDENTICAL to P0. Most of this namespace is PropertySerial, which runs alone at any N, so the walls barely move. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

P41 Phase 3 Task 3 (+ Task 4, the spawn comment). Design Decision 5's rule: the forced GC was >= 10% of the per-test median (62.4 of 120.8 ms, 52%, FileBasedIntegrationTests single arm @ b057610, measured), so the blocking GC.Collect(2, Forced) + WaitForPendingFinalizers is removed from all three arms (CompileAndExecute, CompileAndExecuteProject, CompileAndExecuteEntryFile). CompileAndExecuteWithGC keeps its own GC for the callers that ask for it by name. The timing instrument loses its forced-gc component accordingly. Stage-once (Task 2) is not done: copies + delete were 2.5% (< 10%). The spawn comment's uncited ".NET 10 on Linux x64 reflection/interpreted mode issues" is replaced by the measured split @ b9d542c and the note that in-process execution (no spawn) is deferred to an owner ruling on #2180. Before/after, FileBasedIntegrationTests, same instrument (SHARPY_TEST_TIMING via test-memory-harness --timing), threads = runner json (1), worktree build @ b057610 (before) and b057610 + this change (after), run back to back 2026-10-05 ~08:15-08:33 (measured; the machine was NOT quiet - load average 3.7-8.3 from other sessions - so absolute values and p90s are inflated, the A/B pair is the comparison): single arm, median ms/call before after pipeline 6.82 9.19 emit 1.42 1.49 write+copy 2.01 1.84 process (start->exit) 59.81 57.36 cleanup 1.33 1.33 forced-gc 73.00 - total 139.77 68.94 (p90 386.98 -> 146.76) wall_s (lock acquired -> exit) 703 259 counts 3569/0/0 3569/0/0 (name x outcome multiset IDENTICAL to P0) coverage (sum components / sum total) 0.9995 / 0.9994 Process share of the per-call median after the GC removal: 57.36 / 68.94 = 83% (> 50%, so the in-process question is the owner's, per DD5). Quieter single runs for reference: before @ b057610 02:48 - total median 120.78, process 47.61, forced-gc 62.44, wall 389 s; after, 14-class project/entry-file set at 04:55 - single total median 54.24 with process 44.12 (81%), entry-file 65.97/42.80, project 59.36/43.47. A third FileBased "after" run at 04:53-05:00 was inflated in EVERY component (pipeline 18.2, process 140.4, wall 444 s) - external load, not this change - and is not used. Lead's measurement for context (whole-suite matrix @ 116c6e2, GC still forced): the (b) subset of 48 un-collected Integration classes took 669/458/509/527 s at N=1/2/4/8, and LetEmissionInvarianceTests 1350 s at N=2 vs 576 s at N=1 - scaling stalls past 2 because the blocking per-call Gen2 GC stops every thread. This change is the precondition for raising N; RSS without the GC at each thread count is re-measured by the lead. Counts after the change (worktree build, timing ON, name x outcome vs the P0 trx): - Compiler FileBasedIntegrationTests 3569/0/0 IDENTICAL (two runs). - 14 project/entry-file classes + Float32ArithmeticMatrixTests 837/0/0 IDENTICAL, plus the 3 IntegrationTestBaseExecutePathTests guards passing (840 total). - Stdlib FileBasedIntegrationTests 241/0/0 IDENTICAL; single median total 91.40 -> 84.52 (the Stdlib testhost's GC was only ~10 ms/call: GC cost tracks the live heap). dotnet format whitespace --verify-no-changes (repo-relative --include) clean. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

) P41 Phase 4 Task 2 (Design Decision 4), with the totality guard. xUnit v2 never runs two rows of one class concurrently, so the fixture Theory - one class - was a serial long pole whatever the thread cap. Shape: each FileBasedIntegrationTests.cs (Compiler, Stdlib) is now a NAMESPACE `...Integration.FileBasedIntegrationTests` holding an abstract base (CompilerFixtureShard / StdlibFixtureShard: fixture root, Stdlib's additional references, ShardCount) and classes Shard0..Shard{K-1}, each with its own [Theory] over its shard (MemberData resolves on the declaring type, so the Theory cannot live in the base). Every row's FullyQualifiedName still starts `Sharpy.<Proj>.Tests.Integration.FileBasedIntegrationTests.`, so CI shard 1 / shard 7 (trailing-dot filters, dotnet10.yml), `~FileBasedIntegrationTests` and `DisplayName~` select every row unchanged; dotnet10.yml only gets its shard-1 comment updated (it said "one class, cannot be split further"). FileBasedIntegrationTestsBase.DiscoverTestFixtures(path, shard, shardCount): rows whose ShardOf(relative path, '/'-normalised) == shard. ShardOf is 32-bit FNV-1a over the UTF-16 code units (never string.GetHashCode, randomised per process). The full discovery per root is cached in a ConcurrentDictionary<string, Lazy<...>>, so K classes do one disk walk per root instead of 2K (plan warning). The unsharded DiscoverTestFixtures(path) is unchanged and uncached (the totality test's independent side). K: Compiler K=8 - as one class the corpus took 657 s at N=1 @ 116c6e2 vs 576 s for the next long pole (LetEmissionInvarianceTests), so 8 shards keep an N=8 cell from being bounded by one shard. Stdlib K=4 - 241 rows, ~24 s at N=1. Guard: Infrastructure/FixtureShardTotalityTests (Compiler) and Conformance/StdlibFixtureShardTotalityTests (Stdlib), via TestInfrastructure FixtureShardScan (reflection finds every concrete shard class and reads its own GetTestFixtures): class count == a LITERAL (8 / 4, not ShardCount); union of shard rows == unsharded discovery as a testName multiset (non-empty); pairwise disjoint; no shard empty; plus (Compiler) ShardOf pinned to FNV-1a by the published "foobar" vector 0xbf9cf968 (% 1000 = 720) and two fixture paths. Mutations (main-tree build each time, measured): - Compiler Shard3 modulus 8 -> 9 (DiscoverTestFixtures(root, 3, 9)) + Stdlib Shard1 index 1 -> 0: Compiler red 2 failed (disjoint, union) / 6 passed; Stdlib red 2 failed (disjoint, union) / 2 passed. Restored by cp, rebuilt -> Compiler 8/8, Stdlib 4/4 green. - ShardOf swapped to string.GetHashCode(Ordinal) + Shard7 class deleted: red 6 failed (4x ShardOf_IsFnv1a, union, ShardClassCount_IsTheLiteralK). Restored by cp, rebuilt -> 8/8. Behaviour checks @ 3c91f6a + this change (main-tree build, measured): - --list-tests with FullyQualifiedName~FileBasedIntegrationTests: Compiler 3576, Stdlib 244. The RunTestFixture rows are the pre-change 3568 / 240; the difference is the base class's [Fact] TestFixturesDirectory_Exists, which now runs once per shard class (8 / 4 instead of 1). No fixture row is added or lost. - CI compiler-shard-partition logic run locally (same list/sort/cmp over the workflow env): main filter 48044 = sum of shards 48044, PARTITION OK, no shard empty (shard 1: 3576). - CiFilterCoverageConformanceTests green (3/3). - UPDATE_SNAPSHOTS=true over both FileBased sets: 3576 + 244 passed, all 165 tracked *.expected.cs rewritten (mtime), zero git diff; `git status` identical before/after. - Counts vs the P0 trx, RunTestFixture rows by (testName, outcome) multiset with ShardN folded: Compiler 3568 IDENTICAL at N=4 and N=1; Stdlib 240 IDENTICAL at N=4 and N=1; all passed. Wall (harness, lock -> exit): Compiler N=1 193 s, N=4 101 s (max_concurrent 4 in the trx, 8 classes); Stdlib N=1 22 s, N=4 11 s. - dotnet format whitespace --verify-no-changes clean on every changed .cs (repo-relative --include, one file each). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

A collection defined with DisableParallelization = true runs last and alone, so every such definition, and every class in one, costs the whole suite wall clock. #2179 is one 173-class collection that made the suite serial. ProcessGlobalStateConformanceTests gains a literal SerialCollectionRoster: 7 collections, each with its project, why it must run alone (from the P2.1 audit), and its member classes. - Compiler.Tests: ProcessCwd, ConsoleCapture, PropertySerial (44 members). - Stdlib.Tests: ProcessCwd, ConsoleCapture (3), HostTimeZone. - Core.Tests: ConsoleCapture. New facts, all on the same Roslyn scan as the process-global guard: - EveryRunAloneCollection_IsRostered_AndEveryRosterRowIsDefined: the set of run-alone definitions found must equal the roster rows. An unrostered definition fails, and so does a stale row (drain on fix). - EveryMemberOfARunAloneCollection_IsRostered: a class may name a run-alone collection only as a rostered member; a rostered member that no longer names it fails. - PositiveControl_TheRosterIsFoundInFull: roster size and found size are both pinned to literals counted by hand at landing (7 collections, 52 members), so a scan that finds nothing cannot pass. - EveryCollectionAClassNames_IsDefinedInItsOwnProject: any [Collection("X")] with no [CollectionDefinition("X")] in the class's own project fails, because xUnit v2 ignores the definition elsewhere and the attribute is inert. That was HeavyCompilation's state from #679. Mutations, each restored by cp from a backup and checked with cmp. (a1), (a2) and (c) change scanned source, so no rebuild was needed; (b) changes the roster in the test binary, so it was rebuilt: (a1) [Collection("ProcessCwd")] added to the unrostered PathNormalizerTests broken -> red (2 failed, 29 passed: "+ PathNormalizerTests in ProcessCwd"), restored -> green (31 passed) (a2) a new [CollectionDefinition("Extra", DisableParallelization = true)] in Compiler.Tests broken -> red (2 failed, 29 passed: "+ Sharpy.Compiler.Tests: Extra"), restored -> green (31 passed) (b) the Core.Tests ConsoleCapture roster row deleted broken -> red (3 failed, 28 passed: unrostered definition + unrostered member + positive control), restored -> green (31 passed) (c) [Collection("HeavyCompilation")] added to PathNormalizerTests (no definition anywhere) broken -> red (1 failed, 30 passed: "names HeavyCompilation"), restored -> green (31 passed) Guard class: 31/0/0 (27 process-global + 4 roster). Compiler.Tests FullyQualifiedName~Conformance&Category!=Benchmark&Category!=GapDiscovery: 17367/0/0 (17363 at 0aecaa1 + these 4). dotnet format whitespace sharpy.sln --verify-no-changes: clean. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

P41 sibling cell (lead-added to the Defect Class table): the LetEmissionInvarianceTests corpus Theory is a fixture-corpus Theory over the same discovery, so xUnit v2 ran it as one serial class - 576 s at N=1 @ 116c6e2, 1350 s at N=2 under contention in the P1.3 matrix, and the suite's floor once the fixture corpus was sharded (ebe4523, 101 s at N=4). Same cure. Shape: LetEmissionInvarianceTests is now a NAMESPACE holding - Controls : FileBasedIntegrationTestsBase - the four non-Theory tests (positive control, site finder, comparison exemption, census) and the inherited TestFixturesDirectory_Exists, ONCE; - abstract LetTwinShard (deliberately not a FileBasedIntegrationTestsBase, whose [Fact] every shard would replicate) with the cell body, and Shard0..Shard7, each carrying only [Theory] LetTwin_EmitsByteIdenticalCSharp over CellsOfShard(n); - internal static Instrument - the unchanged corpus/site/twin/compile/allowlist helpers. Rows are selected by FileBasedIntegrationTestsBase.ShardOf over the fixture's '/'-relative path (the same function and key as the fixture corpus). K=8: matches the fixture corpus so neither bounds an N=8 run. FileBasedIntegrationTestsBase.NormalizeCSharp is widened protected -> public so the static Instrument can reach it without deriving from that base. One behaviour change, attributed: each cell used to find its fixture with ComparableFixtures().Single(...), a full walk of the fixture tree PER CELL (~2400 walks). The shards' discovery and FixtureByName now share one cached walk (Lazy). The unsharded ComparableFixtureNames() and the census still walk fresh (the totality test's independent side). Ratchet across shards (measured, temporary allowlist lines, restored by cp, zero git diff): type_shorthand/list_shorthand (identical twin), basics/hello_world (no site) and nope/does_not_exist (stale) -> exactly 3 failures, each once: Shard2 list_shorthand ("twin now emits identical C# ... delete its line"), Shard2 hello_world ("no let site: delete its entry"), Controls.Census ("every allowlist entry must name a comparable fixture"). The per-row checks run in the one shard owning the row; the stale-entry check stays aggregate in Controls. Totality guard: Infrastructure/LetEmissionShardTotalityTests (via FixtureShardScan) - shard class count == LITERAL 8 (not Instrument.ShardCount), union == unsharded ComparableFixtureNames as a multiset (non-empty), pairwise disjoint, none empty. Mutation: Shard3 -> CellsOfShard(2): red 2 failed (disjoint, union) / 2 passed; restored by cp, rebuilt -> 4/4 green. Filters / counts (main-tree build @ ebe4523 + b335dd6 + this change, measured): - --list-tests FullyQualifiedName~LetEmissionInvarianceTests: 2402 before = 2402 after; the listed-name multiset with Shard<n>/Controls folded is IDENTICAL (2397 cells + 5 facts; no replicated Fact). - CI compiler-shard-partition logic run locally: main 48048 = union 48048, PARTITION OK; shard 6 (Conformance) holds all 2402 LetEmission rows. - CiFilterCoverageConformanceTests, FixtureShardTotalityTests, LetEmissionShardTotalityTests, ProcessGlobalStateConformanceTests: 46/46 green. - docs/design/gap-discovery-contracts.md:51 names the class descriptively (not a filter): unchanged. - Counts vs the P0 trx by (testName with Shard<n>/Controls folded, outcome) multiset: 2402 IDENTICAL at N=4 and at N=1, all passed. - Walls (harness, lock -> exit): before (unsharded class, lane-a2 worktree @ b9d542c + the P3.3 file, N=1) 466 s; after N=1 57 s, N=4 43 s. At N=1 sharding adds no concurrency, so the 466 -> 57 s is the cached fixture lookup; N=4 adds the sharding's share (57 -> 43 s). - dotnet format whitespace --verify-no-changes (repo-relative --include) clean on the three changed .cs files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

maxParallelThreads 1 -> 2 in src/Sharpy.Compiler.Tests/xunit.runner.json. The value is the best wall clock of a whole-project re-measurement after P41 Phases 2-4 (no per-call forced GC, HeavyCompilation dissolved, fixture and LetEmission corpora sharded, LetEmission discovery cached), with peak tree RSS inside the recorded memory budget. Measured @ af6f309 (worktree, full sln build, `Category!=Benchmark`, thread count set by the RunSettings override `-- xUnit.MaxParallelThreads=N`, heap limit 5 GB; machine 36 GB / 14 cores with a local model loaded; .claude/tmp/p41-p51/): N | trx span | peak tree / testhost RSS | counts 1 | 2042 s | 8200 / 6743 MB | 48241/1/3 2 | 1641 s | 7197 / 6090 MB | 48241/1/3 4 | 1981 s | 7126 / 6083 MB | 48241/1/3 8 | 4707 s | 7447 / 6349 MB | 48240/2/3 The Phase 0 control (N=1, 116c6e2) was 5325 s / 7106 MB: -62% at N=1, -69% at N=2. N=4 and N=8 lose to N=2: classes with internal parallelism oversubscribe the cores (ReparseEquivalenceConformanceTests busy 65 s at N=1 -> 654 s at N=4; at N=8 the run-alone PropertySerial phase took 2063 s for BooleanPropertyTests' 2 tests). The red at every N is DispatchSiteInventoryTests' guard-citation check, a regression from ebe4523 fixed separately (refs #2179); the second N=8 red is a concurrency-only compiler nondeterminism (Fuzz Compilation_IsDeterministic seed 2025), recorded for filing, and it does not reproduce at N=1/2/4. Heap-limit axis at N=2 (System.GC.HeapHardLimit edited in the built runtimeconfig): 5 GB 1641 s, 7197 / 6090 MB; 4 GB 1789 s (+9%), 5959 / 5077 MB; 3 GB 1877 s (+14%), 5466 / 4160 MB; zero OutOfMemoryException at every value. The plan's rule (lowest limit with no OOM and <=~5% wall loss) keeps 5 GB, so runtimeconfig.template.json is unchanged. Stdlib.Tests keeps 1 here; its cap is raised in its own commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…2179) Regression from ebe4523, caught by the P5.1 whole-suite run at every thread count (N=1 included): DispatchSiteInventoryTests.GuardCitations_ResolveToRealTests_AndGuardedByTestsScanTheirSite reported every refusal-net:FileBasedIntegrationTests row - 42 rows, distinct site keys counted in .claude/tmp/p41-p51/w-n1/memory-harness-tests.log and again here - as "no such test class in any test assembly": CollectGuardCitationViolations resolved a citation by Type.Name only, and FileBasedIntegrationTests is now the namespace of its Shard0..7 classes. My pre-commit filters for ebe4523 (the shard/totality/CI-coverage classes) did not include this roster. Fixed at the resolver, not at the 42 citations, so a sharded group stays citable: - a citation X resolves if a type is named X (unchanged; the LSP source-scan branch unchanged), OR if a namespace whose last segment is X has the SHARD shape: >=1 concrete test class (a non-abstract class with a [Fact]/[Theory] method, inherited included) and EVERY concrete test class there derives from one abstract base declared in that same namespace. Deliberately strict (recorded in a remarks comment): Conformance.LetEmissionInvarianceTests fails it - its Controls class does not derive from LetTwinShard - and nothing cites that name today; relax to "at most one non-shard controls class" only when a row needs it. - guarded-by rows: none cites a group today. A group has no one source that scans the site, so a guarded-by citation of a group is refused with its own message ("a sharded test group, which backs only a refusal-net row ..."); FindTestSourceFile is unchanged. Synthetic control (next to the existing synthetic roster test): GuardCitations_SyntheticRoster_ResolvesAShardedTestGroup_ButNotABroadOrHelperNamespace - refusal-net:FileBasedIntegrationTests resolves; refused as unknown: Transforms (Properties.Metamorphic.Transforms, helper types only), Integration (many unrelated test classes, no shared base) and LetEmissionInvarianceTests (shards + Controls, the strict choice); guarded-by:FileBasedIntegrationTests refused with the group message; exactly 4 violations. Preconditions asserted: no type is named FileBasedIntegrationTests / Transforms / Integration / LetEmissionInvarianceTests; the group namespace holds test classes, Transforms holds types but no test class, Integration holds more than one test class. Mutation (main tree, one script: edited build green -> mutate -> red -> cp restore -> green): resolver reverted to the name-only rule -> red, 2 of 3 GuardCitations failed: the real roster with 42 distinct 'FileBasedIntegrationTests' violations, and the new synthetic control; the pre-existing synthetic test stays green. Restored by cp, rebuilt -> 3/3 green. Pre-commit filter (Compiler.Tests, N=4): FullyQualifiedName~DispatchSiteInventory | ~Census | ~Roster | ~Conformance.CiFilterCoverage | ~Totality | ~Sharpy.Compiler.Tests.Infrastructure. -> 821 passed / 3 failed / 0 skipped @ a24387b + this change + lane B's UNCOMMITTED ProcessGlobalStateConformanceTests.cs / SpyTestCollections.cs edits. The 3 reds are all ProcessGlobalStateConformanceTests ("+ Sharpy.Stdlib.Tests: WallClockWindow ... not rostered"), i.e. lane B's in-progress WallClockWindow collection, not this change (which touches neither file nor any collection). by-name reflection over test classes elsewhere: CiFilterCoverageConformanceTests.ReflectTests uses FullyQualifiedName (unaffected); FixtureShardScan selects by base type; the other GetTypes() calls enumerate compiler/AST types. No other roster cites FileBasedIntegrationTests or LetEmissionInvarianceTests by class name. dotnet format whitespace --verify-no-changes (repo-relative --include) clean. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…hange (refs #2179) P41 Phase 5 Task 3 (docs + wrapper), from the P5.3 inventory. Figures (measured, #2179): Compiler.Tests whole project at the committed maxParallelThreads 2 = 1641 s trx span @ af6f309 (2042 s at 1; 5325 s at 1 before P41 @ 116c6e2); Lsp 107 s, Cli 31 s, Stdlib 66 s, Core 5 s (P0 control @ 116c6e2) -> whole-solution gate ~30 min. GapDiscovery share: the 8 classes carrying [Trait("Category","GapDiscovery")] are 509 s of 2850 s summed test time (17.9%) in the N=2 trx @ af6f309 (class-level sum from that trx; one class carries the trait per method, so this is an upper bound); the slowest classes are MetamorphicCorpusSweep 171 s, ReparseEquivalence 114 s, CPythonDifferentialParse 114 s, BlockScopeRedeclaration 113 s - the sweeps are no longer the long pole. Peak tree RSS at N=2 is 7.2 GB, so the "5-10 GB per run" band stays (CLAUDE.md:24, implement-plan, bench_ab.py:480 untouched). - CLAUDE.md:26 - "~22 min" -> "~30 min (Compiler.Tests ~27 at maxParallelThreads 2 @ af6f309)"; the GapDiscovery advice now gives their share and says filtering saves minutes, not the long pole. Minimal in-place edit (rules-only file, never regenerated). - docs/design/verification-contract.md §6 - "~22 min" -> "~30 min (maxParallelThreads 2, @ af6f309)". - .claude/skills/implement-plan/SKILL.md :49, :126; .claude/skills/compiler-audit/SKILL.md :17; .claude/agents/implementer.md :48; .claude/agents/verification-expert.md :33 - "~22 min" -> "~30 min". - .claude/scripts/dotnet-serialized - lock waiter TIMEOUT 2700 -> 4500 s (75 min), rationale in the comment with its sources: a queued run must outlast one whole-solution gate (~1850 s) plus one more Compiler.Tests run; the lead proposed 3600 s, but that leaves ~4% over gate + an N=2 Compiler.Tests run (3491 s) and is SHORT of gate + an N=1 run (~3900 s), which the measurement matrices queue; 4500 s keeps ~15% over the latter. The old 2700 s was already short of the pre-P41 suite (two queued runs exited 124 on 2026-10-05). "~17 min" for Compiler.Tests in the #1384 comment -> "~27 min @ af6f309"; "45 minutes" in the env-knob comment made number-free. STALL_SECONDS unchanged (not re-measured). - build_tools/bench_ab.py:661 - "the 45-minute timeout" -> "the lock's waiter timeout" (comment). Not changed: the CI shard-balance comment (waits for the P5.3 CI run); historical/dated records (plans, docs/design/test-harness/*, the 2026-08-12 incident docstring in test_dotnet_serialized_lock.py). Checks: bash -n on the wrapper clean; pytest build_tools/tests/test_dotnet_serialized_lock.py 35 passed (env-knob timeouts, unaffected by the default). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

test_timer_fires_after_interval (Spy/threading/threading_module_tests.spy:323-330) starts a 50 ms threading.Timer, then gives it a fixed 200 ms window to set a flag. Sharpy's Timer callback is a System.Threading.Timer callback (Sharpy.Stdlib Threading/Timer.cs:31), so it needs a free thread-pool thread. In the Stdlib matrix at xUnit MaxParallelThreads=8 (@ b335dd6) the pool was saturated and the test went red (Assert.True failure on line 330). It is flaky, not deterministic: - an N=8 repeat was green; - run alone at N=1 it was green 4/4, 3 of them with every core busy. The generated test is not edited and its tolerance is not widened. ThreadingModuleTestsModuleTests instead joins a new run-alone collection, WallClockWindow (DisableParallelization = true, defined in Stdlib.Tests via its partial twin in Spy/SpyTestCollections.cs). The new collection is rostered in ProcessGlobalStateConformanceTests with the reason, and the hand-counted positive-control constants go 7 -> 8 collections and 52 -> 53 members. Findings ledger: - (1) Test design: the .spy test should wait on an Event that the callback sets, with a generous timeout. The class then leaves the roster. - (2) Product: Sharpy's Timer runs on a thread-pool timer, while CPython's runs on a dedicated thread. That is an unlisted surface, so under docs/design/python-fidelity-scope.md R-CM it is a docs/deviations.yaml row by default, not an issue. Verified in worktree p41-lane-b-stdlib at 1cf19a1 plus these two files (byte-identical, cmp): - build 0 warnings / 0 errors; ProcessGlobalStateConformanceTests 31/0/0. - Stdlib.Tests whole at N=4: 4169/0/0, 22 s. Against the P0 trx, the only differences are the 7 explained rows (4 StdlibFixtureShardTotalityTests + 3 replicated TestFixturesDirectory_Exists from the K=4 shard classes, ebe4523). - dotnet format whitespace sharpy.sln --verify-no-changes: clean. Mutations, each restored by cp and checked with cmp: (1) the twin's [Collection("WallClockWindow")] removed (no rebuild; the guard scans source) broken -> red (2 failed, 29 passed: "- ThreadingModuleTestsModuleTests in WallClockWindow" + positive control), restored -> green (31 passed) (2) the WallClockWindow roster row deleted (Compiler.Tests rebuilt) broken -> red (3 failed, 28 passed: unrostered definition + unrostered member + positive control), restored and rebuilt -> green (31 passed) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

src/Sharpy.Stdlib.Tests/xunit.runner.json: maxParallelThreads 1 -> 4. Stdlib.Tests is raised in its own commit, after the Compiler cap (a24387b), because it carries the higher concurrency risk (the spy-generated module tests call stdlib globals in-process). Measured with .claude/scripts/test-memory-harness --threads N --trx on src/Sharpy.Stdlib.Tests, filter Category!=Benchmark, in worktree p41-lane-b-stdlib @ b335dd6 (the matrix) and @ 1cf19a1 + 8c44659's two files (the confirmation runs). Machine with the local model loaded: base used memory 18.8-24.6 GB, swap flat in every cell. | N | wall (harness / trx) | peak tree RSS | peak testhost RSS | counts | |---|---|---|---|---| | 1 | 142 s / 140 s | 843 MB | 529 MB | 4169/0/0 | | 2 | 33 s / 33 s | 924 MB | 578 MB | 4169/0/0 | | 4 | 26 s / 25 s | 922 MB | 580 MB | 4169/0/0 | | 8 | 23 s / 22 s | 912 MB | 590 MB | 4168/1/0 (the timer test, below) | Repeats at N=4 (@ b335dd6): 4169/0/0 three times, 23 s / 23 s / 25 s, peak tree 838/887/932 MB. N=8 repeat: 4169/0/0, 41 s. Perturbed order: xUnit v2 has no random-order switch, so this run used MaxParallelThreads=3 + ParallelAlgorithm=aggressive (different scheduling and interleaving): 4169/0/0, 28 s. With WallClockWindow (8c44659) at N=4: 4169/0/0, 22 s. With this json and no RunSettings override: 4169/0/0, 21 s. The bin copy of the json reads 4, so the cap is honoured from the file. Against the P0 control trx (.claude/tmp/p41-control/Stdlib.trx @ 116c6e2, 4162 passed), every green cell is identical by test name x outcome except for 7 explained rows. Names are normalised for the P4.2 Shard<k> classes and the checkout path embedded in display names. - 4 StdlibFixtureShardTotalityTests (ebe4523). - 3 extra TestFixturesDirectory_Exists rows: each of the K=4 shard classes inherits the Fact. The 4162 fixture rows are unchanged. The N=8 red was ThreadingModuleTestsModuleTests.TestTimerFiresAfterInterval. Its 50 ms Timer has a fixed 200 ms window, and Sharpy's Timer callback runs on the thread pool, so it starves when the pool is saturated. Triage: - the N=8 repeat was green; - run alone it was green 4/4, 3 of them with every core busy. It is a wall-clock-window hazard of the test, not a stdlib-global collision. The class now runs alone in WallClockWindow (8c44659); the test-design fix and the Timer deviation are ledger rows. Chosen N=4: 5.6x faster than N=1. N=8 buys about 3 s more but is the cell that hit the timing hazard. Memory is not the constraint here: under 1 GB tree RSS at every N. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…(refs #2179) The twin comment in Spy/SpyTestCollections.cs and the roster reason in ProcessGlobalStateConformanceTests name #2251 (threading.Timer is a thread-pool timer, not a Thread). Once that lands, the spy test waits on an Event and ThreadingModuleTestsModuleTests leaves the collection. Text only: the roster guard is 31/0/0, the Stdlib ThreadingModuleTests filter is 43/0/0, the build has 0 warnings, and format is clean. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

Verifier finding 2b (in-contract): the three shard totality guards read each shard's GetTestFixtures() - what a shard OFFERS - not what xUnit RUNS. A `[Theory(Skip = "...")]` on one shard's method collapses its ~440 rows to one skipped row, and a `[Trait("Category", ...)]` on one shard lets a CI filter drop it (the main filter excludes Category=GapDiscovery/Benchmark); both left union/disjoint/non-empty green. FixtureShardScan.ShardMethodViolations(shardBase, testMethodName) checks every concrete shard class: it declares the test method with exactly one [Theory] whose Skip is empty, exactly one [MemberData] naming its own GetTestFixtures (MemberType null or the class itself), and no trait-bearing attribute (ITraitAttribute, spelled with its arguments) on the method or the class that the shard base and its ancestors do not carry. New fact EveryShard_RunsItsRowsAsOneUnskippedUncategorisedTheory in FixtureShardTotalityTests (Compiler, RunTestFixture), StdlibFixtureShardTotalityTests (Stdlib, RunTestFixture) and LetEmissionShardTotalityTests (LetTwin_EmitsByteIdenticalCSharp); each also asserts the number of classes checked equals its literal K, so an empty scan cannot pass. Mutations (main tree @ dada5fd, one script: green -> mutate -> build -> red -> cp restore -> build -> green, measured): 1. Compiler Shard3 `[Theory]` -> `[Theory(Skip = "x")]`: Compiler ~Totality 1 failed / 650 passed - only the new fact ("Shard3.RunTestFixture: Skip = "x" (its rows would not run)"); union, disjoint and non-empty stayed green, i.e. the gap was real. 2. Stdlib Shard1 + `[Trait("Category", "Benchmark")]`: Stdlib ~Totality 1 failed / 4 passed - only the new fact ("Shard1: trait(s) the shard base does not carry: TraitAttribute(Category, Benchmark)"). Restored both by cp, rebuilt: Compiler 651/651, Stdlib 5/5 green. Pre-commit filter ~Totality | ~DispatchSiteInventory | ~ProcessGlobalState | ~CiFilterCoverage: Compiler.Tests 702/0/0, Stdlib.Tests 5/0/0 @ dada5fd + this change (measured). dotnet format whitespace --verify-no-changes (repo-relative --include) clean on all four files. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

…2179) PR #2252's run 37378225480 @ 8682582 (maxParallelThreads 2) left shard 5 (Properties) the long pole. Measured per-shard trx spans, in seconds: 1 fixtures 124 · 2 Integration P–V + CodeGen 305 · 3 Semantic P–V 108 · 4 Semantic rest 133 5 Properties 492 · 6 Conformance 150 · 7 everything else 259 Shard 5's span was a 287 s parallel phase, led by CPythonDifferentialParse (178 s) and AmbiguityHotspot (169 s), both in Properties.Parser, followed by a 204 s run-alone PropertySerial phase. Its job took 554 s; the next slowest job took 373 s. New buckets, still 7 jobs: - 3: all of Semantic (old 3 + 4); - 4: Properties.Parser; - 5: Properties minus Properties.Parser. Shards 1, 2, 6 and 7 are unchanged; 7 already negates `Semantic.` and `Properties.`. Expected spans (derived from the same trx, not measured): about 240 s for Semantic, about 210 s for Properties.Parser, and about 300 s for the rest of Properties. That makes shard 2 (305 s) the long pole, roughly 3 minutes off the CI wall. Verified locally on the main-tree build, replaying the compiler-shard-partition job: - the main filter lists 48051 tests, and the union of the 7 shards is identical as a multiset; - shard sizes are 3576 / 3428 / 11074 / 60 / 3001 / 16628 / 10284; - dropping shard 5's `!~Properties.Parser.` makes the replay report 60 double-run rows, so the instrument is not vacuous; - CiFilterCoverageConformanceTests 3/0/0, and the YAML parses. The partition-invariant comments now name the new negation (`!~Properties.` covers 4+5). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HzQeHzwNfujh3DnznBXsQr

antonsynd and others added 23 commits October 5, 2026 02:15

bench: weekly cross-language benchmark results 2026-10-05

56da0e0

Merge origin/dev (weekly benchmark results) into dev

8682582

antonsynd mentioned this pull request Oct 5, 2026

test: CompileAndExecute pays a per-test process + file-copy + forced-Gen2-GC tax (~150 ms × ~11.5k) — stage the runtime once or run in-process, then re-measure the GC #2180

Closed

antonsynd mentioned this pull request Oct 5, 2026

test: restore measured parallelism — maxParallelThreads=1 and the blanket HeavyCompilation collection date from a 9.9k-test suite; 11.5k independent CompileAndExecute tests run serially (28 min) #2179

Closed

antonsynd merged commit 5b98c8e into mainline Oct 5, 2026
25 checks passed

antonsynd deleted the dev branch October 5, 2026 22:28

Navigation Menu

Repository navigation

P41: measured test parallelism — dissolve HeavyCompilation, drop per-call GC, shard corpus Theories, caps 2/4 (#2179, #2180) - #2252

P41: measured test parallelism — dissolve HeavyCompilation, drop per-call GC, shard corpus Theories, caps 2/4 (#2179, #2180)#2252
antonsynd merged 24 commits into
mainlinefrom
dev

antonsynd commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	passed / failed / skipped	sha	wall
Before	57873 / 0 / 4	`116c6e2`	Compiler.Tests alone 5325 s
After	57937 / 0 / 4	`a1504a4`	1925 s for the whole solution

Conversation

antonsynd commented Oct 5, 2026

Summary

Measured

For CI review

Filed from the round

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant