Repository navigation
fix(tests): the Oracle lane stays inside Oracle Free's session limit, so the campaign live-throttle fact stops hanging (#571) - #579
Conversation
… so the campaign live-throttle fact stops hanging (#571) Cause, measured on 2026-11-12 in the full `oracle` lane (Category=Oracle& Category!=Bench) on Oracle Free 23.26.3: the container runs with processes=200, about 85 of them its own background processes. While the campaign suite ran, its host held 90 sessions and classes that had already finished still held 15 in their idle pools (191 of 200 in v$process). The campaign's consumers then met ORA-12516/ORA-50201 after claiming; their items stayed Processing until the ten-minute stale-claim sweep, past the fact's four minutes. Without this change the lane failed 2 of 2 locally (dump: Processing=2 and Processing=7), never alone (6 of 6 isolated runs passed), which is the "about half the time" #571 saw on CI after #558 made a tick's releases go out as one batch. - OracleFixture.CreateSchemaAsync releases every idle pooled session before it creates the next class's schema (the collection runs one class at a time), and bounds each schema's pool at MaxPoolSize = 48. - OracleSessionBudgetTests proves both, each red with its half removed. - The live-throttle fact now says where a campaign that did not finish stopped: the row's watermarks and counters and its items by state. The engine is unchanged: a consumer that fails after its claim is still repaired by the stale-claim sweep, as designed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AFdNdQkwGfRCunWBAcuRC
|
CI on
Generated by Claude Code |
Verifier verdict (run 2026-11-13)Head verified: Verdict: changes requested. 0 BLOCKER, 0 MAJOR, 1 MINOR, 3 NIT.
Finding
Steps 1–6
Deep review
Edge-case probes (throwaway facts in the Oracle collection, deleted, never pushed)
NITs (optional)
Could not run: the hosted 10-of-10 (I may not dispatch or re-run CI on this repository). Labels: Generated by Claude Code |
- OracleFixture.CreateSchemaAsync's summary says it is also called mid-class, per test or per twin, so it releases the current class's idle sessions too, reopened on demand, and never closes a connection under its holder (NIT 2). - The live-throttle fact's timeout still reports "did not complete within 4 minutes" when the dump itself cannot be read, naming why, so the worst run (sessions exhausted) keeps its diagnosis (NIT 3). Part of #571; its hosted 10-of-10 criterion is still to be measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AFdNb9LBiHeVocqFAvJV5
|
Addressed verifier findings: MINOR 1 (closing keyword), NIT 2 ( Head:
Run on
Not mine to fix: the required PRE-PR SELF-CHECK (whole PR)
Generated by Claude Code |
Verifier verdict (run 2026-11-13, daytime)Head verified: Verdict: verified. 0 BLOCKER, 0 MAJOR, 0 MINOR, 1 NIT (optional). All of the morning's findings are addressed:
Steps 1–6
Deep review (delta)
Edge-case probes (throwaway edits on a real Oracle, reverted,
|
| # | Probe | Result |
|---|---|---|
| P1 | Timeout 25 s, waiting for a state the campaign never reaches, Dump intact |
the campaign did not complete within 4 minutes. state=Completed tps=100000 … executions=3000. The dump is attached. Pass |
| P2 | Same, with Dump throwing InvalidOperationException |
the campaign did not complete within 4 minutes. (the state could not be read: InvalidOperationException: PROBE dump failure). Pass: the timeout is still the failure. |
| P3 | Timeout 50 ms, so the cancel lands inside CreateCampaignAsync |
A raw OperationCanceledException: ORA-01013 is raised, without the dump. This is the NIT below. |
| P4 | The fix's facts, three consecutive runs | 3/3, 3/3, 3/3. Pass |
NIT (optional)
- The dump covers only the final
WaitForStateAsync. A hang earlier, in the "wait until the pacer visibly paces" loop (Task.Delay(…, timeout.Token)) or inCreateCampaignAsync, still ends as a bareTaskCanceledException/ORA-01013 with no state (P3). If Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571 ever recurs there, that run will say less. Moving thetryup to cover the whole fact body afterCreateCampaignAsyncwould close it. Not needed for this PR's claim.
Could not run: the hosted 10-of-10 on #571. That is now correctly left open.
Labels: changes-requested removed, agent-verified added. Auto-merge applies (Tier A, every check green, head 5f125ef reproduced in this run).
Owner: safe to merge. Summary: the Oracle integration lane bounds each schema's connection pool to 48 and releases idle pooled sessions when it creates the next schema. That keeps Oracle Free under its 200-process limit, so the campaign live-throttle fact stops hanging on ORA-12516. Test infrastructure only; the engine is unchanged.
Generated by Claude Code
|
Auto-merged under ADR-0014. To revert: #571 stays open until the hosted Generated by Claude Code |
What and why
mainhas been red on theoraclejob about half the time since #529 and #558. The failing fact isCampaignOracleTests.Thousands_release_paced_across_leader_handovers_and_a_live_throttle_lifts_the_ceiling, which times out at 4 minutes (#571). This PR names the cause from a failing run and fixes it in the test infrastructure. The engine is unchanged.Cause (measured 2026-11-12, local, the full lane
Category=Oracle&Category!=Bench, Oracle Free 23.26.3)mainplus the new dump only, the full lane failed 2 of 2 runs on this fact. The fact passed 6 of 6 when run alone. That matches Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571: only in the lane, never alone.state=Running … released=3000 succeeded=2998; items: Processing=2, Succeeded=2998; executions=2998. Run 2:Processing=7,executions=2993. So the items were claimed and their handler never ran.v$sessionduring the campaign suite, sampled live as SYSTEM in lane 2:processes=200, andv$processwas at 191;GP_CAMP_15) held 90 sessions;GP_TGT_CEILING10,GP_CAMP_122,GP_CAMP_111,GP_SUSPEND1, plus one unnamed;Processinguntil the stale-claim sweep, which isStaleClaimAfter = 10 minby default. That is past the fact's 4 minutes. This is the engine's designed repair path, not a defect.Fix (
tests/Goldpath.IntegrationTests/OracleFixture.cs)CreateSchemaAsynccallsOracleConnection.ClearAllPools()before creating a schema. Every Oracle class creates its schema through it, and the collection runs one class at a time (DisableParallelization = true). The classes before it have therefore finished, and their idle pooled sessions are released instead of lingering until ODP.NET's own idle timer closes them. It is also called mid-class (per test or per twin), so it releases the current class's idle sessions too; those are reopened on demand, and a connection in use is never closed under its holder.MaxPoolSize = 48. 48 plus the database's ~85 processes stays well under 200. A burst past the bound waits for a pooled session instead of failing with ORA-12516.OracleSessionBudgetTestsis a deterministic proof of the mechanism. The hang itself is probabilistic.MaxPoolSize = 48.Reproduce
main+ the dumpProcessing=2main+ the dumpProcessing=7In runs 2–4, one other fact also failed:
NotificationOracleTests.The_renewal_mail_actually_lands_and_the_evidence_stamps, a 3-minute timeout onMailKit … The SMTP server has unexpectedly disconnectedfrom thesmtp4devcontainer. It fails the same way with and without this change, and when run alone. CI'soraclejob passes it (#571's run table lists only the campaign fact). I read it as this sandbox's SMTP container, not this change. In run 1, the three Notification facts failed in 1 ms instead, on the session exhaustion. Run 4 overlapped for about 3 minutes with an isolated Notification run I started by mistake; its result is the same as run 3's.Pre-PR self-check
v$session).oraclejob: not verified by me. I cannot dispatch or re-run CI. This PR's ownoraclerun is the first hosted data point. Locally the lane passed 2 of 2 with the fix and failed 2 of 2 without it; the fact passed 6 of 6 alone. For that reason this PR does not close Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571 (see the last line).### Fixedline if the cause is in the engine: it is not; the cause is the test fixture's use of the shared container. I added a### Fixedentry anyway, because theoraclelane is a CI gate, and it says the engine is unchanged.StaleClaimAftershortened to mask it. The fact's assertions are unchanged; it only gained the dump on timeout.MaxPoolSizehas its own<summary>, andCreateSchemaAsync's summary is extended in place. The new privateDumpsits before the fact, with its own summary.ClearAllPoolsis safe because the Oracle collection is non-parallel. A connection in use when it runs is discarded on return, not closed under its holder.Gates run (head
a9ea70f; re-run on5f125efafter the verifier's NITs)dotnet build Goldpath.sln -c Releasescripts/docs-freshness.sh,ledger-check.sh,schema-honesty.sh,skills-parity.sh,template-pins.shOracleSessionBudgetTests5f125ef:OracleSessionBudgetTests+ the live-throttle fact on a real OraclePart of #571. It closes once the hosted
oraclejob has passed 10 of 10, with the run ids recorded on the issue.Generated by Claude Code