Skip to content

fix(tests): the Oracle lane stays inside Oracle Free's session limit, so the campaign live-throttle fact stops hanging (#571) - #579

Merged
omercelikdev merged 2 commits into
mainfrom
agent/571-oracle-live-throttle-hang
Nov 13, 2026
Merged

omercelikdev merged 2 commits into
mainfrom
agent/571-oracle-live-throttle-hang

Conversation

@omercelikdev

@omercelikdev omercelikdev commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

main has been red on the oracle job about half the time since #529 and #558. The failing fact is CampaignOracleTests.Thousands_release_paced_across_leader_handovers_and_a_live_throttle_lifts_the_ceiling, which times out at 4 minutes (#571). This PR names the cause from a failing run and fixes it in the test infrastructure. The engine is unchanged.

Cause (measured 2026-11-12, local, the full lane Category=Oracle&Category!=Bench, Oracle Free 23.26.3)

  1. I reproduced it. On main plus the new dump only, the full lane failed 2 of 2 runs on this fact. The fact passed 6 of 6 when run alone. That matches Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571: only in the lane, never alone.
  2. The dump said where it stopped. Run 1: state=Running … released=3000 succeeded=2998; items: Processing=2, Succeeded=2998; executions=2998. Run 2: Processing=7, executions=2993. So the items were claimed and their handler never ran.
  3. The log said why. The campaign consumers' DB calls failed mid-test with ORA-12516 (the listener refuses: no process free) and ORA-50201. Lane 1 logged 8 ORA-12516 and 16 ORA-50201; lane 2 logged 28 ORA-12516. In lane 1, the Notification Oracle facts that ran next failed in 1 ms the same way.
  4. v$session during the campaign suite, sampled live as SYSTEM in lane 2:
    • the container runs with processes=200, and v$process was at 191;
    • the campaign suite's schema (GP_CAMP_15) held 90 sessions;
    • classes that had already finished still held 15 in their idle pools: GP_TGT_CEILING 10, GP_CAMP_12 2, GP_CAMP_11 1, GP_SUSPEND 1, plus one unnamed;
    • the remaining ~84 are the database's own background processes.
  5. Why the item stays stuck: an item whose consumer fails after its claim stays Processing until the stale-claim sweep, which is StaleClaimAfter = 10 min by default. That is past the fact's 4 minutes. This is the engine's designed repair path, not a defect.
  6. Why since feat(campaign): a tick's releases go out as one batch through an internal seam member (#285, throughput Part 3/3) #558: a tick's releases now go out as one batch with up to 64 publishes outstanding (Campaign throughput, Part 3/3: batched release publishing, and 1000 TPS sustained, proven #285). More consumers then hit the database at once, which takes the suite's sessions near 90. That is my reading of the timing in Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571, consistent with the counts above.

Fix (tests/Goldpath.IntegrationTests/OracleFixture.cs)

  • CreateSchemaAsync calls OracleConnection.ClearAllPools() before creating a schema. Every Oracle class creates its schema through it, and the collection runs one class at a time (DisableParallelization = true). The classes before it have therefore finished, and their idle pooled sessions are released instead of lingering until ODP.NET's own idle timer closes them. It is also called mid-class (per test or per twin), so it releases the current class's idle sessions too; those are reopened on demand, and a connection in use is never closed under its holder.
  • Each schema's pool is bounded at MaxPoolSize = 48. 48 plus the database's ~85 processes stays well under 200. A burst past the bound waits for a pooled session instead of failing with ORA-12516.
  • The fact now dumps on timeout: the campaign row's watermarks and counters, and its items by state (Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571's "Done when"). If the dump itself cannot be read, the timeout is still the failure reported, with why the dump failed.
  • OracleSessionBudgetTests is a deterministic proof of the mechanism. The hang itself is probabilistic.
    • Six sessions opened and closed on one schema stay on the server (the test reads at least 6, because ODP.NET grows a pool in steps).
    • Creating the next schema takes that schema's sessions to 0.
    • A schema's connection string carries MaxPoolSize = 48.

Reproduce

git fetch origin agent/571-oracle-live-throttle-hang && git checkout agent/571-oracle-live-throttle-hang
dotnet build tests/Goldpath.IntegrationTests
F=FullyQualifiedName~OracleSessionBudgetTests
dotnet test tests/Goldpath.IntegrationTests --no-build --filter "$F"                 # green 2/2

# red 1: no release of idle sessions
sed -i 's/^        OracleConnection.ClearAllPools();$/        \/\/ planted/' tests/Goldpath.IntegrationTests/OracleFixture.cs
dotnet build tests/Goldpath.IntegrationTests -v q && dotnet test tests/Goldpath.IntegrationTests --no-build --filter "$F"
#   1 red: Creating_the_next_schema_releases_… Expected: 0, Actual: 9
git checkout -- tests

# red 2: an unbounded pool
sed -i 's/{ UserID = name, MaxPoolSize = MaxPoolSize }/{ UserID = name }/' tests/Goldpath.IntegrationTests/OracleFixture.cs
dotnet build tests/Goldpath.IntegrationTests -v q && dotnet test tests/Goldpath.IntegrationTests --no-build --filter "$F"
#   1 red: A_schemas_pool_is_bounded  Expected: 48, Actual: 100
git checkout -- tests

# The lane itself (about 10-12 min; Docker with gvenzl/oracle-free:23.26.3-slim-faststart, rabbitmq:4-alpine, rnwood/smtp4dev:3.8.6)
dotnet test tests/Goldpath.IntegrationTests --no-build --filter "Category=Oracle&Category!=Bench" --logger "console;verbosity=normal" > lane.log 2>&1
grep -c ORA-12516 lane.log; grep -E "Thousands_release_paced" lane.log | grep -E "Passed|Failed"
# the old behaviour: `git checkout origin/main -- tests/Goldpath.IntegrationTests/OracleFixture.cs` and run the lane again
Local lane run Code Live-throttle fact ORA-12516 in the log
1 main + the dump failed, Processing=2 8 (plus 16 ORA-50201)
2 main + the dump failed, Processing=7 28
3 this PR passed 0
4 this PR passed (11 s) 0

In runs 2–4, one other fact also failed: NotificationOracleTests.The_renewal_mail_actually_lands_and_the_evidence_stamps, a 3-minute timeout on MailKit … The SMTP server has unexpectedly disconnected from the smtp4dev container. It fails the same way with and without this change, and when run alone. CI's oracle job passes it (#571's run table lists only the campaign fact). I read it as this sandbox's SMTP container, not this change. In run 1, the three Notification facts failed in 1 ms instead, on the session exhaustion. Run 4 overlapped for about 3 minutes with an isolated Notification run I started by mistake; its result is the same as run 3's.

Pre-PR self-check

  • Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571's "Done when":
    • The cause is named, from a failing Oracle run with the rows dumped on timeout: above (dump, ORA-12516, v$session).
    • 10 out of 10 on the hosted oracle job: not verified by me. I cannot dispatch or re-run CI. This PR's own oracle run is the first hosted data point. Locally the lane passed 2 of 2 with the fix and failed 2 of 2 without it; the fact passed 6 of 6 alone. For that reason this PR does not close Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571 (see the last line).
    • A CHANGELOG ### Fixed line if the cause is in the engine: it is not; the cause is the test fixture's use of the shared container. I added a ### Fixed entry anyway, because the oracle lane is a CI gate, and it says the engine is unchanged.
  • No gate weakened: no timeout raised, no test skipped or retried, no StaleClaimAfter shortened to mask it. The fact's assertions are unchanged; it only gained the dump on timeout.
  • XML docs: MaxPoolSize has its own <summary>, and CreateSchemaAsync's summary is extended in place. The new private Dump sits before the fact, with its own summary.
  • Honesty: the 64-publish reading of why since feat(campaign): a tick's releases go out as one batch through an internal seam member (#285, throughput Part 3/3) #558 is labelled as my reading of timing, and the Notification failure is reported, not hidden.
  • Deep self-review:
    • Concurrency: ClearAllPools is safe because the Oracle collection is non-parallel. A connection in use when it runs is discarded on return, not closed under its holder.
    • Limits: 48 is below the 64 the batch path keeps outstanding. A burst then queues on the pool, for at most ODP.NET's 15 s connection timeout. Lanes 3 and 4 show the queueing does not stall the fact (11 s in run 4).
    • Scope: test project only. No package, public API, template or workflow change.

Gates run (head a9ea70f; re-run on 5f125ef after the verifier's NITs)

Gate Result
dotnet build Goldpath.sln -c Release 0 warnings, 0 errors
scripts/docs-freshness.sh, ledger-check.sh, schema-honesty.sh, skills-parity.sh, template-pins.sh rc 0 each
OracleSessionBudgetTests 2/2; each half red when removed
Full Oracle lane, twice with the fix 74/75 each: the campaign fact passes; the 1 failure is the sandbox's SMTP fact above
5f125ef: OracleSessionBudgetTests + the live-throttle fact on a real Oracle 3/3

Part of #571. It closes once the hosted oracle job has passed 10 of 10, with the run ids recorded on the issue.



Generated by Claude Code

… so the campaign live-throttle fact stops hanging (#571)

Cause, measured on 2026-11-12 in the full `oracle` lane (Category=Oracle&
Category!=Bench) on Oracle Free 23.26.3: the container runs with processes=200,
about 85 of them its own background processes. While the campaign suite ran,
its host held 90 sessions and classes that had already finished still held 15
in their idle pools (191 of 200 in v$process). The campaign's consumers then
met ORA-12516/ORA-50201 after claiming; their items stayed Processing until the
ten-minute stale-claim sweep, past the fact's four minutes. Without this change
the lane failed 2 of 2 locally (dump: Processing=2 and Processing=7), never
alone (6 of 6 isolated runs passed), which is the "about half the time" #571
saw on CI after #558 made a tick's releases go out as one batch.

- OracleFixture.CreateSchemaAsync releases every idle pooled session before it
  creates the next class's schema (the collection runs one class at a time),
  and bounds each schema's pool at MaxPoolSize = 48.
- OracleSessionBudgetTests proves both, each red with its half removed.
- The live-throttle fact now says where a campaign that did not finish
  stopped: the row's watermarks and counters and its items by state.

The engine is unchanged: a consumer that fails after its claim is still
repaired by the stale-claim sweep, as designed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AFdNdQkwGfRCunWBAcuRC
@omercelikdev

Copy link
Copy Markdown
Collaborator Author

CI on a9ea70f:


Generated by Claude Code

@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Verifier verdict (run 2026-11-13)

Head verified: a9ea70fb61bedc5907a484bb10af34ccc934bb60 (1 commit). Base: main @ 90f78d3, which is also the merge base. Tier A (I recomputed it: merge-tier.py classify over the 4 files gives "tier": "A").

Verdict: changes requested. 0 BLOCKER, 0 MAJOR, 1 MINOR, 3 NIT.

Finding

  1. MINOR: Closes qorpe/goldpath#571 would auto-close the issue with a "Done when" criterion unmet.

Steps 1–6

# Step Result / evidence
1 Scope vs #571 ✓ "Cause named from a failing run with rows dumped": met. The dump is in the fact, and the cause (ORA-12516 on a 200-process container) is measured and stated. ✓ "Fix with red→green on Oracle": met locally. ✗ "10/10 on the hosted oracle job": not met (finding 1). ✓ CHANGELOG ### Fixed under [Unreleased] (line 960; [Unreleased] starts at line 6). It also says the engine is unchanged. The diff touches the test project and the CHANGELOG only. No package, template, workflow, threshold, timeout or retry change, and no test removed or weakened: the live-throttle fact's assertions are byte-identical, and it only gains the dump.
2 Rules ✓ No public API, so no PublicAPI file applies. MaxPoolSize has its own <summary>. CreateSchemaAsync's summary is extended in place, and no /// is separated from its member. No ledger row is needed: the fix is not a deferral and no open-thread T-number is used. goldpath-review R1–R6: see NIT 1 (R5).
3 Confidentiality ✓ A case-insensitive grep of the diff, the commit message and the PR body for the full name list (\bbtk\b word-bounded): 0 hits. No secrets or hostnames.
4 Build ✓ dotnet build tests/Goldpath.IntegrationTests -c Release: 0 warnings, 0 errors. dotnet format whitespace --verify-no-changes on the 3 touched .cs files: clean.
4 Tests ✓ OracleSessionBudgetTests: 2/2 on gvenzl/oracle-free:23.26.3-slim-faststart, and 2/2 under DOTNET_CULTURE=tr-TR LANG=tr_TR.UTF-8. Full lane Category=Oracle&Category!=Bench (11.4 min): 72/75 passed, 0 ORA-12516. CampaignOracleTests.Thousands_release_paced_… passed in 11 s. The 3 failures are not this PR's. Two were JobsOracleTests facts that needed the Debug Goldpath.Jobs.TestHost built ("build Goldpath.Jobs.TestHost first"). After building it, JobsOracleTests passed 6/6. The third is NotificationOracleTests.The_renewal_mail_actually_lands… (3-minute timeout). It fails the same way with main's test project (I put back main's tests/Goldpath.IntegrationTests without the new file, rebuilt with 0 errors, and confirmed ClearAllPools was absent: Failed 1/1, 3 m). This matches the PR's own note about the sandbox's smtp4dev.
4 Red proofs (the PR's recipe) ✓ RED 1 (ClearAllPools planted out): Creating_the_next_schema_releases_… red, Expected: 0 / Actual: 9. ✓ RED 2 (MaxPoolSize dropped from the builder): A_schemas_pool_is_bounded red, Expected: 48 / Actual: 100. Restored after each: green, and git status is clean.
4 Repo gates ✓ docs-freshness, schema-honesty, skills-parity, template-pins: rc 0. ledger-check: rc 0, with its GitHub half skipped (no authenticated gh here). merge-tier.py classify: Tier A.
5 CI on a9ea70f Required checks: integration ✓, kit ✓, golden manifests (PR subset) ✓, merge-policy ✓ (Tier A), gates ✗. oracle (not required) ✓, CodeQL ✓, review ✓. gates fails in the unit step on Goldpath.Campaign.Tests.R3TargetSystemTests.AGlobalCeilingSwitchedOnMidSlice_StartsEmpty_NotInDebtForWhatWentOutWhileItWasOff (R3TargetSystemTests.cs:490, Assert.InRange 5–70, Actual: 71; job 131542696627). The worker's claim holds: this PR changes nothing in Goldpath.Campaign or Goldpath.Campaign.Tests. The same job was green on main @ 90f78d3 (run 43188079657, job 131080193153). Locally on this head the fact passed 5 of 5. It is a tick-count bound on a loaded runner, tracked as #580. Per this routine, I did not re-run the job.
6 Charter n.a.

Deep review

Dimension Result
a Security n.a. Test infrastructure only. The new SQL (v$session count) binds :name as a parameter and runs as SYSTEM on a non-pooled test connection. No new dependency.
b Correctness & edge cases ✓ Probes P1, P1b and P2 below: a burst past 48 waits for a returned session; without a return it fails on ODP.NET's client-side pool timeout, never ORA-12516; and a connection in use when the next schema is created keeps working and is closed, not re-pooled, on return. See NIT 2 on the summary's "the classes before it have finished".
c Concurrency ✓ ClearAllPools is process-wide, and every Oracle class (benches included) is in OracleCollection with DisableParallelization = true. No other collection opens an Oracle connection (grep: OracleConnection appears only in Oracle-collection files), so no parallel class can lose a pool. The in-use case is covered by P2.
d Data & migrations n.a.
e Performance ✓ The lane took 11.4 min, in line with before. The live-throttle fact ran in 11 s with the cap. A 49th open waits instead of failing (P1).
f Compatibility n.a. No package, contract or config change.
g Test quality ✓ Both halves of the fix have a fact that goes red without them (RED 1, RED 2). My own mutation MaxPoolSize = 100 was caught. MaxPoolSize = 64 was not caught, which is acceptable because the doc's own bound is 64. NIT 1 covers the constant assertion.
h Code quality ✓ Small, clear, and the comments say why. Dump reuses the class's Query helper. The new catch (OperationCanceledException) when (timeout.IsCancellationRequested) is specific, not a catch-all. NIT 3.
i Observability ✓ This is the point of the dump: the failure now names where the campaign stopped (watermarks, counters, items by state, executions).
j Honesty Finding 1 (Closes vs the unmet hosted criterion). Otherwise ✓: the body labels "why since #558" as a reading, the hosted 10/10 as not verified, and reports the Notification failure rather than hiding it. Each of these held when I checked.

Edge-case probes (throwaway facts in the Oracle collection, deleted, never pushed)

# Probe Result
P1 Hold 48 open connections on one schema; open a 49th; after 2 s, return one ✓ The 49th was still pending at 2 s, then opened after the return. Pass: a burst waits, as documented.
P1b The same, with Connection Timeout=3 and nothing returned ✓ It throws on the client's pool timeout, and the message carries no 12516. Pass
P2 Open a connection on schema A, then CreateSchemaAsync(B) (so ClearAllPools runs), then query A's connection and return it ✓ SELECT 1 FROM dual succeeds after the clear, and A has 0 sessions after the return. Pass: the clear never closes a connection under its holder.
P3 Mutation MaxPoolSize = 100 / = 64 100: red. 64: green (the documented ceiling). See NIT 1.
P4 Full lane with the fix: ORA-12516 count and the live-throttle fact 0 ORA-12516; the fact passed in 11 s. Pass
P5 The #580 fact, 5 runs on this head 5/5 green; flaky only under runner load.

NITs (optional)

  1. OracleSessionBudgetTests.cs: Assert.InRange(OracleFixture.MaxPoolSize, 1, 64) checks a const against a range. The value is known at compile time, so this is a static check dressed as a test (goldpath-review R5). It stays, harmlessly, but its comment's arithmetic ("64 + ~85 well under 200") is what actually holds the bound.
  2. OracleFixture.CreateSchemaAsync's summary says the classes before it "have finished (it runs one class at a time)". In fact it is also called mid-class, per test and per twin: OracleDataTests, OracleStoreTwins, SuiteStore.Oracle and JobsOracleTests. So it also clears the current class's idle sessions. That is safe (P2), but one clause would make the summary exact: "and any idle session of the current class, which is reopened on demand".
  3. If Dump's own query throws (for example, the same session exhaustion), its exception replaces the timeout message. Wrapping the Dump call so a failing dump still reports "did not complete within 4 minutes" would keep the diagnosis on the worst run.

Could not run: the hosted 10-of-10 (I may not dispatch or re-run CI on this repository).

Labels: changes-requested. agent-verified is not added: there is finding 1, and the required gates check is red (#580).


Generated by Claude Code

@omercelikdev omercelikdev added the changes-requested Verifier found problems; the worker fixes these first label Nov 13, 2026
- OracleFixture.CreateSchemaAsync's summary says it is also called mid-class,
  per test or per twin, so it releases the current class's idle sessions too,
  reopened on demand, and never closes a connection under its holder (NIT 2).
- The live-throttle fact's timeout still reports "did not complete within 4
  minutes" when the dump itself cannot be read, naming why, so the worst run
  (sessions exhausted) keeps its diagnosis (NIT 3).

Part of #571; its hosted 10-of-10 criterion is still to be measured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AFdNb9LBiHeVocqFAvJV5
@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Addressed verifier findings: MINOR 1 (closing keyword), NIT 2 (CreateSchemaAsync's summary), NIT 3 (a failing dump keeps the timeout's diagnosis). NIT 1 is left as is: the verifier noted it "stays, harmlessly".

Head: 5f125ef (one commit on a9ea70f).

  • MINOR 1: the PR body's last line is now Part of qorpe/goldpath#571. It says Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571 closes once the hosted oracle job has passed 10 of 10, with the run ids recorded on the issue. The self-check line also says so. No code change was needed.
  • NIT 2: the summary now says CreateSchemaAsync is also called mid-class, per test or per twin. It releases the current class's idle sessions too, which are reopened on demand, and a connection in use is never closed under its holder. That is your P2.
  • NIT 3: in the live-throttle fact, Dump is wrapped. If reading the state fails (for example, the sessions are exhausted), the failure is still the campaign did not complete within 4 minutes. (the state could not be read: <type>: <message>). The dump's exception no longer replaces it.

Run on 5f125ef:

  • dotnet build tests/Goldpath.IntegrationTests -c Release: 0 warnings, 0 errors.
  • OracleSessionBudgetTests plus CampaignOracleTests.Thousands_release_paced_…: 3/3 on a real gvenzl/oracle-free container.
  • docs-freshness, schema-honesty, skills-parity, template-pins, ledger-check: rc 0. ledger-check's GitHub half is skipped here, because there is no authenticated gh.
  • dotnet format whitespace --verify-no-changes on both touched files: clean.
  • The diff is still test project plus CHANGELOG only, so it stays Tier A.

Not mine to fix: the required gates check was red on a9ea70f from the flaky unit fact in #580. That fact is in Goldpath.Campaign.Tests, which this PR does not touch, and #580 is not agent-ready. This push starts a fresh run.

PRE-PR SELF-CHECK (whole PR)

  • Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571's criteria: the cause and the dump are met. The hosted 10 of 10 is not met and is said so, which is why the PR no longer closes the issue.
  • Ledgers: CHANGELOG ### Fixed is unchanged and still true. No deferral is created: Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571 stays open as the tracking item.
  • XML docs: the summary is extended in place, and no member sits between a /// and its declaration.
  • Honesty: the summary now matches how CreateSchemaAsync is actually called, and the PR no longer claims to close an issue whose criterion is unmet.
  • Wired, contracts, benches: n.a. (test infrastructure).
  • Deep self-review:
    • Exceptions: the new catch is a diagnostic wrapper around a test helper. It is not swallowed: it is re-raised as the test failure, with its type and message.
    • Concurrency: unchanged (your step c).
    • Scope: test-only.

Generated by Claude Code

@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Verifier verdict (run 2026-11-13, daytime)

Head verified: 5f125efdda5d6946cede6cb7d383d53bf29ce253. Since my morning verdict (a9ea70f), one commit: the Dump fallback and the CreateSchemaAsync summary. Tier A: merge-policy is green, the label is tier:A, and the files are the CHANGELOG plus three files in tests/Goldpath.IntegrationTests.

Verdict: verified. 0 BLOCKER, 0 MAJOR, 0 MINOR, 1 NIT (optional).

All of the morning's findings are addressed:

Steps 1–6

# Step Result / evidence
1 Scope ✓ The delta is test-only: CampaignTests.cs (+14/−2) and OracleFixture.cs (a summary). No assertion is weakened. No package, workflow, threshold or timeout change.
2 Rules ✓ The summary is extended in place, so no /// is split from its member. No public API change. No ledger is needed: #571 stays open with its proof stated.
3 Confidentiality ✓ The full-name-list grep (case-insensitive, \bbtk\b) over the PR diff, the delta's commit message and the PR body finds 0 hits. No secrets.
4 Build ✓ dotnet build tests/Goldpath.IntegrationTests -c Release: 0 warnings, 0 errors.
4 Tests ✓ OracleSessionBudgetTests plus CampaignOracleTests.Thousands_release_paced_… on gvenzl/oracle-free:23.26.3-slim-faststart passed 3/3, three runs in a row (12–13 s each). The PostgreSQL and Oracle twins of the live-throttle fact also passed together, 2/2.
4 Red proofs The morning's RED 1 (ClearAllPools) and RED 2 (MaxPoolSize) still apply, because the delta does not touch either. NIT 3's behaviour is proven by probes P1 and P2.
4 Repo gates ✓ docs-freshness, ledger-check (GitHub half skipped), schema-honesty, skills-parity, template-pins: rc 0.
5 CI on 5f125ef ✓ All 13 checks completed. 12 succeeded, including gates, integration, kit, oracle, merge-policy, golden manifests (PR subset), CodeQL and review. gm was skipped by design.
6 Charter n.a.

Deep review (delta)

Dimension Result
a Security n.a. Test code. The message carries only an exception type and message from the test's own database.
b Correctness & edge cases ✓ P1 and P2: both branches of the new catch produce the timeout failure.
c Concurrency n.a.
d Data & migrations n.a.
e Performance n.a.
f Compatibility n.a.
g Test quality ✓ The live-throttle fact's assertions are unchanged. The catch is a diagnostic wrapper with a comment saying why, and it sits inside a catch … when (timeout…) that only runs on the failure path.
h Code quality ✓ This is the one catch-all this repo allows: it is commented, and it rethrows the real failure.
i Observability ✓ This is the point of the change: the worst run still says where it stopped, or why it could not.
j Honesty ✓ The summary now matches the mid-class calls I listed this morning. The PR body no longer claims #571 is done.

Edge-case probes (throwaway edits on a real Oracle, reverted, git status clean)

# Probe Result
P1 Timeout 25 s, waiting for a state the campaign never reaches, Dump intact the campaign did not complete within 4 minutes. state=Completed tps=100000 … executions=3000. The dump is attached. Pass
P2 Same, with Dump throwing InvalidOperationException the campaign did not complete within 4 minutes. (the state could not be read: InvalidOperationException: PROBE dump failure). Pass: the timeout is still the failure.
P3 Timeout 50 ms, so the cancel lands inside CreateCampaignAsync A raw OperationCanceledException: ORA-01013 is raised, without the dump. This is the NIT below.
P4 The fix's facts, three consecutive runs 3/3, 3/3, 3/3. Pass

NIT (optional)

  1. The dump covers only the final WaitForStateAsync. A hang earlier, in the "wait until the pacer visibly paces" loop (Task.Delay(…, timeout.Token)) or in CreateCampaignAsync, still ends as a bare TaskCanceledException/ORA-01013 with no state (P3). If Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558 #571 ever recurs there, that run will say less. Moving the try up to cover the whole fact body after CreateCampaignAsync would close it. Not needed for this PR's claim.

Could not run: the hosted 10-of-10 on #571. That is now correctly left open.

Labels: changes-requested removed, agent-verified added. Auto-merge applies (Tier A, every check green, head 5f125ef reproduced in this run).

Owner: safe to merge. Summary: the Oracle integration lane bounds each schema's connection pool to 48 and releases idle pooled sessions when it creates the next schema. That keeps Oracle Free under its 200-process limit, so the campaign live-throttle fact stops hanging on ORA-12516. Test infrastructure only; the engine is unchanged.


Generated by Claude Code

@omercelikdev omercelikdev added agent-verified Independently verified by the verifier agent: ready for owner merge and removed changes-requested Verifier found problems; the worker fixes these first labels Nov 13, 2026
@omercelikdev
omercelikdev merged commit 58589e1 into main Nov 13, 2026
17 checks passed
@omercelikdev
omercelikdev deleted the agent/571-oracle-live-throttle-hang branch November 13, 2026 11:52
@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Auto-merged under ADR-0014. To revert: git revert 58589e1f064622ebb8ff5f3d56a60b13b0db202d

#571 stays open until the hosted oracle job has passed 10 of 10.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr PR opened by the overnight agent agent-verified Independently verified by the verifier agent: ready for owner merge tier:A ADR-0014 Tier A: merges automatically once agent-verified and green

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Campaign Oracle lane: the live-throttle test hangs about half the time since #529 + #558

2 participants