Skip to content

feat(bench): the Campaign benches and S-FLEET-02's enumeration leg run on Oracle behind the provider switch (G7 Oracle 8/12 Part 2/2, #557) - #578

Open
omercelikdev wants to merge 2 commits into
mainfrom
agent/557-campaign-oracle-benches
Open

omercelikdev wants to merge 2 commits into
mainfrom
agent/557-campaign-oracle-benches

Conversation

@omercelikdev

Copy link
Copy Markdown
Collaborator

What and why

G7 Oracle 8/12, Part 2 of 2 of #411. oracle RFC §6 asks for an Oracle column on each bench page, "measured, not promised". Umbrella #287 names S-FLEET-02's enumeration leg on Oracle. #556 (merged as #560) did Bulk and Jobs; this PR does Campaign, reusing #556's provider-switch shape exactly. No package change.

  • Tests:
    • CampaignBenchTests and CampaignPacerEnumerationBenchTests are written once on SuiteStore. They are PostgreSQL as before: a container of their own, and the enumeration bench stays in the non-parallel broker-bench collection.
    • New Oracle twins CampaignBenchOracleTests and CampaignPacerEnumerationBenchOracleTests each run on a fresh schema of the Oracle collection's pinned container (OracleFixture.Image).
    • Each twin carries Category=Bench and Category=Oracle, so neither per-PR lane runs it.
    • Every BENCH-CAMPAIGN line an Oracle run prints ends in provider=oracle (BenchProvider.Tag).
    • The enumeration leg reads the item table's size from USER_SEGMENTS on Oracle (the table, its LOB and index segments) and from pg_table_size/pg_indexes_size on PostgreSQL, as before.
  • Budgets: on Oracle the facts assert correctness (every target enumerated, every outcome counted, and the throttled phase slower than the first) and print their numbers. The pacer's ±10% band binds PostgreSQL only, as in G7 Oracle 8/12 Part 1/2: Run the Bulk and Jobs benches on Oracle behind a provider switch, and add the Oracle column to their pages #556.
  • scripts/bench-campaign.sh takes GOLDPATH_BENCH_PROVIDER=postgres|oracle, default postgres, whose behaviour and output are unchanged.
  • bench.yml's provider=oracle dispatch runs bench-campaign.sh all after Bulk and Jobs, one after another, and the step fails if any of the three did. The PR is Tier C (bench-campaign.sh is a Tier C path in merge-tiers.yml, and bench.yml is a workflow).

The numbers (packages/Goldpath.Campaign/ops/campaign-benchmarks.md, "Oracle column")

Profile: the overnight agent's cloud sandbox, Linux x64, 4 logical CPUs / 15 GB, Docker 29.9, co-located, Debug (as the script runs). Oracle Free 23.26.3-slim-faststart, PostgreSQL 17-alpine. Code a326f1c. One local run each, PostgreSQL the same night on the same machine; no run id.

Proof PostgreSQL 17 Oracle Free 23.26.3
1M enumeration (engine loop) 53.7 s = 18,636 rows/s 117.4 s = 8,516 rows/s
Pacer at 200 TPS, 15 s 166.7/s 166.7/s
Live throttle → 50 50.8/s 50.0/s
100k sink 10,997/s 12,627/s
S-FLEET-02 enumeration leg, 1M through the pacer 56.0 s = 17,868 rows/s; 152 B/item 122.3 s = 8,177 rows/s; 251 B/item

Nothing is an order of magnitude apart. Oracle enumerates at about half PostgreSQL's rate on one machine. The page gives a line of reading per row, including why Oracle's 251 B/item (allocated extents) is a coarser figure than PostgreSQL's.

Reproduce

git fetch origin agent/557-campaign-oracle-benches && git checkout agent/557-campaign-oracle-benches
dotnet build tests/Goldpath.IntegrationTests

# Green: the four Oracle facts (about 6 min; needs Docker and gvenzl/oracle-free:23.26.3-slim-faststart)
GOLDPATH_BENCH_PROVIDER=oracle scripts/bench-campaign.sh all          # rc 0; every BENCH-CAMPAIGN line ends in provider=oracle
GOLDPATH_BENCH_CAMPAIGN_ENUM_COUNT=20000 GOLDPATH_BENCH_PROVIDER=oracle scripts/bench-campaign.sh enum   # quick: rc 0

# Red (the issue's proof): the Oracle class wired to PostgreSQL must fail rather than measure PostgreSQL.
# (`oracle is null ? null! :` keeps the primary-constructor parameter read: an unread one is CS9113, warnings are errors.)
sed -i 's/    : CampaignPacerEnumerationBenchTests(SuiteStore.Oracle(oracle, "GP_EBENCH"), output);/    : CampaignPacerEnumerationBenchTests(oracle is null ? null! : SuiteStore.PostgreSql(), output);/' \
  tests/Goldpath.IntegrationTests/CampaignPacerEnumerationBenchTests.cs
GOLDPATH_BENCH_CAMPAIGN_ENUM_COUNT=20000 GOLDPATH_BENCH_PROVIDER=oracle scripts/bench-campaign.sh enum
#   rc 1: "bench-campaign: GOLDPATH_BENCH_PROVIDER=oracle, but the run printed no Oracle bench line; nothing was measured on Oracle"
git checkout -- tests

# Refusals
for m in broker smoke fleet02; do GOLDPATH_BENCH_PROVIDER=oracle scripts/bench-campaign.sh $m; echo rc=$?; done   # rc 2 each, the reason named
GOLDPATH_BENCH_PROVIDER=sqlserver scripts/bench-campaign.sh; echo rc=$?                                         # rc 2

# Neither per-PR lane runs a Campaign bench; the monthly PostgreSQL set is the four facts it was
dotnet test tests/Goldpath.IntegrationTests --no-build --list-tests --filter "Category!=Bench&Category!=Oracle" | grep -c "BenchOracleTests\|BenchTests"   # 0
dotnet test tests/Goldpath.IntegrationTests --no-build --list-tests --filter "Category=Oracle&Category!=Bench" | grep -c "BenchOracleTests\|BenchTests"    # 0
dotnet test tests/Goldpath.IntegrationTests --no-build --list-tests --filter "Category=Bench&Category!=Oracle" | grep "Campaign\(Bench\|PacerEnumeration\)"  # the 4 PostgreSQL facts

I ran every line above. My first red attempt used plain SuiteStore.PostgreSql() and also exited 1, but from the build (CS9113), not from the provider check; that is why the plant is written as above. Its run printed the PostgreSQL lines and then the refusal.

Pre-PR self-check

  • Acceptance criteria (G7 Oracle 8/12 Part 2/2: Run the Campaign benches and the S-FLEET-02 enumeration leg on Oracle, and add the Oracle column to campaign-benchmarks.md #557):
    • The provider choice of G7 Oracle 8/12 Part 1/2: Run the Bulk and Jobs benches on Oracle behind a provider switch, and add the Oracle column to their pages #556's scripts, defaulting to PostgreSQL; Oracle runs both bench classes on OracleFixture's pinned image; broker/smoke/fleet02 refused in words.
    • Campaign: the 1M enumeration, the 200 TPS pacer with a live throttle and the 100k sink each have an Oracle column.
    • S-FLEET-02 enumeration leg: CampaignPacerEnumerationBenchTests ran on Oracle at Campaign snapshot is no longer bound to the leader tick, and the 35M enumeration is measured #288's size (1M). The 35M size is not run on Oracle: that is about an hour at the measured rate on this profile, and the page says so with the reason (time).
    • Every Oracle number states its profile (machine, image tag, build configuration), the date and the code; no budget is promised. No number is an order of magnitude worse; each difference has a one-line reading.
    • bench.yml's Oracle dispatch covers the Campaign benches. Tier C.
    • Ledgers: CHANGELOG [Unreleased]; ga-gate-status.md G7 names the page; scenario-campaign.md S-FLEET-02 notes the Oracle enumeration evidence (as the snapshot's rate, not the scenario's verdict). coverage-matrix.md has no Bench column, so it is unchanged.
  • No gate weakened: the PostgreSQL facts keep every assert they had. The Oracle-only branch asserts correctness instead of the band, as G7 Oracle 8/12 Part 1/2: Run the Bulk and Jobs benches on Oracle behind a provider switch, and add the Oracle column to their pages #556's do.
  • Honesty:
    • The page says the PostgreSQL pacer fact misses its own ±10% band on this profile (166.7 < 180; the CI profile measured 140). That fact's PostgreSQL code is unchanged by this PR.
    • The Oracle size figure is allocated extents.
    • The columns are comparable only to each other, not to the older rows.
  • Bench isolation: the Oracle twins run in the Oracle collection, one class at a time. The PostgreSQL enumeration bench stays in the non-parallel collection. In bench.yml the three scripts run one after another. The two local runs were taken one after the other with nothing else running.
  • XML docs: every new constructor, class and helper has its own <summary>. No member was inserted between an existing comment and its declaration.
  • Deep self-review:
    • Security: test and script code only. The Oracle SQL is a constant over the bench's own schema (no input).
    • Edge cases: a provider value outside the two is refused (exit 2). An Oracle run that measured nothing, or measured PostgreSQL, exits 1 (proven). A mode Oracle cannot run is refused before anything starts.
    • Compatibility: the PostgreSQL output lines are byte-identical (the tag is empty there), so bench-fleet02.yml and anything grepping them is unaffected.
    • Code quality: #556's idioms (SuiteStore, BenchProvider.Tag, the twin-class pattern). No suppression.

Gates run (head d49294b)

Gate Result
dotnet build Goldpath.sln -c Release 0 warnings, 0 errors
scripts/docs-freshness.sh rc 0
scripts/ledger-check.sh rc 0
scripts/schema-honesty.sh rc 0
scripts/skills-parity.sh rc 0
scripts/template-pins.sh rc 0
Oracle all (4 facts) / PostgreSQL same facts green / 3 of 4 green: the pacer fact misses its band on this profile, as explained above

What I could not verify

Closes #557

#411 stays open: its Bulk column waits on #559 (open-threads T68).



Generated by Claude Code

…n on Oracle behind the provider switch (G7 Oracle 8/12 Part 2/2, #557)

- CampaignBenchTests and CampaignPacerEnumerationBenchTests are written once on
  SuiteStore (#556's shape): PostgreSQL as before, and CampaignBenchOracleTests /
  CampaignPacerEnumerationBenchOracleTests on a fresh schema of the Oracle
  collection's pinned container, each carrying Category=Bench and Category=Oracle
  so neither per-PR lane runs them.
- Every BENCH-CAMPAIGN line an Oracle run prints ends in provider=oracle; the
  enumeration leg reads the item table's size from USER_SEGMENTS on Oracle.
- On Oracle the facts assert correctness (every target enumerated, every
  outcome counted, the live throttle bites) and print their numbers; the
  pacer's precision band binds PostgreSQL only (oracle RFC §6).
- scripts/bench-campaign.sh takes GOLDPATH_BENCH_PROVIDER=postgres|oracle:
  all and enum run on Oracle, broker/smoke/fleet02 are refused in words
  (no broker-on Oracle shape before G5, #415), and an Oracle run that printed
  no provider=oracle line exits 1.
- bench.yml's provider=oracle dispatch runs the Campaign benches after Bulk and
  Jobs, one after another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AFdNdQkwGfRCunWBAcuRC
…n one machine (#557)

ops/campaign-benchmarks.md gains the Oracle column from the run of a326f1c on
the overnight sandbox (Debug, Oracle Free 23.26.3, PostgreSQL 17 the same
night): 1M enumerated at 8,516 rows/s (engine loop) and 8,177 rows/s through
the pacer against 18,636 and 17,868; the governor at 166.7/s on both; the live
throttle at 50.0 against 50.8; the 100k sink at 12,627 against 10,997/s.
CHANGELOG [Unreleased], ga-gate-status G7 and scenario-campaign S-FLEET-02
name the page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018AFdNdQkwGfRCunWBAcuRC
@omercelikdev omercelikdev added the agent-pr PR opened by the overnight agent label Nov 12, 2026
@github-actions github-actions Bot added the tier:C ADR-0014 Tier C: decisions, releases, gates, policy, agent layer; the owner merges label Nov 12, 2026
@omercelikdev

Copy link
Copy Markdown
Collaborator Author

CI note: the red oracle check (run 43311679419, job 131509067562) is #571, not this PR.

merge-policy is the Tier C gate: it needs owner-approved.


Generated by Claude Code

@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Verifier verdict (run 2026-11-13)

Head verified: d49294b493debb738950c20318da8decc1f4b8b0. Base: main@90f78d3, which is the merge base. Tier C (.github/workflows/bench.yml, scripts/bench-campaign.sh): the owner merges it, never an auto-merge.

Verdict: no blocking findings. 0 BLOCKER, 0 MAJOR, 0 MINOR, 2 NIT. All four Oracle facts ran green on a real Oracle at full size. The issue's red proof reproduces exactly, and a mixed-provider run is refused too.

Steps 1–6

# Step Result / evidence
1 Scope vs #557 ✓ AC1: the provider switch defaults to postgres; oracle runs both twins (all) or the enumeration leg (enum); broker/smoke/fleet02 are refused in words with rc 2. ✓ AC2: there is an Oracle column for the 1M enumeration, the 200 TPS pacer with the live throttle, and the 100k sink. ✓ AC3: the S-FLEET-02 leg ran at #288's size (1M). 35M was not run, and the reason (time) is stated. ✓ AC4: profile, image tag, Debug configuration, date and code sha are stated, and no budget is promised. Nothing is an order of magnitude apart; each difference has a reading. ✓ AC5: bench.yml's Oracle dispatch runs bench-campaign.sh all after Bulk and Jobs, serially, and the step fails if any of the three does. ✓ AC6: CHANGELOG [Unreleased], ga-gate-status G7, and scenario-campaign.md S-FLEET-02 (worded as the snapshot's rate, not the scenario's verdict). coverage-matrix is unchanged, correctly. No package change, no dependency, no threshold, no test weakened: the PostgreSQL facts keep every assert they had.
2 Rules / ledgers ✓ Every new constructor and class has its own <summary>; nothing is inserted between an existing /// and its member. No PublicAPI change (test code only). No open-threads row is added, so no T-number can collide (main ends at T68; #575 holds T69). The PR closes #557 and leaves #411 open on #559, as the issue asks. goldpath-review rule set applied: no R1–R6 pattern found.
3 Confidentiality ✓ The denylist grep (case-insensitive, \bbtk\b) over the diff and both commit messages: 0 hits. No secrets or hostnames.
4 Build ✓ dotnet build tests/Goldpath.IntegrationTests: 0 warnings, 0 errors.
4 Green, real Oracle ✓ GOLDPATH_BENCH_PROVIDER=oracle scripts/bench-campaign.sh all on gvenzl/oracle-free:23.26.3-slim-faststart: rc 0, 4/4 passed, 6 min 5 s, and every BENCH-CAMPAIGN line ends in provider=oracle. My numbers, on a 4-CPU sandbox shared with another Oracle run: enumeration 1M in 149.9 s = 6,669 rows/s; sink 13,329 outcomes/s; pacer 166.7/s (2,500 items); throttle 50.0/s; pacer-enum 1M in 118.3 s = 8,454 rows/s, 2 slices, 249 B/item. That is the same shape as the page's column (the engine-loop enumeration is slower here because of the shared CPU).
4 Red (the issue's proof) ✓ The PR's plant (the Oracle enumeration twin wired to SuiteStore.PostgreSql()), with ENUM_COUNT=20000: the fact passes on PostgreSQL, and the script exits rc 1 with "the run printed no Oracle bench line; nothing was measured on Oracle". Restored: rc 0, tree clean.
4 Lanes ✓ --list-tests with Category!=Bench&Category!=Oracle and with Category=Oracle&Category!=Bench: 0 Campaign benches each. Category=Bench&Category!=Oracle: exactly the 4 PostgreSQL facts. The PostgreSQL all filter does not pick up the twins (CampaignBenchTests is not a substring of CampaignBenchOracleTests), and the Oracle filter (Integration.CampaignBenchOracleTests.) matches the namespace Goldpath.Tests.Integration. The CI oracle job ran 73 tests, the same count as main.
4 Gates ✓ docs-freshness, ledger-check (GitHub half skipped: no authenticated gh), schema-honesty, skills-parity, template-pins, merge-tier.py selftest: rc 0 for all. bash -n scripts/bench-campaign.sh: ok.
5 CI on d49294b ✓ Required checks green: gates, integration, kit, golden manifests (PR subset), all ten gm / *, CodeQL, review, alert-rules. merge-policy is red by design (Tier C without owner-approved). oracle (not required) is red on 1 of 73: CampaignOracleTests.Thousands_release_paced_across_leader_handovers_and_a_live_throttle_lifts_the_ceiling, TaskCanceledException at CampaignTests.cs:160. The worker's claim holds: this is main's known hang (#571, fixed by #579); the PR does not touch that test, and its twins are not in that lane.
6 Charter n.a.

Deep review

Dimension Result
a Security ✓ Test and script code only. The USER_SEGMENTS query is a constant over the bench's own schema, with no input. The workflow change adds no permission, secret or action, and inputs.provider is a choice that reaches run: only through if:. No dependency change.
b Correctness & edge cases ✓ An unknown provider (sqlserver, and ORACLE in upper case) and an unknown mode are refused with rc 2. A run that matched no test prints no line and is refused (rc 1). A failing fact propagates its own exit code (P2). A mixed run is refused (P1). The size query counts the table segment and its LOB segments as "heap", and every index on the table, including the LOB index (user_indexes.table_name), as "indexes"; it is stated as allocated extents. See NIT 1 for the refusal's wording.
c Concurrency ✓ Both twins are in OracleCollection, so they run one at a time; the PostgreSQL enumeration bench stays in the non-parallel broker-bench collection. bench.yml runs the three scripts sequentially. The schema prefixes GP_CBENCH and GP_EBENCH are unique in the repo, plus a process-wide counter.
d Data & migrations n.a. Each run takes a fresh schema on a container that is discarded with the run.
e Performance ✓ This is a bench; it adds about 6 minutes to the dispatch-only Oracle job, inside the unchanged 90-minute timeout. Two schemas of 1M items fit well under Oracle Free's user-data limit (~250 MiB each).
f Compatibility ✓ PostgreSQL output lines are byte-identical (the tag is empty there), so bench-fleet02.yml and anything grepping them is unaffected. The PostgreSQL facts' assertions are unchanged; EnsureCreated moved into SuiteStore.StartAsync, which does the same.
g Test quality ✓ The provider guard is proven red (the plant) and refuses a mixed run (P1). On Oracle the facts assert full enumeration, every outcome counted, and that the throttle bit. See NIT 2 on the ceiling.
h Code quality ✓ It reuses #556's idioms (SuiteStore, BenchProvider.Tag, the twin-class pattern). No suppression, no dead code. The oracle is null ? null! : trick lives only in the PR's reproduce recipe, not in code.
i Observability ✓ Every Oracle line is tagged, and the job summary step is unchanged.
j Honesty ✓ The page states that the PostgreSQL pacer fact misses its own ±10% band on this profile (166.7 < 180), that the Oracle size is allocated extents, and that the columns compare only with each other. The 30M ≈ 1 hour estimate checks out (30M ÷ 8,177/s ≈ 61 min). The "could not verify" list (no workflow dispatch, no 35M) is accurate.

Edge-case probes (mine, throwaway, nothing pushed)

# Probe Result
P1 Mixed run: CampaignBenchOracleTests wired to PostgreSQL, the enumeration twin left on Oracle, all with ENUM_COUNT=20000 rc 1, refused ✓. The PostgreSQL lines (enumeration 17,701 rows/s, pacer 183.3/s) are printed untagged and caught by the grep -v provider=oracle half. See NIT 1.
P2 Assert.Equal(count + 1, …) planted in the enumeration fact, Oracle enum rc 1 with Assert.Equal() Failure: Values differ shown ✓. A red fact is never masked by the provider check.
P3 Oracle enum (20,000) under DOTNET_CULTURE=tr-TR LANG=tr_TR.UTF-8 rc 0 ✓. The USER_SEGMENTS values are converted with InvariantCulture. The printed lines use the current culture (20.000, 6,2 s), but they did so before this PR, on PostgreSQL too.
P4 GOLDPATH_BENCH_PROVIDER=ORACLE (upper case) and an unknown mode bogus on Oracle rc 2 each, with the reason named ✓

NITs (optional)

  1. scripts/bench-campaign.sh: the refusal says "the run printed no Oracle bench line; nothing was measured on Oracle". In the mixed case (P1) the run did print Oracle lines; what is wrong is that some lines were not Oracle's. Something like "…but N bench line(s) were not measured on Oracle" would describe both cases.
  2. CampaignBenchTests.Pacer_holds_200_tps_…: on Oracle the fact returns before both InRange checks, so the upper bounds go too (the pacer never exceeding its ceiling, 210 / 60). Those are the governor's correctness, not a performance budget. CampaignOracleTests in the per-PR lane does guard over-release on Oracle (coarsely, ReleasedThrough < 500 at 10 TPS), so this does not block. Keeping Assert.True(phase1Rate <= 210) and phase2Rate <= 60 on the Oracle branch would make the column's "the governor holds" sentence a tested claim.

Not run

  • bench.yml with provider=oracle on the CI profile (I cannot dispatch a workflow), and the 35M enumeration on Oracle. The PostgreSQL all set was not re-run end to end; its code path is unchanged except for SuiteStore, and P1 ran three of its facts on PostgreSQL through the twin.

Labels: agent-verified added. Tier C: the owner merges after adding owner-approved. I do not merge.

Owner: safe to merge. Summary: no package change. GOLDPATH_BENCH_PROVIDER=oracle scripts/bench-campaign.sh (and bench.yml's Oracle dispatch) now runs the Campaign benches and S-FLEET-02's 1M enumeration leg on Oracle Free, and campaign-benchmarks.md gains an Oracle column: about half PostgreSQL's enumeration rate, with the same pacer behaviour.


Generated by Claude Code

@omercelikdev omercelikdev added the agent-verified Independently verified by the verifier agent: ready for owner merge label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr PR opened by the overnight agent agent-verified Independently verified by the verifier agent: ready for owner merge tier:C ADR-0014 Tier C: decisions, releases, gates, policy, agent layer; the owner merges

Projects

None yet

Development

Successfully merging this pull request may close these issues.

G7 Oracle 8/12 Part 2/2: Run the Campaign benches and the S-FLEET-02 enumeration leg on Oracle, and add the Oracle column to campaign-benchmarks.md

2 participants