Skip to content

feat(campaign): a tick's releases go out as one batch through an internal seam member (#285, throughput Part 3/3) - #558

Merged
omercelikdev merged 4 commits into
mainfrom
agent/285-batched-release
Nov 11, 2026
Merged

omercelikdev merged 4 commits into
mainfrom
agent/285-batched-release

Conversation

@omercelikdev

@omercelikdev omercelikdev commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

#285 is campaign throughput, Part 3/3. A tick's releases were published one awaited broker round trip per item: at 1000 TPS with a 250 ms tick, that is 250 sequential confirms in every tick. This PR implements the owner's decision of 2026-11-09 on #285:

  • The seam gains an INTERNAL batch member. IGoldpathBatchPublisher lives in Goldpath.Messaging. Both engines implement it, and Goldpath.Campaign reaches it through InternalsVisibleTo. Effects:
    • No line moves in any PublicAPI.*.txt (ADR-0013, GA G1).
    • An adopter changes nothing (ADR-0015).
  • No outbox: the engine keeps at most 64 publishes outstanding on the wire.
    • MassTransit uses the scoped IPublishEndpoint, and only when the engine itself composed no outbox (MassTransitPublishMode).
    • Wolverine uses GoldpathWire.PublishBatchAsync on its one confirming channel, under the existing gate.
  • Outbox on: every publish is a row in the one scoped DbContext, so the rows are added one at a time.
    • The outbox size guard has no batch member, so an outbox host takes this path by construction.
    • The configureBus registration carries no mode, so it is never assumed outbox-free.
  • The campaign's three publish paths (a tick's releases, ripe retries, orphan re-releases) each hand their messages over as one batch, then mark.
    • Publish-then-mark is unchanged: a batch that faults part way marks nothing.
    • EngineTests.CrashBetweenPublishAndMarkRepublishesButNeverDoubleClaims now runs through the batch member, because the fixture's publisher carries it, and it stays green.

Acceptance criteria of #285, each with its evidence

  • Releases are published without one awaited round trip per item, and publish-then-mark and the claim guard are kept.

    • BatchedReleaseTests: each of the three paths publishes exactly one batch, with zero single publishes. They go red (3 facts) with main's engine file put back.
    • The crash fact is green through the batch path.
  • No new MassTransit. type in any PublicAPI.*.txt. No PublicAPI file changed: git diff origin/main --stat -- '**/PublicAPI*' is empty. Everything new is internal.

  • The per-tick query count is measured before and after, and written in the bench report. It is 7 commands per steady tick for one hungry campaign, on main and on this branch alike.

    • TickQueryCountTests runs the pacer on a tick-exact clock and pins 7.
    • I ran it on main (2782d3c) in a separate worktree: green at 7.
    • No query was removed.
  • Part 1's bench at Tps = 1000 for ≥15 minutes is green against S-FLEET-02a: NOT MET on the only profile I have. The numbers are published in ops/campaign-benchmarks.md, and RFC §6 carries the measured line, saying it is not met. Local sandbox, 4 CPUs, everything co-located, no run id:

    Run Code Mean Within ±5% Above 1000 Exactly-once
    smoke 120 s main 2782d3c 584.4/s 0.0% 0 71,179 ×3, 0 twice
    smoke 120 s this branch 86615f6 969.2/s (p50 1000) 72.5% 10 97,328 ×3, 0 twice
    broker 900 s this branch 86615f6 809.0/s 37.6% 50 724,500 ×3, 0 twice

    Why it is still red: two causes, neither in the publish path.

    1. The profile. The consumers execute about 811/s co-located, the backlog grows, and the pacer's ticks run late.
    2. A definition for the owner. A late tick releases what it missed, through Campaign burst control: no one-second window above the ceiling (S-FLEET-03 strict) #292's one-tick bank, so one wall-clock second can hold five bursts. The max, 1250, is exactly 5 × 250. Campaign burst control: no one-second window above the ceiling (S-FLEET-03 strict) #292's burst rule, which S-FLEET-02a cites, is stated over a second of ticks, and it holds (BurstControlTests).

    This is recorded as open-threads T67, with its trigger and proof, and in the G10c status row, because S-FLEET-02 is judged by the same words.

  • The S-FLEET-02a evidence: field names the run. It names the branch, the commit, the date, and "local run, no run id".

Reproduce

git fetch origin && git checkout agent/285-batched-release
dotnet test tests/Goldpath.Campaign.Tests -c Release          # 468/468
dotnet test tests/Goldpath.Messaging.Tests -c Release --filter "FullyQualifiedName~BatchPublishTests"   # 9/9
dotnet test tests/Goldpath.Messaging.Conformance -c Release --filter "FullyQualifiedName~S2_22|FullyQualifiedName~S2_23"   # 4/4 (both engines, Docker)

# RED 1, the per-item loops back (3 red: every BatchedReleaseTests path)
git show origin/main:packages/Goldpath.Campaign/GoldpathCampaignEngine.cs > packages/Goldpath.Campaign/GoldpathCampaignEngine.cs
dotnet test tests/Goldpath.Campaign.Tests -c Release; git checkout packages/Goldpath.Campaign/GoldpathCampaignEngine.cs
# RED 2, the MassTransit mode ignored (2 red: the outbox / no-mode sequential theory)
sed -i 's/=> mode is { ConcurrentOnTheWire: true }/=> true/' packages/Goldpath.Messaging/IIntegrationEventPublisher.cs
dotnet test tests/Goldpath.Messaging.Tests -c Release --filter "FullyQualifiedName~BatchPublishTests"; git checkout packages/Goldpath.Messaging
# RED 3, unbounded (2 red: the bound, the cancellation window)
sed -i 's/start += MaxConcurrency)/start += items.Count)/; s/var end = Math.Min(start + MaxConcurrency, items.Count);/var end = items.Count;/' packages/Goldpath.Messaging/GoldpathBatchPublishing.cs
dotnet test tests/Goldpath.Messaging.Tests -c Release --filter "FullyQualifiedName~BatchPublishTests"; git checkout packages/Goldpath.Messaging

# The bench (Docker; run nothing else beside it)
scripts/bench-campaign.sh smoke     # 2 min; on main: ~584/s, 0% in band; here: ~969/s, ~72% in band
scripts/bench-campaign.sh broker    # 15 min; here: 809.0/s, 37.6% in band, 50 above (RED, see T67)

I saw exactly the failures listed for each red run, and nothing else.

Gates run

Gate Result
dotnet build Goldpath.sln -c Release 0 warnings, 0 errors
Goldpath.Campaign.Tests 468/468
Goldpath.Messaging.Tests 122/122
Conformance S2_22, S2_23 (MassTransit and Wolverine, real RabbitMQ and PostgreSQL) 4/4
Conformance, full suite running at PR time; result in a follow-up comment
docs-freshness, ledger-check, schema-honesty, skills-parity, template-pins rc 0 ×5
Mutation, diff mode (GOLDPATH_MUTATE_SINCE=origin/main scripts/mutation-gate.sh Goldpath.Messaging) running at PR time (798 mutants); result in a follow-up comment and stryker/README.md

Pre-PR self-check

  • Acceptance criteria: each is listed above with its evidence. The bench criterion is not met, and that is said plainly, ledgered in T67 and reflected in the G10c row.
  • Ledgers:
  • XML docs: every new member is documented. PublishItemsAsync is inserted before RollQuotaDayIfNeededAsync's own /// block, not between that block and its declaration. GoldpathWire.PublishBatchAsync is inserted before PublishAsync's ///. Both re-read.
  • Honesty:
    • I removed a claim I had first written in S2_23's summary, that a concurrent outbox publish "would throw there". A planted concurrent outbox path stayed green, because the Wolverine outbox branch never awaits, so it proved nothing.
    • The summary now says what the fact proves.
    • The structural guarantee is pinned by the wiring theory instead: an outbox host resolves the size guard, which has no batch member.
  • Wired: The_default_engine_composes_the_mode_from_the_outbox builds the host through AddGoldpathMessaging and checks two things: the mode follows the outbox, and the resolved publisher is the engine's (no outbox) or the guard (outbox). The bench runs the real host end to end.
  • Contracts: no public surface, schema or OpenAPI changed.
  • Benches: run sequentially, nothing beside them, and the published numbers carry profile, date, commits and "no run id". The conformance and mutation runs were started only after the bench finished.
  • Deep self-review:
    • Security: no new input. The messages are the campaign's own, and no secrets are logged.
    • Concurrency:
      • With no outbox, at most 64 publishes are outstanding (unit test). A failure is rethrown only after every started publish has settled, so nothing runs against a disposed scope (unit test).
      • A publish that throws synchronously does not abandon the rest of its window (unit test).
      • The Wolverine wire holds its gate for the whole batch, so another publisher waits up to one batch. That is stated in the XML doc.
      • With the outbox: one at a time, structurally.
    • Partial failure: a batch that faults part way leaves nothing marked, and the next tick republishes; the claim guard's duplicate safety covers it (the crash fact, through the batch path).
    • Cancellation: no further window starts after cancellation (unit test).
    • Edge cases: an empty batch publishes nothing, and a null batch is refused (unit test).
    • Performance: measured above. One list of messages is allocated per batch, bounded by ReleaseBatchSize.
    • Compatibility: behaviour on an outbox host is unchanged, one at a time. A third-party engine without the member falls back to one at a time.
    • Code quality: one helper replaces the three loops.

Not verified, and observations

Refs #285 — not closed by this PR: the bench criterion is still open under T67 (owner decision, 2026-11-11).


Generated by Claude Code

…rnal seam member (#285)

Owner decision on #285 (2026-11-09): the batch goes through a NON-PUBLIC member on the messaging
seam, implemented by both engines and reached by Goldpath.Campaign through InternalsVisibleTo.

- Goldpath.Messaging: internal IGoldpathBatchPublisher and GoldpathBatchPublishing (the entry
  point: the engine's member, else one at a time; BoundedAsync keeps at most 64 publishes
  outstanding and rethrows only after every started one has settled).
- MassTransit engine: concurrent on the wire only when the engine composed no outbox
  (MassTransitPublishMode); with the bus outbox, or on the configureBus registration, one at
  a time. Wolverine engine: envelopes built first, then GoldpathWire.PublishBatchAsync on the
  one confirming channel; with the outbox, rows added in order. The outbox size guard carries
  no batch member, so an outbox host publishes a batch one event at a time by construction.
- Campaign: ReleaseBatchAsync, ReleaseRipeRetriesAsync and ReleaseOrphansAsync publish their
  messages as one batch, then mark (publish-then-mark unchanged).
- No PublicAPI line changes.

Proof: BatchedReleaseTests (red with the per-item loops back), BatchPublishTests (bound,
outbox/no-mode sequential, failure settles, cancellation, wiring), conformance S2_22/S2_23 on
both engines, TickQueryCountTests (7 commands per steady tick, the same on main).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery count, T67 (#285)

- ops/campaign-benchmarks.md: before/after on the overnight sandbox (no run id): smoke 584.4/s
  -> 969.2/s (0% -> 72.5% of windows in band); 15 minutes 809.0/s, 37.6% in band, 50 windows
  above 1000 - still RED; exactly-once over 724,500 items; 7 commands per steady tick on both.
- RFC goldpath-campaign §6: the measured line, saying the bar is not met on that profile.
- scenario-campaign S-FLEET-02a evidence names the run; ga-gate-status G10c notes it.
- open-threads T67: what remains (the co-located profile; wall-clock seconds vs seconds of
  ticks for "zero windows above 1000", for the owner).
- CHANGELOG [Unreleased] ### Changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@omercelikdev omercelikdev added the agent-pr PR opened by the overnight agent label Nov 10, 2026
@github-actions github-actions Bot added tier:C ADR-0014 Tier C: decisions, releases, gates, policy, agent layer; the owner merges labels Nov 10, 2026
…100+-message outbox (#285)

The full conformance run on 499f74b was red in one place: BusDifferentialTests' S3 record of
the new S2_23 (a 200-event batch through the outbox) differed on store.outbox-state.rows and
.delivered (oracle 1, candidate 0). Bisected on real RabbitMQ + PostgreSQL: equal at 1, 2, 50
and 99 events in one outbox, different from 100 — MassTransit's delivery pass stops at its
MessageDeliveryLimit (100) and keeps the row, marked delivered, until the duplicate-detection
window; Wolverine removes it. Engine bookkeeping, not a seam promise: the deliveries (every
event once) are compared line by line and equal. Two allow-list entries for S2_23, with the
measurement as their reason. BusDifferentialTests + OutcomeComparatorTests: 36/36.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mpleted batch; the diff-mode score (#285)

Diff-mode mutation run of Goldpath.Messaging at 499f74b: 92.11 %, gate green (73 mutants,
43.5 min). On #285's lines, `start < items.Count` -> `<=` survived: with a count that is an
exact multiple of the bound, a cancellation during the last window failed a batch whose every
publish had completed. New fact kills it (planted, red). The empty-batch early return is
equivalent. Recorded in stryker/README.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Follow-up to the PR body, with the two runs that were still going when it was opened. The head is now 98124ce.

1. Full conformance suite (both engines, real RabbitMQ + PostgreSQL), on 499f74b: 106/107. Fixed in 0ea78e4.

  • What failed: BusDifferentialTests.S3_…(S2_23). The new fact's S3 record differed on store.outbox-state.rows / .delivered: the oracle (MassTransit) had 1, the candidate (Wolverine) had 0. Delivery was not involved: every event was delivered exactly once on both engines.
  • The cause, bisected on the real containers:
    • The records are equal at 1, 2, 50 and 99 events in one outbox, and differ from 100.
    • From 100, MassTransit's delivery pass stops at its MessageDeliveryLimit (100). It then keeps the OutboxState row, marked delivered, until the duplicate-detection window expires. Wolverine removes it.
    • That is engine bookkeeping, not a seam promise.
  • The fix: two allow-list entries scoped to S2_23_*, with the measurement as their reason. S2_23 keeps 200 events, so the over-limit path stays exercised.
  • After the fix: BusDifferentialTests + OutcomeComparatorTests 36/36.
  • Observation, not changed here: a campaign batch of 100+ through the bus outbox will leave that delivered row until the window ends. That is MassTransit's normal cleanup.

2. Mutation, diff mode (GOLDPATH_MUTATE_SINCE=origin/main scripts/mutation-gate.sh Goldpath.Messaging) at 499f74b: 92.11 %, gate GREEN. 73 mutants tested, 43.5 min.

  • One survivor on Campaign throughput, Part 3/3: batched release publishing, and 1000 TPS sustained, proven #285's lines was a real gap: start < items.Count → <= in BoundedAsync. With a count that is an exact multiple of 64, a cancellation during the last window failed a batch whose every publish had completed, and the caller would have republished it.
    • Killed by A_cancellation_during_the_last_window_does_not_fail_a_completed_batch in 98124ce: planted → red; restored → Goldpath.Messaging.Tests 123/123.
  • The other survivor is equivalent: the empty-batch early return.
  • The remaining survivors are on lines this PR did not change (MassTransitMessagingEngine.cs, which diff mode mutates whole).
  • Recorded in stryker/README.md.

Generated by Claude Code

@omercelikdev

Copy link
Copy Markdown
Collaborator Author

Verifier verdict (run 2026-11-11)

Head verified: 98124ce14da1bc9968a98306b07f8402796a7699 (4 commits). It merges cleanly into today's main@2782d3c, and also cleanly together with #529, which touches the same engine file. Tier C (docs/rfc/**, the conformance allow-list), so it is not auto-merged.

Verdict: no blocking findings. 0 BLOCKER, 0 MAJOR, 0 MINOR, 1 NIT.

What I checked

Step Result
1. Scope vs #285 AC1 (no round trip per item; publish-then-mark and the claim guard kept): ✓, BatchedReleaseTests plus the crash fact through the batch path. AC2 (no MassTransit. type in PublicAPI.*): ✓, no PublicAPI file changed, and everything new is internal. AC3 (per-tick query count measured): ✓, 7, pinned by TickQueryCountTests. AC4 (15 min at 1000 TPS green): ❌ not met and disclosed, ledgered in T67 and the G10c row: owner question. AC5 (the evidence field names the run): ✓. Nothing outside scope: no workflow, permission, dependency or version change. The only new suppression-like item is the two-entry differential allow-list, scoped to S2_23_* with a measured reason (MassTransit's MessageDeliveryLimit keeps a delivered OutboxState row from 100 messages).
2. Rules / ledgers CHANGELOG [Unreleased] ### Changed ✓. RFC §6 measured line ✓. campaign-benchmarks.md ✓, with profile, date, commits and "no run id". T67 is unique: main ends at T64, #552 holds T65/T66, #560 holds T68. Each new internal member has XML docs, and none is inserted between an existing /// and its member (checked PublishItemsAsync and GoldpathWire.PublishBatchAsync). InternalsVisibleTo is commented with the owner decision and ADR-0013/0015.
3. Confidentiality The diff, the commit messages and the PR body: 0 hits. No secrets or hostnames.
4. Build On the merged tree, dotnet build Goldpath.sln -c Release: 0 warnings, 0 errors.
4. Tests Goldpath.Campaign.Tests 468/468. Goldpath.Messaging.Tests 123/123. BatchPublishTests 10/10 under DOTNET_CULTURE=tr-TR. Conformance S2_22 + S2_23 on both engines, with real RabbitMQ and PostgreSQL (Docker): 4/4.
4. Red proofs (the PR's recipe) RED 1 (main's engine file put back): 3 red, all three BatchedReleaseTests paths ✓. RED 3 (unbounded): 2 red, the bound and the cancelled window ✓. RED 2 as written does not compile (CS9113: Parameter 'mode' is unread, warnings as errors): see the NIT. The same fault planted compilably (… || mode is null, then => mode is not null) turns the theory's null row red and then its False row red ✓. Restored, and the tree is clean.
4. Bench (smoke, 2 min, nothing beside it) 959.9/s mean, p50 1000, p95 1094, max 1250; 60.0% of windows in band, 17 above; 95,676 executed once, 0 twice. That is consistent with the PR's 969.2/s and 72.5% on a 4-CPU co-located box, and it is still red against S-FLEET-02a, as T67 says.
4. Gates docs-freshness, ledger-check, schema-honesty, skills-parity, template-pins: rc 0 ×5.
5. CI on 98124ce Every check green except merge-policy, which is ✗ by design (Tier C, no owner-approved).
6. Charter n.a.

Deep review

Dimension Result
a Security ✓ No new input. The messages are the campaign's own, the envelope headers are built per message from the ambient tenant and correlation as on the single path, and nothing is logged. No dependency change.
b Correctness & edge cases ✓ Empty batch: completes without a publish, and the mark is a no-op as before. Null: refused. An exact multiple of 64 with cancellation in the last window: killed by the new fact. Partial failure: nothing is marked and the next tick republishes, which the claim guard makes safe (the crash fact). P1–P3 below.
c Concurrency ✓ No outbox: at most 64 outstanding (P3: exactly 64 in flight). Every started publish settles before a rethrow. With an outbox, one at a time structurally: Wolverine branches on state.OutboxContext, MassTransit on the composed mode, and the configureBus path (no mode) is sequential. Wolverine holds its gate for the whole batch, so the relay waits up to one batch; that is stated in the XML doc. RabbitMQ.Client 7 publishes concurrently on one channel with confirmation tracking enabled (CreateChannelOptions(true, true), unchanged), and S2_22 proves it on a real broker.
d Data & migrations n.a. No schema change. With MassTransit's bus outbox, a 100+ batch keeps one delivered OutboxState row until the dedupe window ends. That is engine cleanup, disclosed in the follow-up.
e Performance ✓ Measured: about 584/s on main against about 960/s here (smoke). One list per batch, bounded by ReleaseBatchSize. The query count is unchanged at 7.
f Compatibility ✓ No public API, contract or config change. Behaviour on an outbox host is unchanged (one at a time). A decorator or third engine without the member falls back to one at a time.
g Test quality ✓ RED 1, RED 3 and a compilable RED 2 are killed. Diff-mode Stryker on Goldpath.Messaging scored 92.11% (PR follow-up), and its one real survivor became a fact. P1–P3 are mine.
h Code quality ✓ One helper replaces the three loops. HeadersOfTheAmbientContext() removes the duplication between the single and batch paths. The comments say why (the scoped DbContext, the confirm tracking, the gate).
i Observability ✓ No log or metric removed. The bench's exactly-once accounting is unchanged.
j Honesty ✓ The PR states that the bench criterion is unmet, with its numbers, and they reproduce here. It also removed an unproven claim from S2_23's summary on its own. The one exception is the RED 2 recipe line (NIT).

Edge-case probes (throwaway facts in Goldpath.Messaging.Tests, deleted, never pushed)

# Probe Result
P1 BoundedAsync over 130 items (three windows) Each published exactly once (130 distinct). Pass
P2 Items 3 and 10 fail in the first window of 130 Rethrows boom 3 after the window settles. 62 published, and none ≥ 64: no second window starts. Pass (the caller treats it as unpublished-maybe-delivered, as documented)
P3 Exactly 64 items, held open 64 outstanding at once: one window, never 65. Pass

NIT (optional)

  1. The PR body's RED 2 recipe does not compile. sed 's/=> mode is { ConcurrentOnTheWire: true }/=> true/' leaves the mode parameter unread, so CS9113 fails the build, and "I saw exactly the failures listed for each red run" cannot be true for that line. Use => mode is not null (red: the False row) or … || mode is null (red: the null row) in the PR body.

Question for the owner (needs-owner)

Should this PR close #285 although the 15-minute, 1000 TPS bench criterion is unmet, or should #285 stay open until T67's proof runs on the reference profile?

  • My recommendation: merge the code, which is verified and is a clear improvement (about 584/s → about 960/s on the same box, exactly-once intact).
  • But change Closes qorpe/goldpath#285 to Refs #285, so the issue and T67 close together, when a green run on the CI reference profile or the Lima VM names its run id.

Labels: agent-verified and needs-owner. Tier C, so not auto-merged.

Owner: safe to merge — summary: the campaign engine publishes each tick's releases, ripe retries and orphan re-releases as one batch, with at most 64 broker confirms outstanding instead of one round trip per item. Outbox hosts are unchanged, and no public API moves.


Generated by Claude Code

@omercelikdev omercelikdev added agent-verified Independently verified by the verifier agent: ready for owner merge needs-owner Needs an owner decision; the question is in the last comment labels Nov 11, 2026
@omercelikdev omercelikdev added owner-approved ADR-0014: the owner approves merging this Tier C agent PR and removed needs-owner Needs an owner decision; the question is in the last comment labels Nov 11, 2026
@omercelikdev
omercelikdev merged commit 02128c5 into main Nov 11, 2026
30 of 33 checks passed
@omercelikdev
omercelikdev deleted the agent/285-batched-release branch November 11, 2026 06:30
omercelikdev added a commit that referenced this pull request Nov 11, 2026
…d included (main red after #529 + #558) (#563)

#558 pinned 7 commands per steady tick, measured on a main without R3-S8. #529 (R3-S8, #290) landed first
and reads the live ceilings in one query a tick, by design (campaign RFC R3.2), so main counts 8 and
TickQueryCountTests is red on 02128c5. The extra command is that read: SELECT ... FROM "GoldpathCampaignCeilings",
as the interceptor's SQL log shows for a steady tick. The pin and the two prose mentions say 8 now.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr PR opened by the overnight agent agent-verified Independently verified by the verifier agent: ready for owner merge owner-approved ADR-0014: the owner approves merging this Tier C agent PR tier:C ADR-0014 Tier C: decisions, releases, gates, policy, agent layer; the owner merges

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants