fix(partitions): complete polls on the owning shard by diegomrsantos · Pull Request #4119 · apache/iggy

diegomrsantos · 2026-09-10T09:47:59Z

A disk poll can remain pending while purge replaces a partition's message history and resets its offsets to zero. Previously, the detached read worker retained shared handles that could advance consumer offsets and group progress after purge. If fresh messages reused those numeric offsets, the stale completion could make a later Next poll skip messages from the new history.

Poll workers now return owned read results to the partition owner. The owner validates the captured history before accepting progress or authorizing a successful reply.

Implementation

PollHistoryId is a process local u64 identity from a checked counter. History replacement invalidates earlier reads, including when the namespace and numeric offsets are reused.
Disk polls reserve capacity before I/O in a dedicated completion lane. Running reads and queued results retain that capacity until dequeue or discard. sharding.poll_completion_capacity is independent of the ordinary inbox, and exhaustion rejects before reading.
Resident reads finish inline under the same history and admission rules. The owner services the completion lane alongside ordinary work.
Automatic commit admission, capacity reservation and progress updates occur synchronously on the owner. A successful poll reply can precede durable offset replication. Once a prepare has been assigned, its continuation must begin immediately on the same pump. A missing partition trips an invariant instead of silently discarding it.
A reply receiver already disconnected when completion arrives prevents admission. Disconnection can still race with that check, so the API documents recovery from explicit offsets after a lost reply.
The owner loop reuses its shutdown receive future, avoiding repeated receive setup on each pump turn. Completion metrics are shared during shard setup, and individual polls copy their history identity without accessing the global counter.

Verification

At c7e6fb165, 1,348 affected tests pass on macOS and 1,371 on ARM64 Linux, with eight existing ignored tests on each platform. Strict Clippy with all features and all targets passes for the six affected crates on both platforms. Formatting, dependency sorting with --no-format, TOML formatting and the source diff checks pass. GitHub workflows for the measured commit require maintainer approval. The PR subsequently advanced to 8f1c0e260 through a master merge containing #4152; its changes are confined to connectors and their tests. The server code and build inputs used by this comparison are unchanged. The test and benchmark results below remain attributed to c7e6fb165.

The new continuation regression proves an automatic commit prepare has been assigned but is not yet in the journal before tombstoning its namespace. It fails with the previous silent drop behavior and passes with the invariant restored. Existing suites cover delayed polls after purge, stale history with valid reused numeric offsets, consumer group progress, completion capacity, caller cancellation and reply release before replication.

The Linux boot smoke test uses IGGY_SHARDING_CPU_ALLOCATION=all because the container denies NUMA memory binding. Partition and shard documentation, including private items, builds successfully. The broader server documentation build encounters existing broken links outside these review edits.

Performance

The measured candidate c7e6fb165 was compared with upstream master 585ab6922 in the ARM64 Docker Desktop Linux VM. After high Docker memory use was reported, the existing benchmark was rerun with only its runtime container active and no concurrent builds. Earlier observations are retained separately and excluded from the rerun estimates.

The rerun completed 120 measured runs and 2,304,000 successful nonempty polls, plus 20 excluded pilot runs. All workload and existing resource checks passed. There are 12 balanced randomized pairs per case. Positive values below mean slower polls.

Case	Mean time change, 95% interval	p99 change, 95% interval
Explicit offset	+0.10%, [-1.35%, +1.69%]	-0.02%, [-2.79%, +2.88%]
Explicit offset with writes	+2.32%, [+0.41%, +4.36%]	+3.40%, [-0.62%, +7.63%]
Next	-0.48%, [-1.41%, +0.44%]	-0.84%, [-2.37%, +0.69%]
Consumer group	+0.45%, [-0.59%, +1.45%]	+0.86%, [-0.89%, +2.65%]
Resident	-3.77%, [-5.27%, -2.38%]	-2.33%, [-6.29%, +1.23%]

Mean intervals use 20,000 bootstrap resamples of whole pairs. They are pointwise 95% intervals without adjustment across cases. p99 is the geometric mean of paired ratios of each run's p99, not a pooled percentile.

The concurrent write case has higher mean time, while resident reads have lower mean time. The other mean intervals span zero. The earlier Next p99 increase was not reproduced. This supports neither a blanket claim that all earlier reports still hold nor a claim that Docker memory caused their differences. Disk reads may use the guest OS cache, and these results describe this laptop VM. Native macOS performance and standalone disk batch and fsync results were not refreshed.

Full report, raw results, frozen methods and validation logs.

The benchmark tooling is reviewed separately in #4122. The earlier measurements and discussion remain historical evidence for their recorded revisions.

Metric retirement

frame_drops_total{variant="partition_auto_commit"} is retired because the inbox operation it measured no longer exists. frame_drops_total{variant="partition_poll_completion"} reports disk completion capacity and return failures, while partition_consumer_offsets_denied_total counts consumer offset capacity denials. These measure different events and are not a direct replacement for the retired label. Unexpected poll replies now log the actual reply instead of a missing reply field that was always false.

hubcio · 2026-09-10T09:54:14Z

#4092 will be merged first :)

hubcio · 2026-09-10T09:57:17Z

@diegomrsantos have you joined our discord? if yes, what's your username?

codecov · 2026-09-10T10:21:01Z

Codecov Report

❌ Patch coverage is 96.38233% with 95 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.31%. Comparing base (585ab69) to head (8f1c0e2).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
core/partitions/src/iggy_partition.rs	98.00%	5 Missing and 18 partials ⚠️
core/shard/src/poll/completion.rs	92.50%	10 Missing and 10 partials ⚠️
core/simulator/src/bin/simulator-ui.rs	0.00%	14 Missing ⚠️
core/shard/src/router.rs	80.39%	8 Missing and 2 partials ⚠️
core/server/src/consumer_group.rs	93.98%	1 Missing and 7 partials ⚠️
core/server/src/dispatch/partition.rs	95.76%	7 Missing and 1 partial ⚠️
core/simulator/src/lib.rs	93.04%	8 Missing ⚠️
core/shard/src/poll.rs	98.31%	2 Missing ⚠️
core/partitions/src/iggy_partitions.rs	98.41%	0 Missing and 1 partial ⚠️
core/shard/src/poll/test_support.rs	98.48%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #4119      +/-   ##
============================================
- Coverage     86.71%   86.31%   -0.40%     
+ Complexity     1457     1455       -2     
============================================
  Files          1268     1272       +4     
  Lines        212608   209121    -3487     
  Branches     177794   174306    -3488     
============================================
- Hits         184356   180507    -3849     
- Misses        23765    23877     +112     
- Partials       4487     4737     +250

Components	Coverage Δ
Rust Core	`87.21% <95.97%> (-0.59%)`	⬇️
Java SDK	`67.58% <ø> (-0.02%)`	⬇️
C# SDK	`77.11% <ø> (ø)`
Python SDK	`91.34% <ø> (ø)`
PHP SDK	`85.67% <ø> (ø)`
Node SDK	`96.26% <ø> (+1.73%)`	⬆️
Go SDK	`69.48% <ø> (+0.07%)`	⬆️

Files with missing lines	Coverage Δ
core/common/src/http/messages/poll_messages.rs	`85.00% <ø> (ø)`
core/common/src/traits/message_client.rs	`100.00% <ø> (ø)`
core/common/src/types/message/polling_strategy.rs	`94.00% <ø> (ø)`
core/configs/src/server_config/sharding.rs	`95.18% <100.00%> (+1.27%)`	⬆️
core/consensus/src/impls.rs	`93.09% <100.00%> (+0.03%)`	⬆️
core/partitions/src/consumer_offset_capacity.rs	`99.00% <100.00%> (-0.03%)`	⬇️
core/partitions/src/lib.rs	`0.00% <ø> (ø)`
core/partitions/src/poll_plan.rs	`93.22% <100.00%> (-0.51%)`	⬇️
core/partitions/src/state_transfer.rs	`71.73% <100.00%> (+3.19%)`	⬆️
core/sdk/src/clients/consumer.rs	`82.69% <ø> (ø)`
... and 24 more

... and 145 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

diegomrsantos · 2026-09-10T13:12:20Z

#4092 will be merged first :)

Thanks for letting me know. I'll rebase this one onto #4092

A disk poll that completes after purge can restore an offset from the deleted history and cause Next to skip fresh messages. Exercise the production disk read, purge, and offset paths with a deterministic schedule. Keep the regression intentionally failing and include a passing control with automatic commit disabled.

An offset from purged history can also fit within the fresh history, so an upper bound check alone cannot establish that consumer progress is valid. Cover both fresh history sizes and verify the complete disk read before purge. Describe the completion output as an automatic commit candidate. Keep the regressions failing until the production fix is implemented.

Document why the reproduction delays completion, reuses message offsets, and covers an old offset within the fresh range. Clarify that the tests observe automatic commit candidates before server admission.

Explain the old poll and fresh messages before discussing reused offsets, so the comments can be understood without tracing the helper first.

Reservations stay on the owning shard and belong to one consumer. Express those constraints in the token API and remove unnecessary ownership requirements from capacity admission. Extract synchronous poll admission and share queue cleanup while preserving history validation and overload behavior. Validated with affected crate tests on macOS and Linux, plus the Linux cluster consumer offset quota regression.

Poll completion must reject a full persistence queue before recording consumer progress or assigning an operation. Apply the persistence admission guard on the owning shard and cover rejection followed by retry. Adapt recovery and consumer group test fixtures to the current polling and persistence APIs, including reads through the shard message pump. Co-authored-by: Codex <noreply@openai.com>

hubcio · 2026-09-11T08:26:26Z

@diegomrsantos which one should be reviewed/merged first, this one or #4122?

diegomrsantos · 2026-09-11T08:34:15Z

I’d prioritize this one, since it fixes the correctness issue. #4122 contains the benchmark tooling and works against master without this fix. Neither PR depends on the other, so they can be reviewed independently, but I’d merge this one first.

Poll completion crosses asynchronous reads, owner admission, and offset replication, so maintainers need the lifetime and ordering contracts at these boundaries. Document the flow, history identity, capacity guards, and reply semantics on the relevant types, fields, and functions. Co-authored-by: Codex <noreply@openai.com>

diegomrsantos · 2026-09-11T10:55:56Z

Added a short overview of poll completion and concise docs for the new types, fields, and functions. These explain history validation, reservation lifetimes, admission checks, and why poll replies can precede durable offset commits. This update changes documentation only.

Reservation bookkeeping can use Rc and Cell because guards stay on the owner thread, outside the shard inbox's Send requirement. Maintainers also need to preserve capacity checking and acquisition as one operation. Document these constraints on the completion message, reservation types, and capacity admission method. Co-authored-by: Codex <noreply@openai.com>

diegomrsantos · 2026-09-11T12:15:46Z

Added concise docs explaining the reservation ownership boundary. Previously, AutoCommitSubmit carried an existing guard through the shard inbox, whose Send bound applied even to sends to the same shard. Now the inbox carries read results, and the owner creates and retains guards in local request or replication state. This permits Rc and Cell while preserving synchronous counter updates. The docs also explain why capacity checking and acquisition must stay in one owner operation: individual atomics would not protect admission split across owner turns. This update changes documentation only.

A missing owner reply does not prove that a poll was rejected: its eventual completion can still advance consumer progress. Returning an empty success hid that uncertainty. Return ShardCommunicationError when the owner reply is unavailable, clarify acceptance and delivery contracts, and cover timeout followed by an undeliverable rejection in a regression test.

Reads rejected before reaching the partition owner must remain safe for SDK retries. Return TransientNotAccepted for routing and inbox submission failures while preserving ShardCommunicationError for missing replies after submission. Merge current master, preserve both sets of regression tests, and cover refusal paths, lost replies, and consumer group ownership.

Groups record their last polled offset even when automatic commits are disabled. Keep a real disk read pending across purge and verify that its stale completion cannot restore group progress after new messages reuse the old offsets. Assert nonempty old results and both unset group offsets before a fresh read, then check that a fresh group completion succeeds without an automatic commit.

Blocking on automatic commit replication could deadlock a test with a paused replica send. Return an owned poll future and let the owner pump accept the read and continue replication. Exercise the helper with replication paused, then verify its commit reaches every replica after release. Update the simulator demo to drive the future and register its client before sending requests.

Exercise history validation after a completed group read passes through the real completion sender, inbox and owner pump. Replace the partition with new data at the same offsets and verify stale rejection preserves both group offsets before a fresh completion succeeds through the same route. Share resident message setup with the timeout test. Disabling the history check in an isolated copy makes the regression fail on stale last-polled progress.

A completion arriving after its caller times out could advance progress and make the next poll skip messages the caller never received. Check the reply connection before owner admission. Update the timeout regression to require unchanged progress, no queued or assigned automatic commit, and redelivery of the unseen messages. Document that cancellation racing with the check remains an unknown outcome. Validated with all 99 shard tests, strict shard Clippy, formatting checks, and the shard documentation build.

Recreating the shutdown receive on every pump turn repeats backoff and receive registration while the channel is empty. Pin the receive once and preserve its priority in the owner select loop. Cover shutdown wakeup after an ordinary completion, draining queued completions, and disconnection of the shutdown sender.

Missing owner replies used HTTP status 504 as the JSON error ID, hiding the Iggy error reported by binary transports. Keep HTTP 504 and render ShardCommunicationError with ID 11001. Cover timeout and TransientNotAccepted responses in HTTP serialization tests.

Stale poll completions triggered reservation reconciliation before their history was rejected, doing maintenance for a result that cannot be accepted. Check history and partition availability before reconciliation. Add a regression that preserves pending accounting after stale rejection and verifies a fresh completion still reconciles before admitting its commit.

History identity should remain unique without depending on allocation lifetime. A checked process counter prevents reuse when a partition is recreated. Represent histories as copied u64 values and allocate IDs only at the existing history creation points. Cover concurrent creation and counter exhaustion with correctness tests.

Preserve the owner completion path while incorporating the current master changes. The conflict was an upstream documentation correction inside the obsolete poll handler removed by this branch.

Completed reads should retain room to return to their owner when unrelated inbox traffic is saturated. Reserve a dedicated completion slot before disk I/O and retain it until dequeue or discard. Bound running reads plus queued results with the independent poll_completion_capacity setting, service both lanes fairly, and release reservations during shutdown and cancellation.

Disk polls carried unrelated metric handles, and partition offset state retained shared ownership that is no longer needed. Share frame drop metrics through existing completion lane state and own the durable offset table and capacity trackers directly. Document recovery from lost poll replies using a checkpoint for each partition. Validate with workspace Clippy, 509 affected unit tests and Rustdoc link checks. Formatting and TOML checks pass.

Make the minimum completion capacity explicit in fixtures and explain why one slot is sufficient.

spetz

Some final comments before it's ready to merge, could you also please confirm that the initial reports still hold after all the improvements?

An accepted poll can own an assigned prepare that has not reached the journal. Require its continuation to run immediately on the owning pump and assert that the partition remains available instead of silently losing the prepare. Report unexpected poll replies accurately and retire the obsolete automatic commit frame drop metric. Cover the continuation invariant with a regression that fails under the previous silent drop behavior.

spetz

LGTM, thank you for all the work!

numinnex · 2026-09-14T10:24:11Z

/dst

This was referenced Sep 10, 2026

feat(cluster)!: let topics require durable acks and flatten config #4092

Merged

feat(bench): add poll measurements and comparison tooling #4122

Draft

diegomrsantos force-pushed the fix/owner-poll-completion branch from b0f7638 to c4e45d4 Compare September 10, 2026 15:47

diegomrsantos and others added 7 commits September 11, 2026 10:11

test(partitions): explain purge poll regression schedule

a8e7650

Document why the reproduction delays completion, reuses message offsets, and covers an old offset within the fresh range. Clarify that the tests observe automatic commit candidates before server admission.

test(partitions): add context to purge poll test comments

131e357

Explain the old poll and fresh messages before discussing reused offsets, so the comments can be understood without tracing the helper first.

fix(partitions): complete polls on the owning shard

0c53871

diegomrsantos force-pushed the fix/owner-poll-completion branch from c4e45d4 to d89b81f Compare September 11, 2026 08:22

diegomrsantos mentioned this pull request Sep 11, 2026

Purge can record completion after consumer offset directory sync fails #4128

Open

diegomrsantos marked this pull request as ready for review September 11, 2026 08:39

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Sep 11, 2026

diegomrsantos and others added 2 commits September 11, 2026 12:15

Merge branch 'master' into fix/owner-poll-completion

e416b6c

diegomrsantos added 2 commits September 11, 2026 15:14

Merge branch 'master' into fix/owner-poll-completion

c46067f

spetz requested changes Sep 11, 2026

View reviewed changes

diegomrsantos added 2 commits September 11, 2026 21:07

spetz and others added 4 commits September 11, 2026 22:22

Merge branch 'master' into fix/owner-poll-completion

33218df

test(shard): characterize poll completion after timeout

ad9fc76

spetz requested changes Sep 11, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Sep 11, 2026

diegomrsantos added 4 commits September 12, 2026 00:41

Merge branch 'master' into fix/owner-poll-completion

0d862e3

hubcio reviewed Sep 12, 2026

View reviewed changes

Comment thread core/partitions/src/iggy_partition.rs

hubcio reviewed Sep 12, 2026

View reviewed changes

Comment thread core/shard/src/poll.rs Outdated

diegomrsantos and others added 7 commits September 12, 2026 22:22

chore(server): merge master into owner poll completion

cfc6b5d

Preserve the owner completion path while incorporating the current master changes. The conflict was an upstream documentation correction inside the obsolete poll handler removed by this branch.

Merge branch 'master' into fix/owner-poll-completion

e7f06ff

refactor(shard): name completion capacities in test fixtures

b4fe6fa

Make the minimum completion capacity explicit in fixtures and explain why one slot is sufficient.

Merge branch 'master' into fix/owner-poll-completion

ebe6465

spetz requested changes Sep 13, 2026

View reviewed changes

Comment thread core/server/src/dispatch/partition.rs

Comment thread core/partitions/src/iggy_partitions.rs Outdated

Comment thread core/shard/src/metrics.rs Outdated

diegomrsantos and others added 2 commits September 13, 2026 10:38

Merge branch 'master' into fix/owner-poll-completion

8f1c0e2

spetz approved these changes Sep 13, 2026

View reviewed changes

hubcio approved these changes Sep 13, 2026

View reviewed changes

spetz merged commit af237a1 into apache:master Sep 13, 2026
103 checks passed

github-actions Bot removed the S-waiting-on-author PR is waiting on author response label Sep 13, 2026

diegomrsantos mentioned this pull request Sep 14, 2026

Disk polling: performance investigation and optimization tracking #4174

Open

5 tasks

Conversation

diegomrsantos commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation

Verification

Performance

Metric retirement

Uh oh!

hubcio commented Sep 10, 2026

Uh oh!

hubcio commented Sep 10, 2026

Uh oh!

codecov Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

diegomrsantos commented Sep 10, 2026

Uh oh!

hubcio commented Sep 11, 2026

Uh oh!

diegomrsantos commented Sep 11, 2026

Uh oh!

diegomrsantos commented Sep 11, 2026

Uh oh!

diegomrsantos commented Sep 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spetz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spetz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

numinnex commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

diegomrsantos commented Sep 10, 2026 •

edited

Loading

codecov Bot commented Sep 10, 2026 •

edited

Loading