Track the performance investigations following #4119, starting with excess read bytes and repeated batch processing. Owner service, tail latency and task construction need fresh diagnostics before choosing an optimization.
Read sizing and batch processing can share counters. Owner service and tail latency can share stage timestamps. Each issue should reach its own measurement and decision; finding that an optimization is not worthwhile is a valid outcome.
The latest report and raw evidence found higher mean poll time with concurrent writes and lower mean time for resident reads. The other mean intervals span zero, which does not establish equivalence.
| Case |
Baseline µs per poll |
Candidate µs per poll |
Mean time change |
Pointwise 95% interval |
| Explicit offset |
258.42 |
258.67 |
+0.10% |
[−1.35%, +1.69%] |
| Explicit offset with writes |
291.93 |
298.71 |
+2.32% |
[+0.41%, +4.36%] |
| Next |
303.86 |
302.40 |
−0.48% |
[−1.41%, +0.44%] |
| Consumer group |
330.16 |
331.63 |
+0.45% |
[−0.59%, +1.45%] |
| Resident |
133.44 |
128.41 |
−3.77% |
[−5.27%, −2.38%] |
Positive changes mean more elapsed time per completed poll. The comparison covers the whole candidate and does not identify the cause of the difference under writes. Tail results and differences between campaigns are in #4178. Historical copy, checksum and task construction evidence is retained beside the relevant investigation in #4175, #4176 and #4179.
Start by inspecting the archived run, resource and producer records, then collect a small fixed diagnostic set on current code with and without writes. Share counts of bytes read, bytes hashed, batches visited and pump turns, together with:
read start → completion → enqueue → owner acceptance → caller resumption
For each proposed change, compare one revision with the same revision plus only that change. Keep instrumentation separate from clean timing, record exact binary identities, validate returned work, and measure elapsed time, CPU and memory. Assess variation with identical binaries where needed and declare a worthwhile effect and fixed budget before confirmation. Coordinate representative Linux workloads with existing benchmark work.
Preserve output, integrity, durability and owner acceptance contracts, including offsets, timestamps, contiguous results and consumer progress. For independent arrival workloads, account for scheduling delays, errors and timeouts: a sequential client reduces its load when polling slows. Keep write progress and other poll modes as guardrails.
The tracker closes when every child has a documented result and decision or is explicitly removed from scope.
Shared evidence, methods and workload details
Latest timing campaign
Baseline 585ab692210b4e900cdc1153e75842378a33370e; candidate c7e6fb1655bd48c18c2fb567917f88ec1bdf7fcc. Twelve independently launched pairs per case used balanced randomized order. All 120 measured runs completed 2,304,000 nonempty polls without recorded errors, timeouts or cancellations. The separate 20 run pilot is excluded.
Mean estimates are ratios of arithmetic means. Pointwise percentile intervals use 20,000 bootstrap resamples of whole pairs, without adjustment across cases or secondary metrics. The fixed sample was not extended according to its results.
Historical CPU profiles used in #4175, #4176 and #4179
Baseline dc2b382097ed7fb6cfeb8a31fed0fbc74af4433e; candidate a699ff91c6b9126502ea112ac676d96d8d107975. Two captures per revision and workload each completed 33,000 polls, with opposite version orders. Linux perf sampled software cpu-clock at 499 Hz; sample periods were normalized by completed polls. Tables report arithmetic means across captures.
These are diagnostic CPU estimates, not elapsed latency or predicted savings. Profiling can perturb execution, and no confidence intervals are assigned to this small set. The detailed reports and raw captures remain archived locally; relevant observations are reproduced in the children. They are separate from the published timing archive. #4177 records a different diagnostic with its own revision and sample size.
Workloads and environment
All these workloads used one shard, one sequential TCP consumer and payloads of 256 bytes. Explicit offset polls returned one message without automatic commits. In the latest campaign, Next and resident polls returned one message with automatic commits; group polls returned 100. Explicit offset runs measured 33,000 polls, and the other latest cases measured 10,000. Separate fixtures supplied 100 warmup polls. Preparation, preload and shutdown were outside the consumer window.
The producer targeted 50,000,000 payload bytes per second in batches of 100. In the latest campaign it achieved 50,154,102.84 to 50,505,824.75 bytes per second, within the 95–105% guard. Written messages per completed poll averaged 57.45 for baseline and 58.81 for candidate. Mixed server CPU includes serving writes and is not consumer CPU alone.
Measurements ran in Docker Desktop's ARM64 Linux VM on an M1 Pro MacBookPro18,1, with server work pinned to CPU 0 and fixtures on a guest volume. Reads could hit the OS cache. The latest disk cases used replicated durability, a flush threshold of one, 16 MiB segments and disabled preallocation. These results do not establish cold storage latency, physical disk traffic, multishard scaling or behavior under many concurrent consumers.
Exact offsets were validated for explicit offset cases. Raw Next and group records do not establish exact returned offsets or payload identity; correctness tests provide separate evidence.
Source scope
Child source links are pinned to merged PR head 8f1c0e260076f9fbe49539e904e43895bc24c3fe. Shutdown receive reuse, reserved completion capacity, numeric history IDs and ownership simplifications are already present there. Historical profiles do not measure their remaining costs.
Track the performance investigations following #4119, starting with excess read bytes and repeated batch processing. Owner service, tail latency and task construction need fresh diagnostics before choosing an optimization.
Read sizing and batch processing can share counters. Owner service and tail latency can share stage timestamps. Each issue should reach its own measurement and decision; finding that an optimization is not worthwhile is a valid outcome.
The latest report and raw evidence found higher mean poll time with concurrent writes and lower mean time for resident reads. The other mean intervals span zero, which does not establish equivalence.
Positive changes mean more elapsed time per completed poll. The comparison covers the whole candidate and does not identify the cause of the difference under writes. Tail results and differences between campaigns are in #4178. Historical copy, checksum and task construction evidence is retained beside the relevant investigation in #4175, #4176 and #4179.
Start by inspecting the archived run, resource and producer records, then collect a small fixed diagnostic set on current code with and without writes. Share counts of bytes read, bytes hashed, batches visited and pump turns, together with:
read start → completion → enqueue → owner acceptance → caller resumption
For each proposed change, compare one revision with the same revision plus only that change. Keep instrumentation separate from clean timing, record exact binary identities, validate returned work, and measure elapsed time, CPU and memory. Assess variation with identical binaries where needed and declare a worthwhile effect and fixed budget before confirmation. Coordinate representative Linux workloads with existing benchmark work.
Preserve output, integrity, durability and owner acceptance contracts, including offsets, timestamps, contiguous results and consumer progress. For independent arrival workloads, account for scheduling delays, errors and timeouts: a sequential client reduces its load when polling slows. Keep write progress and other poll modes as guardrails.
The tracker closes when every child has a documented result and decision or is explicitly removed from scope.
Shared evidence, methods and workload details
Latest timing campaign
Baseline
585ab692210b4e900cdc1153e75842378a33370e; candidatec7e6fb1655bd48c18c2fb567917f88ec1bdf7fcc. Twelve independently launched pairs per case used balanced randomized order. All 120 measured runs completed 2,304,000 nonempty polls without recorded errors, timeouts or cancellations. The separate 20 run pilot is excluded.Mean estimates are ratios of arithmetic means. Pointwise percentile intervals use 20,000 bootstrap resamples of whole pairs, without adjustment across cases or secondary metrics. The fixed sample was not extended according to its results.
Historical CPU profiles used in #4175, #4176 and #4179
Baseline
dc2b382097ed7fb6cfeb8a31fed0fbc74af4433e; candidatea699ff91c6b9126502ea112ac676d96d8d107975. Two captures per revision and workload each completed 33,000 polls, with opposite version orders. Linux perf sampled softwarecpu-clockat 499 Hz; sample periods were normalized by completed polls. Tables report arithmetic means across captures.These are diagnostic CPU estimates, not elapsed latency or predicted savings. Profiling can perturb execution, and no confidence intervals are assigned to this small set. The detailed reports and raw captures remain archived locally; relevant observations are reproduced in the children. They are separate from the published timing archive. #4177 records a different diagnostic with its own revision and sample size.
Workloads and environment
All these workloads used one shard, one sequential TCP consumer and payloads of 256 bytes. Explicit offset polls returned one message without automatic commits. In the latest campaign, Next and resident polls returned one message with automatic commits; group polls returned 100. Explicit offset runs measured 33,000 polls, and the other latest cases measured 10,000. Separate fixtures supplied 100 warmup polls. Preparation, preload and shutdown were outside the consumer window.
The producer targeted 50,000,000 payload bytes per second in batches of 100. In the latest campaign it achieved 50,154,102.84 to 50,505,824.75 bytes per second, within the 95–105% guard. Written messages per completed poll averaged 57.45 for baseline and 58.81 for candidate. Mixed server CPU includes serving writes and is not consumer CPU alone.
Measurements ran in Docker Desktop's ARM64 Linux VM on an M1 Pro MacBookPro18,1, with server work pinned to CPU 0 and fixtures on a guest volume. Reads could hit the OS cache. The latest disk cases used replicated durability, a flush threshold of one, 16 MiB segments and disabled preallocation. These results do not establish cold storage latency, physical disk traffic, multishard scaling or behavior under many concurrent consumers.
Exact offsets were validated for explicit offset cases. Raw Next and group records do not establish exact returned offsets or payload identity; correctness tests provide separate evidence.
Source scope
Child source links are pinned to merged PR head
8f1c0e260076f9fbe49539e904e43895bc24c3fe. Shutdown receive reuse, reserved completion capacity, numeric history IDs and ownership simplifications are already present there. Historical profiles do not measure their remaining costs.