test: qualify Linux NVMe read depth without hiding amplification

BENCHMARK.md

@@ -45,7 +45,7 @@ The main controls are grouped below. See `benchmarks/cache/main.rs` for defaults
 
 Request and latency collection are controlled independently by `CACHE_BENCH_REQUEST_STATS` (complete request counters, default false), `CACHE_BENCH_L1_LATENCY_SAMPLE_INTERVAL`, `CACHE_BENCH_L2_LATENCY_SAMPLE_INTERVAL`, and `CACHE_BENCH_MUTATION_LATENCY_SAMPLE_INTERVAL` (independent: 0 off, 1 full, greater values sample with that mean interval; default 0), `CACHE_BENCH_IO_LATENCY` (full engine latency, default false), and `CACHE_BENCH_STATS_SHARDS` (default 16). These instrument the library itself. `CACHE_BENCH_READ_LATENCY_SAMPLE_INTERVAL` remains the independent benchmark observer and should be held constant across comparisons. The effective statistics settings are printed with each run. Compare disabled, counters-only, full and sampled modes on the same workload, keeping actual hit/overload populations in view.
 
-`CACHE_BENCH_ACTIVITY_COUNTERS=true` adds cache accounting on the measured request path. Use the same setting for baseline and candidate runs.
+`CACHE_BENCH_ACTIVITY_COUNTERS=true` adds cache accounting on the measured request path. Use the same setting for baseline and candidate runs. The `result phase=read_io` line then also reports `in_flight_peak`, `configured_in_flight`, `slot_wait_ns`, `payload_bytes`, and `read_amp` (engine read bytes divided by served payload bytes).
 
 For device measurements, use a data set larger than host RAM and no larger than half of L2 capacity. Run baseline and candidate in alternating order at least five times, compare medians, and retain every sample. Throughput does not replace correctness, overload, memory, or latency checks.
 
@@ -116,7 +116,7 @@ Repeat sizes in `CACHE_SOAK_VALUE_BYTES` to weight a production distribution. Us
 
 Optional fill admission uses `_FILL_CONTROL=disabled|observe|adaptive` under the same three prefixes. Enabled modes require explicit `_FILL_BYTES_PER_SECOND` and `_FILL_OPERATIONS_PER_SECOND` ceilings. The harness prints the effective options and adds a `type=fill_control` report beside cache records. Compare the modes in alternating order with identical ceilings and traffic; record rejected and hypothetical fills alongside completed throughput. High ceilings help measure instrumentation overhead, while lower ceilings and injected storage stalls exercise admission behavior. Buffered macOS results do not qualify Linux NVMe or cgroup throttling.
 
-Select the backend with `_IO_ENGINE=posix|io-uring`. POSIX workers bound concurrent operations. io_uring ring count and aggregate in-flight limit are independent; changing one does not rewrite the other. Ring count must not exceed the in-flight limit. IOPOLL requires `_IO_MODE=direct`; SQPOLL CPU requires an idle timeout. Only the selected backend's settings are read. Each harness prints the resulting `IoEngineOptions`, and buffer estimates use its actual concurrency. The request benchmark's default read-wait capacity follows the selected read pool's in-flight limit.
+Select the backend with `_IO_ENGINE=posix|io-uring`. POSIX workers bound concurrent operations. io_uring ring count and aggregate in-flight limit are independent; changing one does not rewrite the other. Ring count must not exceed the in-flight limit. IOPOLL requires `_IO_MODE=direct`; SQPOLL CPU requires an idle timeout. Only the selected backend's settings are read. Each harness prints the resulting `IoEngineOptions`, and buffer estimates use its actual concurrency. The request benchmark's default read-wait capacity follows the selected read pool's in-flight limit. POSIX Immediate L2 reads still cap `CACHE_BENCH_CLIENTS` at the POSIX read-worker count, so a worker sweep must set clients at least as high as the worker count.
 
 `CACHE_BENCH_STATS` is now `CACHE_BENCH_ACTIVITY_COUNTERS`. Machine-readable reports use `version=2`, renaming the cache record field `statistics_enabled` to `activity_counters_enabled`; the counter population is unchanged.
 
@@ -132,7 +132,7 @@ Capacity variables follow the library's terms: `_CAPACITY_MIB` is L2, `_L1_CAPAC
   /var/tmp/cache2-qualification
 ```
 
-The runner records the revision, machine, filesystem, block device, complete configuration, raw logs, medians, and checksums. It exercises buffered and direct POSIX I/O, a worker-count sweep, mixed turnover, and final warm recovery.
+The runner records the revision, machine, filesystem, block device, complete configuration, raw logs, medians, and checksums. It exercises buffered and Direct POSIX I/O, a POSIX Direct worker sweep with matching client depth, an experimental io_uring Direct in-flight sweep (one ring at 16/32/64 concurrent gets), mixed turnover, and final warm recovery. Qualification builds with `--features io-uring` and sets `CACHE_BENCH_ACTIVITY_COUNTERS=true` so `result phase=read_io` records in-flight peak, slot wait, payload bytes, and read amplification. IOPOLL is an optional follow-up (`CACHE_BENCH_IO_URING_READ_IOPOLL=true` with Direct I/O), not part of the default matrix.
 
 A release pass requires:
 

CONFIGURATION.md

@@ -200,7 +200,7 @@ runtime.read_admission = ReadAdmission::Wait {
 
 A full wait queue, memory pressure, or deadline expiry returns explicit `ErrorKind::Overloaded`. Making the timeout longer does not enlarge the queue and cannot compensate for severely undersized execution capacity. If busy misses or queue-full overloads dominate, add execution capacity or reduce L2 demand. If memory misses dominate, adding capacity can make the problem worse; add managed headroom, reduce retained buffers, or improve L1 promotion instead.
 
-Size POSIX workers from peak concurrent L2 reads, not total requests. For a strict-hit bias, provision close to that peak and use immediate admission or a very short wait. For a memory/thread bias, use fewer workers plus a short deadline and accept explicit overload. Validate p99 as well as hit rate.
+Size POSIX workers from peak concurrent L2 reads, not total requests. For io_uring, size `max_in_flight` to that same concurrent depth and keep ring count low until driver CPU is the measured limit. For a strict-hit bias, provision close to that peak and use immediate admission or a very short wait. For a memory/thread bias, use fewer workers plus a short deadline and accept explicit overload. Validate p99 as well as hit rate.
 
 Direct I/O expands reads to 4 KiB boundaries within the selected Region. Small or unaligned records therefore need more buffer and device bytes than their payload size suggests. Include that amplification when setting memory headroom and read concurrency.
 
@@ -361,6 +361,17 @@ For low overhead start with counters and, if needed, full L2 lookup and I/O late
 - Reserve managed-memory headroom for concurrent aligned L2 reads and for Region-backed values retained by callers.
 - Validate direct-I/O read amplification if Direct mode is used.
 
+### Linux NVMe Direct I/O
+
+Use this profile when the data set exceeds host RAM and the goal is device behavior rather than page-cache behavior.
+
+- Keep Buffered POSIX as the portable production default; switch to Direct only after measuring amplification and warm-close cost on the target filesystem.
+- Size POSIX `read_workers` to the application's concurrent L2 get depth. Worker count is both thread count and admission depth.
+- For experimental io_uring, start with one read ring and set `max_in_flight` to that same concurrent get depth. Extra rings do not add execution slots.
+- Keep write execution modest; more write workers cannot exceed sequential device fill.
+- Profile IOPOLL separately with `IoUringPoolOptions::io_poll` and Direct I/O on a polling-capable filesystem; do not enable it from ring or in-flight sweeps.
+- Watch `requests_in_flight_peak`, `slot_wait_ns`, `l2_read_busy_misses`, and Direct versus served-byte amplification instead of throughput alone.
+
 ## Diagnostic map
 
 Use `Cache::snapshot()` for regular telemetry and `Cache::detailed_snapshot()` for periodic diagnosis.
@@ -379,6 +390,8 @@ Use `Cache::snapshot()` for regular telemetry and `Cache::detailed_snapshot()` f
 | `reinsert_skipped` rises while `reinsert_budget_skipped` is zero    | Reinsertion staging/validation pressure           | Reduce concurrent reclaim or foreground pressure; do not increase the byte budget indirectly |
 | `reinsert_budget_skipped` rises                                     | Hot live bytes exceed the fixed reclaim allowance | Treat retention as best effort; change capacity/workload geometry rather than worker count   |
 | High `l1_bypasses` with useful candidates                           | L1 contention, slots, or byte pressure            | Inspect L1 occupancy, retained bytes, shards, capacity, and oversized values                 |
+| Read in-flight peak stays below configured capacity                 | Caller concurrency below engine depth             | Match concurrent L2 gets to `read_workers` or `max_in_flight`; extra rings add no slots      |
+| Direct read bytes much larger than `served_bytes`                   | Size-class and 4 KiB alignment amplification      | Treat `read_amp` as device work, not a miss; raise entry size or accept extra bytes          |
 | Managed-memory peak approaches the limit                            | Fixed or transient memory pressure                | Rebalance index, L1, staging, I/O topology, and read headroom                                |
 | `fill_control.pressure` is `Paused` while `health` is `Running`     | Pre-timeout slow background I/O                   | Expected Adaptive load-shed; correlate with device queueing before treating it as a fault    |
 | Pause refusals with little device wait                              | Checkpoint fired without device saturation        | Confirm I/O timeouts and host scheduling; `FillLimits` do not drive pause                    |

README.md

@@ -89,7 +89,7 @@ Changing the append-shard count rebinds recovered Active Regions during a warm o
 
 The default `ReadAdmission::Immediate` returns a miss under read-engine or buffer pressure. `ReadAdmission::Wait` enables a queue bounded by `max_waiters` and a positive `timeout`. Queued requests retain their read descriptor and allocate a buffer after admission. Queue saturation, memory pressure, and timeout return explicit overload.
 
-Buffered POSIX I/O is the production path. Direct I/O is an explicit Linux mode. io_uring requires the `io-uring` feature and remains experimental; its ring count and aggregate in-flight limit are independent. SQPOLL and IOPOLL are explicit per-pool opt-ins: SQPOLL adds kernel submission polling with configurable idle time and optional CPU affinity, while IOPOLL adds completion polling and requires direct I/O on polling-capable storage.
+Buffered POSIX I/O is the production path. Direct I/O is an explicit Linux mode. io_uring requires the `io-uring` feature and remains experimental; its ring count and aggregate in-flight limit are independent, so size `max_in_flight` (or POSIX `read_workers`) to concurrent L2 gets rather than adding rings. SQPOLL and IOPOLL are explicit per-pool opt-ins: SQPOLL adds kernel submission polling with configurable idle time and optional CPU affinity, while IOPOLL adds completion polling and requires direct I/O on polling-capable storage.
 
 ### Platform support
 

benchmarks/cache/main.rs

@@ -351,7 +351,7 @@ async fn run(config: BenchConfig) -> io::Result<()> {
     println!("C² cache benchmark");
     println!("stats={:?}", config.stats);
     println!(
-        "entries={} index_slots={} index_load={:.1}% resident_entries={} hot_entries={} hot_read_interval={} value={} B data={:.1} MiB l1_capacity={:.1} MiB initial_l1={:.1} MiB managed_memory_limit={:.1} MiB append_shards={} read_wait_capacity={} read_wait_timeout_us={} read_latency_sample_interval={} write_clients={} read_clients={} l2_clients={} l1_entry_eligible={} l1_eviction={:?} engine={:?} mode={:?} activity_counters={}",
+        "entries={} index_slots={} index_load={:.1}% resident_entries={} hot_entries={} hot_read_interval={} value={} B data={:.1} MiB l1_capacity={:.1} MiB initial_l1={:.1} MiB managed_memory_limit={:.1} MiB append_shards={} configured_read_in_flight={} read_wait_capacity={} read_wait_timeout_us={} read_latency_sample_interval={} write_clients={} read_clients={} l2_clients={} l1_entry_eligible={} l1_eviction={:?} engine={:?} mode={:?} activity_counters={}",
         config.entries,
         index_slots,
         index_load,
@@ -364,6 +364,7 @@ async fn run(config: BenchConfig) -> io::Result<()> {
         initial_l1_bytes as f64 / MIB as f64,
         config.managed_memory_limit_bytes as f64 / MIB as f64,
         config.append_shards,
+        read_max_in_flight(config.io_engine),
         config.read_io_wait_capacity,
         config.read_io_wait_timeout.as_micros(),
         config.read_latency_sample_interval,
@@ -579,13 +580,30 @@ async fn run(config: BenchConfig) -> io::Result<()> {
             snapshot.l1_hits,
             snapshot.l1_misses,
         );
+        let payload_bytes = snapshot.served_bytes;
+        let io_bytes = snapshot
+            .io
+            .read
+            .buffered
+            .bytes
+            .saturating_add(snapshot.io.read.direct.bytes);
+        let read_amp = if payload_bytes == 0 {
+            0.0
+        } else {
+            io_bytes as f64 / payload_bytes as f64
+        };
         println!(
-            "result phase=read_io requests={} buffered_operations={} buffered_bytes={} direct_operations={} direct_bytes={} busy_misses={} memory_misses={}",
+            "result phase=read_io requests={} buffered_operations={} buffered_bytes={} direct_operations={} direct_bytes={} in_flight_peak={} configured_in_flight={} slot_wait_ns={} payload_bytes={} read_amp={:.3} busy_misses={} memory_misses={}",
             snapshot.io.read.requests_succeeded,
             snapshot.io.read.buffered.operations,
             snapshot.io.read.buffered.bytes,
             snapshot.io.read.direct.operations,
             snapshot.io.read.direct.bytes,
+            snapshot.io.read.requests_in_flight_peak,
+            read_max_in_flight(config.io_engine),
+            snapshot.io.read.slot_wait_ns,
+            payload_bytes,
+            read_amp,
             snapshot.l2_read_busy_misses,
             snapshot.l2_read_memory_misses,
         );

scripts/qualify-linux-nvme.sh

@@ -228,6 +228,8 @@ record_command() {
   echo "backing_source=$backing_source"
   echo "device_qualification=$device_qualification"
   echo "performance_gates_configured=$performance_gate_count/4"
+  echo "benchmark_features=io-uring"
+  echo "benchmark_activity_counters=true"
   echo "[performance-gates]"
   for gate in "${performance_gates[@]}"; do
     echo "$gate=${!gate:-unset}"
@@ -319,29 +321,44 @@ run_benchmark_profile() {
   local read_workers=$4
   local write_workers=$5
   local runs=$6
+  shift 6
+  local extra_env=("$@")
   local log=$report_directory/benchmark-$profile.log
   local run
   for ((run = 1; run <= runs; run++)); do
     echo "profile=$profile run=$run/$runs"
-    CACHE_BENCH_DIR="$cache_directory" \
-    CACHE_BENCH_IO_ENGINE="$engine" \
-    CACHE_BENCH_IO_MODE="$mode" \
-    CACHE_BENCH_POSIX_READ_WORKERS="$read_workers" \
-    CACHE_BENCH_POSIX_WRITE_WORKERS="$write_workers" \
-      cargo +1.98.0 bench --locked --package benchmarks --bench cache --quiet 2>&1 | tee -a "$log"
+    env \
+      CACHE_BENCH_DIR="$cache_directory" \
+      CACHE_BENCH_IO_ENGINE="$engine" \
+      CACHE_BENCH_IO_MODE="$mode" \
+      CACHE_BENCH_POSIX_READ_WORKERS="$read_workers" \
+      CACHE_BENCH_POSIX_WRITE_WORKERS="$write_workers" \
+      CACHE_BENCH_ACTIVITY_COUNTERS=true \
+      "${extra_env[@]}" \
+      cargo +1.98.0 bench --locked --package benchmarks --bench cache --features io-uring \
+      --quiet 2>&1 | tee -a "$log"
   done
   summarize_profile "$profile" "$runs" "$log"
 }
 
 echo "building release benchmark targets"
-cargo +1.98.0 build --locked --release --package benchmarks --benches
+cargo +1.98.0 build --locked --release --package benchmarks --benches --features io-uring
 
 run_benchmark_profile posix-buffered posix buffered 4 4 "$benchmark_runs"
 run_benchmark_profile posix-direct posix direct 4 4 "$benchmark_runs"
 
 for workers in 1 2 4 8 16; do
   run_benchmark_profile "posix-direct-workers-$workers" \
-    posix direct "$workers" "$workers" 1
+    posix direct "$workers" "$workers" 1 \
+    "CACHE_BENCH_CLIENTS=$workers"
+done
+
+for inflight in 16 32 64; do
+  run_benchmark_profile "io-uring-direct-inflight-$inflight" \
+    io-uring direct 1 4 1 \
+    "CACHE_BENCH_CLIENTS=$inflight" \
+    "CACHE_BENCH_IO_URING_READ_RINGS=1" \
+    "CACHE_BENCH_IO_URING_READ_MAX_IN_FLIGHT=$inflight"
 done
 
 echo "starting ${soak_seconds}s POSIX/buffered turnover soak"
@@ -353,7 +370,8 @@ CACHE_SOAK_FINAL_WARM_VERIFY=true \
 CACHE_SOAK_REQUIRE_PATH_COVERAGE=true \
 CACHE_SOAK_IO_ENGINE=posix \
 CACHE_SOAK_IO_MODE=buffered \
-  cargo +1.98.0 bench --locked --package benchmarks --bench cache_soak --quiet 2>&1 \
+  cargo +1.98.0 bench --locked --package benchmarks --bench cache_soak --features io-uring \
+  --quiet 2>&1 \
   | tee "$report_directory/soak-posix-buffered.log"
 
 if ! grep -q '^warm_verification .* errors=0$' \