refactor: separate cache lifecycle, runtime, and persistence - #60
Conversation
Narrow PositionedIo to byte-offset reads and writes, and move extent management, durability, locking, identity, and mapping access into StorageFile. Rename the concrete file resources and their counters to describe the owned descriptors and file I/O paths. Implement positioned I/O directly on DataFileHandles and remove RuntimeFileBackend, including its unsupported lock methods. Update driver state, variables, fixtures, and test doubles to the same vocabulary without changing dispatch or buffer ownership. Validation: workspace all-target checks, 276 library tests, repository lint, and Linux io-uring cross-target check.
Move path operations to io::fs and name the OS implementation OsFileSystem. RegionPaths now consistently denotes paths; DataFileHandles denotes descriptors for one open data file. Move descriptor cloning onto StorageFile and remove the filesystem capability that silently skipped runtime startup. Fault-injecting files now follow the same startup path and can fail descriptor cloning at the file boundary. Validation: all-target workspace check, 276 library tests, and both extended recovery qualification tests.
Use RegionStore for the live L2 index, region allocation, and record operations. Move the open/close coordinator to cache::session as CacheSession, and update its owning field and call sites. Name per-region records RegionDesc and read descriptions ReadDesc, including their variables and methods. Use RegionState, AppendStaging, AppendShardGate, AppendWorkerControl, and RegionManagerLock for their concrete roles. Replace the single-field region-access wrapper with the same atomic generation array. Validation: 276 library tests and all-target workspace checks. Persistent state discriminants, layouts, synchronization, and public APIs are unchanged.
Move L1/L2 request coordination and background workers into CacheRuntime. CacheSession now orders recovery, the durable RUNNING barrier, worker startup, shutdown, and CLEAN publication explicitly. Narrow RegionPersistence to file ownership and recovery image publication. Move live-state construction and freezing into RegionStore, remove the FileRegionRuntime forwarding wrapper, and pass validated configuration through the session instead of retaining it in the persistence layer. Preserve admission fences, write completion ordering, health checks, and file-lock retention when issued writes cannot be fenced. Update fixtures to exercise the same startup and shutdown path as production.
Name the shared timeout controller IoRecovery and its per-operation fence IoRecoveryAttempt. Place the protocol under io so engines and Region workers share it without implying that it belongs to one engine. Name the persisted EMPTY/RUNNING/CLEAN enum SessionState and update the associated calls and locals. Keep state encodings, recovery deadlines, admission fences, public health values, and diagnostic event names intact.
Use AppendShardState for each shard's reservations, staged span, and rotation state. Name submitted writes PendingWrite and pair them with WriteCompletion and WriteSubmitError, matching the existing read path. Update fields, locals, submission calls, and fixtures without changing buffer ownership, completion validation, or index publication ordering.
Remove the duplicate Unix import guard left by folding the positioned-I/O wrapper into DataFileHandles and group the shared recovery import with the other I/O capabilities. Retain the kernel's registered-files wording in the SQPOLL diagnostic.
Document the session/runtime/store/persistence boundaries and distinguish Regions, append shards, and index partitions. Explain the remaining I/O capabilities, resource ownership, and separate restart and timeout states. Update the close-path description and use Desc in the contribution naming conventions so future changes follow the same domain vocabulary.
Integrate adaptive fill admission from 13a2dc6 into the renamed runtime and I/O recovery modules. Preserve fill pressure observations, worker pacing, reclaim suppression, and the new public configuration/snapshots. Adapt upstream fixtures to CacheSession and PositionedIo while retaining their behavioral assertions. Validate all workspace and extended recovery tests, lint, feature checks, and Linux io-uring cross-compilation.
Distinguish record credits from completed I/O operations throughout fill control. Name budget acquisition and refund explicitly, and distinguish foreground fill admission from background flush pacing. Use limits for retained FillLimits and max_flush_bytes for the burst floor supplied by Region geometry. Preserve the packed-credit representation, refill arithmetic, oversized-flush handling, and public configuration.
Let CacheRuntime own FillController and IoRecovery independently. Replace the broad recovery attempt with BackgroundIoAttempt, which borrows both policies for admission, slow-I/O observation, timeout recovery, and final publication. Move this coordination to io/background.rs and pass each attempt into Region writes instead of exposing the runtime's policies. Keep timeout and cancellation behavior, recovery fences, and completion ordering intact. Centralize stopping both policies in the runtime, remove the unused observation timeout argument, and name I/O observations and background waits by their role. Exercise recovery in Disabled, Observe, and Adaptive modes; verify that post-close completion cannot reopen fills and failed publication keeps recovery fenced after its I/O observation is released.
Describe independent fill control and timeout recovery, the per-operation BackgroundIoAttempt, and the runtime's coordinated stop boundary. Distinguish record and byte flush budgets from observed I/O request counts.
Include da279f2's NVMe qualification scripts, benchmark depth and read amplification reporting, and tuning documentation. The update does not change library ownership or background I/O policy boundaries.
|
@leiysky could you review the ownership boundaries and naming below, especially the lifecycle/persistence split and the I/O interfaces? This summarizes the final migration against Main responsibilities
The old and new flowchart TD
C[Cache] -->|requests| R[CacheRuntime]
C -->|close| S[CacheSession]
S -->|start / shutdown| R
S -->|recover / persist| P[RegionPersistence]
R -->|L2 operations| L[RegionStore]
R --> W[L1 / staging / workers / I/O engines]
L --> I[RegionIndex]
L --> M[RegionManager]
Method ownership
The barriers remain: durable RUNNING before workers start; durable data and recovery image before CLEAN. An unfenced issued write still prevents CLEAN publication and requires retaining the lock. I/O and domain names
The narrow I/O interfaces remain useful for capability boundaries and fault injection.
Background timeout recovery is separate: Fill-control methods now state their actual operation: Suggested review order: |
|
@leiysky I'm going to merge this PR but you may take a look to avoid concept mismatch or further conflicts. |
Integrate io_uring configuration guidance, shared benchmark defaults, and availability diagnostics from #62. The changes merge without conflicts with the lifecycle and I/O refactoring. Validation: cargo x check, cargo x test, cargo x lint, and Linux workspace checks with no default features and all features.
Summary
Separate cache lifecycle, request execution, live L2 state, and persistence into
CacheSession,CacheRuntime,RegionStore, andRegionPersistence. Remove the single-implementationRegionBackendtrait and theFileRegionRuntimeforwarding wrapper, and move modules, methods, fields, and fixtures to match their actual roles.Split positioned I/O from file ownership and filesystem path operations.
DataFileHandlesnow implements positioned I/O directly. UseRegionDescandReadDesc, consistent append/read/write vocabulary, and distinct names for persistent session state and background I/O recovery. Document the resulting ownership and naming boundaries.Design Notes
FillControllerandIoRecoveryas independent runtime-owned policies.BackgroundIoAttemptborrows both through observation, recovery, validation, and publication. Flush budgets explicitly count records and byte units separately from observed I/O requests.Validation:
cargo x check,cargo x test(including extended recovery, process-kill, and fill/recovery fence tests),cargo x lint, andcargo check --target x86_64-unknown-linux-gnu --features io-uring --all-targets.