Skip to content

refactor: separate cache lifecycle, runtime, and persistence - #60

Merged
tisonkun merged 15 commits into
mainfrom
codex/simplify-region-store
Sep 21, 2026
Merged

tisonkun merged 15 commits into
mainfrom
codex/simplify-region-store

Conversation

@tisonkun

@tisonkun tisonkun commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Separate cache lifecycle, request execution, live L2 state, and persistence into CacheSession, CacheRuntime, RegionStore, and RegionPersistence. Remove the single-implementation RegionBackend trait and the FileRegionRuntime forwarding wrapper, and move modules, methods, fields, and fixtures to match their actual roles.

Split positioned I/O from file ownership and filesystem path operations. DataFileHandles now implements positioned I/O directly. Use RegionDesc and ReadDesc, consistent append/read/write vocabulary, and distinct names for persistent session state and background I/O recovery. Document the resulting ownership and naming boundaries.

Design Notes

  • Keep public APIs, configuration, snapshots, persistent encodings, resource bounds, and recovery semantics unchanged relative to main. RUNNING remains durable before workers start; CLEAN remains the final publication after quiescence and a durable image. Unfenced writes retain file ownership.
  • Keep fault injection at actual file and filesystem operations, with session fixtures following the production startup path.
  • Keep FillController and IoRecovery as independent runtime-owned policies. BackgroundIoAttempt borrows both through observation, recovery, validation, and publication. Flush budgets explicitly count records and byte units separately from observed I/O requests.

Validation: cargo x check, cargo x test (including extended recovery, process-kill, and fill/recovery fence tests), cargo x lint, and cargo check --target x86_64-unknown-linux-gnu --features io-uring --all-targets.

Narrow PositionedIo to byte-offset reads and writes, and move extent management, durability, locking, identity, and mapping access into StorageFile. Rename the concrete file resources and their counters to describe the owned descriptors and file I/O paths.

Implement positioned I/O directly on DataFileHandles and remove RuntimeFileBackend, including its unsupported lock methods. Update driver state, variables, fixtures, and test doubles to the same vocabulary without changing dispatch or buffer ownership.

Validation: workspace all-target checks, 276 library tests, repository lint, and Linux io-uring cross-target check.
Move path operations to io::fs and name the OS implementation OsFileSystem. RegionPaths now consistently denotes paths; DataFileHandles denotes descriptors for one open data file.

Move descriptor cloning onto StorageFile and remove the filesystem capability that silently skipped runtime startup. Fault-injecting files now follow the same startup path and can fail descriptor cloning at the file boundary.

Validation: all-target workspace check, 276 library tests, and both extended recovery qualification tests.
Use RegionStore for the live L2 index, region allocation, and record operations. Move the open/close coordinator to cache::session as CacheSession, and update its owning field and call sites.

Name per-region records RegionDesc and read descriptions ReadDesc, including their variables and methods. Use RegionState, AppendStaging, AppendShardGate, AppendWorkerControl, and RegionManagerLock for their concrete roles. Replace the single-field region-access wrapper with the same atomic generation array.

Validation: 276 library tests and all-target workspace checks. Persistent state discriminants, layouts, synchronization, and public APIs are unchanged.
Move L1/L2 request coordination and background workers into CacheRuntime.
CacheSession now orders recovery, the durable RUNNING barrier, worker
startup, shutdown, and CLEAN publication explicitly.

Narrow RegionPersistence to file ownership and recovery image publication.
Move live-state construction and freezing into RegionStore, remove the
FileRegionRuntime forwarding wrapper, and pass validated configuration
through the session instead of retaining it in the persistence layer.

Preserve admission fences, write completion ordering, health checks, and
file-lock retention when issued writes cannot be fenced. Update fixtures
to exercise the same startup and shutdown path as production.
Name the shared timeout controller IoRecovery and its per-operation fence
IoRecoveryAttempt. Place the protocol under io so engines and Region
workers share it without implying that it belongs to one engine.

Name the persisted EMPTY/RUNNING/CLEAN enum SessionState and update the
associated calls and locals. Keep state encodings, recovery deadlines,
admission fences, public health values, and diagnostic event names intact.
Use AppendShardState for each shard's reservations, staged span, and
rotation state. Name submitted writes PendingWrite and pair them with
WriteCompletion and WriteSubmitError, matching the existing read path.

Update fields, locals, submission calls, and fixtures without changing
buffer ownership, completion validation, or index publication ordering.
Remove the duplicate Unix import guard left by folding the positioned-I/O
wrapper into DataFileHandles and group the shared recovery import with
the other I/O capabilities. Retain the kernel's registered-files wording
in the SQPOLL diagnostic.
Document the session/runtime/store/persistence boundaries and distinguish
Regions, append shards, and index partitions. Explain the remaining I/O
capabilities, resource ownership, and separate restart and timeout states.

Update the close-path description and use Desc in the contribution naming
conventions so future changes follow the same domain vocabulary.
Integrate adaptive fill admission from 13a2dc6 into the renamed runtime
and I/O recovery modules. Preserve fill pressure observations, worker
pacing, reclaim suppression, and the new public configuration/snapshots.

Adapt upstream fixtures to CacheSession and PositionedIo while retaining
their behavioral assertions. Validate all workspace and extended recovery
tests, lint, feature checks, and Linux io-uring cross-compilation.
@tisonkun tisonkun changed the title refactor: remove the region lifecycle backend trait refactor: separate cache lifecycle, runtime, and persistence Sep 20, 2026
Distinguish record credits from completed I/O operations throughout fill
control. Name budget acquisition and refund explicitly, and distinguish
foreground fill admission from background flush pacing.

Use limits for retained FillLimits and max_flush_bytes for the burst floor
supplied by Region geometry. Preserve the packed-credit representation,
refill arithmetic, oversized-flush handling, and public configuration.
Let CacheRuntime own FillController and IoRecovery independently. Replace
the broad recovery attempt with BackgroundIoAttempt, which borrows both
policies for admission, slow-I/O observation, timeout recovery, and final
publication. Move this coordination to io/background.rs and pass each
attempt into Region writes instead of exposing the runtime's policies.

Keep timeout and cancellation behavior, recovery fences, and completion
ordering intact. Centralize stopping both policies in the runtime, remove
the unused observation timeout argument, and name I/O observations and
background waits by their role.

Exercise recovery in Disabled, Observe, and Adaptive modes; verify that
post-close completion cannot reopen fills and failed publication keeps
recovery fenced after its I/O observation is released.
Describe independent fill control and timeout recovery, the per-operation
BackgroundIoAttempt, and the runtime's coordinated stop boundary. Distinguish
record and byte flush budgets from observed I/O request counts.
Include da279f2's NVMe qualification scripts, benchmark depth and read
amplification reporting, and tuning documentation. The update does not
change library ownership or background I/O policy boundaries.
@tisonkun

Copy link
Copy Markdown
Contributor Author

@leiysky could you review the ownership boundaries and naming below, especially the lifecycle/persistence split and the I/O interfaces?

This summarizes the final migration against main. Public APIs, configuration, snapshot fields, and on-disk formats remain unchanged.

Main responsibilities

Before After Responsibility
RegionStore<B: RegionBackend> CacheSession Orders open, recovery, startup, shutdown, and durable state publication.
RegionDataPlane CacheRuntime Executes L1/L2 requests and owns I/O engines, staging, and background workers; moved from region/runtime/ to cache/runtime/.
FileRegionCore RegionStore Holds the live L2 index, Region manager, append gates, generation checks, and health latch.
FileRegionBackend RegionPersistence Owns files and locks, loads recovery state, and persists recovery images.
FileRegionRuntime Removed Its wrapper and forwarding responsibilities are absorbed by the concrete types above.
RegionBackend Removed There is one concrete Region storage implementation; lifecycle no longer needs a backend protocol.

The old and new RegionStore have different responsibilities: the old lifecycle coordinator becomes CacheSession; the new name replaces FileRegionCore.

flowchart TD
    C[Cache] -->|requests| R[CacheRuntime]
    C -->|close| S[CacheSession]
    S -->|start / shutdown| R
    S -->|recover / persist| P[RegionPersistence]
    R -->|L2 operations| L[RegionStore]
    R --> W[L1 / staging / workers / I/O engines]
    L --> I[RegionIndex]
    L --> M[RegionManager]
Loading

Method ownership

  • FileRegionRuntime::install → RegionStore::from_recovery.
  • anonymous_runtime / map_clean_runtime → RegionPersistence::cold_regions / recover_regions; these construct L2 state, not a complete runtime.
  • start_runtime / attach_data_plane → CacheSession directly calls CacheRuntime::start.
  • stop_fast / freeze_warm → CacheSession orders CacheRuntime::shutdown, then, for warm close, RegionStore::freeze → RegionPersistence::persist_frozen → publish_clean.

The barriers remain: durable RUNNING before workers start; durable data and recovery image before CLEAN. An unfenced issued write still prevents CLEAN publication and requires retaining the lock.

I/O and domain names

Before After Meaning
IoBackend PositionedIo Offset-based reads/writes only.
ControlIoBackend + file-management methods from IoBackend StorageFile Adds size, allocation, sync, locking, identity, and handle duplication.
FileBackend / RuntimeFileSet CacheFile / DataFileHandles Owned OS file vs. engine handles; the forwarding RuntimeFileBackend wrapper is removed.
RegionFileSystem / SystemRegionFileSystem FileSystem / OsFileSystem Path operations, moved to io/fs.rs. Handle duplication moves to StorageFile.
RegionRuntime / ReadDescriptor RegionDesc / ReadDesc Descriptive data, rather than execution objects.
RegionMetadataState RegionState The same Free/Active/Sealed state applies to live and persisted Regions.
RegionStaging / RegionShard / ShardMutation AppendStaging / AppendShardGate / AppendShardState Distinguishes append buffers, synchronization, and mutation state.
RegionSpanFlight / RegionSpanCompletion PendingWrite / WriteCompletion Matches PendingRead / ReadCompletion; submit_span becomes submit_write.

The narrow I/O interfaces remain useful for capability boundaries and fault injection.

RecoveryState → SessionState clarifies that EMPTY/RUNNING/CLEAN describes the durable cache open/close lifecycle, not recovery progress. Values and encoding are unchanged; CLEAN remains only a prerequisite for validating a restart image.

Background timeout recovery is separate: BackgroundRecovery → IoRecovery no longer owns fill control. CacheRuntime owns IoRecovery and FillController independently, while RecoveryAttempt → BackgroundIoAttempt borrows both through observation, validation, and publication. wait_with_recovery becomes wait_background.

Fill-control methods now state their actual operation: try_flush / refund_flush → try_acquire_flush_budget / refund_flush_budget. Budget ops / units become records / byte_units, distinguishing record credits from observed I/O request counts. Pacing and recovery policies are preserved.

Suggested review order: cache/session.rs → cache/runtime/mod.rs → region/mod.rs → region/persistence/mod.rs, then io/file.rs and io/background.rs.

@tisonkun
tisonkun requested a review from leiysky September 21, 2026 05:53
@tisonkun

Copy link
Copy Markdown
Contributor Author

@leiysky I'm going to merge this PR but you may take a look to avoid concept mismatch or further conflicts.

Integrate io_uring configuration guidance, shared benchmark defaults, and availability diagnostics from #62. The changes merge without conflicts with the lifecycle and I/O refactoring.

Validation: cargo x check, cargo x test, cargo x lint, and Linux workspace checks with no default features and all features.
@tisonkun
tisonkun enabled auto-merge (squash) September 21, 2026 06:54
@leiysky leiysky mentioned this pull request Sep 21, 2026
4 tasks
@tisonkun
tisonkun merged commit 99bb3b8 into main Sep 21, 2026
8 checks passed
@tisonkun
tisonkun deleted the codex/simplify-region-store branch September 21, 2026 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants