feat(drive): ttl ephemeral-bytes fee reclassification — processing-priced, flagless, refundless

[QuantumExplorer]

Issue being fixed or feature implemented

Stacked on #4581 (time-range TTL). That PR shipped the lifecycle — lazy budgeted drainage on grovedb's flat-subtree drop — but TTL'd index bytes still billed at the perpetuity storage rate, which defeats the point: windowed data with a hard one-week life cap was prepaying decades of retention. This PR is the economic payoff the design describes: TTL'd bytes bill to processing, carry no storage flags, and refund nothing.

What was done?

Fee tables (rs-platform-version)

• FeeStorageVersion gains ttl_ephemeral_disk_usage_credit_per_byte. FEE_STORAGE_VERSION2 prices it at 270 credits/byte — 1% of the 27,000 storage rate, ~27× a pro-rata week of epoch-distributed retention, so it's a safe over-charge, not a subsidy. V1 carries 0 (unreachable pre-PV14: the ttl grammar does not parse).
• FEE_VERSION3 = FEE_VERSION2 + the new storage table, wired into PV14. It keeps fee_version_number: 1 deliberately — the persisted number tags the refund algorithm, which is unchanged (same aliasing precedent as FEE_VERSION2).

Ephemeral operation class (rs-drive::fees::op)

• New LowLevelDriveOperation variants: EphemeralGroveOperation and CalculatedEphemeralCostOperation. Grove ops normally collapse into ONE batch whose cost is consumed as a unit, so ephemeral ops ride a second grovedb batch (apply_batch_low_level_drive_operations splits 3-way); its captured cost is consumed on its own terms: storage_fee = 0, processing_fee += added_bytes × 270 (checked arithmetic). A SectionedStorageRemoval surfacing in an ephemeral batch is a CorruptedCodeExecution — TTL'd elements have no flags, so refundable removal there means a classification bug.
• Estimation routes through the same split with the same layer info, so estimated >= actual holds per class (regression-tested).

Walker routing — the insert (top v2), delete (top v2), and update (v1 bucketed branch) walkers detect ttl on a sub-level, collect that sub-level's ops into a local vec, retag them ephemeral, and pass None storage flags down (actual writes and estimation layers both).

Reference walker v1 (the one real subtlety): add_reference_for_index_level_for_contract_operations v0 builds the terminal reference element from the document info's own flags, ignoring the flags the walker passed down. Those historically diverge (immutable doctypes: walker passes None, element still gets flags), so v0 is kept verbatim for replay and a v1 — terminal ref takes the walker's flags — is wired into DRIVE_DOCUMENT_METHOD_VERSIONS_V4 (PV14-only, table unreleased). The v1 update walker's prebuilt reference is likewise rebuilt flagless on the ephemeral branch. Without this, TTL'd references carry flags and their removal turns sectioned/refundable.

Docs — book/src/drive/time-range-ttl.md fee sections rewritten from "planned" to shipped; meta-schema ttl description now states the fee semantics.

How Has This Been Tested?

New ttl_index_bytes_bill_to_processing_without_refunds: a TTL'd contract, its standing twin (identical minus ttl), and an index-free twin, all receiving the same document with owner-carrying flags:

• TTL insert storage fee exactly equals the index-free contract's — index bytes contribute zero storage;
• processing strictly exceeds the standing twin's (the ephemeral rate lands there);
• TTL delete refunds exactly equal the index-free contract's, strictly below the standing twin's — flagless ephemeral bytes refund nothing;
• estimation stays an upper bound in both fee classes through the split batch.

Full drive lib suite (3563 green — the v1 reference walker touches every PV14 index insert), drive-abci time-range proof + ranked batteries, cargo check --all-targets on drive/drive-abci/queries, fmt, clippy.

Breaking Changes

None released — everything is gated on still-unreleased PV14 (FEE_VERSION3, document method versions v4). v0 walker behavior is preserved verbatim for historical replay.

Checklist

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation
• I have added "!" to the title and described breaking changes in the corresponding section if my code contains any

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: 32d7fa5d-55b4-4e17-b93d-a7fc4122fee6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📖 Book Preview built successfully.

Download the preview from the workflow artifacts.
To view locally: download the artifact, unzip, and open index.html.

Updated at 2026-09-02T08:49:06.140Z

[PastaPastaPasta]

Pushed the fixes from an xhigh-effort review of #4581 + #4583 (10 finder angles, one verifier per finding, then a gap sweep), merged on top of the branch's newer "drain once per write, before queuing" commits.

Which drain fix to keep

The branch's drain_expired_time_range_levels sweep and the review's post-batch drain both fix the per-index budget multiplication and the intra-document ordering (index B's drain dropping paths index A's queued removals targeted). They differ on what else they cover:

	pre-queue sweep (branch)	post-batch requests (this push)
per-index budget → per level	yes	yes
ordering within one document	yes	yes
ordering across documents of one batch transition	no	yes
delete-only writes drain	yes	yes (kept)
drops batched (one propagation per drain)	no, one commit per drop	yes

The cross-document case is the one that matters: a documents batch [delete D, create E] converts every transition first and applies one batch at the end, so a later transition's drain still runs after an earlier transition's removals were queued, wherever the drain sits inside the walker. ttl_removals_queued_before_a_draining_write_in_one_batch_still_apply reproduces it with five filler groups so the delete's own budget stops short of D's group: on adc87a05 alone it fails with

GroveDB(InvalidPath("could not get key for parent of subtree for batch at path […/hashtag/zz] for key 0x00"))

and drive-abci would turn that into an InternalError that strips the valid transition unpaid. With the requests run after the batch it passes. So the merge keeps the post-batch design and folds in the branch's delete-walker drain (as a request) and its two tests, with two fixture adjustments described below.

Correctness

• Drainage no longer races the transition's own batch. The walkers emit a TimeRangeTtlDrainRequest (a new LowLevelDriveOperation variant, one per TTL'd level via request_expired_time_range_drains) instead of draining inline; apply_batch_low_level_drive_operations runs the requests after both grovedb batches, deduplicated per level. See the table above.
• Flat drops are batched. The drain collects its DeleteTree(DropFlat) ops and applies them as one grovedb batch (one root-hash propagation per drain instead of one per drop; each drop was a standalone commit with full propagation, ~1–2M credits of unbilled work). Only the indexed-tree deletes under ranked levels, which have no batched form, stay immediate, and everything beneath such a node is removed immediately too so ordering holds.
• FeeVersionFieldsBeforeVersion4 keeps its frozen storage layout. The new ttl_ephemeral_disk_usage_credit_per_byte field had changed the bincode layout of the pre-1.4 platform-state shape; real testnet V0 bytes failed with UnexpectedEnd, and should_deserialize_state_stored_in_version_0_from_testnet would fail once Rust CI runs on this branch (it has not: the base is not a v*-dev branch). Added FeeStorageVersionBeforeVersion14 per the feat: addresses on the Platform chain #2866 precedent; the fixture test passes.
• The insert walker filters expired buckets (live_time_range_index_keys), like the update walker already did. award_document_to_winner re-inserts the contested document with its contest-start $createdAt; on mainnet the poll is two weeks and the TTL cap one week, so that insert always landed in an expired bucket and could re-create a path with a pending prefix-drop record (grovedb's skipped_live leak). finalize_block now warns, not debug-logs, when a flush skips a live path.
• Ephemeral ops are visible to the batch bookkeeping. batch_insert_empty_tree_if_not_exists dedup and the batch_delete_up_tree_while_empty existing-ops fold matched GroveOperation only, so retagged EphemeralGroveOperations were invisible; the update walker also allocated its local vec per index, so two indexes sharing a TTL'd level could not see each other's deletes and left an empty bucket standing. One shared vec per walk now, and grove_op_ref() matches both classes.
• Legacy partition helpers are ephemeral-aware. grovedb_operations_batch, _batch_consume, grovedb_operations_consume, combine_cost_operations and _with_leftovers silently dropped ephemeral ops (only a debug_assert! guarded one of them). Latent today, but a future DocumentOperation routed through convert_drive_operations_to_grove_operations would have written the row and lost every TTL'd index entry in release builds.
• flags_len in the insert walker's stateless bucket-tree probe used the document's flags instead of index_storage_flags (over-estimate only).

Cleanup

• One expiry predicate: time_range_entry_state returns Live / ExpiredStanding / ExpiredGone built on TimeRangeTransform::bucket_expired (which had no callers); the removal walkers no longer re-derive the horizon inline, and expired_entry_path_exists starts below the bucket the state check already probed.
• One ephemeral routing helper, LowLevelDriveOperation::with_ephemeral_routing / push_retagged_ephemeral, replaces the three copy-pasted blocks; the update walker builds its reference once with the level's flags.
• add_reference_for_index_level_for_contract_operations v0 and v1 share one body parameterised by TerminalReferenceFlagsSource; the dispatcher reports known_versions: [0, 1].
• SubtreePath::from(&[Vec<u8>]) replaces the Vec<&[u8]> conversions in the ttl module.

Docs and tests

• ttl is documented as immutable after registration (the previous wording implied it could be declared on a live grid); find_first_time_range_change now prints the ttl values, and should_return_invalid_result_if_time_range_ttl_changed pins the rejection, which is what keeps flagged pre-TTL entries out of ephemeral levels.
• The book's fee section records the second-batch ancestor re-propagation surcharge (billed as ephemeral processing, in both estimate and actual) and the 670-vs-27,400 per-byte comparison; the cleanup section describes the post-batch, per-level, batched drain.
• ttl_partial_drain_resumes_across_writes_and_removals_stay_exact asserted the bucket was gone after a write whose pruning had already removed it; it now uses 14 groups so the bucket demonstrably survives the deletes and only the next write's drain finishes it. ttl_shared_grid_drains_once_per_write gained a survivor document, since the improved pruning otherwise empties the bucket before the drain is measured.
• PR title lowercased to pass the semantic-title check.

Verified

cargo test -p drive --lib (3566 passed), the TTL e2e suite (24), drive-abci time_range (20) and ranked (18), rs-dpp index_level and time_range suites, platform-version fee tests, clippy and rustfmt clean on drive, drive-abci, dpp and platform-version.

Not done: the ttl_index_bytes_bill_to_processing_without_refunds processing assertion is still only > (the surcharge alone satisfies it), and the TTL matrix still lacks unique, indexOnly, null-timestamp and overlapping-window cases.

---

🤖 Posted autonomously by Claude on behalf of pasta.

[QuantumExplorer]

Folded into #4581 — the base branch was fast-forwarded to this branch's head (adc87a054b), so every commit here (the fee reclassification, the reference-walker v1, and the fee tests) now ships in #4581 as one PR. Closing.

🤖 Generated with Claude Code