feat(drive): ttl ephemeral-bytes fee reclassification — processing-priced, flagless, refundless

book/src/drive/time-range-ttl.md

@@ -67,10 +67,13 @@ That single property pays off three times:
 
 - `ttl` is an optional key of the `timeRange` map, in seconds, parsed
   into the transform. It is **not part of the grid identity**:
-  [`TimeRangeTransform::storage_key`] excludes it, so declaring or
-  changing a TTL never forks the storage level, and query-side grid
+  [`TimeRangeTransform::storage_key`] excludes it and query-side grid
   matching ([`TimeRangeGridSpec`]) continues to compare
-  `(range, step, phase)` only.
+  `(range, step, phase)` only. Like the rest of the transform it is
+  **immutable once the contract is registered**: contract updates reject
+  any change, including adding or removing a TTL — entries written before
+  a TTL carry storage flags, and an ephemeral level must stay flagless
+  (refund-free).
 - **`ttl ≥ range`.** `$createdAt` is consensus-assigned from block time,
   so writes only ever target windows containing *now*; this invariant
   guarantees no bucket that can still receive entries (or serve as the
@@ -95,6 +98,14 @@ TTL'd index continues drainage of the oldest expired bucket (start
 `SystemLimits::max_time_range_ttl_drop_operations_per_write` O(1) drop
 operations and resuming exactly where the previous write's budget ran
 out. When nothing is expired, the check is a single bounded range read.
+The walkers only *request* the drain; it runs once the state
+transition's own batch has applied — one drain per level, however many
+indexes share it or how many documents the transition carries — so it
+can never remove a subtree a queued removal of the same transition still
+targets. Its flat drops are collected and applied as one grovedb batch
+(a single root-hash propagation per drain); only the indexed-tree
+deletes under ranked levels, which have no batched form, execute
+immediately.
 The operation count of a full bucket scales with its distinct groups,
 and write volume scales with group volume, so drainage keeps pace
 roughly one window behind; after a quiet spell the backlog amortizes
@@ -151,7 +162,8 @@ A time-range bucket is *not* flat, so the platform drains it
    prefixes when ranked);
 4. the emptied bucket is flat-dropped.
 
-Every step is O(1); the *number* of steps scales with the window's
+Every step is O(1) in the subtree it drops (the batched drops share one
+root-hash propagation); the *number* of steps scales with the window's
 distinct groups, and that count is what
 `SystemLimits::max_time_range_ttl_drop_operations_per_write` bounds.
 **Every write** into a TTL'd index continues drainage where the previous
@@ -186,6 +198,17 @@ there is nothing to refund, which is also where TTL writers collectively
 pre-pay the drainage described below. Cost estimation routes through the
 same split, so estimated and actual fees stay in the same class.
 
+Two consequences show up in a fee breakdown. Per byte, an ephemeral
+write costs 400 (the ordinary written-byte processing rate,
+`storage_processing_credit_per_byte`) + 270 = **670 processing credits**,
+against 400 processing + 27,000 storage for an ordinary byte. And because
+the ephemeral batch is applied *second*, the ancestor chain it shares
+with the standing batch (document-type tree up through the contract
+tree, contracts root and root key) is re-opened, re-hashed and re-written
+once more per state transition; that second propagation is billed as
+ephemeral processing — roughly 0.2–0.25M credits per TTL'd write — in
+both the estimate and the actual, so `estimated >= actual` still holds.
+
 Drainage itself and the walkers' TTL bookkeeping reads are **unbilled**:
 their costs go to scratch accounting, never to the triggering user. That
 is load-bearing for the `estimated >= actual` fee invariant — the

packages/rs-dpp/src/data_contract/document_type/index/time_range.rs

@@ -94,9 +94,13 @@ pub struct TimeRangeTransform {
     /// means entries live forever, exactly as before the key existed.
     ///
     /// Deliberately **not part of the grid identity**: [`Self::storage_key`]
-    /// excludes it, so a TTL never forks the storage level, and query-side
-    /// grid matching (`TimeRangeGridSpec`) compares `(range, step, phase)`
-    /// only. Contract validation requires `ttl >= range` (a window still
+    /// excludes it and query-side grid matching (`TimeRangeGridSpec`)
+    /// compares `(range, step, phase)` only. Like the rest of the transform
+    /// it is nonetheless immutable once the contract is registered: contract
+    /// updates reject any change, including adding or removing a TTL,
+    /// because entries written before a TTL carry storage flags and an
+    /// ephemeral level must stay flagless (refund-free). Contract validation
+    /// requires `ttl >= range` (a window still
     /// able to receive consensus-timestamped writes, or to serve as the
     /// `oldest` selector's window, can never expire) and caps it at
     /// `SystemLimits::max_time_range_ttl_seconds` — the cap is what makes

packages/rs-dpp/src/data_contract/document_type/index_level/find_first_change.rs

@@ -197,15 +197,24 @@ impl IndexLevel {
     /// dictates how many index entries each document produces and under
     /// which bucket keys, so changing it after creation would leave already
     /// stored documents indexed under stale buckets — it is immutable.
+    /// That includes `ttl`, which the storage key leaves out: entries
+    /// written before a TTL carry storage flags, and an ephemeral level
+    /// must stay flagless.
     ///
     /// Returns `None` if the transform is the same everywhere.
     #[cfg(feature = "validation")]
     pub(super) fn find_first_time_range_change(&self, new: &IndexLevel) -> Option<String> {
         if self.time_range() != new.time_range() {
             let fmt = |t: Option<&super::TimeRangeTransform>| match t {
                 Some(t) => format!(
-                    "Some(on: {:?}, range: {}s, step: {}s, phase: {}s)",
-                    t.source, t.range_seconds, t.step_seconds, t.phase_seconds
+                    "Some(on: {:?}, range: {}s, step: {}s, phase: {}s, ttl: {})",
+                    t.source,
+                    t.range_seconds,
+                    t.step_seconds,
+                    t.phase_seconds,
+                    t.ttl_seconds
+                        .map(|ttl| format!("{}s", ttl))
+                        .unwrap_or_else(|| "None".to_string()),
                 ),
                 None => "None".to_string(),
             };

packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs

@@ -2159,4 +2159,84 @@ mod tests {
             );
         }
     }
+
+    /// A time-range `ttl` is immutable across contract updates even though
+    /// it is not part of the grid identity — old and new resolve to the
+    /// same `storage_key` level, so the diff helper must compare the whole
+    /// transform. Entries written before a TTL carry storage flags, and an
+    /// ephemeral level must stay flagless.
+    #[test]
+    fn should_return_invalid_result_if_time_range_ttl_changed() {
+        let platform_version = PlatformVersion::latest();
+        let document_type_name = "test";
+
+        let index_with_ttl = |ttl_seconds: Option<u64>| Index {
+            name: "test".to_string(),
+            properties: vec![IndexProperty {
+                name: "$createdAt".to_string(),
+                ascending: false,
+            }],
+            unique: false,
+            null_searchable: true,
+            contested_index: None,
+            countable: IndexCountability::NotCountable,
+            range_countable: false,
+            summable: None,
+            range_summable: false,
+            ranked_countable: false,
+            ranked_countable_at: vec![],
+            ranked_summable: false,
+            ranked_averageable: false,
+            time_range: Some(TimeRangeTransform {
+                source: "$createdAt".to_string(),
+                range_seconds: 3600,
+                step_seconds: 3600,
+                phase_seconds: 0,
+                ttl_seconds,
+            }),
+            terminal: None,
+            preallocated: false,
+            skip_if_absent: false,
+        };
+        let fmt_ttl = |ttl: Option<u64>| {
+            ttl.map(|ttl| format!("{}s", ttl))
+                .unwrap_or_else(|| "None".to_string())
+        };
+
+        for (old_ttl, new_ttl) in [
+            (None, Some(86_400)),
+            (Some(86_400), None),
+            (Some(86_400), Some(172_800)),
+        ] {
+            let old_index_structure = IndexLevel::try_from_indices(
+                &[index_with_ttl(old_ttl)],
+                document_type_name,
+                platform_version,
+            )
+            .expect("failed to create old index level");
+            let new_index_structure = IndexLevel::try_from_indices(
+                &[index_with_ttl(new_ttl)],
+                document_type_name,
+                platform_version,
+            )
+            .expect("failed to create new index level");
+
+            let result =
+                old_index_structure.validate_update(document_type_name, &new_index_structure);
+
+            let expected_path = format!(
+                "$createdAt#3600#3600 -> (timeRange: \
+                 Some(on: \"$createdAt\", range: 3600s, step: 3600s, phase: 0s, ttl: {}) -> \
+                 Some(on: \"$createdAt\", range: 3600s, step: 3600s, phase: 0s, ttl: {}))",
+                fmt_ttl(old_ttl),
+                fmt_ttl(new_ttl),
+            );
+            assert_matches!(
+                result.errors.as_slice(),
+                [ConsensusError::BasicError(
+                    BasicError::DataContractInvalidIndexDefinitionUpdateError(e)
+                )] if e.index_path() == expected_path
+            );
+        }
+    }
 }

packages/rs-drive-abci/src/abci/handler/finalize_block.rs

@@ -110,10 +110,21 @@ where
         .flush_pending_prefix_drops(&platform_version.drive.grove_version)
     {
         Ok(report) => {
-            if report.reclaimed_records > 0 || report.skipped_live > 0 {
-                tracing::debug!(
+            // A skipped record means a dropped path was re-created before
+            // its reclamation ran — a violation of the flat-drop path-reuse
+            // contract that leaks the old prefix data until the path is
+            // dropped again, so it is worth a warning rather than a debug
+            // line.
+            if report.skipped_live > 0 {
+                tracing::warn!(
                     reclaimed_records = report.reclaimed_records,
                     skipped_live = report.skipped_live,
+                    "flushed pending prefix drops; some dropped paths were live again and \
+                     were skipped"
+                );
+            } else if report.reclaimed_records > 0 {
+                tracing::debug!(
+                    reclaimed_records = report.reclaimed_records,
                     "flushed pending prefix drops"
                 );
             }