Size the SSD tier's memtable for a cache, not for a write-heavy database

[bjmeetsfo]

The SSD tier sets a 64 MB memtable. RocksDB reserves each write-ahead log at roughly 1.1x the memtable size, so every cache instance reserves a 70.4 MB log file — regardless of how much is actually cached. A store split across three prefixes opens three instances, so it reserved 211 MB of blocks for a cache holding 8 MB of data.

The difference is invisible to anything that measures written bytes: the same store reports 35.6 MB apparent and 269.2 MB allocated. du sees it; stat's size field does not.

Measured downstream on a real ingest path, 400 adds of a 4 KB payload, with the before arm run twice:

memtable	bytes written	blocks allocated	add p50
64 MB (before)	25.9 MB	262.5 MB	125.4 ms
8 MB (after)	27.9 MB	79.6 MB	116.8 ms
64 MB (repeat)	28.3 MB	264.7 MB	118.4 ms

70% less reserved disk. The two before arms land 0.8% apart, so the effect is far outside the noise, and latency is flat — the 8 MB arm sits below both controls. Bytes actually written are unchanged, which is the point: this is space reserved and never used.

Why not simply turn the log off

A cache is rebuildable, so journalling it looks like pure waste. It is not safe here. In the consumer, a hot page can live only in the cache until it is evicted or dumped; an entry evicted before the next flush used to read back as missing, which is an acknowledged write disappearing, and a spill handler exists specifically to close that hole. Dropping the log would reopen it.

So the log stays and only its reservation shrinks. set_sync(false) is unchanged and the log is still written on every admission: crash behaviour is byte-for-byte what it was. This change is about reserved space, not durability.

Shape

The size becomes a field with a default plus a setter, following the existing SetCapacity shape, so a write-heavy tier that would rather flush less often can ask for more. Only the default moves.

One note for reviewers: this file declares StorageEngineRocksDb across four separate blocks and contains three unrelated SetCapacity methods on three different structs. Anchoring edits by string search puts the setter on the wrong struct — it compiles until it reaches the missing field, and fails there. The edits here are anchored inside located line ranges for that reason.

Tests

318 + 6 pass, 0 failed, cargo check --lib clean.