A size read from a file or a stream no longer decides an allocation

[torstei]

From the full-read sweep: places where a length taken from a file or a stream sized a buffer before anything checked it.

Problem

• A chunk's stored and uncompressed lengths came from the .rings record and sized a buffer; zstd::stream::decode_all had no output limit. One corrupt record, or one hostile frame (a megabyte of zeros is a few dozen bytes), was a multi-gigabyte allocation and an abort.
• A records stream's len= sized the entry payload the same way, and a header with no terminating NUL was read without limit (read_until(0)).

Change

Chunk reads (query, store, grain build and extend, import): format::read_frame refuses a frame that would run past the end of the trunk, or that no chunk could be (over 4 GiB), before it allocates; format::decode_frame refuses output beyond the uncompressed length the index records. read_chunk_raw keeps its shape: the read still happens before the seqlock race check, so a read that fails mid-collapse stays retryable rather than becoming an error.

Record streams (records reader, timber-filter, the consumer protocol): records::read_payload takes a payload as it arrives instead of reserving the claim, so a stream that claims more than it delivers fails short; records::read_header stops at 1 MiB, far past any metadata record.

Behaviour changes

• A chunk whose frame expands to more than its index records is now an error. Fewer is unchanged (the store's own equality check still applies).
• A records header over 1 MiB is refused.
• An empty frame still decodes to nothing, as decode_all did; my first draft refused it and a test caught that.

Checked with the CLI

With one chunk's index record patched:

comp_len = 2^50:    Error: reading a stored chunk
                    Caused by: chunk 0 records 1125899906842624 stored bytes, which no chunk has — the index may be corrupt
uncomp_len = 2^40:  Error: decompressing a stored chunk — the .trunk may be corrupt
                    Caused by: a chunk records 1099511627776 bytes, which no chunk has

Both exit 1 instead of attempting the allocation.

Tests

read_frame: a frame where the index says, past the end, a start beyond the file, a length no chunk has, an overflowing range, just over the bound. decode_frame: exact, a megabyte-of-zeros bomb refused when the index says less, a claim no chunk has, garbage, empty. read_payload and read_header: a claim far larger than the stream, exact and short, an endless header; an entry claiming 9 TB through the real Reader is a truncated stream.

Not here

The newline-free input caps, the intake tick and the import verify/overlap bounds are separate changes.

🤖 Generated with Claude Code