A crash partway through an append stopped the metaserver coming back

[bjmeetsfo]

The metadata log is append-only text, one record per line. A process that dies partway through writing one leaves a partial line at the end. That is what a crash looks like, not corruption.

Reading the log refused the whole file on any line that did not parse:

let record = serde_json::from_str::<MetaMutationRecord>(&line).map_err(io::Error::other)?;

and starting a metaserver passes that straight up (with_mutation_log -> log.load()? -> .transpose()? in the backend). So the process failed to start. A metaserver whose metadata was entirely durable except for a fraction of the last record would not come up at all.

Why dropping the last record is safe

It was never acknowledged. A writer only returns once a sync covering its bytes has completed, so nothing was ever promised to anybody about the record still being written. Dropping it loses nothing that was reported durable, and everything before it is kept.

Why the middle is different

A line that does not parse with records after it is not a torn tail. Those later records were acknowledged, and stopping at the damage would discard them silently. That is still refused, and the error says why it is not simply a crash:

meta mutation log <path>: line 2 does not parse and 1 more line(s) follow it,
so it is not a torn tail: <parse error>

The distinction is the whole point: the tail is expected, the middle is not.

Not silent either way

The dropped partial record is logged with the file, the line number, how many records were recovered, and what failed to parse. A metaserver that came back one record short says so.

Testing

• A log cut off mid-record brings back every shard and server that was acknowledged before the crash.
• A log damaged in the middle is refused rather than half-read, and the error names why.

Both were checked against a mutation:

mutation	which test fails
refuse any bad line (the old behaviour)	the torn-tail test, on "a torn last record must not stop the metaserver starting"
skip any bad line wherever it sits	the middle-damage test, on "damage with acknowledged records after it must be refused"

Suites: 328 metadata tests, 47 metaserver binary tests, cargo check --all-targets clean.