Writers waiting on the metadata log share a barrier

[bjmeetsfo]

Every recorded change to the metadata takes a durability barrier, and the barrier is nearly the whole cost. Of the 2.6ms an append takes, 2.1ms is the write and the sync; opening the file is 5us and copying the change is 0.2us.

So writers did not go faster together than alone. Measured end to end -- registering shards through the metaserver, not appending to the log directly:

writers	before	after
1	499/s	490/s
4	467/s	858/s
16	478/s	1941/s

Flat before, because each writer took its own barrier and the next one waited. A cluster coming back -- every datanode registering every shard it holds -- paid one barrier per shard however many datanodes were doing it.

Why it is safe

The barrier a writer needs is not its own. The file is append-only and the writes are ordered by the write lock, so any sync beginning after a writer's bytes reached the file covers them. Numbering the records lets a writer notice that somebody else's sync already covered it and return without taking another.

The part that is easy to get wrong, and the reason it is written this way: how far the log has been written is read BEFORE the sync and published only AFTER it returns. The other order would tell a writer its bytes were durable while the sync was still running.

The first version of this was slower for one writer

Splitting the write from the sync meant a record opened the file twice where it used to open it once. End to end that cost a lone writer more than sharing saved it -- 499 registrations a second became 363 -- and one writer is the ordinary case for a metaserver.

Both handles are now opened once and kept, which is what the write-ahead log next door already does. Syncing is per file rather than per handle, so the barrier handle still covers what the write handle wrote, and the barrier still runs outside the write lock.

Nothing in the tree rewrites this file: the only nearby truncate belongs to snapshot export, which writes a temporary file and renames it.

Testing

Eight writers, twelve records each, and every record every writer was told had landed is in the log exactly once.

How many barriers that took is deliberately not asserted. The barrier counter is process-wide and every other test in the binary adds to it, so a count read in a test says nothing about this log -- and a test that passes alone and fails in the suite is worse than no test. That is how the first version of this test behaved.

Measurements were taken twice with the arms interleaved, because the box is shared. The second pair ran under load 12.4 and the absolute numbers moved together while the shape held.

Suites: 327 metadata tests, 246 consensus tests, 47 metaserver binary tests, all green.