Apply the guards the raft propose path was skipping - #231
Conversation
The mute is the incident lever: while it is set the metaserver is meant to refuse every recorded metadata mutation, so an operator can stop it making changes mid-incident without a restart. On a raft-backed metaserver it did nothing at all. The check lived only in SingleNodeMeta's public methods, and MetaRaftCluster proposes straight past them -- neither the propose path nor the apply path consulted the mute, and raft/cluster_meta.rs did not mention it anywhere. Set it, and every mutation still went through, on the deployment mode a real cluster runs. Gated in mutation_status, the single funnel for raft mutations. Checked before proposing rather than while applying: replay has to reapply what was already accepted, including changes recorded before the mute was set. The check does not clone the cluster's metadata. read_meta() deep-copies the whole state to answer one question, which on the mutation path would make every write pay for a full copy; peek_meta_change_muted reuses the same replica selection and reads one bool. A cluster no replica can answer for is left to propose and fail on its own terms, because refusing there would turn "cannot tell" into "muted". What stays permitted while muted lives on MetaMutation, so the rule cannot drift between the backend that checks it and the one that does not: the lever itself, or muting would be a one-way door, and the retention purge, which the single-node path has always allowed.
apply_mutation dispatches straight to the apply_ functions, so every guard that lives in a public method is skipped when the raft backend proposes. The mute was one instance; it was not the only one. Probing the raft path against each guard the single-node path applies found two that cause real harm: A reserved name could be taken. Reserved names exist to hold a name back from creation, and on a raft-backed metaserver the reservation held nothing back at all. A namespace could be dropped out from under a live table. The single-node path refuses that precisely so a drop cannot strand tables in a namespace that no longer exists; the raft path dropped it and left the table behind. The guards move into admission_refusal, expressed once against state either backend can consult, and the single-node path now defers to it as well so the two cannot drift apart again -- drift is what caused this. Judged before proposing and never while applying: replay has to reapply what was already accepted, and a name reserved today must not invalidate a namespace legitimately created before it. peek_meta_change_muted generalises into with_readable_meta so the admission check also answers without copying the cluster's metadata on every write. One difference is left alone: setting a namespace to the state it already has succeeds here and is refused on the single-node path. It causes no harm beyond a no-op being accepted, and turning a success into an error is wire-visible for anything already calling it.
|
Merge-order note. This PR and the other one listed below both rewrite the
Both report Resolving it by simply taking #231's side compiles, and then hangs. Worth knowing that only the The resolution that works is to factor the guard so it takes |
|
The resolution for the conflict with #231 is written and measured — branch I have not touched this PR's branch. That branch is #231 with this change What it does: each judgement gains a form that takes the state to read rather Why it is not simply "take one side": resolving in favour of #231 leaves and on the composed branch: Only the Both tests from this PR are carried over verbatim; Two things this does not cover. #235 also touches the same function — it threads |
The problem
apply_mutationdispatches straight to theapply_*functions:Every guard that lives in a public method is therefore skipped when the raft
backend proposes. #230 found this for the mute. It is not the only one.
I probed the raft path against each guard the single-node path applies:
apply_)not_modified)Two of these cause real harm:
from creation; on a raft-backed metaserver the reservation did nothing.
path refuses this precisely so a drop cannot strand tables in a namespace
that no longer exists. The raft path dropped it and left the table behind.
Both tests fail on the unmodified path — the first with "a reserved namespace was
created anyway", the second with "a namespace was dropped out from under a live
table".
The change
The guards move into
admission_refusal(&MetaMutation)— expressed once,against state either backend can consult — and the single-node path now defers
to it too, so the two cannot drift apart again. Drift is what caused this.
Judged before proposing and never while applying: replay has to reapply what
was already accepted, and a name reserved today must not invalidate a namespace
legitimately created before it.
peek_meta_change_mutedfrom #230 generalises intowith_readable_meta, so theadmission check also answers without cloning the cluster's metadata on every
write.
What I did not change
not_modified— setting a namespace to the state it is already in succeeds onthe raft path and is refused on the single-node path. It causes no harm beyond a
no-op change being accepted, and turning a success into an error is a
wire-visible behaviour change for anything already calling it. That is your call,
not mine; the guard is one line away in
admission_refusalif you want it.Verification
cargo check --all-targets— 0 errorscargo test --bin metaserver -- --test-threads=1cargo test --lib -- --test-threads=1The
data_node::…jitter_backoff…andengine::…recovery_validates_…failuresreproduce on an unmodified tree at this base and are untouched here.
Correction to the verification note above
I described two failures as reproducing on an unmodified tree. Re-checked on
a quiet machine with disk headroom, run in isolation against unmodified
main:data_node::…jitter_backoff…engine::…recovery_validates_all_timestamped_kv_page_familiesOnly the first is pre-existing. My original evidence for the second came from a
run taken immediately after the disk hit 100%, so it was environmental — that
test is sensitive to disk pressure and load, not broken on
main.The conclusion this change is not responsible for either failure is unchanged.
The evidence offered for half of it was wrong, and the record should say so.