Close the window between the emptiness check and the namespace drop - #285
Close the window between the emptiness check and the namespace drop#285bjmeetsfo wants to merge 2 commits into
Conversation
|
Merge-order note. This PR and the other one listed below both rewrite the
Both report Resolving it by simply taking #231's side compiles, and then hangs. Worth knowing that only the The resolution that works is to factor the guard so it takes |
The namespace state change has to decide and act without letting go: the emptiness check used to run under a read lock that was released before the change was applied, and a table created in that window was stranded with its namespace already Dropped. The judgement it needs now lives in admission_refusal, which takes its own read lock. Calling that under the write lock this path must hold is a read acquired inside a write, on the same lock and the same thread, and it does not return -- measured, the namespace drop test never finishes. So each judgement gains a form that takes the state to read rather than fetching it: admission_refusal_in, reserved_name_refusal_in and namespace_not_empty_in. The propose path takes a read lock and calls it; the namespace path passes the write guard it already holds. add_namespace and add_table now go through admission_refusal too, so one judgement serves every caller.
|
The resolution for the conflict with #231 is written and measured — branch I have not touched this PR's branch. That branch is #231 with this change What it does: each judgement gains a form that takes the state to read rather Why it is not simply "take one side": resolving in favour of #231 leaves and on the composed branch: Only the Both tests from this PR are carried over verbatim; Two things this does not cover. #235 also touches the same function — it threads |
|
This PR also collides with #285, and that resolution is written and tested — branch No branch of yours has been touched. That one is #231, then #285 composed onto The collision is over the same two lines. This PR makes the drop clock come from Git's automatic merge produces something that does not compile, which is the The trap is what comes next. The quickest way to make it compile is to delete The resolution that keeps both: the lock-holding helper takes |
dc31d64 to
dd10297
Compare
# Conflicts: # crates/temporalstore-rust/src/meta/registration.rs
dd10297 to
0d06faa
Compare
|
Rebuilt on main now that the shared admission check has landed. This branch and that change both rewrote the same region, and the resolution is So each judgement has a form that takes the state to read: the propose path It also carries the recorded time through to the drop stamp. Main threads the |
A namespace could be dropped out from under a live table
set_namespace_statechecked "namespace still holds a live table" under aread lock, released it, then recorded and applied the change. A table created
in that window was stranded.
Demonstrated on
main, deterministically:This is not theoretical.
serve_with_stream_handlerdoesthread::spawnperconnection, so a
drop_namespaceand anadd_tablearriving together isordinary traffic. It is also the same end state as the raft-path defect in #231,
reached a different way — there the guard was absent, here it was released too
early.
The change
One write lock across the check, the record and the apply.
The apply body moves into
apply_namespace_state_locked, which takes analready-held
&mut MetaState. The public path checks and applies under a singlelock; replay keeps entering through
apply_set_namespace_stateand reapplyingunconditionally, so the rule that guards must not live in the apply path is
unchanged.
record_mutationstays before the state moves, so a crash between the tworeplays the change rather than losing it. It does not touch
self.inner, soholding the lock across it cannot deadlock.
The cost, and why it is affordable here
Holding the lock across
record_mutationmeans one fsync with readers blocked.That is affordable in this method and would not be in most: the only callers are
freeze_namespace,unfreeze_namespaceanddrop_namespace— operator actionsreached from admin routes, never a background loop, never per-request. Topology
reads are the hot path and they are not on it.
I checked the rest of the metaserver for the same shape —
meta.rs,registration.rs,table_ops.rs,state_setters.rs,proxy_groups.rs— andthis was the only method that released its lock between checking and applying.
Tests
rounds, each racing a drop against a table creation on separate threads, with a
durable log so the fsync sits in the window. On the unfixed code this catches
the bug 6 runs out of 6: a reliable reproduction, not a flaky one.
refusing a non-empty namespace, stops refusing once the table is deleted, and
freeze/unfreeze still work. It passes on the old code too, which is what a
guard against over-correction should do: holding the lock longer must not buy
atomicity by making the namespace undroppable.
Verification
cargo check --all-targets— 0 errorscargo test --bin metaserver -- --test-threads=1cargo test --lib -- --test-threads=1