Repository navigation
fix: stop the view walk hanging on its shared span context - #333
Merged
Merged
Conversation
Repository::materialize_view_entries renders files on rayon workers that share one lazily built span context. Building that context runs rayon jobs as well, so the worker building it could end up waiting for a worker that had picked up another file of the same walk and was parked on the context still being built. Neither could finish, and the rest of the pool parked behind them. - SharedSpanContext builds outside its cell and keeps the first result, so no caller waits for another caller's build. - The view walk builds the context before it fans out when the view has a span-rooted file, so a walk still builds exactly one context.
Priming asked has_visible_span_root about every file of the walk. For a file without a span root that loads its origin change, which the walk already does per file on its workers, so a view of mostly graph-origin files paid for the check a second time on one thread. Look for an indexed root first. The regression test held up a worker of the global rayon pool, so with a single worker the second render had nothing to run on and the test failed. It now uses a two-worker pool of its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Repository::materialize_view_entriescan hang forever on a span-backed view, taking every rayon worker with it.The walk renders files in a
par_iter, and those workers share oneSharedSpanContext. The context was built lazily, insideOnceLock::get_or_init, by whichever worker first reached a span-rooted file. Building it runs rayon jobs of its own, so while that worker waits for them it picks up other work from the pool:get_or_initagain on the cell it is still filling, orEither way nothing finishes. A sample of a hung process shows
OnceLock::initializetwice on one stack, with apar_iteritem of the walk in between, and the other workers parked on the same cell.This walk backs the daemon
MaterializeRPC, sandbox provisioning, the remote cache andatomic revise. A hang there parks the process's global rayon pool, so it does not recover.Reproduction
A span import of hyperfine (1,097 commits, 75 files), a fresh
Repositoryhandle per call, release build, 10 rayon threads, 150 calls per round:After the fix, 150 calls each at 2, 4, 8 and 16 threads also completed. Every rendered file matched the working copy byte for byte, and 100 calls built exactly 100 contexts.
Fix
SharedSpanContextnever makes one caller wait for another's build. The context is built outside the cell and the first result is kept. A caller that finds no context builds its own instead of parking behind a build that may be waiting for it.SharedSpanContext::primebuilds it on the calling thread when the walk covers a span-rooted file, so a walk still builds exactly one context. Without this the change above would be correct but wasteful: every worker that arrived early would build its own.primeonly looks for an indexed span root. It does not load the origin change of a graph-origin file: the walk already does that check per file on its workers, and repeating it up front would serialise that work for views made mostly of graph-origin files. It also reports nothing itself. A failure is stored and surfaces from the walk at the path that hits it, with the same error text as before.What does not change
atomic-core: the span patch format,SpanRenderContext, its verification and the causal proof are untouched. The context is still built bySpanRenderContext::newwith the same transaction, store and visibility closure.SharedSpanContext(project_tree, the batch content readers,plan_file_moves), which are sequential.The only behavioural difference is when the context is built: before the parallel walk starts, instead of by the first worker to reach a span-rooted file. The struct's doc comment claimed it was safe to share across rayon workers; it now says what is actually guaranteed.
Tests
New
view_entries_span_context_tests:render_does_not_wait_for_a_span_context_build_in_progressreproduces the hang deterministically. A change store wrapper holds the first build inside the store until a second render has finished. On the old code the second render waits for the build, the build gives up after 60 s, and the test fails. On the new code it passes in about a second. Both renders run in a two-worker pool of the test's own, so the result does not depend on the size of the global pool; it passes withRAYON_NUM_THREADS=1.prime_builds_the_span_context_once_and_only_for_span_rooted_fileschecks that priming builds nothing and loads nothing for graph-origin files, builds once for span-rooted ones, and that later primes and renders reuse that build.view_entries_walk_renders_mixed_views_on_a_worker_poolruns the walk in an 8-worker pool over a view mixing graph-origin and span-rooted files and compares the bytes.A stress test of the walk itself was not reliable enough to keep: on a small fixture the race did not fire in 150 walks, which is why the regression test drives the build through the store instead.
End to end, through the real
MaterializeRPC:agent_sandbox_bench --span --files-east 120 --files-west 120 --lines 64 --edits 1200 --iterations 300(tonic over a Unix socket, release build) completed all 600 RPC calls on a 1,200-change span history, and the benchmark verified every sample.Locally on macOS:
cargo test --workspace --features atomic-cli/adoption-test-injection: 11,409 passed, 0 failed. This ran on the first commit; after the second,cargo test -p atomic-repository -p libatomicwas rerun: 1,964 passed, 0 failed.cargo test -p atomic-cli --features "experimental-span-import adoption-test-injection" span: 27 passed, 0 failed. CI does not build this feature.cargo fmt --all -- --checkandcargo clippy --workspace -- -D warningsclean;cargo clippy -p atomic-repository --all-targets -- -D warningsclean.Not covered:
atomic revisewas not driven end to end, only the walk it shares. The end-to-end run was not repeated on the code before the fix. Windows is covered by CI only.