Skip to content

fix: stop the view walk hanging on its shared span context - #333

Merged
geekgonecrazy merged 2 commits into
devfrom
fix/span-context-parallel-init
Nov 12, 2026
Merged

geekgonecrazy merged 2 commits into
devfrom
fix/span-context-parallel-init

Conversation

@vinceblock99

@vinceblock99 vinceblock99 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Repository::materialize_view_entries can hang forever on a span-backed view, taking every rayon worker with it.

The walk renders files in a par_iter, and those workers share one SharedSpanContext. The context was built lazily, inside OnceLock::get_or_init, by whichever worker first reached a span-rooted file. Building it runs rayon jobs of its own, so while that worker waits for them it picks up other work from the pool:

  • it can pick up another file of the same walk and call get_or_init again on the cell it is still filling, or
  • a second worker can take one of the build's jobs, pick up a file of the walk while it waits, and park on the cell. The build then waits for a job that sits on a parked thread.

Either way nothing finishes. A sample of a hung process shows OnceLock::initialize twice on one stack, with a par_iter item of the walk in between, and the other workers parked on the same cell.

This walk backs the daemon Materialize RPC, sandbox provisioning, the remote cache and atomic revise. A hang there parks the process's global rayon pool, so it does not recover.

Reproduction

A span import of hyperfine (1,097 commits, 75 files), a fresh Repository handle per call, release build, 10 rayon threads, 150 calls per round:

Rounds Result
Before 5 all 5 hung, after 2, 28, 28, 54 and 83 calls
After 3 all 450 calls completed

After the fix, 150 calls each at 2, 4, 8 and 16 threads also completed. Every rendered file matched the working copy byte for byte, and 100 calls built exactly 100 contexts.

Fix

  • SharedSpanContext never makes one caller wait for another's build. The context is built outside the cell and the first result is kept. A caller that finds no context builds its own instead of parking behind a build that may be waiting for it.
  • The view walk builds the context before it fans out. SharedSpanContext::prime builds it on the calling thread when the walk covers a span-rooted file, so a walk still builds exactly one context. Without this the change above would be correct but wasteful: every worker that arrived early would build its own.

prime only looks for an indexed span root. It does not load the origin change of a graph-origin file: the walk already does that check per file on its workers, and repeating it up front would serialise that work for views made mostly of graph-origin files. It also reports nothing itself. A failure is stored and surfaces from the walk at the path that hits it, with the same error text as before.

What does not change

  • Nothing in atomic-core: the span patch format, SpanRenderContext, its verification and the causal proof are untouched. The context is still built by SpanRenderContext::new with the same transaction, store and visibility closure.
  • Rendered bytes, visibility, path ownership and conflict handling.
  • The other users of SharedSpanContext (project_tree, the batch content readers, plan_file_moves), which are sequential.

The only behavioural difference is when the context is built: before the parallel walk starts, instead of by the first worker to reach a span-rooted file. The struct's doc comment claimed it was safe to share across rayon workers; it now says what is actually guaranteed.

Tests

New view_entries_span_context_tests:

  • render_does_not_wait_for_a_span_context_build_in_progress reproduces the hang deterministically. A change store wrapper holds the first build inside the store until a second render has finished. On the old code the second render waits for the build, the build gives up after 60 s, and the test fails. On the new code it passes in about a second. Both renders run in a two-worker pool of the test's own, so the result does not depend on the size of the global pool; it passes with RAYON_NUM_THREADS=1.
  • prime_builds_the_span_context_once_and_only_for_span_rooted_files checks that priming builds nothing and loads nothing for graph-origin files, builds once for span-rooted ones, and that later primes and renders reuse that build.
  • view_entries_walk_renders_mixed_views_on_a_worker_pool runs the walk in an 8-worker pool over a view mixing graph-origin and span-rooted files and compares the bytes.

A stress test of the walk itself was not reliable enough to keep: on a small fixture the race did not fire in 150 walks, which is why the regression test drives the build through the store instead.

End to end, through the real Materialize RPC: agent_sandbox_bench --span --files-east 120 --files-west 120 --lines 64 --edits 1200 --iterations 300 (tonic over a Unix socket, release build) completed all 600 RPC calls on a 1,200-change span history, and the benchmark verified every sample.

Locally on macOS:

  • cargo test --workspace --features atomic-cli/adoption-test-injection: 11,409 passed, 0 failed. This ran on the first commit; after the second, cargo test -p atomic-repository -p libatomic was rerun: 1,964 passed, 0 failed.
  • cargo test -p atomic-cli --features "experimental-span-import adoption-test-injection" span: 27 passed, 0 failed. CI does not build this feature.
  • cargo fmt --all -- --check and cargo clippy --workspace -- -D warnings clean; cargo clippy -p atomic-repository --all-targets -- -D warnings clean.

Not covered: atomic revise was not driven end to end, only the walk it shares. The end-to-end run was not repeated on the code before the fix. Windows is covered by CI only.

Repository::materialize_view_entries renders files on rayon workers that
share one lazily built span context. Building that context runs rayon
jobs as well, so the worker building it could end up waiting for a worker
that had picked up another file of the same walk and was parked on the
context still being built. Neither could finish, and the rest of the pool
parked behind them.

- SharedSpanContext builds outside its cell and keeps the first result,
  so no caller waits for another caller's build.
- The view walk builds the context before it fans out when the view has
  a span-rooted file, so a walk still builds exactly one context.
Priming asked has_visible_span_root about every file of the walk. For a
file without a span root that loads its origin change, which the walk
already does per file on its workers, so a view of mostly graph-origin
files paid for the check a second time on one thread. Look for an
indexed root first.

The regression test held up a worker of the global rayon pool, so with
a single worker the second render had nothing to run on and the test
failed. It now uses a two-worker pool of its own.
@geekgonecrazy
geekgonecrazy merged commit 5bef3d2 into dev Nov 12, 2026
10 checks passed
@geekgonecrazy
geekgonecrazy deleted the fix/span-context-parallel-init branch November 12, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants