digibyted can deadlock or abort on shutdown: OracleSigningOrchestrator destroyed while a BlockConnected callback is in flight

[JohnnyLawDGB]

Summary

digibyted can deadlock (Linux) or abort with exit code -6 (macOS) during shutdown, because OracleSigningOrchestrator::Shutdown() destroys the orchestrator while one of its own validation-interface callbacks is still in flight on the scheduler thread.

This is what makes rpc_blockchain.py fail intermittently in CI, but the race is not test-only — see Impact below.

Symptoms in CI

rpc_blockchain.py fails in _test_stopatheight (test/functional/rpc_blockchain.py:519), with two different-looking failures that share one cause:

• Linux: AssertionError: Predicate ... not true after 120.0 seconds — the node never exits.
• macOS: AssertionError: [node 0] Node returned unexpected exit code (-6) vs (0) when stopping — SIGABRT during shutdown.

It is nondeterministic and not platform-specific. Two CI runs of an identical tree (only the commit SHA differed) traded places:

Run	macOS 14 ARM64	Ubuntu 22.04
31337196786	FAIL (-6)	pass
31440462238	pass	FAIL (120s)

It also reproduces on develop with no PR involved: run 28624106866 (push to develop, 2026-07-02). Roughly 3 of the last 8 long-running CI jobs.

Root cause

OracleSigningOrchestrator registers itself as a validation-interface subscriber (src/oracle/signing_orchestrator.cpp:195) and unregisters in Stop() (:203). Shutdown() then destroys it immediately (src/oracle/signing_orchestrator.cpp:842):

void OracleSigningOrchestrator::Shutdown()
{
    if (g_signing_orchestrator) {
        g_signing_orchestrator->Stop();   // UnregisterValidationInterface(this) — non-blocking
        g_signing_orchestrator.reset();   // destroys the object a queued callback is about to use
    }
}

src/validationinterface.h:27 warns about precisely this:

/** Unregister subscriber. DEPRECATED. This is not safe to use when the RPC server or main message handler thread is running. */

and :32-35:

// Alternate registration functions that release a shared_ptr after the last notification is sent. These are useful for race-free cleanup, since unregistration is nonblocking and can return before the last notification is processed.

Both conditions are violated at the callsite. OracleSigningOrchestrator::Shutdown() runs at src/init.cpp:280, which is:

• before StopRPC() / StopHTTPServer() (src/init.cpp:285-288) — so RPC threads are still connecting blocks;
• before node.scheduler->stop() (src/init.cpp:303) — so the scheduler can still dispatch queued callbacks;
• before GetMainSignals().FlushBackgroundCallbacks() (src/init.cpp:334) — so the queue has not been drained.

-stopatheight makes the collision near-certain: StartShutdown() is called from KernelNotifications::blockTip() (src/node/kernel_notifications.cpp:63-66) during block connection, so a BlockConnected notification for that same block is already queued when shutdown begins.

The queued callback then runs BlockConnected → OnBlockConnected → CleanupOldSessions, whose first statement is std::lock_guard<std::mutex> lock(m_sessions_mutex) (src/oracle/signing_orchestrator.cpp:1011-1013) — a member mutex (signing_orchestrator.h:151) of an object that no longer exists.

Why one bug produces two symptoms

Locking a destroyed std::mutex is undefined behavior, and the platforms differ in how they express it:

• glibc/Linux: the futex word sits in freed memory; the thread parks and never wakes. The scheduler thread never finishes, so node.scheduler->stop() (init.cpp:303) never returns → the node hangs → 120s timeout.
• macOS/libc++: the same operation traps rather than parking → abort() → exit code -6.

Evidence from a live hung node

Log ordering, at microsecond resolution — the callback runs 69µs after the orchestrator is torn down:

.925733 [shutoff]   [signing_orchestrator.cpp:205] Oracle: MuSig2 signing orchestrator stopped
.925802 [scheduler] [signing_orchestrator.cpp:1120] Oracle: TickEpochSession h=207 epoch=5 state=0 is_oracle=0

Thread states from /proc/<pid>/task/* on a node caught mid-hang (all 10 threads sleeping, node alive):

4067307  b-shutoff      state=S  wchan=futex_wait_queue   <- blocked in scheduler->stop()
4067345  b-scheduler    state=S  wchan=futex_wait_queue   <- blocked inside the orchestrator callback

b-scheduler's last log line is TickEpochSession h=207; b-shutoff's is the preceding DumpAddresses, whose next step in Shutdown() is node.scheduler->stop(). Each waits on the other.

Reproduction

Solo it almost always passes; it needs CI-like load to lose the race. On an 8-core box, 6 concurrent copies reproduced it on the first round on two separate attempts:

for i in $(seq 1 6); do test/functional/rpc_blockchain.py & done; wait

A failing run leaves its datadir behind; node0/regtest/debug.log ends right after Flushed 0 addresses to peers.dat, with the node still alive.

Reachability outside -stopatheight (measured — narrower than it looks)

An earlier revision of this section claimed that any shutdown beginning while a block connects hits the same window. That overstates it, and the correction matters for severity.

UnregisterValidationInterface() removes the subscriber, so a notification dispatched after Stop() never reaches the orchestrator at all. The dangerous case is narrower: an emission already in flight — the scheduler mid-dispatch of a BlockConnected whose slot list still includes the orchestrator — at the moment reset() runs.

-stopatheight correlates those two events by construction. StartShutdown() fires from blockTip() inside block connection, so the emission for that block is in flight exactly as shutdown proceeds. An ordinary stop has no such correlation, and I could not produce it:

Trials	Workload	Hangs	Aborts	Callback-after-teardown
300	digibyte-cli stop vs. a node continuously connecting blocks, unpatched	0	0	0

So I am not claiming operator-visible impact. On the evidence I have, practical exposure looks confined to -stopatheight, and this is a latent use-after-free on an API the header marks unsafe rather than a demonstrated production hazard. 300 trials of one workload is not proof of impossibility either — anything that keeps emissions in flight longer (a slow subscriber, heavy load, a large wallet) widens the window — but nobody should prioritise this as an operator-facing bug on my say-so.

Candidate fix

Honour the documented contract by draining the queue before destroying the object:

void OracleSigningOrchestrator::Shutdown()
{
    if (g_signing_orchestrator) {
        g_signing_orchestrator->Stop();
        SyncWithValidationInterfaceQueue();   // in-flight callback finishes first
        g_signing_orchestrator.reset();
    }
}

Alternatives worth considering: register via RegisterSharedValidationInterface() (the header's own suggestion for race-free cleanup), or move the destruction after node.scheduler->stop(), matching how node.peerman is unregistered early (init.cpp:296) but destroyed only after the scheduler stops (init.cpp:309).

I have the SyncWithValidationInterfaceQueue() variant building and under test locally, and can open a PR if the approach looks right. Happy to defer if a maintainer prefers one of the alternatives.

Unrelated CI note

Functional-test logs are never captured on failure, which is why none of the CI runs above have usable post-mortem evidence:

##[warning]No files were found with the provided path: src/test-suite.log.
No artifacts will be uploaded.

src/test-suite.log is the automake unit test log; functional output lives under the runner's temp dir and is discarded. Uploading that directory on failure would make these self-diagnosing. Happy to file separately if you'd like it tracked on its own.

---

Investigation assisted by AI tooling; all findings were reproduced and verified locally against develop (16159311b3).

[JohnnyLawDGB]

Verification of the candidate fix

The SyncWithValidationInterfaceQueue() variant above now survives the repro that reliably broke the unpatched binary.

Same harness, same 8-core box, 6 concurrent copies of rpc_blockchain.py per round:

Build	Result
unpatched develop (16159311b3)	reproduced on round 1, on two separate attempts
with SyncWithValidationInterfaceQueue()	0 failures in 60 runs (10 rounds × 6 concurrent)

The one-line change is the only difference between the two builds.

I'll open a PR unless a maintainer would rather take one of the alternatives (RegisterSharedValidationInterface, or moving the destruction after node.scheduler->stop()) — happy either way.

[JohnnyLawDGB]

Fix opened as #442, with a regression test.

One correction to my verification numbers above. That first "0 failures in 60 runs" came from a harness whose six concurrent instances shared a port seed, so instances could collide on 127.0.0.1 bind ports and fail for reasons unrelated to this bug. A later run hit exactly that and surfaced it.

Re-verified with each instance given a distinct --portseed, and with failures classified rather than counted as any non-zero exit:

RESULT: 60 runs | _test_stopatheight failures: 0 | unrelated: 0

The conclusion is unchanged — every baseline failure counted against the unpatched build was a confirmed _test_stopatheight shutdown failure, one of them captured mid-hang with the thread states above — but the corrected harness is what #442 cites.

The regression test in #442 turned out sharper than expected: reverting only the SyncWithValidationInterfaceQueue() line makes it fail the contract check and take a SIGSEGV, so it catches the use-after-free itself rather than just the ordering.

[JohnnyLawDGB]

Correction: I overstated the severity above

I've edited the body. The original text claimed that any shutdown beginning while a block connects hits this window, including an ordinary digibyte-cli stop or a systemd SIGTERM. That was reasoning from the mechanism, and testing it did not support it.

UnregisterValidationInterface() removes the subscriber, so a notification dispatched after Stop() never reaches the orchestrator. Only an emission already in flight when reset() runs is dangerous. -stopatheight correlates those by construction — StartShutdown() fires from inside block connection — whereas an ordinary stop lands at an uncorrelated moment.

Measured on an unpatched build: 300 trials of digibyte-cli stop against a regtest node continuously connecting blocks, with -logtimemicros so the ordering was visible. Result: 0 hangs, 0 aborts, and 0 instances of the callback-after-teardown ordering that the -stopatheight path reproduces on the first round.

That doesn't prove it's unreachable — 300 trials of one workload is not a proof, and anything keeping emissions in flight longer would widen the window — but it does mean I have no evidence of operator-visible impact, and nobody should weight this as a production hazard on the strength of my earlier wording.

The fix in #442 stands on its own terms: it's a use-after-free on an API the header explicitly marks unsafe, it's one call, and it clears a CI failure that reproduces on develop.