Repository navigation
fix(dash): initialise ADLX once on a resident thread - #1020
Conversation
On Windows the dashboard's in-process collector initialised and terminated ADLX on every 1s tick, on a different blocking-pool thread each time. That loaded and unloaded amdadlx64.dll (and the d3d9, OPENGL32 and dbghelp it pulls in) repeatedly while the driver's atiadlxx.dll stayed resident. After a while atiadlxx.dll fail-fasts (0xC0000409, FAST_FAIL_FATAL_APP_EXIT), which kills rocm dash with no panic message or stack trace. Initialise ADLX once on a dedicated reader thread that keeps the helper for the process lifetime and answers read requests over a channel. Init is attempted once per process. The init and read steps are injectable so the once-only and same-thread contract is unit-tested on every platform. The reader answers with a Result, so a caller can tell an absent driver from a reader that failed. A failed init answers with no readings and no warning, as before, since that is what a machine with no AMD GPU looks like. A read that panics is caught, reported as a warning and leaves the reader usable; a reader whose thread is gone is reported too. Without that split, one failure would blank temperature, power and clock for the rest of the session and be indistinguishable from having no driver — the reader is built once per process and never replaced. The crash was not reproduced outside the dashboard, so the per-tick init/terminate cycle is the leading cause rather than a proven one. The manual-testing checklist gains a 30 minute long-run check for it. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
d4e1868 to
1a5329d
Compare
juhovainio
left a comment
There was a problem hiding this comment.
I read the diff and found nothing blocking. Moving ADLX init onto one resident thread matches the stated cause (per-tick init/terminate), and the FFI stays off the async workers. This reflects a diff-only read plus green CI, not a runtime check.
Non-blocking points, mostly left inline:
- An init failure is dropped via
.ok()and never retried, so a transient startup failure blanks temp/power/clock for the whole session with no log line. Logging it would make that diagnosable. init()runs outsidecatch_unwind, while read panics are caught. An init panic kills the reader and every later tick warnsThreadGone.- The spawn-failure path has no test, and the "degrades to no readings" comment is stale: it actually yields
Err(ThreadGone)and a warning. snapshot's newDone(Err(gone))arm has no direct test.- A caught read panic still runs the process-wide panic hook. Worth checking that the dash's hook doesn't disturb the terminal.
- The Windows-only paths are covered only by CI compiling them, and the 30-minute soak hasn't been run. The PR says both honestly, and the cause is described as the leading one, not proven.
| // `state` is deliberately held until the request channel closes | ||
| // (never, for the process-wide reader) so it is not torn down | ||
| // and re-initialised between ticks. | ||
| let state = init(); |
There was a problem hiding this comment.
init() runs outside catch_unwind, so a panic here kills the reader and every later tick warns ThreadGone. Consider wrapping it, and logging the init failure (init_adlx drops the Err via .ok()) so a silent blank is diagnosable.
| #[cfg(windows)] | ||
| fn adlx_readings() -> Vec<AdlxReading> { | ||
| fn adlx_readings() -> ReaderAnswer { | ||
| static READER: std::sync::OnceLock<ResidentReader> = std::sync::OnceLock::new(); |
There was a problem hiding this comment.
Init is never retried, so a transient first-tick failure blanks ADLX metrics for the whole session with no warning. It's a documented trade-off; at least log the init error.
| /// Not gated: `snapshot` matches on it and the non-Windows `adlx_readings` | ||
| /// stub returns it, on every platform. | ||
| #[derive(Debug, Clone, Copy, PartialEq, Eq)] | ||
| enum ReaderGone { |
There was a problem hiding this comment.
ReadPanicked isn't "gone" (the reader survives), so the name and doc are slightly misleading; ReaderFault might fit better. Also worth checking that a caught panic doesn't trip a terminal-restoring panic hook in the dash.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 1a5329d492
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change.
No ticket named in the PR description or title; the change is scoped and described on its own terms (a Windows ADLX crash fix), not against an external tracker item.
Blocking
None
Non-blocking
- A panic during
init()is not caught, unlike a panic duringread()—crates/rocm-dash-collectors/src/winperf.rs:482(let state = init();) runs outside thecatch_unwindthat wraps the per-requestread(state)call atwinperf.rs:490. Ifinit_adlx()(e.g.adlx::helper::AdlxHelper::new()) ever panics rather than returningErr, theadlx-readerthread unwinds before entering its request loop,inboxdrops, and every laterResidentReader::read()call (winperf.rs:511) permanently answersErr(ReaderGone::ThreadGone)for the rest of the process. Before this change, each tick created a freshAdlxHelper, so a one-off init panic would simply be retried, and likely succeed, on the next tick. The new resident-thread design trades that self-healing property away for exactly the init step the comment atwinperf.rs:482-485says must not be re-run — the asymmetry is real even thoughinit_adlx()'s only fallible call already returnsResultrather than panicking in the documented case.
Confidence 75 · architectural · Fix: wrapinit()in the samecatch_unwindasread(), mapping a caught panic toErr(ReaderGone::ThreadGone)(or a new variant) before entering the loop, so an init panic degrades the same way a read panic does. - The new
SingleFlight::Done(Err(gone))arm insnapshot()has no direct test —winperf.rs:101(SingleFlight::Done(Err(gone)) => (Vec::new(), Some(gone.warning()))) is new behaviour this diff introduces (the priorDonearm only ever carriedVec<AdlxReading>, never aResult). No test in the file callsWinPerfCollector::snapshot()oradlx_readings()directly (confirmed: neither symbol appears anywhere under#[cfg(test)]), so this exact wiring — "a reader fault turns into no readings plusgone.warning()in the warnings list" — is unverified by anything that runs in CI. The components it wires together are each tested in isolation (ResidentReader'sErrpaths,ReaderGone::warning()), andsnapshot()'s hardcoded call toSelf::run_script()makes mocking the whole function impractical the same wayadlx_readings()'s own FFI chain is documented as untestable atwinperf.rs:1183— but that precedent is never stated for this specific arm.
Confidence 70 · logic · Fix: either extendparse_metrics's existing "take ADLX readings as a parameter" testability pattern to this match (e.g. a small pure helper taking aReaderAnswerand returning(Vec<AdlxReading>, Option<String>)), or add a one-line comment next to theDone(Err(gone))arm pointing at the same impracticality argument already made foradlx_readings().
Decisions for the author
ResidentReader::read()deliberately carries no deadline of its own — tradeoff
AGENTS.md §6's subprocess/FFI checklist says a blocking FFI call wrapped only by an outertimeout()still leaks its thread, and to "give the blocking call its own deadline or cancellation flag instead." This PR'sread()(winperf.rs:511) has none, relying solely onrun_single_flight's outerADLX_TIMEOUTplus the single-flight gate — the same reasoning the pre-existingADLX_TIMEOUTdoc comment already used. The PR body's "Scope" section names this explicitly and ties it to AGENTS.md §6 by number, arguing ADLX's FFI has no cancellation primitive a deadline could act on. That argument holds for this driver, but it is worth the author confirming out loud that the general rule is knowingly not followed here rather than overlooked, since the next engineer touching FFI code in this crate may cite this file as precedent without re-deriving the exception.- A single truly-wedged read now strands two threads instead of one, for the rest of the session — tradeoff
Pre-PR, a wedgedadlx_readings()call leaked one blocking-pool thread and leftADLX_IN_FLIGHTpermanently set, so every later tick also gotStillRunningwith no new attempt — the same terminal "no more ADLX readings this session" end-state as today. Post-PR the dedicatedadlx-readerthread is also stuck mid-read, so every future tick's blocking-pool thread blocks onanswer.recv()forever too, and the one extra OS thread is now held for the process's whole lifetime even on hosts with no AMD driver. The end state for the user is unchanged (no more temp/power/clock either way), but the resource shape is different and already disclosed candidly in the PR's "Scope" section — flagging here only so the tradeoff is a confirmed decision rather than a side effect nobody weighed.
Positive signals
ResidentReader::spawntakesinit/readas injectable closures specifically so the "built once, same thread for every read" contract is unit-testable without Windows or a driver — the same testability patternparse_metricsalready uses for ADLX readings, applied consistently to the new abstraction.- The new tests are mutation-grade, not just happy-path:
resident_reader_initialises_once_and_reads_on_one_threadencodes which thread a read ran on into the returned value itself, so it fails (not just passes vacuously) ifinit()is ever moved inside the per-request loop — verified by tracing the code, matching the PR's own stated mutation table. - The PR description is unusually honest about the limits of its own evidence ("Not verified" section: the crash was never reproduced outside the dashboard, evidence comes from one machine and one driver version) rather than overclaiming a proven root cause.
Deployment notes
This adds one dedicated, permanent OS thread (adlx-reader) for the lifetime of the rocm dash process on Windows, created lazily on the first metrics tick — present even on hosts with no AMD driver or ADLX at all. ADLXTerminate/DLL unload is intentionally never called on this path; the PR explicitly relies on process teardown to release it.
What this covered
Single-context pass over the whole change (crates/rocm-dash-collectors/src/winperf.rs and docs/manual-testing.md, merge-base eb2a8194a2aa4323904b01ab15a573c5b9cb0fa3…head 1a5329d4925eb84528ad776cad91c5a2f41442f4), read completely at both the merge-base and PR-head blobs via git show (not the working tree, to avoid any interference from other processes sharing the clone). This is a deliberate choice given the diff's size (one source file plus a short doc addition), not a degraded or failed fan-out. Also read: AGENTS.md in full; git log/git show history for winperf.rs back through the commit that introduced ADLX support (#715) and the one intermediate touch (#812); the PR description and its single commit message; and live CI status via gh pr view --json statusCheckRollup (every required check green, including windows-build-and-test, clippy, and the dash-crate coverage ratchet).
Did not run: no cargo/rustc toolchain is available in this sandbox, so cargo test, cargo clippy, and the author's own mutation checks could not be executed directly here — findings and the "tests are mutation-grade" positive signal are based on tracing the logic by hand against the code, cross-checked against the green CI results and the author's own stated mutation-evidence table, not an independent run. Impact-analysis searches (consumers, docs, fixtures) were done by direct grep rather than a separate worker pass, given the narrow blast radius (two files, no changed public signatures). Not reconciled against the existing human review thread already on this PR (a separate APPROVED review is present) — this review was produced blind to it, per method.
jussielo-amd
left a comment
There was a problem hiding this comment.
Nothing blocking. Moving ADLX init onto one resident thread matches the stated cause, the helper is built and used on that one thread (only a Sender is shared), and CI on this head is green. This was a static review: no tests were run (no Rust toolchain here), and the cfg(windows) paths and the 30-minute soak are only as verified as the PR says.
Non-blocking (some overlap with the existing threads):
crates/rocm-dash-collectors/src/winperf.rs:~482:init()runs outsidecatch_unwind, so an init panic kills the reader and every later tick warns that the reader thread is gone. Wrapping it likeread()keeps the two failure paths alike.init_adlxdrops the init error via.ok()and init is never retried. A missing driver is fine to treat as silent, but a transientADLXInitializefailure with the driver present then blanks temperature, power and clock for the session with no trace, the case theReaderGonecomment says must not be silent. At least atracing::warn!with the error, or separating "DLL absent" from "init failed".- AGENTS.md section 6 (FFI) asks for the blocking call's own deadline;
ResidentReader::read()waits onanswer.recv()with none. The PR says single-flight is the section 6 answer, which reads as compliance. Either userecv_timeout(asdevice_readable_within_budgetinrocm-core/src/amd_smi.rsdoes) or state in the code that this is a reviewed exception. - The new warnings ("the ADLX reader thread is gone", "ADLX read panicked", and the older "ADLX reading task panicked") use internal wording and give no effect or next step; one phrasing per cause such as "ADLX unavailable: temperature, power and clock hidden; restart rocm dash" would read better in the dash badge.
ReaderGone::ReadPanickedalso isn't "gone". - Stale comments and a test gap: the spawn-failure comment (~500) says reads "degrade to no readings" but they now return
ThreadGone; the test-module comment (~1183) saysadlx_readings()is not unit-testable although the shared logic now is; and theDone(Err(gone))arm ofsnapshothas no test asserting the resulting state (AGENTS.md section 3), only the wording.
🤖 by agenthub on AMD AgentHub
Review follow-ups on the resident ADLX reader. Both findings were real. `read` was wrapped in `catch_unwind` and `init` was not, and the asymmetry mattered more on the init side. A panic escaping `init` unwound the reader thread before its request loop started, so `inbox` dropped and every later tick answered `ThreadGone` — the one verdict that cannot be told apart from "the thread never started", for a cause that is neither. Because the reader is built once per process, that verdict was permanent, where the previous per-tick design would simply have retried init on the next tick. Init is now caught the same way, with its own `InitPanicked` variant rather than being folded into `ThreadGone`: the two are different faults and the remedy differs. This does not restore per-tick retrying — that is the trade this design makes deliberately — but it keeps the thread answering and names the real cause. Verified by mutation: dropping the `catch_unwind` turns the new test red with the reader thread unwinding, which is the reported defect exactly. The `SingleFlight::Done(Err(_))` arm added by this PR was the one piece of new wiring nothing executed: `snapshot` calls `run_script()` unconditionally, so no test can drive it. The match is pure and total, so it is now `adlx_outcome`, tested over every arm — including the one that must stay silent, since an absent driver is an expected state and not a fault to warn about. Mutation-checked by making the fault arm silent. Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Both non-blocking findings fixed —
|
|
Correction to the last line of my previous comment: the push did not dismiss the approvals — this repo does not dismiss stale reviews on push, so #1020 still reads That does mean the two approvals now stand on a commit the approvers have not seen. The delta is small and self-contained (one file: catch |
Problem
On Windows,
rocm dashexits silently after about a minute. No panic message, nobacktrace, nothing in
~/.rocm/logs/. The exit code is0xC0000409.Windows Error Reporting names
atiadlxx.dllas the faulting module, and under WinDbg thecrash is
FAST_FAIL_FATAL_APP_EXIT— the AMD driver callingabort(). That bypassesRust's panic hook entirely, which is why nothing is printed and the terminal is left
unrestored.
The dashboard's in-process Windows collector initialised and terminated ADLX on every 1 s
metrics tick, each time on a different blocking-pool thread. That loads and unloads
amdadlx64.dll(and thed3d9,OPENGL32anddbghelpit pulls in) over and over whilethe driver's
atiadlxx.dllstays resident. ADLX is meant to be initialised once perprocess; after enough cycles the driver aborts.
What changed
ADLX is initialised once, on a dedicated
adlx-readerthread that holds the helper for theprocess lifetime and answers read requests over a channel.
adlx_readings()sends a requestinstead of initialising.
The reader answers with a
Result, so a caller can tell an absent driver from a reader thatfailed:
GPU looks like, and it is expected.
That split matters because the reader is built once per process and never replaced: without
it, a single failure would blank temperature, power and clock for the rest of the session
and be indistinguishable from having no driver at all.
run_single_flightand the 5 sADLX_TIMEOUTare unchanged.How it was checked
initandreadare injectable, so the once-only and same-thread contract is unit-testedon every platform: init runs once across many reads, a failed init degrades without
retrying, a panicking read is reported while the reader survives, a dead reader is reported
rather than answering empty, and each fault warns in words that name ADLX and tell the
faults apart.
Mutation evidence — each check was shown to fail against a deliberately broken input:
init()moved inside the request loopcatch_unwindremoved from the read stepThreadGone, notReadPanicked)read()made to swallow the faultOn the reported hardware, a debug build of
rocm dashdied after 46 s with0xC0000409;the fixed build ran 241 s with no crash and no new Application Error event before being
stopped by hand.
Not verified. 241 s is about 5× the previous time-to-crash and is not proof — the
30-minute long-run check added to
docs/manual-testing.mdis the real one, and it has notbeen run. The crash was never reproduced outside the dashboard: standalone harnesses
re-initialising ADLX every 100 ms for two minutes did not trigger it, so the per-tick
init/terminate cycle is the leading cause rather than a proven sole trigger. Evidence comes
from one machine and one driver version (31.0.21925.1001); whether other Adrenalin versions
abort the same way is unknown.
Platform-gated path, per AGENTS.md §3.
init_adlx,read_adlxand theOnceLockwiring in
adlx_readings()arecfg(windows), so they compile on no developer machinewithout Windows and in no default CI lane. They are covered by inspection only; the unit
tests above cover the shared logic, and
windows-build-and-testis the first thing thatactually compiles them.
No Gherkin scenario: an internal crash fix with no change to command output or exit codes.
Scope
on the channel, and the single resident reader.
run_single_flightstill refuses to startanother call while one is outstanding, so that is bounded at one in-flight request rather
than one per tick.
ResidentReader::read()deliberately carries no deadline of its own,keeping the existing single-flight answer to AGENTS.md §6 rather than queueing a request
per tick behind a wedged reader.
including on hosts with no AMD driver.
ADLXTerminateis never called — Rust does notdrop statics, so the helper is released by process teardown.
Not in this PR
SetUnhandledExceptionFilteror minidump writer recording the fault module and offsetwould make the next one diagnosable.
likelihood. Worth doing if the crash recurs.
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. — no xfail row references this crash; nothing to narrow.docs/architecture.md. — n/a, no new subcommand or subsystem.docs/architecture.mdhas the matching entry. — n/a, no file added, renamed or removed.ADLX read panicked,the ADLX reader thread is gone).snapshotpairs each with no readings, so the state is "ADLX columns blank, with a reason"; an absent driver stays warning-free. Covered by the fault-warning test plus the two reader-fault tests that pin which fault is reported.