add: live gridded reciprocal-space volume (4/4)

[pecomyint]

Stack 4 of 4. Base rsm/03-detector-physics. This branch contains the whole stack — check this one out for beamline testing and record its SHA in the test notes.

Draft until validated on real PVs. Three reviewable commits.

Why

Live 3D today is a point cloud with amnesia: a 100-frame, 1M-point ring buffer that overwrites its oldest frame. Scan a region twice and the second pass evicts the first rather than reinforcing it. PR #129 gave us the accumulator concept offline; this brings it live, so a voxel means the same thing during the scan as in the file you reopen later.

Commit 1 — Qt-free volume persistence

The accumulator runs in a pvaccess consumer, and this repo documents that mixing pvaccess with PyQt5 core-dumps (the two-process note in consumers/ioc_rsm_parameter). So the write path had to lose Qt first.

volume_io.py imports no Qt and produces the same layout Workbench already reads. HDF5Writer becomes a plain class with an on_finished callback; HDF5Handler re-wraps it as the signal — the split its own docstring already described. hkl_3d_viewer and area_det_viewer used HDF5Writer directly as a threaded QObject and now use HDF5Handler. log_manager was pulling PyQt5 into everything via LogMixin for one type annotation and one .instance() call; both deferred. A subprocess test asserts the write path stays Qt-free.

Empty voxels become NaN with a coverage volume alongside — Gridder3D leaves un-hit bins at zero, indistinguishable from a measured zero. volume_result_to_metadata used volume.min()/max(), which would have returned [nan, nan] the moment empty voxels became NaN, breaking every colour scale that trusts it; now finite-only.

Coverage does not make a flux-weighted mean recoverable: from sum(I/m) and N you cannot recover sum(I) and sum(m) separately unless the monitor was constant across a voxel's contributions.

gridder_access.py is the one place touching Gridder3D._gdata/_gnorm, avoiding .data (which copies the whole numerator and divides on every access — 134 MB per snapshot at 256³). Its guard pins the semantics, sum and count, not just that the attributes exist.

Commit 2 — Fail-closed metadata binding

If a frame's angles belong to a different frame, every pixel lands in the wrong place and the result still looks like a diffraction volume. There is no visual tell. So a frame whose required metadata is missing, stale, non-finite or duplicated is rejected and counted, never gridded with a substituted value.

Channels are classified: STATIC geometry legitimately arrives once and is latched; REQUIRED_DYNAMIC (circle positions, monitor) must be fresh; OPTIONAL never blocks. Rejections are counted per reason, so "the preview is empty" resolves to a cause. Scope is frame-bound and trigger-latched metadata only — fly-scan interpolation and pulse-ID joins are deliberately not attempted, because guessing at them is the exact mis-binding this prevents.

Commit 3 — Accumulator, session, consumer, dock

Bounds are locked before the first frame: Gridder3D latches fixed_range on first use, and rebinning mid-scan would change what a voxel means. Changing bounds needs a new accumulator — Clear() keeps the latched range.

Two grids over the same accepted samples: the full-resolution one that gets saved, and an independently gridded coarse one for the preview. One extra C call per frame, versus sweeping hundreds of MB of accumulator every snapshot in the process running the hot loop. Coarse shape is chosen per axis to fit 4 MiB while preserving aspect ratio.

Aggregation matches the offline builder exactly — unweighted mean, monitor-normalized per frame. Summing would brighten voxels wherever the scan path sampled more often, an artifact of the trajectory, not the scattering.

Nothing drops silently: out-of-range, non-finite and masked samples are counted separately and shown on screen. Masked pixels are excluded, not zeroed.

HpcRsmGridProcessor is an alternative to HpcRsmProcessor, not an addition — that one attaches three float64 Q arrays to every frame (~100 MB/frame at 2048²); this computes Q where it is consumed and publishes only the preview. It subclasses rather than forks the Q path. Single-instance by nature, so drops are counted and the preview marked incomplete: a drop-aware preview, not a lossless record. Full-resolution state never crosses PVA; saving happens in the consumer, confined to OUTPUT_PATH, refusing overwrite, and only from stopped so it cannot capture a torn accumulator.

The HKL3D dock adds bounds/resolution, Estimate/Start/Stop/Clear/Save and live counters. Bounds lock while running. Estimate is labelled as observed bounds — later scan motion can still fall outside, which is why the out-of-grid count sits beside it. Uncovered voxels render transparent.

The assertion that matters

Live and offline agree for identical frames, bounds and resolution — coverage exactly equal, means to 1e-12. build_volume gained an optional fixed_bounds so the comparison is possible at all.

444 tests pass (227 at #129), ruff check src/dashpva/ clean.

Blocked on beam

• Sustained-rate soak with real PVs; record achievable rate and drop count
• Confirm which metadata/ca/ key is I0
• Half-voxel origin visually in Workbench
• PR 3's axis-0 and tilt-sign gates

Known limits

Single consumer caps throughput; no mid-acquisition snapshot (save requires stop); no full-volume transfer over PVA; assumes shared storage visible to both the consumer and Workbench.

[Osayi-ANL]

The bug that I mentioned in PR #137 also prevents me from running DashPVA run and crashes

[pecomyint]

The crash you hit here came from the same root cause as #137 — fixed in 2493b8e + 8dc4067 on rsm/02-naxis-issue132, which this branch now contains.

A bad profile no longer makes import dashpva.settings raise, so DashPVA run starts and reports the problem instead of tracebacking. The config tree also no longer corrupts lists or empty tables, which is what produced the bad profile in the first place.

This branch was rebased (it had a duplicate cherry-pick of the fix), so please git pull before re-reviewing. 480 tests pass, ruff clean.

[Osayi-ANL]

• Version bump from 1.0.5 to 1.1.0 this is a minor update to the HKL setup and the hdf5 writing path -- update pyproject.toml and uv.lock
• Make apply and save stick to the bottom of the ioc window to not have us scroll all the way down to save -- optional
• Remove underscore in front of 'Save' in the apply save button in the HKL setup GUI
• Fix 'IOC out of sync' issue when removing PV's from the list in the HKL Setup and hitting apply save it seems to cause an error
• Although we add a label to each PV in the HKL Setup when saving, the default SAMPLE_CIRCLE_AXIS_1, SAMPLE_CIRCLE_AXIS_2, and SAMPLE_CIRCLE_AXIS_3 in the HDF5 file still gets saved. I assume the intended hierarchy of the save tree structure is LABEL or RECORD NAME if label is not available, but it seems as though it saves within the SAMPLE_CIRCLE_AXIS. What is the intended save structure? I would suggest making the save structure naming convention within the HDF5 match the labels or record name in the HKL Setup, instead of just the default structure name.

[pecomyint]

Went through each item:

• Version bump — done, 1.0.5 → 1.1.0 in pyproject.toml + uv.lock (9d77d11).
• Apply & Save scrolling out of view — fixed. It was inside the same QScrollArea as the axis tables; moved the Reload/Retry/Apply & Save row into the window's own layout below the scroll area, so it's always visible (91924cf).
• Underscore in front of "Save" — found the cause: the button was built as QPushButton("Apply & Save"), a single unescaped & followed by a space. Qt's mnemonic parser consumes the & and assigns the mnemonic to the next character — here a space — which some Qt styles render as a literal underscore glyph instead of an invisible underline. Escaped it to "Apply && Save" (91924cf).
• "IOC out of sync" when removing a PV and hitting Apply & Save — traced the two places this can legitimately fire (_activate_snapshot/_mark_out_of_sync): either a genuine RAW_CONFIG != snapshot mismatch after reload, or the child IOC process timing out/failing on restart. Both already surface the underlying exception text in the popup and the inline notice, but I don't have that text from your report, so I can't tell which one you hit or write a real fix without guessing. Could you paste the exact error shown in the "IOC out of sync" dialog, and whether it happens on every removal or only sometimes?
• HDF5 saves under SAMPLE_CIRCLE_AXIS_1/2/3 instead of the label — the numbered group name is intentional (it's the stable, beamline-generic layout the whole canonical refactor is built around), but the code already writes LABEL, RECORD_NAME, ANGLE_UNITS, and a convenience NAME dataset inside each numbered group (hkl_axes.canonical_axis_metadata, since 51cfa30). That only populates when the profile has IOC_RSM_PARAMETER — on a legacy profile (no IOC_RSM_PARAMETER yet, e.g. 19-ID/4-ID/8-ID/SSRL/11-ID configs), there's no LABEL to write, so the group is bare. If the profile in your screenshot was canonical and still missing LABEL/NAME, let me know which config and I'll dig further — otherwise this is working as designed.

Current tip is 9d77d11. Re-requesting your review.

[Osayi-ANL]

• Change these '{"CENTER_CHANNEL_PIXEL": [300.0, 300.0], "DISTANCE": 400.644, "PIXEL_DIRECTION_1": "z-", "PIXEL_DIRECTION_2": "x-", "SIZE": [28.38, 28.38], "UNITS": "mm"}' to a single input in the GUI. You don't have to change the dict structure it was saved in just how we grap the values.

• Sample Orientation input in the IOC_RSM_PARAMETER is not information that is used in 6IDB. We already have the Primary, Inplane, Sample directions which we don't necessarily use, but can be used in place of the input. So it is just redundant.

• The HKL Setup does not populate based on the profile selected instead it selects the profile toggled in the setup config. Try opening the config and viewing an selected profile. Then open the HKL Setup. It opens that profile even though it is not selected.

[pecomyint]

Thanks for the error text — that was the whole ballgame. Reproduced it exactly, and it's not what either of us assumed.

The "IOC out of sync" error

It was a profile switch, not a failed save, and nothing to do with removing PVs.

The tell is in your own message: both the prefix and the record names differ — os:Nu:* vs 6idb1:m29_RBV:*. A single profile failing to round-trip can't produce that; I tested that path first and it round-trips clean. Two different profiles can. I built the two-profile scenario against a real SQLite database and the first five lines of my output match yours character for character, PV names included.

What happened: HKL Setup binds to whichever profile was active when it opened, because the IOC builds its EPICS record database from that profile at launch. But settings.reload() re-resolves the active profile on every call. If something selects a different profile while the editor is open — another DashPVA window, a Workflow profile change — the two diverge, and the old code reported that as "the IOC did not activate that snapshot" and offered a Retry that could never succeed.

Your save went through fine. Nothing was corrupted and no rollback was needed. Do you remember having a second window open, or switching profiles in Workflow around then? That would confirm it end to end.

What changed (b500b8b)

Three genuinely different situations were sharing one wrong message. They're now separate, and written for someone reading them at a beamline rather than in the source:

A different profile is now active — names both profiles, says the angle IOC is still serving the original profile's PVs so anything reading them right now is unaffected, and tells you to reopen HKL Setup. Retry is hidden, because it cannot work.

HKL Setup is editing '6IDB motors', but '6IDB canonical' is now the active profile — most likely selected in another DashPVA window.

Nothing was lost: your settings for '6IDB motors' are saved, and the angle IOC is still serving '6IDB motors' PVs, so anything reading them right now is unaffected.

To work on '6IDB canonical', close HKL Setup and open it again. To stay on '6IDB motors', make it the active profile again in the Workflow window.

Saved settings don't match what was read back — the same profile came back with different values, i.e. something else wrote to it concurrently. This one now lists the differing settings in HKL terms under Show Details:

Detector circle 1 - POSITION: saved '6idb1:m29_RBV:Position', read back 'os:Nu:Position'
Detector circle 2 - AXIS_NUMBER: saved '6idb1:m18_RBV:AxisNumber', read back 'os:Delta:AxisNumber'

The angle IOC did not restart — the genuine restart failure, which is the only one where Retry makes sense. Says the PVs still carry the previous values and nothing was rolled back.

Profiles are named throughout, never shown as database row ids.

One thing worth flagging: the reloaded=… saved=… detail in your report doesn't exist in any branch — I grepped rsm/02 through rsm/04, main, and my working tree. You're running a locally patched build, so it's worth re-syncing before the next test. That detail was genuinely useful though, which is why the equivalent is now shipped.

Two adjacent bugs fell out of tracing this:

• set_locator(None) wrote the literal string 'None' into the environment and the state file, so the next read came back with 'None' as a real, unresolvable locator. Production always passes a real id so it never bit us, but the tests use it as a reset and it silently wasn't resetting.
• The out-of-sync text claimed "the profile is saved" even on the Reload path, which never saves anything.

Your two new items

DETECTOR_SETUP as separate inputs — done. It's one input per value now, generated from DETECTOR_SETUP_FIELDS, so the optional entries (PIXEL_SIZE, DETECTOR_SHAPE, ROI, BINNING, DETROT/TILT/TILTAZIMUTH, per-field units) each get their own row with a placeholder showing the expected shape — row, column, degrees, and so on. Blank means the key is simply absent from the profile, which is how the optional ones are meant to be expressed. Per your note the stored table is unchanged, right down to int-vs-float types, and any key this build doesn't recognise is carried through rather than dropped.

SAMPLE_ORIENTATION — I'd push back on removing this one. It isn't redundant with the primary/inplane/normal directions; it's passed straight through as sampleor= to the xrayutilities QConversion (utils/rsm_geometry.py:310), where it sets which frame the sample azimuth is referenced to. The three direction vectors define the reference frame's axes; sampleor says how the sample circle stack is mounted relative to it. det, sam, and an explicit [xyz][+-] give materially different Q for the same angles.

You're right that 6IDB never needs to change it — but other beamlines in this stack do, and dropping it would silently pin everyone to one convention. If the goal is "stop asking me for something I never touch", I'd rather move it into a collapsed advanced section alongside the other things 6IDB leaves at defaults. Want me to do that?

507 tests pass, ruff clean. Current tip is b500b8b — git pull before re-reviewing.

[Osayi-ANL]

Yes, I think an advanced section or tab to place the 'Sample Orientation' dropdown is necessary to not overstimulate the user. Let's do it that way.

• The protected_static_values change in 51cfa30 (rsm/02-naxis-issue132) reasserts profile values back onto the IOC on a timer, so caput can't be used from the terminal. We want all PVs assertable from the terminal.

I already approved #137, so I have a patch ready: it removes protected_static_values and the reassert loop. Static records still get set once at IOC startup. They track the ioc value and if the ioc is changed via caput or the gui a save dialog comes up and lists the changes. So caput changes can be detected by the ioc to be applied and saved to the profile.
Do you agree? I think that we should still allow caput's from the terminal to also be saved to profiles.

• I don't think we need to save the AXIS_NUMBER as a dataset just a scalar is fine since it doesn'tchange

• There are three ways we save a name associated to a PV(Label, Record Name, and Spec Motor Name). I don't think it needs to be that much maybe removing spec motor name.

[pecomyint]

@Osayi-ANL — status update after re-reviewing current stacked head 86e2605. Thank you for the fixes added since the original handoff. Peco is taking over the remaining RSM work; this edited checklist records the merge gate and avoids duplicate implementation.

Resolved on the current head

• Selected-profile inspection no longer changes the active settings locator.
• Sample Orientation is under Advanced.
• Dirty-window close uses Save / Discard / Cancel and keeps the editor open on unsuccessful save paths.
• Detector round-trip and live construction preserve canonical literals, including PIXEL_SIZE without raw SIZE, and overlay only the six IOC-backed live fields. ROI, binning, tilt, detector rotation, pixel size, and frame-axis order remain available without turning all detector fields into PVs.
• Focused raw absence/type and detector-parity coverage was added.

Remaining takeover blockers

• Finish raw-baseline IOC adoption. Keep both the exact loaded raw snapshot and normalized semantic baseline. Preserve absent keys, raw types, and unknown extensions across every writable record. Loaded axes need ephemeral stable identities so reorder/rename plus concurrent IOC changes merge against the original axis rather than list position. Apply and close must use the same pending-change calculation, including IOC-only changes; divergent edits remain conflicts and write nothing.
• Finish legacy HDF5 compatibility. Canonical axes write ordinal scalar AXIS_NUMBER. Legacy axes must retain their configured channel, validate that observations are invariant, and collapse that series to one scalar. Restore the public optional SpecMotorName IOC/HDF5 contract and its regression coverage.
• Connect the live-grid viewer to the consumer. Reuse control/status channels resolved through settings.py; add a namespaced rsm_grid request/acknowledgement envelope; and replace the viewer-local accumulator with an injectable remote client. Full Q arrays must not cross PVA.
• Prevent stale geometry and metadata. Bind required dynamic angles and I0 from the current frame only; cache only static fields. Snapshot static geometry on the first accepted frame and stop fail-closed if UB, directions, energy, or detector geometry changes. Invalidate cached area-viewer geometry on static changes and clear stale Q after conversion failure.
• Make throughput telemetry truthful and bounded. Replace the unbounded seen-ID set with bounded deduplication and a monotonic high-water mark. Report gap events and actual missing frames separately. Use disjoint session counters for binding and processing rejection, expose upstream pvapy misses separately, keep RPC/PVA names and values consistent, and enforce the frame-accounting invariant in tests.
• Guard memory and concurrency. Apply the shared grid-memory limit before allocation, bound preview payloads, and serialize frame, control, and save access. Save remains stop-only and must not race accumulation.

Restack and gates

• Move fixes to the lowest owning PR: editor/configuration in add: arbitrary-axis HKL chain and profile-driven IOC RSM editor (2/4, closes #132) #137, detector resolution in add: unbinned detector calibration, unit normalization, and tilt (3/4) #138, and live integration in add: live gridded reciprocal-space volume (4/4) #139. Preserve the unrelated Workbench Scan Info change for a separate PR.
• Merge CI-only add: github actions ci #141 first, then perform one bottom-up restack, update stale descriptions/test claims, run locked Ruff plus Python 3.11/3.12 tests, and re-request review on refactor: safe configuration resolution and shared RSM geometry (1/4) #136–add: live gridded reciprocal-space volume (4/4) #139.
• Keep scope keV-only.
• Keep the five hardware results unclaimed until tested against an exact SHA: detector axis order, tilt/detrot sign, actual I0 key, sustained rate/drop counts, and Workbench half-voxel origin.