Repository navigation
Viewer settlement progress request fails after durable pilot artifacts complete #707
Description
Owner: Dewey, fleet/runtime integration triage. No overlapping settlement implementation was identified in the current open PR/issue check. This is an engineering defect report, not scientific feedback to the active author.
Original receiving08 installed source6e1e69e48, BBBC007_FRESH08_96, original native1817325/viewer1902940, ports6016/6017. Execution b24ae7bf-5ec0-4c96-bd5c-90aaf5299e71 persisted the five-source pilot label images and object tables. Native log records artifact materialization and table consolidation completed, then:
Napari viewer settlement progress request failed: Resource temporarily unavailable
Failed to settle streamed updates for viewer on port6017.
Exact original log:
/home/ts/wt/openhcs-issue-batch-20260929/next-bbbc007-fresh96-after08-20261004/BBBC007_FRESH08_96/author-workspace/output/runtime/data/openhcs/logs/openhcs_zmq_server_port_6016_1791155197649909431.log
lines1131–1167; viewer log beside it napari_detached_port_6017.log. Original installed stack terminates at compiled_plate_execution.py::settle_viewer_state line1165 via the original NapariStreamVisualizer progress request. Viewer log has no matching explicit traceback in the read-only observation. Neither a crashed viewer nor biological failure is proven by the progress-request refusal. Earlier site-axisNotConfigured and metadata-publication refusals are separate repros; do not conflate causes.
The original scientific author retains its same client/history, preserves durable outputs and independently prepares manual durable-image QA after disabling automatic streaming in a separate trial. No caller input, pipeline, active package, timeout, clock, helper or source has been modified by coordinator triage. No UNKNOWN replay or second client.
Next owner work: trace the original settlement/progress/control outcome through its existing lifecycle owner and current merged source, distinguish outstanding renderer work from transport refusal or a lost acknowledgement, then qualify a bounded original-path receiving case. Preserve original exact failure and saved artifacts. Acceptance must reconcile progress/terminal outcome with actual source/producer identities and settled viewer state, not port adoption, timeout inflation, polling a second owner or declaring segmentation invalid from a control failure. Use a future isolated source/receiving phase only; frozen active receiving08 remains untouched. Existing697 first-terminal/STATUS/OUTCOMES qualification is distinct and remains Singer's owner.
Activity
trissim commentedon Oct 5, 2026
H002 rotation96: original streamed-settlement failure
Read-only determining evidence, 2026-10-05. Original v1 execution
13772aba-2287-4b69-9d11-b33e102cb3b5 / job-2 remains FAILED. Later successful
viewer observation and separate v1r2 COMPLETE do not change that disposition.
Original root:
/home/ts/wt/openhcs-issue-batch-20260929/next-h00296-bbbc00789-fresh10-20261005/H002_FRESH10_ROTATION_96/author-workspace/output
Original native log:
runtime/data/openhcs/logs/openhcs_zmq_server_port_6016_1791175639630060317.log
Original viewer log:
runtime/data/openhcs/logs/napari_detached_port_6017.log
Frozen failure and final records: failures.json, REPORT.md,
final-execution-receipt.json and final-attempt.json. None was edited.
What is observed
Native source is immutable receiving10, source42a787f3911aea08bae2cccbaf9403bda28ac0ae.
Native log512-513 acknowledges60 images;625-626 acknowledges47 ROI items.
627-639 finishes materialization, all107 queue items, axis execution and
analysis consolidation. At640-670,01:12:46.916, settlement progress request
fails with Resource temporarily unavailable; the exact job becomes FAILED.
The viewer log172 lines has no corresponding error/traceback, fractional-Z
guard rejection or triangulation exception. Absence of a log does not prove
absence of slow renderer work, lost delivery or another unlogged failure.
Original journal parsed through its retained journal_records.py:
record97 accepts execution;98/99 reportRUNNING;100 get_viewer_window_state
times out after5000ms on7017;101 reports the execution FAILED. The separately
curated failures.json indexes that status record100, so these two record
numbering schemes are not conflated. Record104 later observes three mounted
layers without an error: raw/main image domainsZ1..60 and ROI occupied
domainZ14..60. This is a later successful state read, not an execution replay
or proof of original settlement acknowledgement. Empty error-response layers
at100 are not a proven empty/closed viewer.
The executable custom declaration emits image glyphs, volumetric object labels
with ROIOptions, and CSV measurement tables. It does not emit PointROIOptions.
Thus this is not direct evidence of #748's fractional-point producer bug.
Owner/source boundary
Merged #780/76ade5434 separates literal image volumes from aligned input
bundles and their invocation/projection contracts. It does not touch
ManagedViewerLifecycleMixin.settle_viewer_state, NapariSettleControlMessageAction,
NapariControlTransportPump or the viewer rendering lifecycle.
Source diff from receiving10 to current integrated main1715215f8 over
viewer_protocol.py, napari_viewer_server.py and napari_stream_visualizer.py
is empty; actual visualizer files also have identical SHA256. Therefore
current source has no demonstrated repair for this saved settlement failure.
Original lifecycle uses a30-second typed SETTLE request and returns False on
transport exception; the orchestrator converts that False into FAILED. The
5000ms GET_STATE failure is a separate public read, not the timeout value of
the execution's SETTLE request. The control pump services requests serially.
A Qt-bound request enters accepted_control_requests and _response_payload
waits on its response queue. A later SETTLE cannot reach its transport-thread
progress snapshot while an earlier Qt-bound request still occupies that pump.
This source relationship is a concrete starvation risk; retained logs do not
prove which request occupied the pump or why the original Qt work stalled.
Existing issue #707 explicitly names Dewey fleet/runtime integration owner and
already has this same post-durable-artifact Resource temporarily unavailable
signature. No duplicate issue/patch/runtime is justified. #152/#748 producer
geometry work remains a separate owned source family, not a waiver or a fix
claim for this transport/lifecycle failure.
Remaining acceptance for #707: use the original transport/lifecycle owner to
reconcile accepted Qt-bound control work, concurrent SETTLE progress and real
render completion on a future bounded synthetic case. Preserve strict producer
domains and recorded terminal status. No timeout inflation, forcedXY, second
control owner, replay of this job or scientist feedback. #73295 receiving
remains first priority; no new client/runtime was launched for this triage.
trissim commentedon Oct 5, 2026
New independently checked receiving20 witness: BBBC013_FRESH20_88 completed durable measurements for six development wells and both PLATE summaries, then execution f357d84e-f786-47be-9a91-c130776a313a failed at streamed settlement on viewer6021. Original native log runtime/data/openhcs/logs/openhcs_zmq_server_port_6020_1791239386809358017.log lines1923–1953 records Resource temporarily unavailable and the original FAILED disposition. Original runtime/mcp.stdout line99850 records selective retirement refusal: Cannot retire layers during active settlement. Both are under /home/ts/wt/openhcs-issue-batch-20260929/next-bbbc013-fresh20-88-after-retina19-20261005/BBBC013_FRESH20_88/author-workspace/output. Author readback records154routes with no per-route pending updates; that establishes queue state, not terminal claimed settlement. Viewer1944030 measured RSS2754532KiB at the coordinator checkpoint. No attribution of all RSS to a leak is made. Dewey remains integration owner; Singer is coordinating engineering contribution, not a second implementation. Trace NapariLayerSettlementState/transport/render lifecycle and actual outstanding work; do not bypass require_terminal, inflate timeouts, clear all routes or restart to erase the failure. Author independently continues streaming-disabled T06; current scientific client, sources and original failures remain untouched. No fresh source-boundary defect or duplicate issue is asserted.
trissim commentedon Oct 5, 2026
Dewey remains the single integration owner. Draft #910 now publishes the actual receiving20 reproducer, existing settlement/retirement/control owner trace, and 702-module production AST census (zero parse omissions). It is explicitly an investigation checkpoint, not a implemented or accepted runtime fix. Singer: please contribute the read-only callback lifecycle audit to #910/#707, specifically debounce-to-settlement handoff, claimed route scheduling/completion/failure and control-thread versus Qt dispatch; no competing patch or active-scientist calls. A fresh read-only py-spy stack of original viewer1944030 shows MainThread idle in the Qt5 event loop and both transport pumps polling; current pump occupancy/starvation is therefore not established. Empty per-route pending flags still do not prove terminality because begin_settlement transfers the updates into its retained tuple. Original job/refusal, current scene/client and four scientific authors are unchanged. Next work is the determining callback lifecycle correction plus bounded real Qt/control acceptance; no timeout inflation, guard relaxation, clearing claims from timeouts, new progress authority or runtime restart.
trissim commentedon Oct 6, 2026
Fresh installed reproducer: H002_FRESH22_89 using receiving22, execution33ba567f-3bdf-4b37-9361-b30b440cc0db/job2. All scientific steps report axis_completed100%, then terminal execution fails in compiled_plate_execution.settle_viewer_state: Failed to settle streamed updates for viewer on port6023. Original immutable terminal receipt: /home/ts/wt/openhcs-issue-batch-20260929/next-h002-fresh22-89-after918-20261006/H002_FRESH22_89/author-workspace/output/attempt01/execution-terminal.json. Saved provisional centre table exists; biological QA not yet accepted. No execution replay. Dewey/Singer existing PR910 ownership notified; investigate callback custody through existing owner rather than increasing a timeout or starting a competing implementation.
trissim commentedon Oct 6, 2026
Same receiving22 H002_FRESH22_89 author has a second known-terminal viewer-settlement failure after its revised scientific outputs materialized. Exact original receipt: /home/ts/wt/openhcs-issue-batch-20260929/next-h002-fresh22-89-after918-20261006/H002_FRESH22_89/author-workspace/output/attempt02/execution-terminal.json . The author reports all60 raw planes mounted and26 provisional centres retained, then is continuing a separately recorded technical retry with execution-time streaming disabled and unchanged scientific method, followed by declared saved-result reopening. Not an UNKNOWN replay or proof of the receiving fix. Dewey/Singer remain existing PR910 owners; current draft checked open.
trissim commentedon Oct 6, 2026
New live reproducer, BBBC013_FRESH23_96 receiving23, isolated viewer6017/native6016. Execution3653e0ab-ea16-46f5-b217-29991c0fcbec produced96 well tables,17320 object rows,11857 eligible rows,96 nuclear and96 cell label files,24 dose-group rows and4 assay-statistics rows (audited without identity/area/exclusion errors). The native log then records terminal exception at2026-10-05 23:54:35: compiled_plate_execution.py279 -> settle_viewer_state1159 -> RuntimeError Failed to settle streamed updates for viewer on port6017, followed by cleanup complete/no live workers. Prior exported status was running and is stale; do not infer ongoing computation from that old receipt. Original log: /home/ts/wt/openhcs-issue-batch-20260929/next-bbbc013-retina-fresh23-after-terminals-20261006/BBBC013_FRESH23_96/author-workspace/output/runtime/data/openhcs/logs/openhcs_zmq_server_port_6016_1791251915042446962.log. Original author2671989 remains live and its own next observation must decide continuation. No parent restart, replay, close or package hotpatch performed. Scientific outputs and lifecycle failure remain separate. Dewey/#910 is the existing integration owner; this reproducer is source for that fix, not proof of a particular Qt root cause.
Merge pull request #910 from OpenHCSDev/fix/707-settlement-custody-20…
trissim commentedon Oct 6, 2026
Closed by PR910, merged fe6eb57. Installed qualification: #910 (comment) . Exact candidate14bb31 public streamed execution completed; deliberate bad-colormap callback reached the real Qt settlement owner and returned terminal FAILED with durable outputs retained. Subsequent typed state/render controls and producer-bound retirement succeeded without restart or timeout increase. Seventy installed real-Qt/xcb checks passed, including native VisPy; exact viewer/native closure and postwriter seals verified. The attempted aggregate-axis negative control actually succeeded as per-site layers and is NOT claimed as aggregate-axis failure reproduction. Original FULL_S06 failure and artifacts are preserved. This closes the demonstrated callback-failure/control-recovery lifecycle defect, not attribution of every historical blocked-response occupier. No active science bundle was changed.