Repository navigation
Expose compiled materialized results to MCP inventory and viewer review #134
Description
OpenHCS 0.8.6: with PipelineConfig.materialization_results_path set to an absolute output directory, a successful headless run wrote typed measurement CSVs and per-plane ROI ZIPs. The compiled artifact plan listed candidate paths, but openhcs_query_plate_files on the output plate returned zero result records because that directory is outside metadata-declared results_dir. openhcs_stream_plate_files_to_viewer rejected an exact persisted ROI ZIP path as "not found in the plate"; the MCP surface also had no bounded table-read route for the CSV. This prevents same-coordinate raw-plus-result QA without bypassing OpenHCS MCP. Please expose exact runtime materialization receipts/inventory (including absolute configured results paths), bounded schema-bearing table rows, and streaming of persisted ROIs through the existing authorized viewer path. Keep AgentPathPolicy and provenance checks.
Activity
trissim commentedon Sep 29, 2026
Fresh reproduction on the running 0.8.6 MCP/UI route confirms this remains a
blocker after successful scientific execution. Schema-bearing live measurement
previews retain exact disk materialization addresses, but the result-inventory
and existing-file stream routes cannot resolve a separately declared absolute
PipelineConfig.materialization_results_path.A direct request against that result directory fails with
plate_handler_detection_failed; an explicit saved point-ROI archive streamalso fails before file resolution. The default result inventory derives a
different directory through path-planning metadata. This is not missing output
or a reason to relocate files, rerun science, widen read roots, or bypass MCP.
Implementation is now assigned in an isolated main-based worktree, with a
draft PR to be opened at its first commit. Acceptance includes bounded native
CSV/ROI inspection and same-viewer reopening using authoritative materialization
identity/addresses, retained source/axis/object provenance, and AgentPathPolicy
checks. Regression cases must cover relative/default and explicit absolute
destinations, rejected foreign paths/identities, and no science rerun needed.
No duplicate artifact registry, mirror manifest, manual file-extension dispatch,
or caller-side backend ladder. Runtime and fresh-process evidence remain pending.
trissim commentedon Sep 29, 2026
Implementation tracked in draft PR #153. Mencius owns the isolated result-inspection/reopening surface; orthogonal viewer controls are tracked separately in #152 / PR #154. Runtime implementation, focused regressions and fresh-process verification are still pending.
trissim commentedon Sep 29, 2026
PR #153 remains a draft with no runtime Python edits. The isolated worker has traced the mismatch between reconstructed PathPlanningConfig result directories and the richer typed runtime artifact address/materialized-location authority; the checkpoint document records source witnesses and proposed synthetic guards.
Pre-edit scanning is blocked by resources, not a green partial audit: the first complete-scope scan hit NRA's ordinary 20-second deadline (exit 124); a serial explicit-900-second retry exceeded the 768 MiB envelope and was stopped by its worker before findings (exit 143, 68.62 seconds, peak RSS 1604952 KiB). Census and overlay completed, but R1 raw records and detector coverage/omissions remain unavailable. Host memory PSI full subsequently reached 1.14% over 60 seconds. No new heavy loads or Python implementation before main supplies a resource-approved complete-scan path.
Main's direction is also needed to add the existing plate_streaming_service.py consumer to the scoped write set, since it opens microscope context before saved-file identity resolution. capabilities.py remains untouched; coordinate any plate-region/import crossing with Jason before overlap. No science, real-data loading, live GUI/viewer/MCP operations, tests or fresh-process reopening proof have occurred.
trissim commentedon Sep 29, 2026
PR #153 remains draft and documentation-only. Main approved the existing plate_streaming_service.py scope extension; no Python edits have begun.
Common NRA audit status: held by the pre-launch memory gate, not running and not complete. Conditional 2 GiB RSS scan authority requires >=11 GiB available and PSI full avg10 <=1% stable for 10 seconds. The guarded attempt at 2026-09-29T03:58:54Z exited 78 before launching: available 10343272 KiB, PSI 0.00. Receipt: /tmp/openhcs-pr153.LeLQct/scan-recovery-agent.guard. Prior full/raw attempts remain explicitly incomplete.
Loaded NRA declarations confirm compact eligibility (79 detectors, zero AST-retaining context detectors). A same-context compact summary scan is prepared to populate authenticated cache, followed by the required identical-source/config full/raw export. Eligibility is not coverage, and compact presentation is not the required full R1 receipt.
Source lineage also confirms that successful materialization receipts are not yet propagated, existing durable source projections retain image paths only, and PolyStore archive point-kind reopening needs native verification at its dependency owner. No science, real-data loading, viewer operation, MCP connection, native test or submodule gitlink change was performed. Full scan/R1, implementation, native tests and isolated managed/MCP reopening proof remain outstanding.
trissim commentedon Sep 29, 2026
Documentation-only checkpoint pushed to draft PR #153 at f1532c8. It now records actual codec/profile/FULL-gate receipts, not planned completion.
Complete common NRA coverage/full raw R1/source-export evidence remains unavailable. No production Python edits or fresh MCP/GUI operation. Worker remains terminal; no heavy scan while the orthogonal native diagnostic runs. Draft only; no merge/readiness claim.
trissim commentedon Sep 29, 2026
Delivered checkpoint: #153 is merged on remote main (ed129d8), advancing the merged native POINT codec and fixing shared raw/ROI calibration (#171). The actual isolated installed MCP-to-Napari reopening path passed exact payload reconciliation plus six individually opened matched raw/result/combined captures under two numeric windows; the synthetic viewer was closed through public MCP. Frozen H002 science was not rerun or altered.
The broader explicit materialized-directory/location/provenance workflow here is still open and now tracked in draft #173, with the same integration owner. Coordinate writer/progress crossings with active #157; orthogonal controls remain separate #152/#154. No arbitrary-directory implementation, shared-process upgrade or biological acceptance is claimed by the merged checkpoint.
trissim commentedon Sep 29, 2026
Checkpoint shipped: #173 merged as 9a6db7b, independently verified on remote main.
The existing typed file-query backend now accepts an explicit result_directory with kind=result. It reuses native inventory/preview owners, checks directory and traversed file admission including escaping symlinks, and does not fabricate a microscope handler or infer acquisition identity from filenames. Forty focused source and installed checks pass; actual installed public MCP read frozen H002 centre CSV and Z032 native ROI artifacts and rejected an unrelated directory/component filter. Scientific hashes still match; current H002 UI/viewer and desktop were preserved.
New precise retained-file evidence: old results/centres.roi.zip is rejected by the native reader because nucleus_0001.roi has no metadata sidecar. It was not healed, converted through a fallback, or overwritten. CSV rows remain the frozen candidate output, not accepted cell detections.
I retain integration ownership of the remaining full route: observed writer/projection provenance, schema-bearing artifact admission and managed-viewer reopening from explicit materialization locations, plus the separately tracked orthogonal QA (#152/#154). Coordinate the publisher/worker crossing with active #157 rather than duplicating their patch. The installed checkpoint proves persisted-file inspection only; #134 stays open and the blind-analysis goal is not complete.
Full live envelopes and disposition: /home/ts/wt/openhcs-live-point-evidence-20260929/materialized-inspection-20260929. Only the isolated candidate venv was updated, not shared packages/GUI.
trissim commentedon Sep 29, 2026
Native reopening checkpoint #177 merged; remote main verified at abb8b61. Existing ImagePayloadMetadata now persists in native ROI sidecars and drives reopen source/plane/domain/spacing; explicit result_directory is automatically MCP exposed and reuses the admitted inventory from #173. Original source supplies absent calibration. Source 156 / installed 157 focused checks passed with installed module paths verified. Actual installed public MCP on isolated :90/5691 reopened POINT and native-materializer polygon archives from an independent directory, retained exact coordinates/labels and A01/channel1/scale0.65 despite misleading B99/w7 filenames, and produced six personally opened matched captures under two numeric windows. This is synthetic transport only, not biological acceptance or a compile/run-to-reopen pipeline proof. Extra viewer was closed; current H002 processes and frozen science hashes unchanged. Full envelopes, requests, hashes and disposition: /home/ts/wt/openhcs-live-point-evidence-20260929/native-reopening-20260929. Issue stays OPEN under the same integration owner: runtime source-bearing writer-success admission with #157, continuous pipeline journey, historical H002 missing provenance recovery, 3D/orthogonal installed viewer proof and fresh blind biological QA remain unfinished. Resource headroom guard still warns, so no extra agents were started.
trissim commentedon Sep 29, 2026
Read-only frozen H002 reconciliation checkpoint (no science rerun or artifact repair). Same integration owner: this blind-analysis coordinator.
The current isolated installed public MCP returns the complete persisted centre CSV. Every field in its 28 rows and the separate count CSV agrees with the saved terminal runtime receipt for execution e485a4a4-49a1-48c4-9261-f9194dc6e6c2, keyed by object identity rather than cardinality alone. Independently decoding all 44 label ZIPs through native PolyStore yields 828 contour members / 773 distinct object-plane metadata records; per-object retained area sums and weighted XY centroids reproduce the runtime volumes and centres exactly (718717 total voxels). This reconciles extraction metadata, not rasterized polygon pixels or biological identities.
The historical archive limitation is now quantitatively explicit: all 28 standard ImageJ point members preserve the declared float32 XY projection but store rounded Z, maximum 0.4970393447358106-voxel deviation from fractional runtime Z. The point ZIP has no native metadata sidecar and is still strictly rejected. None of the 44 native label ZIPs contains ImagePayloadMetadata source binding. Filenames were not used as acquisition authority; no replacement metadata/points, compatibility reader or scientific Run was introduced.
All 111 frozen science digests passed before and after. Reproducer: 0.58 seconds / 50964 KiB peak RSS. Full installed MCP envelope, per-object comparisons, native rejection, repro helper and limits are retained under /home/ts/wt/openhcs-live-point-evidence-20260929/frozen-H002-reconciliation-20260929.
#154 is merged with native orthogonal navigation separately live-verified, but this is not restored historical 3D point/source review. #134 and #152 remain open for the continuous source-bearing compiled-materialization-to-reopen result journey. Existing H002 GUI/worker/viewer, independent frozen AMBIGUOUS decision and held-out/reference boundaries are preserved. No new agents or heavy runs while the headroom guard warns.
trissim commentedon Oct 1, 2026
Singer owns the remaining declared persisted ROI directory admission investigation; #398/#399 remain complete/closed. Visible diagnostic draft #404 has exact Dalton candidate-2 request/error, 1 passing supported explicit-directory read-only admission (17 records, exact neurons ROI), and 2 red original mixed-output publication→reader cases (simple/nested arbitrary destination names). No ROI geometry/pixels/native operations. Missing relation: RuntimeArtifactMetadataTarget supplies results_dir=None, so native metadata does not admit its result directory to the original reader. Shared function_outputs.py release requested on #394 before source edits; no competing implementation. Existing typed result_directory selection works for admission without invented receipt, but separate ROI provenance/stream acceptance remains untested. Original source failures archived byte-exact; own 160 KiB scratch released. Receipt in #404 identifies canonical path and original stdin:45/stdout:44741-44789; Dalton source/runtime unchanged.
Publish exact source-proposal R0 and declaration behavior receipts for …
Publish every successfully saved result destination through output ow…
Bind graph ROI source metadata through existing archive owners (#134)
refactor(runtime): consolidate producer and projection authorities
Keep publication and output identity on their nominal owners
Preserve native source calibration through viewer receipt replay
Fix scalar viewer admission before aggregate domain validation
Record calibrated public native receiving acceptance