Repository navigation
Declared secondary image sources lose CellProfiler database source-information values #444
Description
At frozen PR #394 source 919fee286f8003a00aafc023ad28767d9713842b, a fresh ordinary 1w/1t Advanced Segmentation execution completes, but its SQLite source-information columns omit shared illumination input images.
Matched fresh native CellProfiler 4.2.8.1 and OpenHCS use the exact same physical input paths and SHA256 values: 10 TIFF acquisitions plus five shared 20585_Illum*.npy arrays, two image sets. The OpenHCS input workspace retains 20 declared source occurrences, including illumination arrays reused across sites. The original native well selection is A01; OpenHCS's imported logical frame is W001. Scientific object counts, IDs and relationship integer data match. The original exact integer gate finds 20 Per_Image fields across two rows, 40 cells. The independent complete source-field inventory finds 90 missing cells total: forty integer geometry/frame/series cells, ten Scaling cells, and forty FileName/PathName/URL/MD5Digest cells.
For each of IllumER, IllumHoechst, IllumMito, IllumPh_golgi and IllumSyto, native exports Frame=0, Series=0, Height=520, Width=696. OpenHCS exports NULL. Preserve the original native and candidate databases and the exact failure; do not exclude these values to call the complete output equivalent.
The generic production frontier is CellProfilerAnalystProjectionBuilder._collect_image_provenance and CPATableRowProjection.collect_image_provenance in openhcs/interop/cellprofiler/analyst_export.py. The former traverses recorded runtime image aliases; a declared secondary input can be borrowed without becoming an output record. Source projections/binding declarations remain the authority for source occurrences. The existing _source_image_projection_values also assumes PIL-readable files, which does not cover these declared NPY image inputs if reached. A source inventory over 703 production Python modules traces the existing owner closure: the builder already owns the compiled source plan and context; declared source occurrences must be consumed from VirtualWorkspaceSourceProjection, and file header/read facts belong to the existing ImageFileFormat family. The final runtime store was not captured, so actual source-only record enumeration remains a separately unobserved detail.
The repair should derive source-information export from the existing source authority and existing pixel/backend capabilities. No fabricated runtime output records, per-alias NULL fills, second source registry, extra pixel copies or relaxed comparison guards. Cover declared secondary inputs, repeated shared physical files across sites, correct source/image-set association and normal TIFF inputs.
This issue is being addressed in draft PR #394. Installed reopening work in #404 and singleton binding work in #437 remain separately owned.
Retained local evidence:
/var/tmp/openhcs-latest-ordinary-advanced-ifc-20261002/source-freeze.json/var/tmp/openhcs-current-native-advanced-ifc-v2-919fee-20261002/cp_tutorial_advanced_segmentation_final/(native DB, input witness and native report)/var/tmp/openhcs-current-native-advanced-ifc-v2-comparison-20261002.log(original exact discrete failure)
The separate IFC case passes the existing strict exported measurement comparison with zero differences. No claim of latest-main all30 parity or Advanced complete-output acceptance follows.
Independent unchanged-output evidence: /var/tmp/advanced-fresh-native-missing-illum-metadata-20261002.json; source ownership/mandatory countercontrols: /var/tmp/advanced-source-occurrence-cpa-owner-closure-20261002.json. Existing scientific database comparison passes zero differences separately, because its declared contract excludes structural source-information families. This does not waive the exact metadata failure.
Activity
docs(perf): record fresh CP comparison and measurement frontier (#384, …
fix: export declared secondary image source occurrences
refactor: derive header image layout from pixel semantics owner
docs: qualify source-only metadata repair against native CP (#444)