Skip to content

Declared secondary image sources lose CellProfiler database source-information values #444

Closed

Description

@trissim

At frozen PR #394 source 919fee286f8003a00aafc023ad28767d9713842b, a fresh ordinary 1w/1t Advanced Segmentation execution completes, but its SQLite source-information columns omit shared illumination input images.

Matched fresh native CellProfiler 4.2.8.1 and OpenHCS use the exact same physical input paths and SHA256 values: 10 TIFF acquisitions plus five shared 20585_Illum*.npy arrays, two image sets. The OpenHCS input workspace retains 20 declared source occurrences, including illumination arrays reused across sites. The original native well selection is A01; OpenHCS's imported logical frame is W001. Scientific object counts, IDs and relationship integer data match. The original exact integer gate finds 20 Per_Image fields across two rows, 40 cells. The independent complete source-field inventory finds 90 missing cells total: forty integer geometry/frame/series cells, ten Scaling cells, and forty FileName/PathName/URL/MD5Digest cells.

For each of IllumER, IllumHoechst, IllumMito, IllumPh_golgi and IllumSyto, native exports Frame=0, Series=0, Height=520, Width=696. OpenHCS exports NULL. Preserve the original native and candidate databases and the exact failure; do not exclude these values to call the complete output equivalent.

The generic production frontier is CellProfilerAnalystProjectionBuilder._collect_image_provenance and CPATableRowProjection.collect_image_provenance in openhcs/interop/cellprofiler/analyst_export.py. The former traverses recorded runtime image aliases; a declared secondary input can be borrowed without becoming an output record. Source projections/binding declarations remain the authority for source occurrences. The existing _source_image_projection_values also assumes PIL-readable files, which does not cover these declared NPY image inputs if reached. A source inventory over 703 production Python modules traces the existing owner closure: the builder already owns the compiled source plan and context; declared source occurrences must be consumed from VirtualWorkspaceSourceProjection, and file header/read facts belong to the existing ImageFileFormat family. The final runtime store was not captured, so actual source-only record enumeration remains a separately unobserved detail.

The repair should derive source-information export from the existing source authority and existing pixel/backend capabilities. No fabricated runtime output records, per-alias NULL fills, second source registry, extra pixel copies or relaxed comparison guards. Cover declared secondary inputs, repeated shared physical files across sites, correct source/image-set association and normal TIFF inputs.

This issue is being addressed in draft PR #394. Installed reopening work in #404 and singleton binding work in #437 remain separately owned.

Retained local evidence:

  • /var/tmp/openhcs-latest-ordinary-advanced-ifc-20261002/source-freeze.json
  • /var/tmp/openhcs-current-native-advanced-ifc-v2-919fee-20261002/cp_tutorial_advanced_segmentation_final/ (native DB, input witness and native report)
  • /var/tmp/openhcs-current-native-advanced-ifc-v2-comparison-20261002.log (original exact discrete failure)

The separate IFC case passes the existing strict exported measurement comparison with zero differences. No claim of latest-main all30 parity or Advanced complete-output acceptance follows.

Independent unchanged-output evidence: /var/tmp/advanced-fresh-native-missing-illum-metadata-20261002.json; source ownership/mandatory countercontrols: /var/tmp/advanced-source-occurrence-cpa-owner-closure-20261002.json. Existing scientific database comparison passes zero differences separately, because its declared contract excludes structural source-information families. This does not waive the exact metadata failure.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions