Skip to content

Export requested source identity from typed measurement provenance (#731) - #734

Merged
trissim merged 4 commits into
mainfrom
fix/spreadsheet-source-provenance-731
Oct 5, 2026
Merged

trissim merged 4 commits into
mainfrom
fix/spreadsheet-source-provenance-731

Conversation

@trissim

@trissim trissim commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes #731. Spreadsheet flags already reached the exporter, but copying existing
Image fields could not export filenames or acquisition coordinates never
projected from the typed measurement provenance.

The existing CellProfilerDatabaseColumnDialect now owns metadata/file-value
projection. CPA consumes the same methods; its replaced path/metadata mapping
is deleted. Spreadsheet export consumes producer-owned per-plane identities,
the existing named-source selection and image-set numbering. It does not reload
pixels, guess source names, change grouping or synthesize absent components.
Original extracted metadata keeps its spelling/value and its existing
participation rule (only actual extraction records). Requested acquisition facts
are folded separately using the existing source consensus. Both new projections
are driven by the original flags: false flags do not synthesize extra columns;
already declared upstream Image features are always retained. Conflicting file values
still go through the original sparse accumulator conflict checks.

Owner and consumer closure

Production changes: database_column_dialect.py, CPA
analyst_export.py:collect_image_provenance, and spreadsheet
_measurement_tables/_source_metadata_measurement_rows.
Existing SourceMetadataFields.component_domains/original_items remain the
source-field authority. Existing SourceImageProvenance.for_source_plane and
for_source_image remain the plane/name authority. No runtime-export schema,
materialization, stream or measurement producer changes; Singer owns #728/#732.

Static family evidence is retained at
/home/ts/wt/openhcs-issue-batch-20260929/engineering722/EXPORT-FAMILY-BEFORE.json.
All matching tracked production Python parsed; dynamic receiver resolution was
read semantically, not claimed proved by AST. IMPL-12/BOUND-2: remove the separate
CPA mapping and make both consumers use the existing dialect and metadata owner.

Qualification state

68 spreadsheet/CPA controls PASS in original export-controls06.log/.time,
handle25254 terminal0, 10.24s whole process, peak421648KiB, swaps0. Includes a tiny object
table with no upstream Image features, all four flag combinations for both explicit/axisless rows, both
independent contributor orders, absent axes, preserved original extraction path
templates and conflicting supplied filenames rejected. All pre-existing exact
CSV/metadata assertions remain unchanged and pass.

Original negatives are retained: controls01 used the old receiving09 native CSV
ABI (5 arguments, current source requires6); controls02 refused the old ignored
checkout extension origin before tests. controls03 used the correct receiving10
native ABI, passed50 and exposed12 role/default-projection regressions. Corrected
the source role fold and tied extra projection to existing requested flags, not
weakened tests or native compatibility. controls04/05 corrected64PASS remain
preserved. Final source checkpoint271229f93 includes the original numbering
owner's reusable declared-source slice selection: no axisless provenance loss,
no new grid/scope, ambiguity rejection and original summary anchor unchanged.
No source edit occurred after the68-control run; the commit packages those bytes.

Family final evidence: engineering722/EXPORT-FAMILY-FINAL.json; no parse
omissions, dynamic receiver limitation remains explicit. Foreign gitlinks and
untracked histories were not staged or changed.

Actual ordinary installed/public MCP acceptance

Whole receiving11 source7300be4d190c225ad7d5b38d8307c5565faaed34 contains the
exact reviewed271 production bytes, installed from ordinary wheel
d23fcf2ef93a7d764a938f07d9d92366b1eeab94a424074350454bc74524805e.
Whole proof:913 RECORD entries,813 tracked source files,790 Python files,
13 packaged/synced skill files. No installed overlay or current scientist change.

Original persistent MCP client7585/native36406 used the existing registered
primary-object/intensity/PLATE spreadsheet declarations, three immutable tiny
12x15 synthetic images, no custom registration or viewer.
Forward session2 job4/16a2f990-5d6d-4e2c-88ae-4390f622fc8f and reverse session3
job6/78829037-80a7-4e71-ae83-90e099c9b32e each COMPLETE/errors[].
All four actual Image/Cells CSVs reconcile one row, declared WellA01/site1/Z1/time1,
blank varying channel, exact producer-named filenames/paths and independent
intensity values430/255 and889/255 (means divided by12). Both orders give identical
measurement values. Original input and complete PipelineDocument hashes unchanged.

Explicit remaining upstream limit: the photometry producer retains only the
first selected image's source identity, despite correct numerical features for
both images. Forward export represents Mask+Process; reverse represents
Mask+Nuclear. These are original virtual workspace filenames with parent .;
this exporter does not fabricate a missing contributor or override it with an
assumed physical path. Singer accepted separate producer triage. This limit is
not represented as complete multi-image provenance or a biological/count claim.

Original session1/job2 STEP_INPUT selector failure is retained; corrected source03
uses declared PIPELINE_START bindings/new destinations, no guard relaxation or
failed-job replay. Local helper02's wrong enum-value assertion and the reverse
CLI's undeclared compile spelling remain explicit predispatch negatives.

Actual receipts:
/home/ts/wt/openhcs-issue-batch-20260929/engineering-pre-first-routing-20261004/receiving11/public88/ACTUAL-RECEIVING.rst;
under its ADMIN734_88/author-workspace/output:
PUBLIC-EXPORT-VERIFIED04.json, verify-export-source04.log/.time,
provenance-forward03.json, provenance-reverse03.json, corrected
export-case03/SOURCE-PACKET.json and original runtime/mcp.{stdin,stdout,timing}.
CSV reconciliation terminal0,0.05s,17100KiB RSS,Swaps0. Typed exact-owned
native close returned errors[] and acknowledged/process_exited/endpoint_terminated
alltrue for36406/create1791168723.62; independent PID/listener checks agree.
Original recorder terminal2 at23:25:20-04:00 retains rejected command history,
not shutdown failure: MCP4192764 independently absent and original scope
inactive/dead. Dewey owns the original7585 handoff/journal seal. All original
values/output evidence was retained before closure; no viewer was started.

@trissim

trissim commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

Independent parent source review at7300be4: read complete PR diff and determining image-number projection; verified production bytes unchanged from271229f AST/source-qualified checkpoint. Existing CellProfilerDatabaseColumnDialect now owns source-file/extraction/acquisition formatting and both CPA and spreadsheet consume it; old CPA mapping is deleted. SourceMetadataFields and SourceImageProvenance remain actual identity/domain owners. Axisless selection is carried by existing CellProfilerImageSetNumbering rather than another numbering registry, preserving ambiguity rejection and artifact-summary anchoring. Search confirms the related new helpers have their declared single owners and expected consumers. Existing complete production-symbol evidence reports zero parse omissions; this review is focused semantic source review, not a fresh global NRA R1 proof. I found no source-level reason to delay this family beyond the already-running installed public export acceptance. No new tests or package rebuild requested; no hosted CI hold.

@trissim
trissim marked this pull request as ready for review October 5, 2026 03:22
Copilot AI balanced review requested due to automatic review settings October 5, 2026 03:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@trissim
trissim merged commit 2bd9fa4 into main Oct 5, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spreadsheet export omits requested source identity when upstream Image features are absent

2 participants