Skip to content

DL3: an output dataset records what produced it (draft, handed off) - #1236

Draft
trissim wants to merge 14 commits into
mainfrom
refactor/DL3-output-provenance
Draft

trissim wants to merge 14 commits into
mainfrom
refactor/DL3-output-provenance

Conversation

@trissim

@trissim trissim commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Lands after DL1 and after four library releases.

  • python-introspect>=0.3.0,<0.4 is pinned (0.3.0 is on PyPI: family members are stored under a tag). ObjectState, formbridge, pyqt-reactive and zmqruntime declare python-introspect<0.3; each needs a release widening that bound before this branch can be installed.
  • The storage part (0.6.3: backends state a file's content hash and size, or why they cannot PolyStore#41) has to be released as PolyStore 0.6.4, which requires arraybridge 0.5.0; OpenHCS main is on arraybridge 0.4.1 until refactor/DL1-device-and-memory-types merges. Pins to set then: polystore>=0.6.4,<0.7, arraybridge>=0.5.0,<0.6, submodule external/PolyStore at the 0.6.4 tag (it points at 5301d35 of the Transport mode fix #41 branch until then; pyproject.toml still says polystore>=0.6.3,<0.7, which is wrong).

State after the third hand-off: stored family members name their class (step A) and the record's owner stores it, with both oversized classes back to main's size and local guardrails at rc=0 (step B). Not started: the recorded-content defects, PolyStore 0.6.4, the dataset-root reader and the acceptance re-run. Exact list: "Third hand-off" in the receipt.

DRAFT: handed off mid-job. The receipt docs/refactor/consolidation-20261009/DL3-output-provenance.md has the design, what is implemented, the acceptance results and the ordered list of what remains.

What this does

Gap 6 of the Cellpose/N2V2 investigation (#1230): an output dataset now says what produced it.

  • The compiler builds a RunDeclaration (complete pipeline text with every parameter written, the saved global config, and the resolved configuration with the scope and class ObjectState reported for each value). It travels on the CompiledExecutionBundle.
  • When execution starts, the parent process builds the RunRecord: per step the function identity (registry key, distribution and version, imports; source hash and text for a custom function), every parameter with its effective value and source (pipeline, function_default, run_time), assigned devices and memory types, and each declared file with the content hash the storage layer reports (PolyStore file_content, 0.6.3: backends state a file's content hash and size, or why they cannot PolyStore#41).
  • The record's owner stores it in openhcs_metadata.json under run_records, in a locked update right after the dataset's metadata is published, with produced_files mapping each saved path to the producer identity a viewer item already carries.
  • Reader: RunRecord.for_dataset, RunRecord.for_output_file, RunRecord.rebuild(); agent capability openhcs_read_run_record (MCP tool and read-run-record CLI command).

State

  • Tests: tests/unit/test_run_record.py (17), tests/unit/test_run_record_capability.py (7), tests/integration/test_run_record_round_trip.py; round trip gives byte-identical TIFF and CSV outputs.
  • Acceptance (CPU, Cellpose 3.1.1.3, 3 BBBC039 fields; N2V2 denoise with a model file): passes, one defect found (a declared path left at its default is recorded as written in the pipeline).
  • guardrails_local.sh: rc=1 (ratchet rose: the record is handled as string-indexed dicts; code was added inside PipelineCompiler and OpenHCSMetadataTarget). Not yet fixed.
  • Not run: smoke_local.sh, parity.
  • Depends on 0.6.3: backends state a file's content hash and size, or why they cannot PolyStore#41 (0.6.3) being merged and released.

DL3 · PR draft · guards 0/1 (guardrails failing) · not ready to merge

🤖 Generated with Claude Code

trissim and others added 14 commits October 12, 2026 14:32
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Function references and registry entries state where a function comes from
(distribution, version, imports; source hash for custom functions). Declared
path kinds say what content a run reads. RunRecord builds, reads and rebuilds
the record. PolyStore 0.6.3 (file_content) pinned. Not wired into the
compiler, the workers or the metadata writer yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The compiled bundle carries a RunDeclaration (complete pipeline document,
saved global config document, resolved configuration with sources). Workers
report the producer identity of each saved file. The parent hashes declared
input files once before workers start and the metadata writer stores the
record and the produced files in the dataset document's one transaction.

Complete rendering (clean_mode=False) now writes the signature default of
every function parameter a pattern leaves out, except those the platform
supplies; range and module-exported callable defaults get formatters.

The record stores the saved global config (global_config_code), not the
effective config: rebuilding then resolves through the same hierarchy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reads the record of what produced an output dataset, or of one output file,
through RunRecord. Long texts are returned as hash and length unless asked for.
The complete pipeline text writes every parameter; the authoring round-trip
test states that for the full rendering.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cord's pipeline text is its own rendering

Item 1. RunRecord, StepRecord, StepCalls, FunctionRecord, RecordedParameter,
DeclaredFile, SoftwareEnvironment, ConfigValuesFromOwner, DatasetRunRecords
replace the nested dicts. A parameter's source and a configuration value's
source are registered class families stored as their registry key. The stored
JSON is python_introspect.to_jsonable of the record and is read back with
python_introspect.dataclass_from_mapping; the identity and file-content
families are read as unions of dataclasses. Resolved configuration is grouped
by the owner that supplied it, so each scope id is written once. The reader
and the openhcs_read_run_record capability return these types; the capability
returns what each step ran by default and the whole record on request.

Item 2. clean_mode=False keeps its meaning (every stored field, no parameter
the pipeline does not state). The record's copy is a separate rendering,
PipelineDocumentCodec.render_with_every_parameter. The two tests that were
weakened for the changed flag are restored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The function identity and the file content of a record are registered families
(defined_by: installed_module / source_text; content_kind: hashed / not_hashed)
and the record's fields are declared by the family, not by a union. The codec
writes {registry key: fields} and reads the class from the key; it no longer
picks a member by the fields present.

UiLiveMeasurementEntryState imports its preview type at run time: the codec
now reads declared field types when encoding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wo oversized classes are back to main's size

- ProducedFiles.stated_by_step replaces OpenHCSMetadataTarget.saved_path_producers.
- RunRecord.store_with_saved_files and RunRecordWriter (an AtomicMetadataWriter)
  store the record in its own locked update right after the dataset's metadata
  is published. OpenHCSMetadataTarget.finalize_completed_dataset and
  AtomicMetadataWriter.reconcile_completed_dataset are main's again.
- The compiler builds the bundle of an empty compilation through one factory,
  CompiledExecutionBundle.with_nothing_to_run.
- StepExecutionObservation.is_empty is derived from the dataclass fields.
- The attribute that holds a custom source namespace is named in one place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…line, what remains)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant