Skip to content

Validate measurement subjects and expose exact CellProfiler selectors (#133, #204) - #205

Merged
trissim merged 24 commits into
mainfrom
feat/exact-label-selection-20260929
Sep 30, 2026
Merged

trissim merged 24 commits into
mainfrom
feat/exact-label-selection-20260929

Conversation

@trissim

@trissim trissim commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Measurement declaration checkpoint (#133, #204)

Implementation owner: Zeno. Integration owner: OpenHCS issue-batch coordinator.
Worktree: /home/ts/wt/openhcs-exact-label-selection-20260929.
Base: openhcsdev/main at dbf1c7a8b (merged PR259), integrated normally through
0e04ad6c7. Earlier main260 c86f562e1 was integrated through b7e6a2fbc after
implementation/acceptance commit 4f4332368. The initial
278-test runtime checkpoint used 32d070c26 (PR255), integrated through
85d8b2ae7; the refreshed journey below also verifies current main260 and the
latest owner cutover. The preceding checkpoint merged
589c33a12 through 684e4bbd3,
and de23449a4 through 21c15ad41 before that.
The output-policy checkpoint used 283b21275.
The earlier checkpoint used 98d9b9d23; the runtime checkpoint integrates new
main through normal merge 9f5d444515. No parent/main-worktree edits, rebases,
resets, installed-source edits, or gitlink changes.

Closes #133. References #204; paired-channel headless execution now passes, but
actual GUI/installed acceptance remains open. This is a source-headless verified
draft, not an installed/live-readiness claim.

Latest integration checkpoint: current main259 verified

Normally merged dbf1c7a8b in the existing worktree at 0e04ad6c7 without
conflicts. Parent main/raw argument hooks and object-label strategy leaves are
unchanged by this worker. No source edit or competing fix for issue257/264, R0,
startup, source binding or the broader archive was introduced.

Explicit shared-file flags were delivered directly on the existing owner PRs:
Socrates PR262
and Lovelace PR263.
The available comms route rejected two attempted sends because sender Zeno is not
registered there; no fake participant was registered and no delivery was claimed.
The separate broader refactoring agent identity remains unconfirmed; Lovelace
is the known R0 owner, not an inferred owner of every archive surface.

Under the explicitly retained finite native slot and nonblocking lock, fresh
admission measured 16.08 GiB available RAM / 30.57 GiB disk, historical-swap
warning only. The worktree import path and exact merged source were verified.
pytest -q tests/integration/test_measurement_declaration_journey.py tests/unit/test_artifact_output_ownership.py tests/unit/test_runtime_value_store.py --tb=short passed 85 cases, 37.74 seconds (43.61 seconds process wall time):
all six real declaration/compile/orchestrator/CSV journeys, all 23 output-policy
cases and all 56 runtime-input controls. No deselection, weakened assertion,
mocked replacement entrypoint, restart or replay of any original blind run.

The existing packaged ratchet against dbf1c7a8b → 0e04ad6c7 again exits
0, 16 changed production files, zero positive deltas. All 25 changed Python
files parse/compile in memory; diff check passes. Coverage limits from the R0
receipt below still apply; this is not a complete NRA R1 proof or full reference
corpus/installed GUI acceptance. The internal policy and source-context owners
still reject invalid inputs independently of output recording ownership.

Supervisor 2418566 and pytest 2418569 exited zero and are absent; no
surviving worker runtime/test handle remains. The finite native slot is explicitly
returned to the parent. Worker-owned self-generated fixture/cache scratch was
removed after termination, retaining command and ratchet reports at
/home/ts/.cache/agent-scratch/openhcs-issue-measurement-20260929/pr205-main259-dependency-20260930.
Final SHA256:

  • journey-policy-input-controls.log:
    a0cc302bb562e24b3f109fc4dea88d2ab6d2e3e7c6abb2921a57d373038a10e0.
  • packaged-ratchet.log:
    50a31a25fd1f09503c5e3448d0b3bed421ac05f189c79ac48a6a4457cb9c9d8b.

Main260 journey and R0 owner correction

Read the complete new docs/refactor rules/index/R0/owner override package in
Lovelace's existing PR263 worktree and the binding expanded-goal receipt. Existing
PR205 keeps its scope. No R0/L0/S1-S8 implementation, parent259 method, issue264
export owner, managed skill or installed source was edited here. Hosted CI is not
a waiting gate; actual local behavior and enforced integration rules remain.

At published source 9271a55f1 (main260 already normally merged), fresh locked
admission measured 15.20 GiB RAM / 30.84 GiB disk, with only authorized
historical swap warning. All nine package roots and both native extensions
resolved inside this worktree. Subject variants: 2 passed, 4 deselected,
3.60 seconds. The full six-case journey plus runtime/catalog/output-policy and
module-local CellProfiler controls: 108 passed, 40.30 seconds. The ten added
backend controls cover all object-intensity features/row axes, secondary distance
tie-breaking and replacement-primary topology/relationships. This is focused
local parity evidence, not the full native CellProfiler reference corpus.
No external dataset/reference acquisition, download or benchmark replay occurred.

The newly applied packaged ratchet exposed a real gap in the earlier focused
structural claim. Original comparison of main260 to 9271a55f1 returned 1:
one foreign absence probe in the CP policy, and class-excess growth of eight lines
in CallableContract, one in CellProfilerRuntimeAdapter, and one in
RuntimeInputBindingRequest. The original report is retained; it is not presented
as a passing check or waived by the earlier index's historical debt total.

Owner correction 5296400d0:

  • ArtifactSpec.require_measurement_feature_owner() owns its required row-owner
    obligation; CP policy calls it, rather than probing another declaration's field.
    Common kind invariants and legitimate heterogeneous CP recording remain intact.
  • CallableMetadata owns policy projection from its existing adapter declaration;
    CallableContract projects that result. Its redundant artifact_key_specs
    override is deleted, using the existing inherited selection unchanged.
  • RuntimeAdapterRequest.runtime_artifact_input() supplies its own exact compiled
    scope/source context. Both CP consumers call it; both duplicated constructors
    are deleted. The native loader retains its existing non-adapter construction.
    No matching roster, alternate store, channel-ignore flag or output exemption
    was added; input management and output recording are still independent.

These are IDEN-1/7, BOUND-2, IMPL-4/5/12 and TIME-1 owner/factoring corrections,
not blank-line/class relocation to evade a size check. Source controls after the
cutover: 159 passed, 7.86 seconds. Strengthened metadata/policy and exact
request-context identity assertions: 79 passed, 2.09 seconds. Counts overlap.
A subsequent ratchet report found only the redundant quoted postponed return
annotation counted as a string subscript; 20bf7e401 uses the nominal postponed
annotation directly, without any change to the tool or its rules.

Final unweakened six-case journey at 20bf7e401: 6 passed, 7.16 seconds,
test process wall time 12.57 seconds. Admission was 16.08 GiB RAM / 30.95 GiB
disk
, under nonblocking validation.lock, without a new environment or GUI.
Missing subject fails before callable entry; its subject-only correction executes
and persists CSV. Exact two-producer selector, both source-document roundtrips,
same-source/paired execution, actual typed channel scopes and omitted-selector
fail-closed/single-producer controls all remain enabled. No xfail or deselection.

Packaged ratchet at 20bf7e401 against c86f562e1: exit 0, 16 changed
production files, zero positive deltas. It ran the existing package directly
from /home/ts/wt/comms-owner-startup-sol-20260929/src, tooling revision
dbd0965d06a8d85310c8b34032ac0baf3421c3dd, using the required interpreter;
no install, copied script or tool edit. Coverage includes the available type-
identity, long-chain/term, foreign-probe, codec, string-subscript and class-excess
measures. This package revision does not include dispatch-arm measures; focused
source/AST/new-kind caller evidence covers this change, not a complete NRA R1
or global semantic proof
. R0 automation/activation remains with Lovelace.

All supervisors/tests 2375676, 2375692, 2375870, 2390917, 2390918, 2398808,
2398809, 2403863, 2403884
are terminal and absent. No worktree process survives.
The finite native slot is explicitly returned to the parent. Actual installed
MCP/GUI verification still belongs to the integration owner; this source-native
journey does not claim installation, activation, merge or biological acceptance.

Owned receipt/scratch root:
/home/ts/.cache/agent-scratch/openhcs-issue-measurement-20260929/pr205-post-main260-20260930.
Logs/reports are retained, including original ratchet failures and final success.
Verified regeneratable cache/tiny synthetic pytest outputs were removed only after
all processes terminated. Final evidence SHA256:

  • journey-runtime-catalog-parity.log:
    071260ee6ba492f7a7f61d742c462c8fd89dd5edfd07e3027505b38c773d43b9.
  • owner-cutover-six-case-journey.log:
    cb171090de4e4b5fc8c0c33fbb4bd73506e115eff5766bd72ebe463d308928f2.
  • packaged-ratchet-final.log:
    2b64440ecc0fc076fa238faf1b76c5e5ed596d6bc22435b132453b50d20f3a3b.

Persisted-format declaration (R0 rule 6)

  • PipelineDocument source: external, unchanged serialization/schema; existing
    exact selectors and subject relations roundtrip through the original authority.
  • Measurement CSV/images: external, unchanged format; the corrected synthetic
    declaration persists through the existing materialization owner.
  • Artifact contracts, policies and runtime input projections: internal,
    process-local state reset at fresh compile/execution; no durable migration,
    compatibility reader or second format is introduced.
  • Catalog DTO: declaration-derived runtime projection, not a persisted store;
    the existing selector field now carries its binding-owned required name.

Earlier finite journey: terminal, working checkpoint

The parent released the finite serial slot after its terminal PR259 attempt.
Every build/test phase acquired validation.lock nonblocking. Raw initial
headroom was 12.99 GiB available RAM / 24.90 GiB disk, and final admission
was 13.07 GiB / 29.88 GiB. Initial RAM >=11 GiB, trial RAM >=8 GiB and disk

=20 GiB were enforced; only the authorized historical-swap warning was waived.
The existing interpreter and all nine imported source roots resolved inside this
worktree. setup.py build_ext --inplace --parallel 1 completed in 2.59 seconds,
building the two existing native extensions locally, without installation,
downloads or shared-source changes.

Actual results, in order (counts overlap; they are not an additive unique total):

  • Subject-only variants: 2 passed, 4 deselected, 3.61 seconds. The real
    headless compiler rejects missing subject before callable entry. Adding only
    the exact image subject compiles, executes and returns the image-owned rows.
  • First complete journey: 1 failed, 5 passed, 37.62 seconds. Actual paired
    execution already passed. The omitted-selector case still failed closed, but
    common output validation masked the original scalar-label ambiguity diagnostic.
    Its original failure is retained in full-six-case-journey.log, SHA256
    850195091fec271e2a8b43d6dc7ce29a365f1ae4a0339cb46398efee3d4ac175.
  • Runtime, output-policy, catalog and identity controls: 89 passed, 3.37
    seconds. This includes exact unscoped/asymmetric admission and coordinate,
    producer, relation and ambiguity negative controls.
  • After the ordering repair, the unweakened complete journey: 6 passed, 7.54
    seconds. A broader compile run exposed three incomplete older test declarations:
    3 failed, 269 passed, 9.38 seconds. Original failures are retained in
    finalized-contract-controls.log, SHA256
    285082789885e46a8762cfe5913ccff507b68c244531e3f86ce84ee96262ff28.
  • Final combined run: 278 passed, 15.59 seconds: all 272 cases in
    test_function_patterns.py, test_artifact_output_ownership.py,
    test_function_step_execution_plan.py, test_path_planner_materialization.py
    and test_function_artifact_outputs.py, plus all six continuous cases in
    tests/integration/test_measurement_declaration_journey.py. No journey case
    was deselected or weakened. The corrected subject case now also reads the
    actual emitted details CSV and asserts pixel_count=4 and
    source_image_name=CountedImage against the returned subject.
    Final log: final-six-case-and-compile-regressions.log, SHA256
    ef97da954948958ece3713fb1bfd5940f2de6f871bdf02554cb53efddb17710a.

The diagnostic repair moves the existing
CallableContract.validate_artifact_input_parameter_bindings() call from the
later path-planner consumer into _compile_invocation, immediately after
declaration finalization and before dependent output validation. The old call is
deleted. Three new policy-parametrized cases prove scalar labels ambiguity wins
at compile while the same declaration still independently fails the artifact
kind's conflicting-subject invariant. No validator, output-policy exemption,
selector, subject requirement or source compatibility predicate was weakened.

The three older fixture repairs declare what their assertions already require:
the artifact-fed illumination_function parameter now exists on the callable;
both sidecar-conflict rows declare artifact subjects; and the channel-grouped
rows declare their exact source image as subject separately from group lineage.
Their component projection, sidecar conflict and channel/producer assertions are
unchanged. The final output-policy file has 23 passing cases, including the
new-kind matrix across native, adapter-recorded and CP-recorded owners and the
legitimate heterogeneous CP recording contract.

NRA/refactor-audit evidence is focused, not a complete NRA semantic/proof scan:
IDEN-1 keeps input lineage and output subject separate; BOUND-2/IMPL-4 retains
validation with the finalized declaration owner; IMPL-5/12 removes the replaced
consumer call in place. Adding an output kind still needs zero compiler/consumer
branches. The catalog still derives its selector from the binding owner, and
paired context still uses declared source relations rather than a global channel
exception. Current caller and AST inspection complements the real owner cases.
All 25 changed Python files against the validated main32 parse and compile
in memory. The two current production deltas contain only one added owner call
and one removed call, with zero added match or if nodes. Caller search finds
one finalized compiler call and the pre-existing explicit authoring wrapper; no
remaining manages_artifact_outputs caller exists. git diff --check passes.
The first AST command compared the moving remote main and tried to read an
upstream-only new benchmark file absent from this older worktree; the check was
corrected to the actual validated main32 baseline, without editing that file.
After the normal main260 merge, the same 25 changed Python files also pass
AST parse/in-memory compilation against current c86f562e1; no fixture or owner
conflict occurred. This structural recheck is not a post-merge journey rerun.

Build/test processes 2304493, 2304586, 2305892, 2306949, 2320857, 2321108,
2328819
and supervisors 2304492, 2305891, 2320856, 2328817 are all terminal
and absent. Final test/supervisor exit codes were zero; no worktree process or
surviving handle remains. The worker explicitly returned the native slot to the
parent. No MCP server, JVM or GUI was launched. This exercises the real compiler,
orchestrator, processing and CSV persistence on self-generated tiny fixtures;
it does not claim installed MCP/GUI acceptance or a replay of any blind attempt.

Owned scratch/log root:
/home/ts/.cache/agent-scratch/openhcs-issue-measurement-20260929/pr205-live-20260930.
Command logs, including both original failures and the final passing run, are
retained. Approximately 11 MiB of verified regeneratable build intermediates,
synthetic pytest outputs and caches were removed after process termination.
Locally built worktree extensions remain; no source, saved session, author input,
reference, installed package or other owner's output was removed.

Historical subject preparation and lineage-boundary review

The newly reported frozen-author failure is precisely #133: a typed PURE_3D
measurement output can carry a feature owner and source lineage without declaring
what it measures. The original attempt remains untouched. No author source,
science, pixels, parameters or references were needed or inspected for this
continuation; no replay or author coaching occurred.

The existing synthetic integration journey declares its own CountFeature
and nominal feature owner, CSV materialization and MainFlowStackOutputSpec
for both row variants. The existing MainFlowArtifactContractProvider binds
their stack/group lineage to the actual current image input. The corrected
declaration is a dataclasses.replace of the invalid
one, adding only ImageMeasurementSubjectRelation(CountedImage.ref()).
Both variants render/parse through PipelineDocumentAuthority; the test asserts
the reconstructed feature owner and real provider's input-qualified scope before
the headless compile boundary; the subject remains the exact image output
ref. The invalid callable still raises if entered. The valid case
uses the real compiler and orchestrator and checks both image and measurement
outputs and four counted synthetic pixels. These assertions and actual CSV
persistence now pass in the current-main finite journey above.

Parent review: output-qualified lineage fixture repaired

The parent correctly identified an independent invalid declaration in checkpoint
94595f271: generic ArtifactSpec.output with
SourceStackLineageSourceRelation(CountedImage output ref) does not make a legal
group-scope input. The earlier four output-policy probes did not exercise that
boundary. Their passing result was not evidence that this valid-subject variant
could reach execution or CSV. No such readiness is claimed.

A bounded source-only probe using the actual CallableContract,
NormalizedFunctionItem, ArtifactDeclarationStepContext and
MainFlowArtifactContractProvider reproduced the failure:
Callable 'count' output group-scope relations reference undeclared inputs
with the exact CountedImage OUTPUT ref. Switching both variants to the existing
MainFlowStackOutputSpec yields three observed outcomes in the final probe:

  1. The original generic output-qualified lineage still fails the real
    CallableContract.group_scope_inputs boundary.
  2. The corrected input-qualified lineage passes that boundary while its missing
    measurement subject still fails output-owner validation.
  3. Adding only ImageMeasurementSubjectRelation(CountedImage.ref()) passes both
    boundaries and retains the exact OUTPUT subject separately from INPUT lineage.

The first probe confirmed the red witness, then used a nonexistent
ArtifactSpec.measurement_subject() test API. That probe error was corrected to
MeasurementsArtifactType.require_output_subject; it was not a runtime defect.
The final probe passed in 0.58 seconds without invoking the callable. It uses
minimal metadata, not CSV/row/native execution, and is explicitly source-contract
evidence. The actual integration fixture retains its feature owner, CSV options,
roundtrip and compile/execute assertions for the released-slot gate.

NRA/refactor-audit finding: IDEN-1 separates invocation source/group ownership
from measurement subject; IMPL-4 reuses the output declaration's existing binding
hook instead of restoring consumer inference. Only the journey fixture and this
receipt were edited in that preparation checkpoint. No guard relaxation, new
owner/registry, parent PR255 file edit, source-binding edit, installed-root use
or production-code change occurred in that checkpoint.

Actual lightweight checks at this checkpoint:

  • All nine imported package paths resolve inside this worktree, using the required
    existing interpreter and eight recorded submodule src paths. PolyStore
    1209068f5 and ZMQRuntime 0f9e840a9 were synchronized from existing local Git
    objects only: no dependency download, donor source edit or installation change.
  • Four direct, real output-policy checks passed: native-return and ordinary
    adapter-recorded owners each reject a missing subject despite the declared
    feature owner and source lineage, then accept the subject-only correction.
    This is owner-hook behavior evidence, not compile/execution/CSV evidence.
  • AST parse and in-memory Python compilation passed for all 24 changed Python
    files
    against current main; git diff --check passed. Focused guards verify
    common kind validation precedes payload-policy validation with no exemption
    branch, and the removed manages_artifact_outputs has no source/test callers.
  • The original exact two-producer paired execution remains enabled, together
    with omitted-selector ambiguity/single-producer cases and real unscoped and
    asymmetric RuntimeArtifactInput regressions. No assertion, channel check,
    selector, source policy or production owner was weakened in this continuation.

NRA/refactor-audit coverage is focused source/caller/AST inspection, not a
complete NRA scan or native proof. Actual owner evidence: IMPL-1/4/10 puts output
invariants on the artifact kind and native/recorded obligations on their declared
policy; the compiler calls ArtifactPlanKeySelector and runtime reuses
MeasurementsArtifactType.require_output_subject. IDEN-1 keeps feature ownership,
source lineage and measurement subject independent; schema/CSV is not a subject.
MEMB-1/2 and IDEN-5 derive the catalog selector from the existing binding's
require_parameter_name, with no copied roster/store. IDEN-7/BOUND-2 keeps exact
producer/group selection before typed contextual compatibility; empty/asymmetric
additional constraints do not alter global source-image matching semantics.
The retained new-kind experiment declares one new artifact kind, requires
zero compiler/consumer changes, and covers all three output policies,
independent input management and both contract/compile boundaries. Its prior
20-case passing evidence, including legitimate heterogeneous CP ownership, is
historical below, not relabeled as a current-main rerun.

Historical first admission: the parent explicitly granted the slot after freezing and
closing the original blind runtime and its finite installed acceptance attempt.
The worker acquired validation.lock nonblocking for the admission check.
It exited 78: available RAM 14.08 GiB exceeds the initial 11 GiB floor,
but /home available disk 19.7195 GiB is below the required 20 GiB floor.
The general resource guard's rounded 20.0 GiB is not an admission proof.
No build/test/MCP/JVM/GUI process started; the admission process exited and
released its lock. Previously recorded worker-owned disposable scratch is
already empty (4 KiB directory), so no other owner's artifacts were removed.

At that admission, the named dependency was at least 0.3 GiB
additional available disk plus build margin. This worktree still lacks the
current-main _tabular_native extension (find_spec returns None). No native
build, environment startup, installation or managed-skill change occurred.
The released-slot continuation above subsequently built the extensions and
validated both subject variants, the complete six-case journey and paired-source
controls. Installed/live verification and merging remain with the parent;
optional CI is not a prerequisite for shipping the reviewed #133 checkpoint.
Parent owns the separate #254 payload ABI/PURE_2D follow-up, including only
FunctionChainInvocationExecutor.main_flow_call_argument in the shared runtime
file. This worker's measurement strategy/input-loader scope does not collide;
no cherry-pick of that unpublished follow-up or installed-root validation occurs.

Finite-slot continuation after parent's terminal PR259 source-live attempt:
parent explicitly returned the slot to PR205. Worker-owned disposable scratch is
/home/ts/.cache/agent-scratch/openhcs-issue-measurement-20260929/pr205-live-20260930,
purpose local native intermediates, command logs, and self-generated tiny pytest
plates/cache only. No source, saved session, author input or reference belongs
there. Keep failed-attempt receipts until classified, then remove verified
regeneratable build/test scratch after preserving results. The build uses the
existing setup.py build_ext --inplace --parallel 1; no installation or download.
Validate required imported paths and fresh RAM/disk floors under the nonblocking
shared lock before build, then both subject variants before the complete six-case
journey. No parent PR259 contextualization method is edited by this worker.

Frozen-harness compile incident: triaged, regression retained

The original server5793 request 4f8e1d55-07f0-4bde-9f37-d24c0ea398f9
failed at 20:24:35 UTC with MeasureCells_4_measurements group "1" having no
declared group-scope source. The original saved technical request's source hash
matches the server's 5fb62ebed47d: it produces exact Cells under channel group
"2", but explicitly dispatches its size/shape measurement under group "1".
The real module already declares the exact labels-owned group lineage and object
subject. This is an invalid authored grouping, not missing backend projection.

The new real-declaration/planner regression preserves that failure and proves
matching-group and declaration-derived controls compile: 3 passed. Combined
with the output-policy review cases, 23 passed in 1.87 seconds after current
main integration. No production relaxation, frozen replay or installation change
was made. Original request/variant identities, owner trace, pattern IDs and
acceptance boundaries are preserved in
docs/plans/measurement_group_scope_triage_20260929.md. This technical case stays
on #204/PR205; it does not resolve the separate paired runtime/live gate.

Implemented

  • Measurement output validation belongs to MeasurementsArtifactType. Compilation
    requires a subject relation before invoking the callable. Schema-bearing rows and
    CSV materialization do not imply a measurement subject.
  • Compile and runtime resolve subjects through the original relation owners. The
    error identifies the output and explains image, object, and artifact relations.
  • The finalized callable contract projects its declared artifact_output_policy.
    Every policy runs the artifact kind's common declaration invariants, then its
    native-return or adapter-recorded payload obligation. The old blanket exemption
    and manages_artifact_outputs field are deleted, with all callers cut over.
    Input management remains an independent capability. CellProfiler's policy
    requires the existing module measurement owner rather than a table-wide subject;
    its original row policy and recording boundary still validate actual subjects.
  • CellProfiler catalog bindings project require_parameter_name(), not the optional
    raw parameter_name. The existing select_object_sets_to_measure selector is
    discoverable separately from the runtime-owned scalar labels parameter.
  • Catalog guidance explains exact-name tuples and consumed compile-time selectors.
    No selector registry, artifact store, fallback, or ambiguity relaxation was added.
  • The secondary module already declares InputGroupLineageSourceRelation from its
    primary-label input to its image input. Runtime input matching now projects typed
    scope identities using that exact relation's source bindings and the existing
    SourceImageSetIdentityPolicy / SourceImageSetIdentityCompatibility owners.
    This permits declared paired-channel context without a global channel exception.
    Missing relations or unrelated visible bindings retain exact coordinate checks.
  • All three production input constructors forward their existing compiled source
    binding plan: native callable loading, adapter record access, and binding requests.
    Producer address/group selection, stack-plane projection, and ambiguity checks
    remain authoritative; the old duplicate fixed-coordinate loops are deleted.

Historical focused evidence (current rerun above)

Using /home/ts/code/projects/openhcs/.venv/bin/python with this worktree and all
eight recorded submodule src directories on PYTHONPATH; all nine imported
package paths were verified inside this worktree. Submodules were initialized to
their recorded gitlinks without installed-source changes.

Architectural review follow-through

Review: #205 (comment).
The review correctly identified a consumer-owned exemption in the previous
checkpoint. Its earlier audit claim below was too strong: skipping every kind
hook was not an adequate expression of CP's heterogeneous table obligation.

  • Red witness at the pre-fix body: the new materialization-bearing output kind
    produced 4 failed, 4 passed, 1.65 seconds. All four adapter-recorded cases
    silently bypassed its invariant; native-return cases rejected the declaration.
    Contract and compile boundaries, with input management both on and off, expose
    the same defect. Corrected declarations also compile without executing a callable.

  • tests/unit/test_artifact_output_ownership.py now exercises the same new kind
    under native, generic adapter-recorded and CP-recorded policies without any
    compiler/consumer edit for that kind. It also checks single-subject requirements,
    common conflicting-subject rejection, CP's required module row owner, and the
    nominal policy boundary. The real CellProfilerInvocationContractProviderFactory
    derives a primary module contract with its original measurement owner and no
    table-wide subject. Real compile_function_pattern admits that contract;
    replacing only its recording owner with native returns rejects it.

  • Final output-policy file: 20 passed. The preceding five-file run covering
    output policy, native artifact outputs, all runtime-store regressions, catalog
    friction and artifact identity reconstruction: 153 passed, 4.95 seconds
    (17 policy cases before the final three nominal-boundary cases were added).

  • test_function_patterns.py plus test_function_step_execution_plan.py:
    80 passed in the 95-case run including the then-15 policy cases. Four generic
    adapter measurement fixtures previously omitted subjects; they now declare valid
    artifact subjects while retaining their exact component/projection assertions.

  • Three selected planner cases: 3 passed, 96 deselected, 1.92 seconds.
    Exact managed group lineage and main-flow preservation retain their assertions.

  • Existing test_track_objects_record_builder_uses_nominal_image_table_ownership:
    1 passed, 393 deselected, 5.08 seconds. The actual module row builder emits
    mixed image/object tracking rows with their declared per-row owners. This is
    provider-free record-building evidence, not complete pipeline/live execution.

  • Existing adapter output runtime-context regression:
    1 passed, 91 deselected, 1.71 seconds. The separate adapter component-output
    chain regression also passes; no MCP/JVM/GUI process or environment was started.

  • All checks use the required interpreter, worktree and recorded submodule paths.
    Latest guard: 11.5 GiB available RAM, historical swap warning at 11.9 GiB.
    These are bounded source/unit checks, not a second heavy validation slot. No
    installed source, shared environment, skill, or frozen runtime was modified.

  • Initial published checkpoint: 12 catalog/identity tests and 6 focused native
    subject tests passed.

  • Full focused rerun: pytest -q tests/unit/test_function_artifact_outputs.py tests/unit/agent/test_mcp_cellprofiler_function_contract_friction.py tests/unit/test_cellprofiler_artifact_identity_reconstruction.py tests/integration/test_measurement_declaration_journey.py -k 'not paired':
    85 passed, 1 deselected, 9.33 seconds (80 unit, 5 real headless journey cases).

  • The real PipelineDocumentAuthority and FunctionStepDocumentAuthority
    render/parse routes preserve select_object_sets_to_measure=("Cells",).
    InProcessCompileInspectionGateway.compile selects exactly the secondary
    producer (source step 1) for runtime labels; authored selectors are consumed.

  • The same-source two-producer journey then runs the real orchestrator and
    verifies the secondary label area is larger than the primary area, the returned
    measurement subject is Cells, and integrated DNA intensity includes the grown
    secondary region rather than just the four bright primary pixels. Feature
    retrieval uses the existing CellProfiler measurement dialect.

  • The paired DNA/actin-channel case also passes roundtrip and compile, but its
    earlier continuous run failed at the secondary step's primary-label input.
    It remains an unweakened regression, deselected in that historical run;
    see the remaining acceptance section. No xfail or fallback was added.

  • Omitted selection with two producers fails closed with the original useful
    ambiguity diagnostic; omitted selection with one producer compiles.

  • Missing and conflicting native subjects fail before execution. The real
    headless compiler rejects schema-bearing DataclassMeasurementColumnarRows
    with CSV options but no subject. The corrected image-subject declaration
    compiles and executes, returning an image-owned table with four counted pixels.

  • Runs used nonblocking batch validation flock and CPU-only mode. The resource
    guard most recently reported 14.7 GiB available RAM and a warning for historical
    11.5 GiB used swap. No broad suite, JVM, MCP server, or extra GUI was started.
    The final run completed and released the lock before Euler's H003 restart;
    no further heavy validation was started after the resource-priority instruction.

Runtime checkpoint after the explicit scope extension:

  • pytest -q tests/unit/test_runtime_value_store.py --tb=short: 48 passed,
    1.53 seconds, provider-free and without a JVM, GUI, or MCP server. New cases
    exercise paired-channel matching, both adapter consumers, conflicting fixed
    site/z/time coordinates, another well or exact producer path, missing source
    declarations, missing context relations, a relation to an unrelated source,
    an absent producer site plane, and unresolved multiple-record ambiguity.
  • Replaying the published matching body on the same new fixture rejects channel
    1 versus 2; the new body selects its exact producer. This is provider-free
    reproduction of the precise owner fault, not complete paired execution.
  • Attempting the complete six-case headless journey under nonblocking flock -n -E 75 returned 75 (lock busy), without starting pytest. Euler's H003 slot
    remains reserved. Paired runtime acceptance is pending this run, not claimed
    from the 48 focused store checks. No deselection is planned for it.
  • The resource guard reported 11.6 GiB available RAM; its warning was historical
    11.5 GiB used swap. Only lightweight source/unit checks proceeded.

Coordinator scope-preservation witness, followed through:

  • Ordinary exact unscoped inputs and producer-only, consumer-only, shared-partial,
    and disjoint-partial fixed scopes were explicitly tested through
    RuntimeArtifactInput.records, not only through a predicate. They already
    passed because the earlier projection retained the exact axis key.
  • A genuinely empty projected-context witness uses a context-carrying object-label
    artifact and an existing source declaration whose plane membership projects
    the axis key. The published candidate then reproduced four failures:
    unscoped, producer-only, consumer-only, and disjoint-partial inputs lost their
    exact address-matched record because bool(shared_keys) was false. This proves
    a focused runtime failure for the witness, not its prevalence in real pipelines.
  • Artifact admission now derives shared additional constraints from the typed
    scope owners, compiled variable-axis projection, and declared image-set policy.
    No shared additional constraint is vacuously satisfied after producer/address/
    group selection. Otherwise the existing typed predicate checks their values.
    The artificial shared axis key was removed from this additional-context test.
  • SourceImageSetIdentityCompatibility is unchanged. Each projected-context
    case asserts it still rejects identities with no shared evidence; exact artifact
    admission has a different, already-proven identity boundary. Wrong well and
    producer paths still fail before this additional-constraint check.
  • pytest -q tests/unit/test_runtime_value_store.py --tb=short: 56 passed,
    1.67 seconds in the final run including the exact-address negative assertions.
    Original conflicting spatial/time,
    exact producer/group, retained-plane, and ambiguity assertions still pass.
  • Only core/runtime_stores.py, its typed scope owner, and their tests were
    modified for this witness. No source-binding owner, shared artifact declaration,
    CellProfiler adapter, or Expose primary-segmentation intermediates through typed diagnostic artifacts #214 diagnostic-output field was modified in this step.

Essential anti-pattern audit: actual ownership evidence

This is a focused source/caller and AST audit against the current archived
refactor-audit pattern catalog, not a complete NRA semantic/proof scan. All
eleven changed production files in the earlier runtime checkpoint were parsed against its merged main: zero added
match, isinstance, getattr/hasattr, or 6+-term boolean-chain AST nodes.
That guard detects listed shapes, not every possible semantic ownership defect.

Pattern IDs Owner, projection, and new-case evidence
IMPL-1/2/3/4, MEMB-1/2 The finalized compiler calls the selector's public validation contract. ArtifactOutputPolicy runs every kind's common hook, then the policy dispatches to the kind's native-return or adapter-recorded obligation. MeasurementsArtifactType owns common relation coherence and the native subject requirement; CP declares its distinct module-row-owner obligation. The new-kind materialization rule requires one declaration and zero compiler/consumer changes, and executes under all three owners. No kind/name roster or concrete-adapter switch was added.
IMPL-5/12/13, TIME-1/3 ArtifactSpecRelation.measurement_subject_for_output resolves original relation-owned subjects once. The output-plan accessor, kind requirement, and native runtime all delegate; the former output-plan extraction body and runtime missing-subject body were deleted in place. The output-plan method remains a domain accessor, not an old reader or compatibility alias.
IDEN-1/2/5, BOUND-2/7, MEMB-3/4 SettingToKeywordBinding.require_parameter_name() owns selector identity, separately from runtime_parameter_name. The only production CellProfilerArtifactBindingSummary constructor projects that owner; its selector field is now required. No selector store, guessed names, raw-row decoder, or attribute-name probe exists in the change. The catalog regression derives every registered binding and exercises new derived-name input and explicit-name output declarations without editing the projection.
IDEN-1/7, IMPL-4/10, IMPL-5/12 Recording ownership no longer answers whether common compile invariants may run. The blanket early return and output flag were removed, not relocated; native/adapter policies have distinct kind-dispatched payload obligations. CP requires its already-declared measurement_feature_owner, leaving actual heterogeneous row validation with the original module row policy and add_measurements. Common conflicting-subject checks stay on the kind. The new-case matrix varies input management independently, and the real CP provider/admission plus mixed tracking-row builder pass.
AGENT-2/3/6/7, TIME-5/9 No second mechanism, format converter, test-only production switch, or dormant feature was added. One pre-existing cardinality fixture received a valid artifact-level subject so its original missing-return assertion remains meaningful. The paired-channel failure is retained, not hidden by weaker expectations; publication proceeds with that exact acceptance gap stated.
IDEN-1/6/7, BOUND-2, MEMB-1/2/3, IMPL-5/12, TIME-1/3 Exact compiled producer/address selection and contextual source compatibility answer different questions. RuntimeArtifactInput derives context membership from edge.spec.source_context_sources() and the referenced original source bindings, then delegates matching to the existing typed identity predicate. RuntimeExecutionAxisScope owns its typed-coordinate projection. No literal channel/site/z/time matching roster, new registry, global ignore flag, alternate reader, or copied predicate was added. All three constructors are migrated; two actual adapter consumers and fail-closed new cases execute. The replaced fixed-coordinate loops are gone.
IDEN-7, BOUND-2, IMPL-5 Artifact admission asks whether an already-selected producer violates any shared additional coordinate constraint, not whether unknown plane identities prove correspondence. The four failing empty/asymmetric witnesses distinguish these questions. ComponentSet.intersection, scope-owned source_components, and policy-owned identity_components() derive the applicable constraint domain. No copied compatibility matcher or weakened global empty-identity semantics was introduced.

Caller review covered the default invocation declaration provider (returns the
finalized CallableContract), every call to the new output-validation hook and
shared subject resolver, all production measurement-subject accessors, and the
catalog DTO constructor/consumers. Structural extract_artifact_declarations
still checks relation references without prematurely applying native runtime
requirements to partially reconstructed CellProfiler contracts. Real compile
inspection reaches the finalized validation boundary and fails before execution.

Review follow-through at dd02a00f6 plus the latest normal main merge covered all
output-policy constructors, decorator calls, compiler hooks and four production
recording consumers. rg finds no remaining manages_artifact_outputs use in
production or tests; there is no compatibility alias. The nine production files
changed for this review were AST parsed: zero added match, attribute-name probes
or 6+-term boolean chains; one added isinstance is the nominal output-policy
declaration boundary, tested for invalid/abstract policies. This is focused
source/caller/new-case coverage, not complete NRA detector or native-proof coverage.

Runtime caller review also covered records, static/dynamic all_records,
_records, _validate_selected_records, _axis_value, projected_values,
composed_value, all three production RuntimeArtifactInput constructors, and
the existing secondary input relation declaration. Compiled group selection and
variable-component projection are unchanged: existing complete-producer-group
and transposed-stack tests still pass, while the new missing-site-plane test fails
closed. This checks this runtime surface; it is not a global provenance audit.

New-case runtime edit count: a context relation and a source declaration supply
membership, with zero changes to the matching consumer or a component roster.
Tests show a visible binding cannot substitute for a missing relation, and a
relation cannot borrow membership from an unrelated source. Partially scoped
identities continue using the existing compatibility predicate, not a new
equality algorithm.

New-case edit count: one owner declaration for a new setting binding, zero new
catalog/compiler branches; the registered/new-binding regression exercises this.
A new subject relation exposes its existing measurement_subject() capability;
both compile and runtime use it unchanged. A new output kind supplies its own
validation hook, with zero edits to the generic compiler consumer. The
duplicate-resolution removal is factoring; no pure file move is claimed.

Remaining acceptance and integration boundaries

  • The original paired-channel failure was before measurement: exact channel-1
    Nuclei was rejected against the secondary source's channel-2 context. This
    regression now passes actual paired execution in the unweakened six-case
    journey, including declared source relation, actual typed channel coordinates,
    exact selector and measured-secondary-region assertions. The 56 runtime-store
    controls also pass in the current 89-case run. No source-binding owner,
    source preparation, ROI, viewer or Artifact-plan inspection writes outside authorized MCP write roots #211 authorization method/test was modified.
  • The finite source-native slot is terminal and explicitly returned to the
    parent. No surviving worker handle exists. Further heavy/installed acceptance
    requires the next named serial handoff; no environment or blind replay is queued.
  • Parent owns the cold-inspection gateway change and PR259's argument ABI and
    object-label contextualization follow-up. This worker does not edit those
    methods, import unpublished parent changes or use foreign installed MCP as
    source acceptance. Its measurement/input-loader scope remains separate.
  • Source-only explanation of the apparent grouped/ungrouped distinction: group
    coordinates are selected by the compiled edge, while the previous predicate
    compared fixed coordinates on either side. Binding-owned channel identities
    expose this failure with GroupBy.NONE; a channel selected as a producer group
    takes another existing path. No Euler data, output, pipeline, or run
    configuration was opened to infer its particular success mechanism.
  • Coordinator-authorized actual UI code-document/live installation validation.
    Source tests do not establish installed readiness. Worker must not install or
    merge, and must not inspect Euler's blind input or output.
  • Verify exact object-label selectors through the installed UI code-document route #204's corrected diagnosis is catalog discoverability, retaining the original
    failure and acceptance verbatim in issue204_catalog_diagnosis_20260929.md.
    Exact author/compile/roundtrip, same-source and paired-channel execution are
    proven here; actual installed GUI interaction remains open, so Verify exact object-label selectors through the installed UI code-document route #204 is not
    claimed closed. Review/merge/installation remain with the integration owner.

Draft PR: #205.

Changed paths

  • openhcs/agent/dto/functions.py
  • openhcs/agent/services/function_catalog_service.py
  • openhcs/core/artifact_key_selection.py
  • openhcs/core/artifacts.py
  • openhcs/core/callable_contract.py
  • openhcs/core/function_patterns.py
  • openhcs/core/pipeline/path_planner.py
  • openhcs/core/runtime_adapters.py
  • openhcs/core/component_group_scope.py
  • openhcs/core/runtime_stores.py
  • openhcs/core/steps/function_runtime.py
  • openhcs/interop/cellprofiler/runtime/adapter.py
  • openhcs/interop/cellprofiler/runtime/artifact_binding.py
  • openhcs/interop/cellprofiler/runtime/module_execution.py
  • openhcs/interop/cellprofiler/runtime/output_record_request.py
  • openhcs/interop/cellprofiler/runtime/output_recording.py
  • tests/unit/test_artifact_output_ownership.py
  • tests/unit/test_function_patterns.py
  • tests/unit/test_function_runtime_source_projection.py
  • tests/unit/test_function_step_execution_plan.py
  • tests/unit/test_path_planner_materialization.py
  • tests/unit/agent/test_mcp_cellprofiler_function_contract_friction.py
  • tests/unit/test_function_artifact_outputs.py
  • tests/unit/test_runtime_value_store.py
  • tests/integration/test_measurement_declaration_journey.py
  • docs/plans/issue133_204_measurement_contract_checkpoint_20260929.md
  • docs/plans/issue204_catalog_diagnosis_20260929.md
  • docs/plans/measurement_group_scope_triage_20260929.md

Owned disposable test output was /home/ts/.cache/agent-scratch/openhcs-issue-measurement-20260929,
owner Zeno, purpose tiny synthetic plate and bounded pytest temporary outputs.
Removed its 608 KiB after preserving evidence and the fully self-seeding source
reproducer. Generated arrays and logs are regeneratable; no source or saved
sessions were stored there. No owned large scratch artifacts were created. Viewer-bind and ROI
materialization implementation surfaces remain with their assigned owners.

Historical local build prerequisite: a0263e82a1 adds the required
_granularity_reconstruct extension, reached by intensity -> shape -> zernike ->
granularity imports. A fresh source import failed with ModuleNotFoundError
before a test could execute. Zeno owns the bounded worktree-only build using the
existing setup.py build_ext --inplace declaration, with native intermediates
under /home/ts/.cache/agent-scratch/openhcs-issue-measurement-20260929/native.
This does not install packages or modify the shared environment or installed
source. The existing build completed in 2.33 seconds with 97,264 KiB maximum
resident memory. Intensity, primary, secondary, and the generated native extension
all import from this worktree; available RAM before the build was 10.8 GiB.
Removed the 136 KiB owned disposable native intermediates after this successful
import check. They are regeneratable. The generated ignored .abi3.so stays in
this worktree for source validation. This is import/build evidence, not a complete
headless or installed journey. The strengthened missing-source regression retains
the context relation and removes only its source declaration: one focused case
passes, with 47 unrelated cases deselected in that one-test check.

@trissim

trissim commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

Architectural review: output recording ownership should not be a blanket validation exemption

Reviewed 688e4e7d88f163d3f21c094ce4a599365dd3faa1, tracing finalized CallableContract, RuntimeAdapterSpec, the CellProfiler adapter/output-recorder family and RuntimeArtifactInput. Also read the subsequent authored scope repair at d7515e76703efff4fe05d22b621a3e7e6b43d532: it leaves the blanket output-validation exemption below unchanged.

The move of native measurement-subject validation onto MeasurementsArtifactType, and the catalog's use of SettingToKeywordBinding.require_parameter_name(), put those answers on their proper existing owners. The remaining gap is CallableContract.validate_artifact_output_declarations, lines 1149–1156: adapter.manages_artifact_outputs makes the method return before any output-kind declaration is validated.

These are two different questions: who records the payload? and what may this output declaration require at compile time? It is correct that CellProfiler's heterogeneous adapter-recorded measurement tables cannot inherit the native single-subject return rule. But that specific difference is currently encoded as a generic consumer-owned exemption from every ArtifactType.validate_output_declaration hook. The compile call site added by this PR appears generic, yet a future adapter-owned output kind's invariant is silently skipped.

Proper factoring: express output validation through the declared output-ownership policy, with meaningful native-return versus adapter-recorded implementations. The shared compile algorithm calls that public contract; artifact kinds retain their common declaration invariants, and the CellProfiler owner supplies only its genuinely different table/subject obligation. Input management and output recording are independent capabilities; their composition should not grow another repeated roster or concrete-adapter switch. RuntimeAdapterSpec is currently a factory plus flags, so moving the same if manages_artifact_outputs into another helper would only relocate the decision (IMPL-4/IMPL-10).

The existing CellProfiler CellProfilerOutputRecorder family and CellProfilerRuntimeAdapter.add_measurements already demonstrate where record-time subject/row validation belongs. Keep that distinct from what can be validated in an output declaration before execution. Add a new-kind regression whose compile-time invariant must hold for both native and adapter-recorded outputs, alongside the legitimate heterogeneous CellProfiler case.

Separately, the documented paired-channel failure is an actual completion gate, not a selector-discovery problem. The trace reaches RuntimeArtifactInput._matches_execution_scope (runtime_stores.py:553–576), which compares the consumer's fixed coordinates to the producer record after exact edge/address selection. A repair should derive cross-source alignment from the existing ArtifactInputProjectionPlan/compiled producer–consumer edge, not globally drop channel checks or guess the newest label. Same-channel success does not validate that distinct source relation. This review does not attribute the pre-existing runtime failure to the new catalog projection. Update for d7515e7: the new matcher now derives membership through spec.source_context_sources() and the corresponding filtered CompiledSourceBindingPlan, and routes compatibility through the existing SourceImageSetIdentityPolicy/Compatibility owners. Its missing-source, unrelated-source, other-site/time/Z and ambiguous-record regressions address important counterexamples; this is materially better than dropping channel checks globally. The receipt still leaves the complete paired execution outstanding, so do not treat the historical failure paragraph as a claim that the newer matcher has no repair.

@trissim

trissim commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Zeno (#205 DTO/service owner): Lovelace's separately assigned #230 checkpoint is
draft #233 at150346666, normally integrated
with main250fd1ecf / recorded PolyStore1209068. Parent requested direct shared-file
coordination; this note identifies the exact existing-owner changes, not another
catalog implementation.

Shared openhcs/agent/dto/functions.py scope is the custom-registration request,
destination query/response/result and control-message member. Shared
openhcs/agent/services/function_catalog_service.py scope is constructor path
policy injection, register_custom_function, and read-only
custom_function_registration_destination. Function parameter/artifact-selector
metadata and your measurement/runtime contracts are untouched by this task.

Existing AgentPathPolicy, ExecutionConnectionSpec, ProcessIdentity and
CustomFunctionManager own admission, routing, PID incarnation and native path
derivation. No new registry or provider. Both caller and native policy are checked
before evaluation/persistence; public MCP fields cannot supply write authority or
the server identity. Post-dispatch exceptions/receipt mismatch stay uncertain,
with no replay/fallback. The original H002 receipt033 and persisted side effect
are preserved and never reused as a fixture.

94 affected source tests pass; after normal main integration23 focused cases
pass. A separate transport-module collection remains blocked by its unbuilt
CellProfiler native extension. Actual stdio owned-vs-shared, delayed-response,
compile/execute acceptance is pending coordinator's finite technical slot; no
installed edit/JVM/new runtime has been performed.

Please flag any in-progress changes to these registration-specific regions or
their constructors before further shared-owner edits; maintain the single
function-catalog authority. Parent retains integration ownership for both PRs.

@trissim
trissim marked this pull request as ready for review September 30, 2026 04:52
@trissim
trissim merged commit fb5fea4 into main Sep 30, 2026
0 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compile inspection should reject measurement outputs missing a subject relation

1 participant