Repository navigation
Retain runtime facts on compiled, source and publication owners - #648
Merged
Merged
Conversation
Share parsed filename results and derived component membership through the existing RuntimePatternDiscoveryCache. Physical discovery, matching and grouped selection reuse the same parser semantic snapshot; grouped selection does not parse names. Retain the context source authority and rebind against strong owners. Observe optional workspace file eligibility and document epochs live while retaining its provider. Consume the producer store ordered load view without re-sorting or re-comparing known cohorts. Make runtime logging lazy and remove duplicate traceback formatting. Existing source/provenance/ambiguity controls: 153 passed. Production consumer probe verifies parse-once behavior and live ownership/document epochs. Whole-pipeline parity and performance remain integration gates; no speedup claim. External persisted result formats unchanged; internal execution caches derive from current declarations.
…time-file-inventory-20261004
Implement P1 of plans/runtime_plumbing using the existing callable contract and compiled invocation. Bind declared return slots, executable callables, parameter bindings and framework devices before runtime; keep live adapter requests and selected artifact admission. Delete the per-call return matcher and Core projection forwarding. Rebuild process-local executable state from declaration-only pickle transport. Preserve authored kwargs, plate calls without image domains, callable-qualified memory errors and debug/error diagnostics.
Reuse the existing runtime resolution snapshot for exact path and correlated component matching. Keep live resolver contexts on their original read path and retain source-position ambiguity and ordering. Refs #648
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runtime repeatedly reconstructs callable bindings, filename membership, source identity joins and the growing output manifest. This change retains those facts on their existing compiled, source and publication owners, removes the returned-output matcher and redundant forwarding/conversion paths, and keeps current pixels and adapters specific to each request.
CallableContractowns its fixed return ABI and shared memory-domain validation.CompiledFunctionInvocationholds the executable, runtime binding template, processing operations and assigned devices; authored kwargs remain authored-only. Constructor transport is derived from declared fields and rebinds executable state in an actual spawned process.RuntimePatternDiscoveryCacheshares physical filename parsing and component membership across discovery, matching and grouped filtering.ProcessingContextretains its source-projection authority while rebinding actual owners and observing optional metadata-file/document changes live.StepOutputManifestStoreupdates an ordered address map only for newly published records, retains path-ordered selections, and invalidates only the affected producer. Existing immutableDurableSourceMetadatafixes the slot captured at publication; provenance and filename aliases remain live. This publication policy was explicitly approved by the user. Compiled patterns derive inherited-domain replacement once; redundant revision counters are removed.RuntimeSourceResolutionSnapshotjoins already-admitted physical paths and correlated component records instead of rescanning the alias map for every source identity. Live callback contexts retain their live resolution path; record correlations, ordering and ambiguity are preserved.ImageFileFormatowns host projection at the disk boundary; memory checkpoints retain their declared framework. Mixed intrinsic-channel promotion and mask broadcasting still need generic ArrayBridge capabilities and are explicitly outstanding P5 work.The seven supplied P0–P5 documents are included unchanged in
plans/runtime_plumbing/, with an implementation map. P0 profiling is already merged in #645. This head merges main e7e79cd and implements P1–P4 plus the core P5 path; the overall performance goal remains active.Validation includes 127 integrated source/projection/manifest/profile/metadata-transport controls, 234 invocation/contract/projection controls, actual pickle/spawn receiving, and production consumers for parse-once and live ownership/document epochs. Actual GTX1080 receiving verifies native NumPy/CuPy mixed stacking, placement, independent pixel/mask buffers, memory checkpoints, and TIFF/NPY/PNG host serialization without replacing the GPU owner. GPU latency and all mixed-channel frontend cases are not claimed.
The saved real 60-plane matching frontier retains every source position: old matching 233.219 ms; candidate first admission plus matching 53.358 ms. The baseline receipt explicitly documents recovery of exact timing fields after a helper filename overwrite. Twenty-one existing lifetime/alias/ambiguity/correlation controls pass. This is component evidence, not an end-to-end speedup.
The ordinary 3D pair on exact main e7e79cd and integrated b379d3c is complete: all four strict comparisons against both retained native observations pass, including zero image tolerance. Source/dependency/environment/input/native ABI guards and owned server/worker shutdown pass. Identical sparse projections and source-qualified preparation precede public READY; server/registry/kernel startup is outside pipeline clocks. No coarse or sampled tracing is installed in the ordinary jobs.
This is one observation per source. Execution improves, but total and RSS are worse; no global speedup or per-patch causation is established. Evidence:
/home/ts/.local/state/openhcs-maintenance/20261004/runtime-plans-main-ab-3d-v2. The earlier sparse-preparation rejection is retained as V1 and launched no scientific job. Fresh original 3D native CellProfiler full-invocation timing completed separately after scientific acceptance: 14.372347 s and 14.109910 s, mean 14.241128 s (one warmup excluded). The integrated execution/native ratio is 2.67×; integrated total/native ratio is 2.08×. All source/dependency/input guards and owned cleanup pass at native terminal. Fresh native outputs are retained; the four scientific comparisons above remain explicitly against the retained V15 goldens. The user explicitly requested merging locally qualified changes despite existing main CI failures. Remaining compile and runtime costs continue under the active performance goal.The earlier P0-only diagnostic matches both retained native references; its coarse/sampled times are diagnostic only. Its py-spy terminal child-process error and detail-hook errors remain retained. Prior negative whole-pipeline observations for #630, #635, #597 and #608 are retained; local validity is not presented as proof of runtime improvement.
Persisted formats: internal reset. Execution-local manifest indexes and compiled derived state are regenerated. No external pipeline, configuration, image or measurement format changes.
Closes #647