Skip to content

Retain runtime facts on compiled, source and publication owners - #648

Merged
trissim merged 13 commits into
mainfrom
perf/runtime-file-inventory-20261004
Oct 4, 2026
Merged

trissim merged 13 commits into
mainfrom
perf/runtime-file-inventory-20261004

Conversation

@trissim

@trissim trissim commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Runtime repeatedly reconstructs callable bindings, filename membership, source identity joins and the growing output manifest. This change retains those facts on their existing compiled, source and publication owners, removes the returned-output matcher and redundant forwarding/conversion paths, and keeps current pixels and adapters specific to each request.

  • CallableContract owns its fixed return ABI and shared memory-domain validation. CompiledFunctionInvocation holds the executable, runtime binding template, processing operations and assigned devices; authored kwargs remain authored-only. Constructor transport is derived from declared fields and rebinds executable state in an actual spawned process.
  • RuntimePatternDiscoveryCache shares physical filename parsing and component membership across discovery, matching and grouped filtering. ProcessingContext retains its source-projection authority while rebinding actual owners and observing optional metadata-file/document changes live.
  • StepOutputManifestStore updates an ordered address map only for newly published records, retains path-ordered selections, and invalidates only the affected producer. Existing immutable DurableSourceMetadata fixes the slot captured at publication; provenance and filename aliases remain live. This publication policy was explicitly approved by the user. Compiled patterns derive inherited-domain replacement once; redundant revision counters are removed.
  • RuntimeSourceResolutionSnapshot joins already-admitted physical paths and correlated component records instead of rescanning the alias map for every source identity. Live callback contexts retain their live resolution path; record correlations, ordering and ambiguity are preserved.
  • Compiled invocation placement uses the actual active source memory type. ArrayBridge-native stacking removes mandatory GPU-host-GPU transfers. ImageFileFormat owns host projection at the disk boundary; memory checkpoints retain their declared framework. Mixed intrinsic-channel promotion and mask broadcasting still need generic ArrayBridge capabilities and are explicitly outstanding P5 work.

The seven supplied P0–P5 documents are included unchanged in plans/runtime_plumbing/, with an implementation map. P0 profiling is already merged in #645. This head merges main e7e79cd and implements P1–P4 plus the core P5 path; the overall performance goal remains active.

Validation includes 127 integrated source/projection/manifest/profile/metadata-transport controls, 234 invocation/contract/projection controls, actual pickle/spawn receiving, and production consumers for parse-once and live ownership/document epochs. Actual GTX1080 receiving verifies native NumPy/CuPy mixed stacking, placement, independent pixel/mask buffers, memory checkpoints, and TIFF/NPY/PNG host serialization without replacing the GPU owner. GPU latency and all mixed-channel frontend cases are not claimed.

The saved real 60-plane matching frontier retains every source position: old matching 233.219 ms; candidate first admission plus matching 53.358 ms. The baseline receipt explicitly documents recovery of exact timing fields after a helper filename overwrite. Twenty-one existing lifetime/alias/ambiguity/correlation controls pass. This is component evidence, not an end-to-end speedup.

The ordinary 3D pair on exact main e7e79cd and integrated b379d3c is complete: all four strict comparisons against both retained native observations pass, including zero image tolerance. Source/dependency/environment/input/native ABI guards and owned server/worker shutdown pass. Identical sparse projections and source-qualified preparation precede public READY; server/registry/kernel startup is outside pipeline clocks. No coarse or sampled tracing is installed in the ordinary jobs.

Metric Main Integrated Change
Compilation 0.971835 s 1.267273 s +0.295438 s
Execution 5.555591 s 5.326673 s −0.228919 s (−4.12%)
Total 6.774921 s 6.837611 s +0.062690 s (+0.93%)
Peak RSS 1640.191 MiB 1700.121 MiB +59.930 MiB

This is one observation per source. Execution improves, but total and RSS are worse; no global speedup or per-patch causation is established. Evidence: /home/ts/.local/state/openhcs-maintenance/20261004/runtime-plans-main-ab-3d-v2. The earlier sparse-preparation rejection is retained as V1 and launched no scientific job. Fresh original 3D native CellProfiler full-invocation timing completed separately after scientific acceptance: 14.372347 s and 14.109910 s, mean 14.241128 s (one warmup excluded). The integrated execution/native ratio is 2.67×; integrated total/native ratio is 2.08×. All source/dependency/input guards and owned cleanup pass at native terminal. Fresh native outputs are retained; the four scientific comparisons above remain explicitly against the retained V15 goldens. The user explicitly requested merging locally qualified changes despite existing main CI failures. Remaining compile and runtime costs continue under the active performance goal.

The earlier P0-only diagnostic matches both retained native references; its coarse/sampled times are diagnostic only. Its py-spy terminal child-process error and detail-hook errors remain retained. Prior negative whole-pipeline observations for #630, #635, #597 and #608 are retained; local validity is not presented as proof of runtime improvement.

Persisted formats: internal reset. Execution-local manifest indexes and compiled derived state are regenerated. No external pipeline, configuration, image or measurement format changes.

Closes #647

trissim added 13 commits October 4, 2026 13:06
Share parsed filename results and derived component membership through the existing RuntimePatternDiscoveryCache. Physical discovery, matching and grouped selection reuse the same parser semantic snapshot; grouped selection does not parse names.

Retain the context source authority and rebind against strong owners. Observe optional workspace file eligibility and document epochs live while retaining its provider. Consume the producer store ordered load view without re-sorting or re-comparing known cohorts. Make runtime logging lazy and remove duplicate traceback formatting.

Existing source/provenance/ambiguity controls: 153 passed. Production consumer probe verifies parse-once behavior and live ownership/document epochs. Whole-pipeline parity and performance remain integration gates; no speedup claim. External persisted result formats unchanged; internal execution caches derive from current declarations.
Implement P1 of plans/runtime_plumbing using the existing callable contract and compiled invocation. Bind declared return slots, executable callables, parameter bindings and framework devices before runtime; keep live adapter requests and selected artifact admission.

Delete the per-call return matcher and Core projection forwarding. Rebuild process-local executable state from declaration-only pickle transport. Preserve authored kwargs, plate calls without image domains, callable-qualified memory errors and debug/error diagnostics.
Reuse the existing runtime resolution snapshot for exact path and correlated component matching. Keep live resolver contexts on their original read path and retain source-position ambiguity and ordering.

Refs #648
@trissim
trissim marked this pull request as ready for review October 4, 2026 18:38
Copilot AI balanced review requested due to automatic review settings October 4, 2026 18:38
@trissim
trissim merged commit 5c20b3d into main Oct 4, 2026
3 of 40 checks passed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retain compiled, source and publication facts through the runtime performance plan

2 participants