Repository navigation
Eliminate repeated runtime source-context projection and scalar metadata allocation outside callable execution #384
Description
The ordinary 1w_1t 3D monolayer pipeline executes in approximately 9.75 seconds on current production sources. A separate lightweight owner-thread trace measured approximately 3 seconds in raw CellProfiler calls, with substantial remaining work in runtime stack/provenance handling, output ownership checks, recording and publication. Server startup, registry/kernel preparation and shutdown are excluded.
The trace records 29,101 component metadata merges and 6,240 leading-plane metadata projections in one well. Full diagnostic profiling identifies image output ownership checks that flatten pixel payloads just to compare source identity, plus repeated allocation/scanning of unrelated scalar metadata for individual component queries. Diagnostic profiler times are not accepted performance measurements.
Implement the generic repair through existing nominal owners: source metadata role views, source provenance, runtime alignment strategies and output context consumers. Preserve exact alias priority, plane ordering/cardinality, contributor identity, mask/axis validation and physical outputs. Avoid per-function shortcuts, mutable identity caching and a second main-flow store. Remove obsolete projection helpers after migration.
Qualification must include independent physical projection controls, saved representative replay, original source failure comparison, unprofiled production timing with current-main controls and CellProfiler output parity. Extend the structural work to the other dominant plumbing phases after measuring this coherent dependency; do not declare the entire performance target achieved by a lookup microbenchmark.
Current working branch: perf/runtime-plumbing-20261001. An initial representative metadata replay preserves every answer and halves component lookup time; end-to-end benefit is not yet established. Four stale runtime value fixture assertions reproduce on unchanged main and will be migrated to the merged explicit filename-extension and typed measurement materialization contracts, retaining the original failure evidence.
Activity
trissim commentedon Oct 1, 2026
The first prototype (2253ec63b, based on 66ed7ef) is not accepted as a performance optimization. All paired production observations are retained; two observations per revision on each of three pipelines did not establish an end-to-end improvement. Candidate-minus-baseline mean execution time was +0.325s on 3D monolayer, +0.365s on advanced segmentation and +1.566s on ImagingFlowCytometry. Server startup, readiness/kernel preparation and shutdown were excluded from every pipeline clock. These are OpenHCS-only measurements, with no native CP speedup claim.
The behavioral gates passed (574 targeted controls and 751 consumer tests), and changed-source R1 passed. The original R0 architectural ratchet failed because this version added complexity to oversized metadata/provenance carriers. Neither correctness nor a component lookup microbenchmark is sufficient to promote this route. No optimization PR has been opened or merged for it.
The larger generic plumbing investigation continues on latest main (10f7e70). One structural lead is repeated materialization rendering: the owner-thread profile observed eight rendering passes for two actual saves, with metadata and observation consumers reconstructing outputs after the save discards them. A lighter production diagnostic is running to bound its payoff before selecting the architecture. Stack/provenance reconstruction remains another substantial term. The intended fix must reuse actual production results at their existing lifetime boundaries, preserve live preview and exact output semantics, and avoid mutable identity caches or a parallel main-flow truth store.
Evidence is retained locally in runtime-plumbing-first-route-disposition-20261001.json, runtime-plumbing-qualified-abba-20261001.json, runtime-plumbing-r0-native314-20261001.log and the individual source/observation/test receipts under /home/ts/code/projects/openhcs-benchmark-runs. This issue remains open; the rejected prototype is not represented as a resolved performance fix.
Preserve raster publication admission for actual saved outputs (#384)
Sync declaration-owned function help from main before final qualifica…
docs: record repaired qualification and formal issue-link audit for #384
refactor(runtime): store cached image payloads directly (#384)
perf(metadata): derive geometry from transaction-owned projections (#384
docs(perf): prioritize shared plumbing and record owner-level evidence (
perf(runtime): reuse compiled callable and resolved image inputs (#384)
refactor: derive broadcast and output contexts from existing owners
docs: retain shared plumbing bounds and rejected cache assumptions
refactor: bind declared output contexts on aligned image stack
docs: record existing-owner gates against synchronized main
docs: retain mixed current-main plumbing benchmark outcomes
docs: retain output equality and current startup integration gates
58 remaining items
Pin merged terminal execution record owner from ZMQRuntime PR17
Capture normalized pipeline declarations before axis compilation
docs: retain qualified runtime samples after cache and client consoli…
refactor: use provenance owner for source plane identity axes
test: use native object artifact source payload authority
refactor: derive image context through one final field update
Own physical measurement batches across declared query and export
Remove scalar label wrapper and dictionary feature selection
refactor: remove unused mixed-row layout normalization family
Let artifact kinds own current-source output admission
Bind the source metadata reader in RelateObjects feature admission
Record current-source full science acceptance and rejected timing scope
Capture resolved request configuration once across compilation
Add bounded exact native median selection networks (#619)