Skip to content

Adopt lazy PolyStore NGFF imports for fresh execution servers #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
trissim merged 3 commits into from
Sep 29, 2026
Merged

Adopt lazy PolyStore NGFF imports for fresh execution servers #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6534cc6
perf: adopt lazy PolyStore NGFF imports
trissim Sep 29, 2026
fa9c2e3
Update PolyStore after lazy NGFF import merge
trissim Sep 29, 2026
200cc5f
Merge current main into lazy NGFF import PR
trissim Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions benchmark/results/perf_lazy_ome_zarr_20260929/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Lazy NGFF imports: official30 fresh-server 1w_1t (2026-09-29)

This report accompanies [PolyStore issue #14](https://github.com/OpenHCSDev/PolyStore/issues/14) and [PR #15](https://github.com/OpenHCSDev/PolyStore/pull/15). The OpenHCS integration code is the same on both sides of the comparison: commit `503686d1b`, including the step-scoped Costes cache. Only the PolyStore submodule changes, from `0efe67f` to `11269ed`. Each observation uses one well, one OpenHCS thread, the `fork` worker start method, and a fresh client-owned ZMQ server. `total_seconds` includes connection, server startup and shutdown, compilation, and execution. Source workspace preparation before the observation timer is included only in the separately measured full CLI wall time.

Importing the Zarr backend class eagerly loaded OME-Zarr and Dask even when the benchmark selected disk materialization. Moving NGFF imports to their consuming reads and writes lowered the cold `openhcs.runtime.zmq_execution_server` module import from **1.523 to 0.900 seconds**. In three fresh endpoint trials, connection fell from approximately **1.81 to 1.17 seconds**. These import and connection timings identify the mechanism; the full production-path sweep measures its transfer.

All 30 cases succeeded on both sides. Every case spent less time outside the two server jobs; the paired median saving was **0.731 seconds**, with a range of **0.601 to 0.828 seconds**. All 110 produced result files had identical paths and bytes.

| Sum over 30 observations (seconds) | Control | Lazy NGFF imports | Difference |
| --- | ---: | ---: | ---: |
| Server compilation jobs | 84.095 | 89.271 | +5.176 |
| Server execution jobs | 87.857 | 84.018 | -3.839 |
| Other measured time (`total - compile - execute`) | 88.976 | 67.294 | **-21.682** |
| Per-observation total | 260.928 | 240.582 | **-20.346** |
| Full CLI wall time | 283.613 | 263.415 | **-20.198** |

The change targets import/startup work. The compilation and execution differences are run variation, not claimed effects of lazy NGFF loading. The native CellProfiler 4.2.8.1 total-phase reference sums to 1,319.357 seconds, and its 5× reference line is 263.871 seconds. The candidate's OpenHCS per-observation sum is 23.289 seconds below that line. Native CP and OpenHCS total timers have different preparation boundaries, so the line is indicative rather than a like-for-like end-to-end ratio. Native execution data and the full four-mode scaling run remain in the existing benchmark artifacts; this startup-only change does not alter native CP or OpenHCS image-processing numerics.

![Fresh-server total-time breakdown and native reference](fresh_server_total_breakdown.png)

![Per-case reduction in time outside server jobs](per_case_nonjob_delta.png)

`control.csv` and `candidate.csv` retain all per-case measurements. `plot.py` regenerates the figures using those CSVs and the [native CP summary](../perf_lazy_catalog_20260929/native_cp_summary.csv). Raw result trees and receipts are under `/home/ts/code/projects/openhcs-benchmark-runs/perf-coloc-step-cache-warm-candidate-full-20260929` and `/home/ts/code/projects/openhcs-benchmark-runs/perf-lazy-ome-zarr-full-candidate-20260929`.

Validation: 142 PolyStore Zarr/NGFF/FileManager tests passed; 43 OpenHCS Zarr-path unit/integration tests passed with two skipped. The full benchmark used the ordinary ZMQ completion path and achieved 110/110 byte parity. Ruff E9/F63/F7 and whitespace checks passed for the PolyStore change.

Fresh benchmark command:

```bash
OPENHCS_CPU_ONLY=true .venv/bin/python scripts/benchmark_cppipe_well_throughput.py \
--manifest benchmark/manifests/official30_portable_axis1.json \
--output-dir /path/to/output \
--native-summary-csv benchmark/results/perf_lazy_catalog_20260929/native_cp_summary.csv \
--mode 1w_1t
```
31 changes: 31 additions & 0 deletions benchmark/results/perf_lazy_ome_zarr_20260929/candidate.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256
ExampleColocalization,1w_1t,1,1,3.5455358028411865,0.0,3.011810302734375,8.903501073364168,0.3320262232625062,1,37.035618890076876,37.035618890076876,12.296796666261757,1238.1640625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleCometAssay,1w_1t,1,1,3.2558608055114746,0.0,2.317523717880249,7.795750766061246,0.43149504459642035,1,13.304211451206356,13.304211451206356,5.740701313458493,1041.89453125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleFly,1w_1t,1,1,3.1798794269561768,0.0,2.140878915786743,7.764661327935755,0.4670978786450956,1,8.094948755111545,8.094948755111545,3.7811333912533605,811.70703125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleFlyURL,1w_1t,1,1,2.8148956298828125,0.0,1.1226205825805664,6.163078103680164,0.890772907175193,1,4.900778113398701,4.900778113398701,4.365480367492719,759.9375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleHuman,1w_1t,1,1,2.763174057006836,0.0,1.1867127418518066,6.205665768124163,0.8426639107620513,1,4.420192994643003,4.420192994643003,3.7247371151888955,725.55078125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,2.7868611812591553,0.0,2.323122501373291,7.0478366878814995,0.43045513071689495,1,946.6214662808925,946.6214662808925,407.4780670073604,1151.5390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,2.6810081005096436,0.0,0.16802477836608887,4.800115841906518,5.951503163546635,1,4.54475614009425,4.54475614009425,27.048130545318923,685.8125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleIlluminationCorrection_Example2,1w_1t,1,1,3.08339262008667,0.0,1.7868974208831787,6.931202474981546,0.5596292144770948,1,48.04419008316472,48.04419008316472,26.886932356429703,900.875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleIlluminationCorrection_Example3,1w_1t,1,1,3.072758913040161,0.0,0.8477485179901123,6.092497589997947,1.1795951025321207,1,2.1520505752414465,2.1520505752414465,2.5385483189562437,717.12890625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.5723140239715576,0.0,21.692100048065186,27.888105599675328,0.04609973205840872,1,61.60158259188756,61.60158259188756,2.8398164518599516,2358.83984375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleNeighbors,1w_1t,1,1,2.5889482498168945,0.0,1.4464046955108643,6.261920489836484,0.6913694369934301,1,4.5466188420541584,4.5466188420541584,3.143393309054705,757.1015625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExamplePercentPositive,1w_1t,1,1,2.7028207778930664,0.0,0.4494204521179199,5.302496148273349,2.225087877704368,1,2.5050158216618,2.5050158216618,5.573880338237319,723.390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleSpeckles,1w_1t,1,1,2.557448387145996,0.0,0.804171085357666,5.388225274626166,1.243516483255844,1,3.179383432958275,3.179383432958275,3.9536157054741667,857.75390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleTrackObjects,1w_1t,1,1,2.567854166030884,0.0,2.7153048515319824,7.354687624145299,0.3682827729033067,1,9.587221414782107,9.587221414782107,3.5308084870739176,1491.51171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleTumor,1w_1t,1,1,2.433227300643921,0.0,0.8098001480102539,5.325696494895965,1.2348725823983644,1,4.360487373080105,4.360487373080105,5.384646302910889,894.8828125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleUntangleAndStraightenWorms,1w_1t,1,1,3.26237154006958,0.0,1.2429263591766357,6.708871386945248,0.8045528945596102,1,3.4416790287941694,3.4416790287941694,2.769012824761457,782.8125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleUntangleWorms,1w_1t,1,1,3.3210535049438477,0.0,2.180208921432495,7.8420076933689415,0.4586716392954466,1,4.618807300925255,4.618807300925255,2.1185159163051637,840.1171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleUntangleWormsBrightField,1w_1t,1,1,3.423585891723633,0.0,3.7209994792938232,9.53094509523362,0.26874499863939283,1,7.377188352867961,7.377188352867961,1.982582473854045,955.33984375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleVitra,1w_1t,1,1,3.1533331871032715,0.0,1.3800678253173828,6.8105818326584995,0.7246020678512839,1,4.122247557155788,4.122247557155788,2.9869891041099876,896.59765625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleWoundHealing,1w_1t,1,1,2.26784348487854,0.0,1.022160530090332,5.21838288474828,0.9783199121488543,1,4.702593941241503,4.702593941241503,4.600641291467122,1087.171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleYeastColonies,1w_1t,1,1,3.4512972831726074,0.0,1.8895881175994873,7.712748416233808,0.5292158596289172,1,12.096297012176365,12.096297012176365,6.401552221625616,1040.890625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
ExampleYeastPatches,1w_1t,1,1,3.4672110080718994,0.0,2.503957748413086,8.367739741690457,0.39936776115082706,1,5.990533472970128,5.990533472970128,2.3924259411991686,1039.546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp_tutorial_3d_monolayer,1w_1t,1,1,3.7307965755462646,0.0,9.506073474884033,15.830886270850897,0.1051959047699449,1,14.058660756796598,14.058660756796598,1.4789135381649365,1339.546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp_tutorial_advanced_segmentation_final,1w_1t,1,1,4.483505725860596,0.0,9.217328071594238,16.570328325033188,0.10849131030518251,1,36.253466514870524,36.253466514870524,3.9331860853033613,1029.8359375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp_tutorial_quality_control,1w_1t,1,1,2.1472067832946777,0.0,0.6336092948913574,5.065476151648909,1.5782596752648115,1,4.787729676347226,4.787729676347226,7.556280684247474,739.578125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.409761667251587,0.0,5.560011625289917,11.475596643984318,0.17985573905123925,1,16.074153264053166,16.074153264053166,2.8910287149291722,923.109375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp_tutorial_pixel_based_classification,1w_1t,1,1,2.502073287963867,0.0,0.9103264808654785,5.467205463908613,1.0985069873494901,1,4.208680049050599,4.208680049050599,4.623264441400478,924.19140625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp_tutorial_translocation_final,1w_1t,1,1,2.7205471992492676,0.0,0.9192965030670166,5.8188305222429335,1.0877883214650934,1,3.004684013314545,3.004684013314545,3.268460179376429,808.8125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp4_supplement_combine_objects,1w_1t,1,1,2.1259372234344482,0.0,0.15667152404785156,4.443546548020095,6.382780828088287,1,1.8334413380362093,1.8334413380362093,11.702454221842054,707.51171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
cp_tutorial_translocation_start,1w_1t,1,1,2.1982362270355225,0.0,0.35207080841064453,4.493525821249932,2.840337727840335,1,2.1760300607420504,2.1760300607420504,6.1806602784403415,744.8671875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b
Loading
Loading