diff --git a/benchmark/results/perf_lazy_ome_zarr_20260929/README.md b/benchmark/results/perf_lazy_ome_zarr_20260929/README.md new file mode 100644 index 0000000000..d0bd18914c --- /dev/null +++ b/benchmark/results/perf_lazy_ome_zarr_20260929/README.md @@ -0,0 +1,35 @@ +# Lazy NGFF imports: official30 fresh-server 1w_1t (2026-09-29) + +This report accompanies [PolyStore issue #14](https://github.com/OpenHCSDev/PolyStore/issues/14) and [PR #15](https://github.com/OpenHCSDev/PolyStore/pull/15). The OpenHCS integration code is the same on both sides of the comparison: commit `503686d1b`, including the step-scoped Costes cache. Only the PolyStore submodule changes, from `0efe67f` to `11269ed`. Each observation uses one well, one OpenHCS thread, the `fork` worker start method, and a fresh client-owned ZMQ server. `total_seconds` includes connection, server startup and shutdown, compilation, and execution. Source workspace preparation before the observation timer is included only in the separately measured full CLI wall time. + +Importing the Zarr backend class eagerly loaded OME-Zarr and Dask even when the benchmark selected disk materialization. Moving NGFF imports to their consuming reads and writes lowered the cold `openhcs.runtime.zmq_execution_server` module import from **1.523 to 0.900 seconds**. In three fresh endpoint trials, connection fell from approximately **1.81 to 1.17 seconds**. These import and connection timings identify the mechanism; the full production-path sweep measures its transfer. + +All 30 cases succeeded on both sides. Every case spent less time outside the two server jobs; the paired median saving was **0.731 seconds**, with a range of **0.601 to 0.828 seconds**. All 110 produced result files had identical paths and bytes. + +| Sum over 30 observations (seconds) | Control | Lazy NGFF imports | Difference | +| --- | ---: | ---: | ---: | +| Server compilation jobs | 84.095 | 89.271 | +5.176 | +| Server execution jobs | 87.857 | 84.018 | -3.839 | +| Other measured time (`total - compile - execute`) | 88.976 | 67.294 | **-21.682** | +| Per-observation total | 260.928 | 240.582 | **-20.346** | +| Full CLI wall time | 283.613 | 263.415 | **-20.198** | + +The change targets import/startup work. The compilation and execution differences are run variation, not claimed effects of lazy NGFF loading. The native CellProfiler 4.2.8.1 total-phase reference sums to 1,319.357 seconds, and its 5× reference line is 263.871 seconds. The candidate's OpenHCS per-observation sum is 23.289 seconds below that line. Native CP and OpenHCS total timers have different preparation boundaries, so the line is indicative rather than a like-for-like end-to-end ratio. Native execution data and the full four-mode scaling run remain in the existing benchmark artifacts; this startup-only change does not alter native CP or OpenHCS image-processing numerics. + +![Fresh-server total-time breakdown and native reference](fresh_server_total_breakdown.png) + +![Per-case reduction in time outside server jobs](per_case_nonjob_delta.png) + +`control.csv` and `candidate.csv` retain all per-case measurements. `plot.py` regenerates the figures using those CSVs and the [native CP summary](../perf_lazy_catalog_20260929/native_cp_summary.csv). Raw result trees and receipts are under `/home/ts/code/projects/openhcs-benchmark-runs/perf-coloc-step-cache-warm-candidate-full-20260929` and `/home/ts/code/projects/openhcs-benchmark-runs/perf-lazy-ome-zarr-full-candidate-20260929`. + +Validation: 142 PolyStore Zarr/NGFF/FileManager tests passed; 43 OpenHCS Zarr-path unit/integration tests passed with two skipped. The full benchmark used the ordinary ZMQ completion path and achieved 110/110 byte parity. Ruff E9/F63/F7 and whitespace checks passed for the PolyStore change. + +Fresh benchmark command: + +```bash +OPENHCS_CPU_ONLY=true .venv/bin/python scripts/benchmark_cppipe_well_throughput.py \ + --manifest benchmark/manifests/official30_portable_axis1.json \ + --output-dir /path/to/output \ + --native-summary-csv benchmark/results/perf_lazy_catalog_20260929/native_cp_summary.csv \ + --mode 1w_1t +``` diff --git a/benchmark/results/perf_lazy_ome_zarr_20260929/candidate.csv b/benchmark/results/perf_lazy_ome_zarr_20260929/candidate.csv new file mode 100644 index 0000000000..0cd92f406d --- /dev/null +++ b/benchmark/results/perf_lazy_ome_zarr_20260929/candidate.csv @@ -0,0 +1,31 @@ +case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256 +ExampleColocalization,1w_1t,1,1,3.5455358028411865,0.0,3.011810302734375,8.903501073364168,0.3320262232625062,1,37.035618890076876,37.035618890076876,12.296796666261757,1238.1640625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleCometAssay,1w_1t,1,1,3.2558608055114746,0.0,2.317523717880249,7.795750766061246,0.43149504459642035,1,13.304211451206356,13.304211451206356,5.740701313458493,1041.89453125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleFly,1w_1t,1,1,3.1798794269561768,0.0,2.140878915786743,7.764661327935755,0.4670978786450956,1,8.094948755111545,8.094948755111545,3.7811333912533605,811.70703125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleFlyURL,1w_1t,1,1,2.8148956298828125,0.0,1.1226205825805664,6.163078103680164,0.890772907175193,1,4.900778113398701,4.900778113398701,4.365480367492719,759.9375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleHuman,1w_1t,1,1,2.763174057006836,0.0,1.1867127418518066,6.205665768124163,0.8426639107620513,1,4.420192994643003,4.420192994643003,3.7247371151888955,725.55078125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,2.7868611812591553,0.0,2.323122501373291,7.0478366878814995,0.43045513071689495,1,946.6214662808925,946.6214662808925,407.4780670073604,1151.5390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,2.6810081005096436,0.0,0.16802477836608887,4.800115841906518,5.951503163546635,1,4.54475614009425,4.54475614009425,27.048130545318923,685.8125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleIlluminationCorrection_Example2,1w_1t,1,1,3.08339262008667,0.0,1.7868974208831787,6.931202474981546,0.5596292144770948,1,48.04419008316472,48.04419008316472,26.886932356429703,900.875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleIlluminationCorrection_Example3,1w_1t,1,1,3.072758913040161,0.0,0.8477485179901123,6.092497589997947,1.1795951025321207,1,2.1520505752414465,2.1520505752414465,2.5385483189562437,717.12890625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.5723140239715576,0.0,21.692100048065186,27.888105599675328,0.04609973205840872,1,61.60158259188756,61.60158259188756,2.8398164518599516,2358.83984375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleNeighbors,1w_1t,1,1,2.5889482498168945,0.0,1.4464046955108643,6.261920489836484,0.6913694369934301,1,4.5466188420541584,4.5466188420541584,3.143393309054705,757.1015625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExamplePercentPositive,1w_1t,1,1,2.7028207778930664,0.0,0.4494204521179199,5.302496148273349,2.225087877704368,1,2.5050158216618,2.5050158216618,5.573880338237319,723.390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleSpeckles,1w_1t,1,1,2.557448387145996,0.0,0.804171085357666,5.388225274626166,1.243516483255844,1,3.179383432958275,3.179383432958275,3.9536157054741667,857.75390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleTrackObjects,1w_1t,1,1,2.567854166030884,0.0,2.7153048515319824,7.354687624145299,0.3682827729033067,1,9.587221414782107,9.587221414782107,3.5308084870739176,1491.51171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleTumor,1w_1t,1,1,2.433227300643921,0.0,0.8098001480102539,5.325696494895965,1.2348725823983644,1,4.360487373080105,4.360487373080105,5.384646302910889,894.8828125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleUntangleAndStraightenWorms,1w_1t,1,1,3.26237154006958,0.0,1.2429263591766357,6.708871386945248,0.8045528945596102,1,3.4416790287941694,3.4416790287941694,2.769012824761457,782.8125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleUntangleWorms,1w_1t,1,1,3.3210535049438477,0.0,2.180208921432495,7.8420076933689415,0.4586716392954466,1,4.618807300925255,4.618807300925255,2.1185159163051637,840.1171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleUntangleWormsBrightField,1w_1t,1,1,3.423585891723633,0.0,3.7209994792938232,9.53094509523362,0.26874499863939283,1,7.377188352867961,7.377188352867961,1.982582473854045,955.33984375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleVitra,1w_1t,1,1,3.1533331871032715,0.0,1.3800678253173828,6.8105818326584995,0.7246020678512839,1,4.122247557155788,4.122247557155788,2.9869891041099876,896.59765625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleWoundHealing,1w_1t,1,1,2.26784348487854,0.0,1.022160530090332,5.21838288474828,0.9783199121488543,1,4.702593941241503,4.702593941241503,4.600641291467122,1087.171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleYeastColonies,1w_1t,1,1,3.4512972831726074,0.0,1.8895881175994873,7.712748416233808,0.5292158596289172,1,12.096297012176365,12.096297012176365,6.401552221625616,1040.890625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +ExampleYeastPatches,1w_1t,1,1,3.4672110080718994,0.0,2.503957748413086,8.367739741690457,0.39936776115082706,1,5.990533472970128,5.990533472970128,2.3924259411991686,1039.546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp_tutorial_3d_monolayer,1w_1t,1,1,3.7307965755462646,0.0,9.506073474884033,15.830886270850897,0.1051959047699449,1,14.058660756796598,14.058660756796598,1.4789135381649365,1339.546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp_tutorial_advanced_segmentation_final,1w_1t,1,1,4.483505725860596,0.0,9.217328071594238,16.570328325033188,0.10849131030518251,1,36.253466514870524,36.253466514870524,3.9331860853033613,1029.8359375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp_tutorial_quality_control,1w_1t,1,1,2.1472067832946777,0.0,0.6336092948913574,5.065476151648909,1.5782596752648115,1,4.787729676347226,4.787729676347226,7.556280684247474,739.578125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.409761667251587,0.0,5.560011625289917,11.475596643984318,0.17985573905123925,1,16.074153264053166,16.074153264053166,2.8910287149291722,923.109375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp_tutorial_pixel_based_classification,1w_1t,1,1,2.502073287963867,0.0,0.9103264808654785,5.467205463908613,1.0985069873494901,1,4.208680049050599,4.208680049050599,4.623264441400478,924.19140625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp_tutorial_translocation_final,1w_1t,1,1,2.7205471992492676,0.0,0.9192965030670166,5.8188305222429335,1.0877883214650934,1,3.004684013314545,3.004684013314545,3.268460179376429,808.8125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp4_supplement_combine_objects,1w_1t,1,1,2.1259372234344482,0.0,0.15667152404785156,4.443546548020095,6.382780828088287,1,1.8334413380362093,1.8334413380362093,11.702454221842054,707.51171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b +cp_tutorial_translocation_start,1w_1t,1,1,2.1982362270355225,0.0,0.35207080841064453,4.493525821249932,2.840337727840335,1,2.1760300607420504,2.1760300607420504,6.1806602784403415,744.8671875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,e6917c0282cb3da492e8675c88098202bb388a7d3442523e847c3b3d5b6acd1b diff --git a/benchmark/results/perf_lazy_ome_zarr_20260929/control.csv b/benchmark/results/perf_lazy_ome_zarr_20260929/control.csv new file mode 100644 index 0000000000..9304dbb288 --- /dev/null +++ b/benchmark/results/perf_lazy_ome_zarr_20260929/control.csv @@ -0,0 +1,31 @@ +case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256 +ExampleColocalization,1w_1t,1,1,3.302696704864502,0.0,3.718935251235962,10.096186046022922,0.2688941679389705,1,37.035618890076876,37.035618890076876,9.958661925552038,1310.671875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleCometAssay,1w_1t,1,1,3.055711030960083,0.0,2.591752529144287,8.60999168176204,0.38583930709239733,1,13.304211451206356,13.304211451206356,5.133287727744198,1217.671875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleFly,1w_1t,1,1,2.9811851978302,0.0,2.148517608642578,8.231432453263551,0.4654371907297491,1,8.094948755111545,8.094948755111545,3.7676902076803973,919.47265625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleFlyURL,1w_1t,1,1,2.655057668685913,0.0,1.139488697052002,6.735906984191388,0.8775865900092942,1,4.900778113398701,4.900778113398701,4.300857152929748,829.60546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleHuman,1w_1t,1,1,2.6445040702819824,0.0,1.2340328693389893,6.7614734950475395,0.8103511866224851,1,4.420192994643003,4.420192994643003,3.581908638309353,798.98828125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,2.55765700340271,0.0,2.846787214279175,8.120627946220338,0.35127318086301246,1,946.6214662808925,946.6214662808925,332.522733533698,1274.73046875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,2.516428232192993,0.0,0.16867399215698242,5.414463059045374,5.928596265566031,1,4.54475614009425,4.54475614009425,26.944024280071062,761.5625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleIlluminationCorrection_Example2,1w_1t,1,1,2.8671979904174805,0.0,1.772667646408081,7.5290824440307915,0.5641215385333392,1,48.04419008316472,48.04419008316472,27.10276242730308,1004.50390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleIlluminationCorrection_Example3,1w_1t,1,1,2.940793514251709,0.0,0.8718390464782715,6.647682448849082,1.1470007039021997,1,2.1520505752414465,2.1520505752414465,2.468403524635073,799.00390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.4476630687713623,0.0,21.533740043640137,28.337062722072005,0.046438751372191105,1,61.60158259188756,61.60158259188756,2.8607005781181623,2426.86328125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleNeighbors,1w_1t,1,1,2.472842216491699,0.0,1.471928358078003,6.772686000913382,0.6793808914081716,1,4.5466188420541584,4.5466188420541584,3.088885961807943,837.9375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExamplePercentPositive,1w_1t,1,1,2.444183111190796,0.0,0.5557022094726562,5.940390051808208,1.7995249667064817,1,2.5050158216618,2.5050158216618,4.507838513075161,801.6953125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleSpeckles,1w_1t,1,1,2.3182339668273926,0.0,0.8259930610656738,5.874337824061513,1.21066392338675,1,3.179383432958275,3.179383432958275,3.849164820896099,902.00390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleTrackObjects,1w_1t,1,1,2.423386573791504,0.0,3.357529878616333,8.586916312109679,0.29783800476918126,1,9.587221414782107,9.587221414782107,2.85543889745907,1571.36328125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleTumor,1w_1t,1,1,2.3061318397521973,0.0,0.8246448040008545,5.943696439266205,1.2126433043031262,1,4.360487373080105,4.360487373080105,5.287715816463916,955.89453125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleUntangleAndStraightenWorms,1w_1t,1,1,3.0381131172180176,0.0,1.2356970310211182,7.267080613877624,0.8092598548801643,1,3.4416790287941694,3.4416790287941694,2.7852126713860743,863.01953125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleUntangleWorms,1w_1t,1,1,3.2455945014953613,0.0,2.2246034145355225,8.542148170992732,0.44951832468925307,1,4.618807300925255,4.618807300925255,2.076238519974411,913.421875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleUntangleWormsBrightField,1w_1t,1,1,3.3344175815582275,0.0,3.7285454273223877,10.139678824227303,0.26820110402091535,1,7.377188352867961,7.377188352867961,1.9785700608094252,1025.49609375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleVitra,1w_1t,1,1,2.9031996726989746,0.0,1.3693935871124268,7.169969641137868,0.7302502431814736,1,4.122247557155788,4.122247557155788,3.0102722810672495,936.21875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleWoundHealing,1w_1t,1,1,2.035460948944092,0.0,1.54510498046875,6.293716668151319,0.6472052142998223,1,4.702593941241503,4.702593941241503,3.0435433195062527,1194.7890625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleYeastColonies,1w_1t,1,1,3.2516937255859375,0.0,2.1811540126800537,8.476604882627726,0.45847289746003217,1,12.096297012176365,12.096297012176365,5.545824339709628,1150.05078125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +ExampleYeastPatches,1w_1t,1,1,3.356388568878174,0.0,2.5256619453430176,9.025991102680564,0.3959358067867579,1,5.990533472970128,5.990533472970128,2.3718667037035064,1094.1796875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp_tutorial_3d_monolayer,1w_1t,1,1,3.613546371459961,0.0,10.241461515426636,17.157875870820135,0.09764231389179245,1,14.058660756796598,14.058660756796598,1.372720166513358,1420.94140625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp_tutorial_advanced_segmentation_final,1w_1t,1,1,4.264207124710083,0.0,9.438591003417969,17.339269490912557,0.10594801699087003,1,36.253466514870524,36.253466514870524,3.84098288629544,1102.30078125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp_tutorial_quality_control,1w_1t,1,1,1.9192872047424316,0.0,0.6170415878295898,5.4689456769265234,1.6206363067316831,1,4.787729676347226,4.787729676347226,7.759168540305045,807.9375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.207782506942749,0.0,5.538364887237549,12.039867510087788,0.18055870647027458,1,16.074153264053166,16.074153264053166,2.902328320962382,996.01953125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp_tutorial_pixel_based_classification,1w_1t,1,1,2.3737740516662598,0.0,0.8111603260040283,6.017411075998098,1.232801910968996,1,4.208680049050599,4.208680049050599,5.188468807126667,1021.8046875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp_tutorial_translocation_final,1w_1t,1,1,2.562145948410034,0.0,0.9290289878845215,6.426769196987152,1.0763926777754111,1,3.004684013314545,3.004684013314545,3.234219870960612,880.37109375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp4_supplement_combine_objects,1w_1t,1,1,1.9982993602752686,0.0,0.15286612510681152,4.984665384981781,6.541671670563208,1,1.8334413380362093,1.8334413380362093,11.993771260670972,778.32421875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 +cp_tutorial_translocation_start,1w_1t,1,1,2.0574376583099365,0.0,0.2561960220336914,4.975822530221194,3.9032612296708247,1,2.1760300607420504,2.1760300607420504,8.493613770692695,835.33203125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,96d7bea1c83b1b08e752ca5c6ed1f7e19a221bf2965d6aafb8b96c6061ce7c98 diff --git a/benchmark/results/perf_lazy_ome_zarr_20260929/fresh_server_total_breakdown.png b/benchmark/results/perf_lazy_ome_zarr_20260929/fresh_server_total_breakdown.png new file mode 100644 index 0000000000..195e784e35 Binary files /dev/null and b/benchmark/results/perf_lazy_ome_zarr_20260929/fresh_server_total_breakdown.png differ diff --git a/benchmark/results/perf_lazy_ome_zarr_20260929/per_case_nonjob_delta.png b/benchmark/results/perf_lazy_ome_zarr_20260929/per_case_nonjob_delta.png new file mode 100644 index 0000000000..24427ad43f Binary files /dev/null and b/benchmark/results/perf_lazy_ome_zarr_20260929/per_case_nonjob_delta.png differ diff --git a/benchmark/results/perf_lazy_ome_zarr_20260929/plot.py b/benchmark/results/perf_lazy_ome_zarr_20260929/plot.py new file mode 100644 index 0000000000..5e44049581 --- /dev/null +++ b/benchmark/results/perf_lazy_ome_zarr_20260929/plot.py @@ -0,0 +1,102 @@ +"""Regenerate startup and total-time figures from the retained CSVs.""" + +from __future__ import annotations + +import csv +from pathlib import Path + +import matplotlib.pyplot as plt +import numpy as np + +HERE = Path(__file__).resolve().parent + + +def rows(path: Path) -> dict[str, dict[str, str]]: + with path.open(newline="", encoding="utf-8") as handle: + return {row["case_name"]: row for row in csv.DictReader(handle)} + + +def nonjob(row: dict[str, str]) -> float: + return ( + float(row["total_seconds"]) + - float(row["compile_seconds"]) + - float(row["execute_seconds"]) + ) + + +control = rows(HERE / "control.csv") +candidate = rows(HERE / "candidate.csv") +native = rows(HERE.parent / "perf_lazy_catalog_20260929" / "native_cp_summary.csv") + +names = ("Control", "Lazy NGFF imports") +observations = (control, candidate) +compile_time = [ + sum(float(row["compile_seconds"]) for row in dataset.values()) + for dataset in observations +] +execute_time = [ + sum(float(row["execute_seconds"]) for row in dataset.values()) + for dataset in observations +] +nonjob_time = [sum(nonjob(row) for row in dataset.values()) for dataset in observations] +native_five_x_line = ( + sum(float(row["median_native_total_phase_seconds"]) for row in native.values()) / 5 +) + +fig, ax = plt.subplots(figsize=(8, 5), layout="constrained") +x = np.arange(2) +ax.bar(x, compile_time, label="Server compilation", color="#697b8c") +ax.bar( + x, + execute_time, + bottom=compile_time, + label="Server execution", + color="#d87a57", +) +bottom = np.array(compile_time) + np.array(execute_time) +ax.bar(x, nonjob_time, bottom=bottom, label="Other measured time", color="#3a8c83") +totals = bottom + np.array(nonjob_time) +for index, total in enumerate(totals): + ax.text( + index, + total - 7, + f"{total:.1f} s", + ha="center", + color="white", + fontsize=10, + fontweight="bold", + ) +ax.axhline( + native_five_x_line, + color="#3f4b56", + linestyle="--", + linewidth=1, + label="Native CP total-phase / 5 (different timer boundary)", +) +ax.set_xticks(x, names) +ax.set_ylabel("Sum of 30 fresh-server observations (seconds)") +ax.set_ylim(0, max(totals) + 24) +ax.legend( + frameon=False, + fontsize=8, + loc="upper center", + bbox_to_anchor=(0.5, -0.13), + ncol=2, +) +ax.set_title("Lazy NGFF imports remove fresh-server startup work") +fig.savefig(HERE / "fresh_server_total_breakdown.png", dpi=180) +plt.close(fig) + +ordered = sorted( + control, key=lambda name: nonjob(candidate[name]) - nonjob(control[name]) +) +deltas = [nonjob(candidate[name]) - nonjob(control[name]) for name in ordered] +fig, ax = plt.subplots(figsize=(10, 9), layout="constrained") +ax.barh(range(len(ordered)), deltas, color="#3a8c83") +ax.set_yticks(range(len(ordered)), ordered, fontsize=7.5) +ax.invert_yaxis() +ax.axvline(0, color="#2f3640", linewidth=0.8) +ax.set_xlabel("Candidate minus control non-job seconds (lower is faster)") +ax.set_title("All 30 fresh-server cases spend less time outside jobs") +fig.savefig(HERE / "per_case_nonjob_delta.png", dpi=180) +plt.close(fig) diff --git a/external/PolyStore b/external/PolyStore index e430c331ad..f94bbbe863 160000 --- a/external/PolyStore +++ b/external/PolyStore @@ -1 +1 @@ -Subproject commit e430c331ad931edc92dfe9d4fcd0d837a3cfeea8 +Subproject commit f94bbbe8631a4c78f7ceaec72fb99671b58c9a18