ENH: Out-of-core architecture rewrite and filter optimizations#1568
Draft
joeykleingers wants to merge 1 commit into
Draft
ENH: Out-of-core architecture rewrite and filter optimizations#1568joeykleingers wants to merge 1 commit into
joeykleingers wants to merge 1 commit into
Conversation
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
March 24, 2026 18:13
b4ef97f to
99b49ed
Compare
joeykleingers
requested review from
JDuffeyBQ,
imikejackson,
jmarquisbq,
mmarineBlueQuartz and
nyoungbq
March 24, 2026 18:17
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
2 times, most recently
from
March 24, 2026 18:51
b4ef97f to
bb09048
Compare
This was referenced Mar 24, 2026
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
April 2, 2026 00:55
102c436 to
b4c1358
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
6 times, most recently
from
April 8, 2026 17:41
2bd614a to
110c054
Compare
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
4 times, most recently
from
April 16, 2026 13:03
35aecd0 to
3a88bbf
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
April 27, 2026 18:18
bdfed87 to
6fbfc8d
Compare
joeykleingers
marked this pull request as ready for review
April 28, 2026 00:08
joeykleingers
changed the title
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
May 5, 2026 13:30
2095f32 to
65c4ba8
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
May 12, 2026 15:03
65c4ba8 to
653cf62
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
3 times, most recently
from
June 1, 2026 15:13
4834fa1 to
d4f634b
Compare
5 tasks
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
June 5, 2026 15:58
d4f634b to
c04347c
Compare
joeykleingers
changed the title
joeykleingers
marked this pull request as draft
June 11, 2026 14:25
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
June 12, 2026 15:32
ec3a5e7 to
3c8903e
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
2 times, most recently
from
June 25, 2026 17:25
ed64934 to
a56f168
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
3 times, most recently
from
July 13, 2026 18:29
c6bb7fe to
ee6b370
Compare
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
July 15, 2026 15:18
8f3727e to
5a8c281
Compare
Adds storage-neutral APIs for choosing, creating, reading, writing, and recovering data stores at runtime. Equips SimplnxCore and OrientationAnalysis filters with bounded direct and scanline execution paths, plus memory budgeting, allocation safeguards, bulk transfer primitives, and DREAM3D/HDF5 integration. 561 files changed, +67105 / -18693 lines. ================================================================================ 1. Storage-neutral data stores and runtime format resolution ================================================================================ Files: src/simplnx/DataStructure, src/simplnx/DataStructure/IO, src/simplnx/Core, src/simplnx/Filter/Actions Extend datastore contracts with bulk, chunk, extent, planned-format, and recovery metadata operations while retaining in-memory implementations. Add runtime format resolvers and DataIOCollection hooks so applications can supply concrete storage backends without coupling simplnx to them. Carry selected formats through array, neighbor-list, geometry, string-store, HDF5, and DREAM3D creation and recovery paths, including metadata-only placeholder stores. ================================================================================ 2. Bounded-memory execution infrastructure ================================================================================ Files: src/simplnx/Filter/IFilter.cpp, src/simplnx/Utilities, src/simplnx/Common/Extent.hpp Add machine-aware shared memory budgeting, defensive preflight and execution allocation handling, extent-based access, slice-buffered transfers, union-find support, and storage-aware algorithm dispatch. Update array, geometry, clustering, meshing, parsing, and parallel utilities to use batched transfers and bounded temporary storage while preserving cancellation and error reporting. ================================================================================ 3. Direct and scanline filter algorithms ================================================================================ Files: src/Plugins/SimplnxCore/src/SimplnxCore/Filters, src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters Split storage-sensitive filters into direct and scanline implementations and select the appropriate path from runtime storage characteristics. Add bounded, chunk-sequential processing for segmentation, statistics, feature analysis, meshing, transforms, sampling, file output, and orientation workflows while retaining optimized in-memory paths. ================================================================================ 4. Tests, documentation, and build integration ================================================================================ Files: src/Plugins/*/test, src/Plugins/*/docs, test, cmake, vcpkg.json Register both algorithm paths, expand exemplar and regression coverage for storage formats, migration, memory safety, bulk I/O, and filter equivalence, and document affected filters. Update CMake, test utilities, plugin metadata, and dependencies for the storage-neutral execution model. Verification: The squashed commit tree is checked against the original range tip. No additional build or test run is performed for this history-only rewrite.
joeykleingers
force-pushed
the
worktree-ooc-architecture-rewrite
branch
from
July 15, 2026 17:22
5a8c281 to
b61d3fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch does two things at once:
Reworks the core out-of-core (OOC) data-storage architecture in
simplnxso that disk-backed storage is supplied by a runtime-registered IO manager through clean interfaces, with the core library holding no reference to any concrete OOC format or symbol. Core gains a bulk-I/O data-store API (copyIntoBuffer/copyFromBuffer), an injectable store-format resolver, IO-manager lifecycle hooks, a tri-state storage-mode preference, a memory-budget manager, and memory-safety guards. The OOC implementation itself (chunked HDF5 stores, the chunk cache, deflate decompression) lives in a separate privateSimplnxOocsource set and is not part of this diff — it plugs into core at runtime through these interfaces.Adds chunk-aware (OOC-optimized) algorithm variants for ~50 filters across the SimplnxCore and OrientationAnalysis plugins, plus shared core utilities that benefit many more. Random-access algorithms that thrash the chunk cache on disk-backed arrays now either stream data with bulk I/O or dispatch to a sequential "scanline / CCL" variant, eliminating per-element virtual dispatch and HDF5 chunk churn. In-core behavior and performance are preserved via a runtime storage-type check, so optimized filters keep two code paths: one for in-memory data, one for disk-backed data.
Scope of this diff
develop…HEAD: 378 files, +42,853 / −13,112.src/simplnx(core library)Plugins/SimplnxCorePlugins/OrientationAnalysisHow to review
The diff is large but highly patterned, and the two layers are independent:
src/simplnx/DataStructure/IO/Generic/*,Core/Preferences.*,Utilities/{MemoryBudgetManager,AlgorithmDispatch,SegmentFeatures,UnionFind,SliceBufferedTransfer}.*, andFilter/IFilter.cpp(Part 1).Part 1 — Core out-of-core architecture (
src/simplnx)1.1 Bulk-I/O data-store API
copyIntoBuffer/copyFromBufferonAbstractDataStore<T>(a flat range form and a tuple-addressed form) are the single mechanism for bulk reads/writes, implemented byDataStore<T>andEmptyDataStore<T>; the OOC implementation supplies its own override.IDataStore::StoreType { InMemory, OutOfCore, Empty }, queried viagetStoreType(). The chunk-traversal API (loadChunk,getNumberOfChunks,getChunkLowerBounds,getChunkUpperBounds,getChunkShape) and the chunk-shape-based OOC detection are removed — bulk I/O plusgetStoreType()replace them entirely.EmptyStringStore,AbstractStringStore/StringStoreplaceholder support, andStringArray::isPlaceholder().1.2 Runtime store-format resolution and IO-manager hooks
OOC capability is supplied entirely at runtime; core never names the concrete format.
IDataStoreFormatResolver: a const, thread-safe policy interface deciding which registered format a soon-to-be-created array uses (""= in-memory).InMemoryFormatResolveris the default policy.ArrayCreationUtilities::ResolveStorageFormatis the single decision point for every array-creation call site, with a fixed precedence: the unstructured/poly-geometry gate (ParentGeometrySupportsOoc) forces in-core, then an explicit per-filter format override, then theDataStructure's resolver.DataStructurecarries a per-instance resolver plus a lazily-seeded process-wide default (neither serialized).IDataIOManagerlifecycle hooks (default no-ops):finalizesImport,onImportFinalize,onRecoveryWrite,onFinalizeStores,setBaseDirectory,shutdownManager.DataIOCollectionfans these out to every registered manager, so an OOC manager participates in import finalization, recovery writes, store read-only transition, and shutdown without core knowing the specifics.CoreDataIOManager: the always-present default manager that registers the in-memory data-store/list-store factories; an OOC manager registers on top at runtime.CreateDataStore/CreateListStoresingle entry points;CreateNeighborListAction/CreateNeighborsthread an explicitdataFormatoverride through to the store.1.3 Storage-mode preference (+ legacy migration)
DataStorageMode { Adaptive, ForceInCore, ForceOutOfCore }(persisted asdata_storage_mode) is the single source of truth for storage placement, replacing thelarge_data_format/force_ooc_datapreferences. The enum is deliberately OOC-vocabulary-free — core states user intent; the registered manager maps it to a concrete format.dataStorageMode()migrates older preference files from the retained legacy keys;useOocData()is a convenience view (true unlessForceInCore).Preferencesseeds the OOC base directory and default format on startup via the IO-collection hooks, and gains aremoveValuehelper.1.4 Memory budget + memory safety
MemoryBudgetManager: tracks the budget governing in-core vs OOC placement.defaultBudgetBytes()(50% of RAM, ≥1 GiB, clamped to the cap),maxBudgetBytes()(max(min(total−6 GiB, 0.95·total), 1 GiB)), andsetBudgetBytes()which clamps to the cap and reports whether it clamped. Total-RAM detection is centralized inMemory::GetTotalMemory(). The default is clamped to the cap so it never exceeds it on <12 GiB machines (e.g. CI runners).nxrunner --memory-budget <GB>: CLI flag wired through to the budget manager so the override takes effect in headless runs, with parsing hardened against NaN/inf/trailing garbage.-264total-RAM hard block):-271— non-blocking preflight warning when an in-core array would exceed currently-available RAM. OOC arrays are excluded (EmptyDataStore::memoryUsage()reports 0 for OOC placeholders; the format is resolved in preflight via the sharedResolveStorageFormathelper).-272— astd::bad_allocsafety net at the singleIFilter::execute → executeImplboundary, turning an out-of-memory condition into a clean pipeline error instead of a crash.1.5 HDF5 streaming write + compression
DatasetIOprovides the streaming OOC write path —createEmptyDataset+writeSpanHyperslabfor arrays too large to be resident — alongside the single-shotwriteSpan; reads usereadChunk/readChunkIntoSpan.createEmptyDatasetbuilds the chunked-deflate creation property list (BuildChunkedDeflateDcpl), so a large OOC array taking the two-step streaming write is compressed identically to the single-shot path. Contiguous storage is still used for compression level 0 and for arrays below the small-array threshold.1.6
.dream3dloader APIDream3dIOexposes aLoadDataStructurefamily:LoadDataStructure,LoadDataStructureMetadata(metadata-only for preflight),LoadDataStructureArrays(array subset), plus resolver-aware overloads that stamp a per-DataStructureresolver before import finalization (so a read-only visualization load can direct arrays to disk for fast first-show). Import is eager or deferred based onanyManagerFinalizesImport(); writes run under a recovery-write guard supplied by the registered manager.ImportH5ObjectPathsActionuses this API: metadata-only on preflight, full load on execute, with a selective shortest-path-first merge of only the requested paths.1.7 Core algorithm infrastructure
AlgorithmDispatch.hpp—DispatchAlgorithm<InCoreAlgo, OocAlgo>(arrays, args…), a free function template selecting between an in-core and an OOC algorithm class at runtime. Priority:ForceInCoreAlgorithm()> any array OOC >ForceOocAlgorithm()> in-core default. RAII test guardsForceOocAlgorithmGuard(bool)andForceInCoreAlgorithmGuardlet a single build exercise both paths.SegmentFeaturesOOC path —executeCCL(): Z-slice connected-component labeling with a 2-slice rolling buffer andUnionFindequivalence tracking (Face and FaceEdgeVertex connectivity, optional periodic BCs), replacing random-access BFS/DFS flood-fill on disk-backed data.UnionFind— vector-based disjoint set with union-by-rank and path-halving.SliceBufferedTransfer— type-dispatched Z-slice buffered tuple copy for morphological / neighbor-replacement transfer phases.Extent— region/range math helper (with unit tests).AlignSectionsOOC path — bulk slice read/write transfer for the align-sections family.1.8 Core utility bulk-I/O conversions
These live in core utilities and benefit every caller; each is guarded by a runtime storage-type check that preserves the original in-core code path:
DataArrayUtilities—ImportFromBinaryFile,AppendData,CopyData, and the mirrorswap_rangesops route through chunked bulk I/O when OOC. (PowersReadRawBinaryandAppendImageGeometry's mirror.)DataGroupUtilities::RemoveInactiveObjects— chunked featureIds renumbering viacopyIntoBuffer/copyFromBuffer.ClusteringUtilities::RandomizeFeatureIds— chunked bulk I/O (both overloads; benefits segmentation filters, SharedFeatureFace, MergeTwins).GeometryHelpers—FindElementsContainingVert/FindElementNeighborsuse 65K-element chunked passes with a current-chunk cache-hit check before falling back to per-element reads.ImageRotationUtilities— Z-slab source cache for nearest-neighbor and a ±2-slice trilinear margin, sliding-window slab updates (memmove + delta reads), and intra-slice parallelism. This is howApplyTransformationToGeometryandRotateSampleRefFrameget their OOC speedups (no plugin algorithm file changes for ApplyTransformation).TriangleUtilities— bulk-load triangles/labels for winding repair.H5DataStore— streaming row-batchFillOocDataStorereplacing full-dataset allocation on import.RectGridGeom/ImageGeomfindElementSizes— route through the resolver-awareCreateDataStore(voxel-sizes array can go OOC); the RectGrid inner loop refactored to per-axis precompute + Z-slicecopyFromBuffer.Part 2 — Filter optimizations
Optimization patterns
Every filter optimization is one of these four shapes:
DispatchAlgorithm<…Direct, …Scanline>: an in-coreDirectclass (unchanged or parallel) and an OOCScanlineclass that streams Z-slices / chunks. The originalFoo.cppbecomes a thin dispatcher.DispatchAlgorithm<…BFS, …CCL>(or an in-file branch): random-access flood-fill in-core, sequential Z-slice connected-component labeling for OOC.std::vectors, with a runtime storage-type check so in-core stays optimal.SliceBufferedTransfer, or an OOC-correctness guard (e.g. disabling threading for OOC stores) / progress-and-cancel additions.Algorithm structure: in-core + out-of-core variants
For dispatch-split filters the original algorithm file is renamed to the in-core variant and the OOC variant is added beside it; the original filename remains as a thin dispatcher:
FillBadDataBFSFillBadDataCCLIdentifySampleBFSIdentifySampleCCL…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Direct…Scanline…Worklist…Scanline…Direct…ScanlineNew shared header
IdentifySampleCommon.hpp(SimplnxCore) provides theVectorUnionFindand per-slice functor shared by the BFS/CCL variants;TupleTransfer.hppgainsquickSurfaceTransferBatch/surfaceNetsTransferBatchbulk APIs used by the mesh Scanline variants.SimplnxCore inventory
DispatchAlgorithm<FillBadDataBFS, FillBadDataCCL>; CCL streams Z-slices with coreUnionFindDispatchAlgorithm<IdentifySampleBFS, IdentifySampleCCL>; scanline labeling viaVectorUnionFindSegmentFeatures::executeCCL()(Z-slice CCL)copyIntoBuffer(256K-tuple)canMergefindClusters; bounded per-cluster peak memorynodeIdsfor rolling 2-plane node buffers;quickSurfaceTransferBatchsurfaceNetsTransferBatchstd::vectoraccumulators (no DataStore); 64K-tuple chunked featureIdsassignBadVoxelsvoting; sparse changed-voxel trackingtriMaskbitset + sparsetriPrefixSumpopcount (~6.4× less memory); 65K-element streamed passesZSliceWorkerparallel Z-slice rasterize → mutex-guarded bulkcopyFromBufferSliceBufferedTransferper-Z commit + Z-slice neighbor readsSliceBufferedTransferper-Z commitSliceBufferedTransferper-Z best-neighbor commitfindShiftsOoc()with per-Z-slice mask readscopyIntoBufferfor coordinate/data outputOrientationAnalysis inventory
DispatchAlgorithm<…Direct, …Scanline>; Direct keeps parallel in-core, Scanline 65K-tuple chunks + cached crystal structures; color key forwarded to the OOC pathDispatchAlgorithm<…Worklist, …Scanline>; bool-mask reads routed through bulk I/OSegmentFeatures::executeCCL()slice-by-slice, replacing DFS flood-fillfindShiftsOoc(), 2-slice buffered quats/phases/mask + cached crystal structuresSliceBufferedTransfer; cached crystal structures[plane−kZ, plane+kZ]slab reads; cached crystal structurescopyFromBuffertriIncluded); feature-level caching; raw-pointer parallel selectorfmt::memory_buffercopyFromBufferfor all cell arrays; chunked Euler interleave; in-place phase validationcopyFromBufferin CopyData template, phase copy, Euler interleavecopyFromBufferfrom raw HDF5 reader bufferscopyFromBufferfor intensity/image outputsCancel + progress
In-core and OOC variants gained
m_ShouldCancelchecks at the top of major loops andThrottledMessenger-based progress reporting with per-phase messages and percent-complete.Part 3 — Performance results
All benchmarks use arm64 Release builds. OOC measurements force the disk-backed path through
DataStorageMode::ForceOutOfCoreorForceOocAlgorithmGuard.CRITICAL filter optimizations (existing unit-test data, filter.execute() only)
Small existing datasets do not expose every asymptotic win: mesh bucketing/pruning and bounded-memory streaming can add fixed overhead at this scale. All 17 filters passed their affected suites in both configurations; the table reports measured values rather than hiding those small-test regressions.
Large synthetic filter benchmarks (filter.execute() only)
HIGH and MEDIUM benchmarks use 8,000,000 cells (200³) except the 2D-only tests (2048²), the generated Channel5 reader dataset (8,000 × 1,000), and CSV's 8,000,000 rows. Figures are medians of five execute-only runs for the MEDIUM batches and repeated runs for HIGH where practical;
>values are conservative timeout bounds. Every listed filter passed its normal affected suite and hidden large benchmark in both configurations.Additional 200³ execute-only OOC benchmarks
Full-test wall-clock OOC benchmarks
Geometry / Mesh / Phase Filters
ImageRotationUtilitiesslab cachePart 4 — Test infrastructure
UnitTestCommon.hpp):CompareDataArrays,CompareDataArraysByComponent,CompareArrays,CompareFloatArraysWithNansstream both arrays in 40K-element chunks viacopyIntoBufferinstead of per-elementoperator[](per-element access on OOC arrays is pathologically slow).ExpectedStoreType()(derives the expectedStoreTypefrom the activeDataStorageMode+ whether an OOC manager is registered) andRequireExpectedStoreType().PreferencesSentineltakes aDataStorageModeand restores the original preferences on destruction.TestFileSentinelreference-counts archive extraction via per-process holder files, so parallel test runs don't delete shared decompressed data prematurely.LoadDataStructuretest helper callsDREAM3D::LoadDataStructure(path)directly.SegmentFeaturesTestUtils.hpp(~638 lines): shared builders/verifiers for the SegmentFeatures family.ForceOocAlgorithmGuard+GENERATE(from_range(k_ForceOocTestValues))runs each case in both in-core and forced-OOC modes — adopted by 23 plugin test files (16 SimplnxCore, 7 OrientationAnalysis).test/:DataStoreFormatResolverTest,DataStorageModeMigrationTest,Dream3dLoadingApiTest,EmptyStringStoreTest,ExtentTest,MemoryBudgetManagerTest,MemorySafetyTest,IParallelAlgorithmTest.RotateSampleRefFrame/RotateEulerRefFrametest paths use slab/chunked bulk I/O.Part 5 — Build system
cmake/SimplnxConfig.hpp.inis a generated PUBLIC, ODR-safe config header.SIMPLNX_TEST_ALGORITHM_PATHcache option (default0):0=Both,1=OOC-only,2=InCore-only; plumbed into every plugin test as a compile definition (cmake/Plugin.cmake). An in-core build can validate both algorithm paths (forcing the Scanline path against in-core data still runs correctly and fast).SIMPLNX_UNIT_TEST_TARGETSglobal property —create_simplnx_plugin_unit_testrecords each test target so consumer (add_subdirectory) builds can attach extra sources/settings.zlibas a direct vcpkg dependency — the OOC layer compiled into consumers uses it directly for parallel deflate-chunk decompression off the global HDF5 mutex (andUnitTestCommonuses it for the in-house tar.gz extractor)./bigobjfor the MSVC build ofDream3dIO.cpp(exceeds the COMDAT limit / C1128 in Debug).Extent,EmptyStringStore,IDataStoreFormatResolver,InMemoryFormatResolver,MemoryBudgetManager,AlgorithmDispatch,SliceBufferedTransfer,UnionFind,IdentifySampleCommon).Part 6 — Documentation
src/Plugins/*/docs/(27 SimplnxCore, 21 OrientationAnalysis; ~+1,177 lines), each adding an## Algorithmsection with### Performanceand paired In-Core / Out-of-Core subsections that explain the dual implementation, memory footprint, and chunk/slab streaming strategy. No docs deleted.Part 7 — Test data archives
Three
download_test_data()entries added:fill_bad_data_exemplars.tar.gz(SimplnxCore)identify_sample_exemplars.tar.gz(SimplnxCore)segment_features_exemplars.tar.gz(referenced by both SimplnxCore and OrientationAnalysis)No existing archive entries were removed.
Related PR
Test Plan
SIMPLNX_TEST_ALGORITHM_PATH=2) passesSIMPLNX_TEST_ALGORITHM_PATH=1) passesSIMPLNX_TEST_ALGORITHM_PATH=0) pass