Skip to content

L5: per-backend processors written once against arraybridge ArrayOperations - #1191

Merged
trissim merged 5 commits into
mainfrom
refactor/L5-arraybridge-operations
Oct 11, 2026
Merged

trissim merged 5 commits into
mainfrom
refactor/L5-arraybridge-operations

Conversation

@trissim

@trissim trissim commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator

Surface L5 (replaces C4). Receipt with the census and parity table: docs/refactor/consolidation-20261009/L5-arraybridge-operations.md.

Library PR: OpenHCSDev/ArrayBridge#10 (0.4.0, CI 14/14 green, not merged or tagged). The submodule points at its branch head 6f6f907; the pin is arraybridge>=0.4.0,<0.5. Merge this after the library is released.

What changed

  • arraybridge: ArrayOperations is a registered family (one leaf per memory type); ArrayOperations.for_memory(MemoryType.X) returns the leaf. Leaves hold only the primitives that differ (max/min/mean/sum, astype, prepend_axis, linspace, ones, assign, outer, floor, accepts). The string *_OPERATIONS roster is deleted. The TensorFlow leaf now names dtypes through its NumPy dtype, which fixes "Cannot interpret 'tf.uint16'" on every decorated TensorFlow call.
  • processors/image_stack_operations.py writes the shared operations once: validation, linear/weight masks, apply_mask, max/mean/min projection + one StackProjection family, spatial binning + one SpatialBin family, crop, equalization and percentile normalization (NumPy/CuPy), composite weights and the weighted composite.
  • Each <backend>_processor.py keeps its registered names, signatures and contracts. Each name delegates to the shared operation, except where the algorithm genuinely differs (census ≠ cells: sharpen, tophat, and per-backend normalization, equalization and composites).
  • StackProjectionMethod gains MIN. NumpyStackProjectionMethod is kept because saved pipelines import it (decision default).

Deletions and certificates

  • derived (by image_stack_operations + ArrayOperations.for_memory; covered by per-backend parity): per-backend copies of create_linear_weight_mask (numpy, cupy, jax, torch, tf), create_weight_mask (all 6), apply_mask (5), max_projection (6), mean_projection (5), create_projection + 6 StackProjection strategy families, 2 SpatialBin families + spatial_bin_2d/3d and crop (numpy, cupy), equalization (numpy, cupy), percentile_utils.py, composite weight validation (numpy, cupy, jax, cle), the weighted composite (cupy, jax), 6 _validate_3d_array, and PyclesperantoMaskDimensionality. arraybridge's *_OPERATIONS singletons and name field: derived from the registry.
  • derived (by torch.iinfo and TensorFlow DType.max): the hand-written integer-maximum tables in torch create_composite and stack_equalize_histogram, and in TensorFlow stack_equalize_histogram.
  • unused: cupy create_composite logging; numpy min_projection (an unregistered public helper with no reader in the repo; create_projection(MIN) serves it).
  • tests-only (parity did not exercise these):
    • dtypes int8/int16/int32/uint32/float64 on any backend; parity used uint16, uint8, float32 and constant stacks.
    • JAX on a GPU (the desktop's JAX CUDA plugin sees no GPU; JAX ran on the CPU).
    • The torch decorated path: it is nondeterministic run to run on main itself. The undecorated torch bodies are exact.
    • The TensorFlow create_composite integer clamp: it takes a list of stacks, and parity passed stacks, so it raised TypeError both before and after.
    • The three OPENHCS_SUBPROCESS_NO_GPU branches (the registry already skips modules of disallowed memory types).
    • _check_pyclesperanto_available.
    • The TensorFlow ≥2.12 version probe (TensorFlow 2.8–2.11 was not exercised).

Verification

  • Per-backend numeric parity: every public declaration, decorated and undecorated, before (b987b7b) and after. numpy 236/236 and jax-CPU 140/140 (here and on the desktop); cupy 282/282, pyclesperanto 188/188, torch 70/70 undecorated (desktop RTX 2070 SUPER). All exact (bitwise, same dtype); no tolerance was needed. TensorFlow, undecorated: 62 same and 8 error→value (create_projection and create_weight_mask failed on main through the broken nested decorator). Decorated TensorFlow now equals undecorated. See the receipt table.
  • Processing/backend unit tests: 104 files, each in its own process. 80 pass; the 24 that fail on the branch fail on b987b7b with identical per-file counts (environment: missing caches/references, Qt).
  • arraybridge suite: 247 passed, 17 skipped.
  • test_main disk + direct: 3/3 viewer-less cases pass. The 9 napari/fiji cases fail with "no Qt platform plugin" and fail identically on b987b7b.
  • official30 parity: 1 passed (30/30), 741 s.

L5 · PRs OpenHCSDev/ArrayBridge#10 + this · guards 4/4 · production −1913 +692 · tests −199 +184

🤖 Generated with Claude Code

trissim and others added 3 commits October 10, 2026 21:21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… ArrayOperations

image_stack_operations holds every operation that two or more backends
implemented identically modulo the array library: stack validation, linear
weight masks, mask application, max/mean/min projection with one
StackProjection family, spatial binning with one SpatialBin family, crop,
histogram equalization and percentile normalization (NumPy/CuPy), composite
weight validation and the weighted composite. Each backend module registers
them with its ArrayOperations.for_memory leaf, keeping registered identities,
signatures and contracts; it keeps its own body only where the algorithm
genuinely differs (sharpen, tophat, torch/JAX/TensorFlow/pyclesperanto
normalization and equalization, numba/torch/TensorFlow/pyclesperanto
composites, pyclesperanto mask, linear mask and float32 mean).

Deleted: percentile_utils.py, six StackProjection and two SpatialBin strategy
families, six _validate_3d_array copies, the OPENHCS_SUBPROCESS_NO_GPU
branches, _check_pyclesperanto_available, the TensorFlow version probe,
PyclesperantoMaskDimensionality, the unregistered numpy min_projection, and
the hand-written torch/TensorFlow integer dtype maxima. StackProjectionMethod
gains MIN. arraybridge pinned >=0.4.0,<0.5; submodule at the PR branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 11, 2026 02:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

trissim and others added 2 commits October 10, 2026 22:49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trissim
trissim merged commit b34b049 into main Oct 11, 2026
6 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants