Skip to content

Add validated illumination correction worker - #172

Closed
arjunrajlab wants to merge 2 commits into
masterfrom
codex/findings-based-illumination-correction
Closed

Add validated illumination correction worker#172
arjunrajlab wants to merge 2 commits into
masterfrom
codex/findings-based-illumination-correction

Conversation

@arjunrajlab

@arjunrajlab arjunrajlab commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a CPU image-processing worker for grid-locked uneven illumination in stitched microscopy TIFFs
  • infer one physical stitched-tile grid and either choose the best reference channel automatically or use a validated manual reference channel
  • fit channel-specific BaSiC (darkfield off/on), folded log-gradient, and split-half affine models
  • make identity/no correction a first-class automatic candidate and select models on independent held-out Z planes
  • conservatively keep single-Z automatic runs unchanged, with a warning; manual algorithms remain available
  • apply fitted models across Z only at the reference XY/time acquisition, preserving unrelated fields, time points, and unselected channels
  • use Pareto/uncertainty-aware artifact ranking, opt-in punctate-channel spot scoring, and explicit preservation/numeric guardrails
  • reject non-finite output and material preserve-dtype clipping before upload
  • write corrected multi-frame TIFF output with explicit indexing, scope, settings, model diagnostics, and software-version metadata
  • register production/test Compose services and add worker documentation plus registry coverage

Review fixes

A five-pass scientific/software review found and fixed same-plane model-selection leakage, the missing identity comparator, cross-acquisition model transfer, unsafe default per-tile gains, unscoped spot scoring, incomplete numeric guardrails, pre-cast-only validation, stale-setting edge cases, dependency gaps, and documentation/provenance drift.

Validation

  • ./build_workers.sh --build-and-run-tests illumination_correction — 30 passed (including real BaSiC fits with darkfield off and on)
  • docker compose config --quiet
  • python3 -m py_compile for worker and test modules
  • built image labels verified: isGPUWorker=false, workerVersion=1.1.0
  • pip check reports only BaSiCPy's intentionally overridden stale scipy<1.13 metadata pin; real fits pass against the worker base's newer SciPy stack

Context

This is a fresh findings-based stitched-TIFF fallback and does not modify the older approach in #151. The documentation notes that the study's preferred workflow uses raw overlapping acquisition tiles when those data are available.

@arjunrajlab
arjunrajlab marked this pull request as ready for review August 30, 2026 12:24
@arjunrajlab

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-30T12:29:02.923808Z 155a53a Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 155a53a525

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1186 to +1187
"P2_spearman": float(spearmanr(raw_sum[keep], corrected_sum[keep]).statistic),
"P2_applicable": True,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Mark undefined rank correlation as inapplicable

When at least ten detected objects have identical integrated intensities, spearmanr returns NaN even for the identity candidate because the rank vector is constant. This code nevertheless sets P2_applicable to true, so check_guardrails rejects the unchanged image and can report that every candidate is unsafe. Detect constant raw_sum or corrected_sum vectors and mark this metric inapplicable, as is already done when too few objects are measurable.

Useful? React with 👍 / 👎.

@arjunrajlab

Copy link
Copy Markdown
Collaborator Author

Superseded by PR 176: #176

@arjunrajlab arjunrajlab closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant