This repository contains the training, evaluation, and uncertainty estimation stack we use to study Neural Cellular Automata (NCA) based segmentation on a diverse set of biomedical datasets. In addition to classical nuclei benchmarks (DSB2018, MoNuSeg) we support augmented “hard” splits, real/synthetic ultrasound (RUS), hand-curated NuInsSeg tissues, and ISIC 2017 Task 1 lesions with multi-annotator labels used to build disagreement-aware baselines.
- End-to-end NCA training / evaluation via
train.pyandevaluate.pywith configurable steps, channel budgets, image sizes, and datasets. - Optional dropout and MC-dropout baseline in the shared NCA update network. Dropout is disabled by default, enabled at training time with
--dropout_rate, and sampled at test time with themc_dropoutuncertainty method. - Unified dataloader (
dataloader.py) for VOC, CamVid, DSB2018, MoNuSeg (with automatic XML rasterization + augmentation), NuInsSeg (with automatic split generation + augmentation), RUS ultrasound, and ISIC 2017 Task 1 (binary lesions with multi-annotator metadata). - Uncertainty tooling:
quality_labels.pycomputes Dice / boundary scores per sample and records annotator disagreement if available.compute_uncertainty.pyimplements single forward entropy, MC dropout, stop-time, stability, flicker, resilience, test-time augmentation (TTA), and the ISIC-specific disagreement baseline derived from annotator variance.evaluate_uncertainty.pyreports Dice@80/90, AURC, AUROC, AUPRC, and adds two fusion baselines (rank-average and validation-tuned weighted fusion) to combine the best signals.plot_uncertainty_examples.pycreates qualitative panels.
- Augmented “hard” splits for DSB2018 (photometric + geometric noise), MoNuSeg, and NuInsSeg (
*_augsplits are generated on-the-fly if missing).
python -m venv nca_env
source nca_env/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
# Ensure the matching CUDA builds of torch/torchvision if training on GPU.| Dataset | Expected Location | Notes |
|---|---|---|
| VOC 2012 | datasets/voc/VOCdevkit/VOC2012 or datasets/VOC2012_train_val |
Standard PASCAL VOC segmentation |
| CamVid | datasets/CamVid |
Requires train/val/test and label folders |
| DSB2018 | datasets/dsb2018 |
Contains stage1_train; _aug splits auto-generated |
| MoNuSeg | datasets/MoNuSeg |
Run python datasets/MoNuSeg/split_monuseg.py first |
| RUS Ultrasound | datasets/US/RUS/{images,annotations}/{train,val,test} |
Provide masks for training/validation splits |
| NuInsSeg | datasets/NuInsSeg |
Run python datasets/NuInsSeg/split_nuinsseg.py --root datasets/NuInsSeg --force once; _aug splits are generated on demand |
| ISIC 2017 Task 1 | datasets/isic/isic2017_task1 |
Keep the official ISIC-2017_<split>_Data and *_Part1_GroundTruth folders. Additional annotator masks (if present) are automatically discovered for disagreement baselines. |
All scripts accept --data_root to override the automatically detected locations when needed.
Train on any supported dataset:
# Example: NuInsSeg baseline
python train.py \
--dataset nuinsseg \
--data_root datasets/NuInsSeg \
--batch_size 4 \
--epochs 50 \
--lr 1e-3 \
--steps_min 32 --steps_max 64 \
--image_size 512 512 \
--num_workers 4 \
--device cuda \
--exp_name nuinsseg_baselineTrain the otherwise identical dropout baseline by passing a non-zero probability:
python train.py \
--dataset nuinsseg \
--data_root datasets/NuInsSeg \
--dropout_rate 0.1 \
--exp_name nuinsseg_mc_dropout--dropout_rate (also available as --dropout) defaults to 0.0. The value is
saved in the checkpoint and is restored automatically by evaluation and uncertainty
scripts. Ordinary validation/evaluation disables dropout, as expected.
Evaluate a checkpoint on a target split:
python evaluate.py \
--checkpoint runs/nuinsseg_baseline/best.pt \
--dataset nuinsseg \
--split test \
--data_root datasets/NuInsSeg \
--batch_size 2 \
--num_workers 4 \
--image_size 512 512 \
--device cudaevaluate.py reports pixel accuracy, mIoU, Dice, and boundary F1.
-
Quality labels (Dice, boundary F1, annotator stats):
python quality_labels.py \ --datasets dsb2018 monuseg rus nuinsseg isic2017 \ --split test \ --runs_dir runs \ --pattern "*best.pt" \ --batch_size 2 \ --num_workers 4 \ --device cuda
-
Uncertainty computation (per-method maps + scalars). Available methods:
single,mc_dropout,stoptime,stability,flicker,resilience,tta,disagreement.python compute_uncertainty.py \ --datasets dsb2018 nuinsseg isic2017 \ --split test test_aug \ --methods single stoptime stability flicker resilience tta disagreement \ --runs_dir runs \ --batch_size 2 \ --num_workers 4 \ --device cuda \ --save_pngFor a checkpoint trained with non-zero
--dropout_rate, run the MC-dropout baseline with:python compute_uncertainty.py \ --datasets nuinsseg \ --split test \ --methods mc_dropout \ --mc_dropout_samples 20 \ --runs_dir runs \ --device cudaMC dropout keeps the rest of the model in evaluation mode, activates only the update-network dropout layer, and uses predictive mutual information as its primary uncertainty map/score. Predictive entropy and variance maps are saved as well. Checkpoints trained with the default
--dropout_rate 0are skipped for this method.--tta_max_transformscontrols how many geometric transforms are used.disagreementrequires multi-annotator metadata (ISIC 2017); it falls back to zeros otherwise.
-
Evaluation + fusion using
evaluate_uncertainty.py. This script loads the quality/uncertainty JSON pairs per run and computes Dice@80/90, AURC, AUROC, and AUPRC. Optional fusion baselines:python evaluate_uncertainty.py \ --datasets dsb2018 nuinsseg isic2017 \ --split test \ --methods single tta stability disagreement \ --fusion_pairs tta,stability tta,single \ --fusion_metric aurc \ --fusion_alpha_steps 21 \ --fusion_val_split val \ --runs_dir runs \ --output runs/unc_summary_test.jsonfusion_rank_<m1>_<m2>: average normalized ranks of each method.fusion_weighted_<m1>_<m2>: learnαon the validation split (grid search) after standardizing each uncertainty score.
-
Qualitative panels:
python plot_uncertainty_examples.py \ --datasets nuinsseg isic2017 \ --split test \ --methods single tta disagreement \ --runs_dir runs \ --output runs/uncertainty_plots
dataloader.ISIC2017Datasetstores multi-annotator masks (when available). Downstream scripts exposemulti_mask_pathsmetadata.quality_labels.pylogs per-sample annotator variance, maximum disagreement, and pairwise Dice statistics.- The
disagreementuncertainty method computes pixel-wise variance and the average mask, serving as an upper bound for how much label ambiguity exists per image.
- DSB2018: requesting
split=test_augortrain_augtriggers_ensure_dsb_augmented_split, which applies geometric and photometric noise to each case. - MoNuSeg:
split=test_auguses_ensure_monuseg_augmented_splitto rasterize XML annotations, apply augmentations, and cache the results. - NuInsSeg:
split=test_augcalls_ensure_nuinsseg_augmented_splitto create noisy counterparts of each split image/mask.
Use --split test for in-distribution evaluation and --split test_aug to stress the models with synthetic perturbations.
The repository includes SLURM helpers such as run_dsb2018.sbatch, run_monuseg.sbatch, and run_isic.sbatch. Update these scripts with your environment-specific parameters (partition, time, etc.) before submitting them to a cluster.
├── NCA.py # Backbone NCA implementation
├── dataloader.py # Dataset builders and augmenters
├── train.py / evaluate.py # Segmentation training and evaluation
├── quality_labels.py # Dice/boundary tracking + annotator stats
├── compute_uncertainty.py # Uncertainty map generation (multiple methods)
├── evaluate_uncertainty.py # Risk/coverage, ROC/AUC, fusion baselines
├── plot_uncertainty_examples.py # Qualitative grids
├── datasets/
│ ├── NuInsSeg/split_nuinsseg.py
│ ├── MoNuSeg/split_monuseg.py
│ └── ...
└── runs/ # Default output directory for checkpoints + metrics
- Missing dataset roots: all loaders call
_ensure_existsand raise descriptive errors. Use--data_rootto point to the correct location or create symlinks underdatasets/. - RUS annotations: ensure
datasets/US/RUS/annotations/{train,val,test}exist. Training will fail otherwise. - NuInsSeg splits: run
datasets/NuInsSeg/split_nuinsseg.pyonce before training; re-run with--forceto rebuild. - Torch install: match the CUDA version on your system. For cluster jobs, load the corresponding module before
pip install.
This repository is part of an ongoing NCA-based uncertainty study. Please contact the maintainers for up-to-date citation and licensing information before redistributing or publishing results derived from this codebase.
Both compute_uncertainty.py and ablation_sweep.py accept:
| Option | Meaning |
|---|---|
| --resilience_mode single | One injection, followed by one recovery stage (default). |
| --resilience_mode double | Two sequential injections, each followed by a recovery stage. |
| --resilience_mode both | Save single and double results from the same original state and first recovery. |
| --no_perturb_rgb | Add noise only to non-input channels; keep original RGB fixed (default). |
| --perturb_rgb | Add noise to all channels, including RGB/input channels. |
The original prediction is always obtained with the original RGB held fixed. During each recovery stage the input channels are clamped to that stage's input: original RGB for latent-only noise, or noisy RGB when RGB noise is enabled. In double RGB mode, the second fresh noise is added to the already recovered state, including its previously perturbed RGB. No clipping is applied to Gaussian noise. This keeps the RGB perturbation effective instead of immediately restoring the original image, and avoids unconstrained RGB drift from learned updates.
For initial rollout length T and recovery length T':
- Single: inject at T, recover T' updates, evaluate at T + T'.
- Double: inject at T, recover T' updates, inject again at T + T', recover another T' updates, evaluate at T + 2T'.
- Each injection uses independent Gaussian noise with the same standard deviation.
- Both endpoints are compared to the original mask at T using 1 - IoU.
- --resilience_relax_steps (computation) and each --t_primes value (ablation) mean recovery steps per injection, not the total recovery budget.
Compute paired single/double results without RGB noise:
python compute_uncertainty.py --datasets dsb2018 --split test --methods resilience --resilience_mode both --no_perturb_rgb --resilience_noise 0.02 --resilience_relax_steps 12 --seed 42Compute the corresponding paired results with RGB noise:
python compute_uncertainty.py --datasets dsb2018 --split test --methods resilience --resilience_mode both --perturb_rgb --resilience_noise 0.02 --resilience_relax_steps 12 --seed 42Results use distinct method IDs:
| Mode | Without RGB noise | With RGB noise |
|---|---|---|
| Single | resilience_single_latent | resilience_single_rgb |
| Double | resilience_double_latent | resilience_double_rgb |
The directory and JSON filename follow the existing convention, for example runs//uncertainty_dsb2018_test_resilience_double_latent/uncertainty_dsb2018_test_resilience_double_latent.json. The old ..._resilience outputs are left in place. Repeating the same configuration overwrites its own output; use separate run folders to retain different seeds or noise/recovery settings.
Evaluate all four variants together after computing both RGB settings:
python evaluate_uncertainty.py --datasets dsb2018 --split test --methods resilience_single_latent resilience_double_latent resilience_single_rgb resilience_double_rgb --output runs/unc_summary_resilience_comparison_test.jsonNew resilience records include Dice and boundary F1 measured on the exact original mask being probed. Evaluation uses these values even if a separate quality file contains a different stochastic prediction. New resilience outputs can therefore be evaluated without first generating quality_*.json. Other uncertainty methods continue to require their quality-label files.
pred_mask and prob_map describe the original prediction. recovered_pred_mask and recovered_prob_map describe the selected endpoint. unc_map is the pixel-wise mask disagreement; the ranking score remains the scalar 1 - IoU, stored in the existing uncertainty-score fields. Saved frames and labels identify both injections and their recovery stages. JSON metadata records the selected mode, RGB choice, seed, noise magnitude, injection times and final evaluation time.
Run the paired ablation grid:
python ablation_sweep.py --checkpoint runs/dsb2018_baseline/best.pt --dataset dsb2018 --resilience_mode both --no_perturb_rgb --sigmas 0.02 --t_primes 12 --seed 42 --output runs/ablation_resilience_latent.jsonRepeat with --perturb_rgb and a different output filename for the RGB comparison. Grid entries include both their method ID and parameters, and contain per-image scores and original Dice. Every sample in a batch is scored. AUROC is recorded as null if the evaluation set contains only one quality class.
Baseline, injection and recovery randomness are isolated. With the same checkpoint, dataset order, batch size, image size, rollout settings and seed, standalone single/double runs match their endpoints in both mode. Additional recovery steps do not change the random sequence used for later images. Keep these settings identical across comparisons. The same seed also pairs the latent noise between RGB settings. Fire masks are reproducible, but GPU reproducibility across hardware or different PyTorch builds is not guaranteed.
The corrected rollout does not reproduce the old compute_uncertainty.py resilience branch exactly: that branch perturbed all channels and allowed RGB to evolve without clamping, including during the initial rollout. Existing checkpoints remain compatible; no retraining is needed. Recompute inference results for the new comparisons.
For a journal comparison, additionally run single perturbation with twice the recovery length to control for the extra update budget in double mode. Plotting/evaluation scripts accept the saved method IDs via --methods; the visualization tool creates separate GIFs with injection labels for each variant.
Run the focused regression tests with:
python -m unittest discover -s tests -p "test_resilience.py" -vUse plot_calibration.py after computing per-image uncertainty results for both validation and test. The script fits a separate mapping for each dataset, run and saved method ID; it does not pool different checkpoints or change NCA weights.
Failure is defined as original-prediction Dice < --failure_threshold. The default threshold is 0.8, matching ablation_sweep.py's default. The existing evaluate_uncertainty.py instead labels the lowest-Dice quantile as failures, so its AUROC can differ. This script recomputes AUROC using the same fixed failure definition as the calibration plot. Choose the threshold before inspecting test results.
First generate paired validation and test results using identical inference settings (including noise magnitude, recovery length and RGB choice):
python compute_uncertainty.py --datasets dsb2018 --split val --methods resilience --resilience_mode both --no_perturb_rgb --resilience_noise 0.02 --resilience_relax_steps 12
python compute_uncertainty.py --datasets dsb2018 --split test --methods resilience --resilience_mode both --no_perturb_rgb --resilience_noise 0.02 --resilience_relax_steps 12Plot the single/double comparison with the logistic baseline:
python plot_calibration.py --datasets dsb2018 --methods resilience_single_latent resilience_double_latent --calibration_split val --split test --failure_threshold 0.8 --n_bins 10 --calibrator logistic --output runs/calibrationBoth --dataset and --datasets accept one or more dataset names. --runs_dir uses the existing runs/* convention; --run_name restricts analysis to one run. Use the corresponding *rgb method IDs after generating RGB-noise results. Legacy methods such as resilience and single are also supported when their matching quality.json contains original-prediction Dice.
New resilience files already contain exact baseline Dice. That value takes precedence over separate quality-label files. The script refuses duplicate IDs, missing Dice, invalid values, detected validation/test sample or group overlap, and conflicting saved protocol settings. Missing result pairs are skipped. Legacy files may lack enough metadata to verify the full protocol; the output records which checks could not be made. Split-local numeric indices alone cannot prove sample independence.
The default calibrator learns p(failure | u) = sigmoid(a*u + b) on validation. Logistic regression uses standardized scores and fixed L2 regularization C=1; --logistic_c changes this value. The saved coefficients a and b apply directly to the original uncertainty scale. No class balancing is applied, since failure prevalence must be retained. Both failure classes must occur in the calibration set; the script refuses to fit if validation has only successes or only failures.
Use --calibrator auto to compare logistic and increasing isotonic regression by cross-validation within validation, then refit the selected mapping on all validation images. Selection uses --selection_metric brier (default) or log_loss. By default isotonic is considered only with at least 100 validation images and 10 examples of each failure class; these are configurable practical thresholds, not guarantees against overfitting. Insufficient data or unusable group folds fall back to logistic. --calibrator isotonic explicitly fits isotonic without automatic selection.
Calibration plots use 10 quantile bins by default; choose --n_bins 5 for smaller test sets, or --strategy uniform for equal-width probability bins. Binning follows scikit-learn calibration_curve. Tied probabilities stay together, and empty bins are omitted, so fewer than the requested number of plotted points can occur. Labels show failures/image count. The horizontal axis is mean predicted failure probability; the vertical axis is observed failure fraction. A second panel shows the test probability histogram.
Each run produces:
- runs/calibration//__calibration.png
- runs/calibration//__calibration.pdf
- runs/calibration//calibration_.json for every method
The JSON contains fitted mapping parameters, validation CV scores, test Brier score/log loss/ECE, raw-uncertainty AUROC, calibrated-probability AUROC, per-bin counts/failures/confidence intervals, validation u/Dice/failure records, and test u/Dice/failure/predicted-probability records. AUROC is null if the test set has only one outcome class. ECE depends on binning and is descriptive; it is not used to select a mapping on test. Brier and log loss assess overall probability quality as well as calibration, as explained in the scikit-learn calibration guide.
For related images, supply --group_key patient_id (or slide_id/video_id) if that field exists in saved records. Alternatively, --group_file groups.json reads:
{
"dsb2018": {
"val": {"case_val_001": "patient_A", "case_val_002": "patient_A"},
"test": {"case_test_001": "patient_B", "case_test_002": "patient_B"}
}
}The mapping must cover every analyzed image. The script checks group independence across the supplied splits; it does not create or repair dataset splits. With groups, validation CV keeps whole groups together and confidence intervals use a cluster bootstrap (--bootstrap_samples 1000, --seed 42). Without groups, 95% Wilson intervals assume independent images. Bins containing fewer than two independent groups have null interval bounds rather than misleading precision. These intervals describe test-sampling uncertainty conditional on the fitted mapping and fixed bin boundaries; they do not include uncertainty from refitting the calibration mapping.
Run the statistical and file-format regression tests with:
python -m unittest discover -s tests -p "test_calibration.py" -v