Repository navigation
Fix/bug hunt 2026 11 - #96
Merged
Merged
Conversation
GpuPatchAugmentation downsampled with F.interpolate(mode='nearest'), which takes the first voxel of each block, so after upsampling the image sat (factor - 1) / 2 voxels away from the unchanged mask (0.5 to 1.5 voxels for factors 2 to 4). 'nearest-exact' samples the block centres: the shift is 0 for odd factors and 0.5 voxel for even ones, as in TorchIO's RandomAnisotropy. Models trained with anisotropy (the VS config uses p=0.25) saw this as label noise on about a quarter of the patches; no retraining is needed for it alone.
fastMONAI points MLflow at its own store only on its first MLflow call. Code that called raw mlflow first in a fresh process searched <cwd>/mlflow.db instead, so `prepare_model_bundle.py --run` failed with "Run ... not found" and tutorial 11f with "No MLflow experiment found", although training had logged the runs. Both now set fastMONAI's store first unless MLFLOW_TRACKING_URI is set. The commented MLflow option in 12c uses find_model_artifacts, which does the same.
fastai skips after_fit when a fit raises or is interrupted (Ctrl-C), so the run the callback started stays active. Only the same callback instance cleaned it up at its next fit. A new callback, as when a notebook cell is run again, raised "Run ... already active" with auto_start=True and silently joined the stale run with auto_start=False. Runs the callback starts are now recorded until after_fit or mark_failed ends them, and the next fit of any ModelTrackingCallback ends such a run as FAILED before starting its own. A caller's own run is still joined.
Predictions were resized to the original shape with tio.Resize, then reoriented and given the original affine. Resize only matches the shape: on many grids its output came out one slice short and was padded at one end, so the mask sat half a slice off, e.g. 0.6 mm on 120 slices of 1.2 mm, and grids that do not divide evenly were slightly stretched. The VS training grids happened to be unaffected, but clinical grids (1.2 mm x 120/160 slices, 0.6 and 0.8 mm) are not. inference, inference_from_config and PatchInferenceEngine now resample the prediction with tio.Resample onto the original image's own shape and affine, which also undoes the reordering. The prediction's edge voxels are repeated once first, so original edge slices beyond the last model voxel keep the edge value instead of 0. Shape-based label restore and probability maps use the same path. inference and inference_from_config no longer need org_size to use caller-supplied images; the argument is still accepted and ignored. Deployed VS masks and vote maps move by up to half a slice on affected grids once the PACS container is rebuilt with this version.
The training contract hashed the whole preprocessing manifest, which records absolute source paths and file times, the preprocessing worker count, which defaults to the CPU count, and the loss factory's module path, which is workflow.models in notebook 01 and vestibular_schwannoma.workflow.models in train_5fold.py. Folds from two machines, or from the notebook and the script, never merged. The contract now keeps the preprocessing settings and the label fingerprint from the manifest, drops preprocess_workers and records the loss factory by name. The workflow revision is now 2, so folds trained before this change do not merge with folds trained after it. The README says when folds from different machines merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.