From 6800d8e5960dc8ef8d58ed8b37e9c5dfed2c9f61 Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Mon, 9 Nov 2026 09:22:48 +0100 Subject: [PATCH 1/8] docs(vs-pacs): point the ror entry at the latest tag The integration team switched the PACS entry to fastmonai-vsseg:latest, so a new release needs no config change. --- research/vestibular_schwannoma/deployment/pacs/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/research/vestibular_schwannoma/deployment/pacs/README.md b/research/vestibular_schwannoma/deployment/pacs/README.md index 1b7b5b1..0acaac1 100644 --- a/research/vestibular_schwannoma/deployment/pacs/README.md +++ b/research/vestibular_schwannoma/deployment/pacs/README.md @@ -76,10 +76,14 @@ in the PACS `config.json` and fill in the series selector: "ENTRY_NAME": { "select": "", "ROR_CONT_OPTIONS": "{}", - "docker_image": "fastmonai-vsseg:" + "docker_image": "fastmonai-vsseg:latest" }, ``` +The entry uses `latest`, so a new release needs no config change. The release archive +carries both tags, and loading it moves `latest` to the new image. The dated tag shows +which release `latest` is. + `"{}"` runs the defaults. Use `"{\"tta\":false}"` to turn TTA off. Don't write `"\"\""` here: that passes two quote characters and is rejected. From 13306482ea1879f4f652db9fa3935f335ea26024 Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Mon, 9 Nov 2026 09:37:06 +0100 Subject: [PATCH 2/8] feat(vs-pacs)!: store each predicted region in its own redcap instance Only the largest region reached REDCap, so a second tumour region (for example participant 300) was lost. Each bundle now owns a block of 100 pr2mask repeat instances and region k goes to base + k, so DynUNet fills 201, 202, ... in pr2mask's "label k of N" order. An empty mask fills 201 with blank columns to record the run. Every instance also gets the new model_version field on the REDCap form, e.g. "dynunet 5-fold, fastmonai-vsseg:". pr2mask has no such field, so the exporter writes it and adds it to the data dictionary. BREAKING CHANGE: DynUNet results move from instance 11 to 201+. The registry now holds block bases (multiples of 100); old results at 11 are removed on the REDCap side. --- .../deployment/pacs/README.md | 100 +++++++------ .../deployment/pacs/pacs_inference.py | 19 ++- .../pacs/redcap_model_instances.json | 2 +- .../deployment/pacs/redcap_output.py | 105 ++++++++------ .../tests/deployment/test_pacs_inference.py | 7 +- .../tests/deployment/test_redcap_output.py | 131 +++++++++++++----- 6 files changed, 240 insertions(+), 124 deletions(-) diff --git a/research/vestibular_schwannoma/deployment/pacs/README.md b/research/vestibular_schwannoma/deployment/pacs/README.md index 0acaac1..9c57e4e 100644 --- a/research/vestibular_schwannoma/deployment/pacs/README.md +++ b/research/vestibular_schwannoma/deployment/pacs/README.md @@ -115,55 +115,63 @@ to prevent overwrites. The final output includes pr2mask's `redcap//output.json` and `output_data_dictionary.zip`, in pr2mask's EAV row format and instrument. `redcap` is an owned output directory and must not already exist. There is no direct REDCap uploader. -The exporter sends only pr2mask's own columns, in the model's permanent `pr2mask` repeat -instance. An empty measurement list is routed with the source PatientID and -ReferringPhysicianName (`EventName:` prefix removed), matching pr2mask routing. The mask -itself goes to the PACS as the `labels` series. + +pr2mask splits the mask into connected regions (26-connected) and numbers them 1, 2, 3, ... +in scan order. The exporter writes each region to its own repeat instance of the `pr2mask` +instrument: region k goes to the model's block base + k. DynUNet's base is 200, so a mask +with one region fills instance 201, and a mask with two regions fills 201 and 202. Instance +200 + k always holds pr2mask's "label k of N" (`mask_label_info`). Only instances that have a +region are written. An empty mask fills instance 201 with blank pr2mask columns, which records +that the model ran and found nothing. It is routed with the source PatientID and +ReferringPhysicianName (`EventName:` prefix removed), matching pr2mask routing. + +Every written instance has all of pr2mask's columns plus `model_version`, for example +`dynunet 5-fold, fastmonai-vsseg:20261110T081500Z`. pr2mask itself has no such field. It is on +the REDCap form, and the exporter adds it to `output_data_dictionary.zip` as a text field. If +pr2mask ever writes its own `model_version`, the exporter's value replaces it. The mask itself +goes to the PACS as the `labels` series, whose DICOM header also records the full bundle hash +and the TTA setting. + +pr2mask stores each region's mask in seven `mask_*` fields. `mask_base64_gzip_compressed` +covers the whole scan, so its length depends on the scan size, not the region: about 84,000 +characters for a 512×512×120 scan. REDCap cuts values longer than 65,535 characters, so a +region's mask only decodes for scans under about 24.5 million voxels (about 93 slices at +512×512). The full mask is in the PACS `labels` series. + +`pr2mask_dictionary.csv` holds the `instrument.csv` rows of a non-empty run of the packaged +pr2mask. An empty mask, where pr2mask lists no columns, takes them from there. When the Fiona +base changes, regenerate it from a smoke test and check that a non-empty run lists the same +columns. Up to 20261026T134845Z the exporter also sent eight `vs_*` fields, among them the mask as `vs_mask_json`. They are no longer sent. ### Stable model destinations and overwrites -`redcap_model_instances.json` assigns each full bundle SHA-256 a permanent positive -integer of at least 10. Instances 1-9 of the `pr2mask` instrument are left to the original -(manual) mask and pr2mask's own region numbering. DynUNet uses instance 11, and the nnU-Net -large container in the separate nnU-Net repository uses 10. Instance 12 is free for the next -model. They moved from 3 and 2 on 2026-10-21, and results imported before that stay at the -old instances. The retired UNet bundle was registered at instance 2. -Keep allocations unique across both projects, and don't renumber or reuse entries after -REDCap imports. Register each new bundle with an unused integer before building a release; -unknown bundles, duplicate allocations and instances below 10 fail closed. -The registry is copied into the container and validated before inference. - -All fields for a model share its repeat instance. Within the same record/event: - -- The same bundle overwrites its previous result, including when TTA, deployment - version, timestamp or the predicted mask changes. -- A different bundle uses a different instance, including different weights of - the same architecture. - -pr2mask writes each region as its own repeat instance (1, 2, 3, ...), which would collide -with the model instances, so regions are not sent as separate instances. pr2mask's columns -in the model instance hold the **largest** region (most voxels; ties keep the lower region -number). pr2mask numbers regions in scan order, so the largest is not always region 1. An -empty mask has no regions, so pr2mask lists no columns; the exporter then takes them from -`pr2mask_dictionary.csv` and sends them blank, so the importer must overwrite blanks to -clear values from an earlier run. - -Since the Fiona base of 2026-10-19, pr2mask also stores the mask in seven `mask_*` fields, -which fill the columns for the largest region. `mask_base64_gzip_compressed` is about 84,000 -characters for a 512×512×120 scan, and REDCap cuts values longer than 65,535 characters, so -it only decodes for small volumes. The full mask is in the PACS `labels` series. - -`pr2mask_dictionary.csv` holds the `instrument.csv` rows of a non-empty run of the packaged -pr2mask. When the Fiona base changes, regenerate it from a smoke test and check that a -non-empty run lists the same columns. - -Install pr2mask's dictionary in the receiving REDCap project and qualify a -PACS-to-REDCap round trip before use. The importer must honor the supplied numeric -repeat instance and update existing values. Live importer behavior and field limits -have not been verified. The record/event must already identify the intended case/scan; -multiple scans routed to the same record/event and model intentionally share a slot. -If earlier prototype exports were imported, reconcile those old instances before -using the permanent mapping. +`redcap_model_instances.json` gives each full bundle SHA-256 a permanent block of 100 +instances, stored as the block's base (a multiple of 100, at least 100). The DynUNet bundle +has block 200 (instances 201-299). Instances below 100 belong to the original (manual) mask +at 1, pr2mask's own region numbering and the earlier one-instance-per-model scheme: the nnU-Net +large container in the separate nnU-Net repository still writes its largest region to 10. +DynUNet wrote its largest region to 11 up to 20261027T083045Z (3 before 2026-10-21; the +retired UNet used 2); those results are removed in REDCap. Block 100 is planned for nnU-Net. +Keep blocks unique across both projects, and don't renumber or reuse them after REDCap +imports. Register each new bundle with an unused block before building a release. Unknown +bundles, duplicate blocks, invalid bases and region numbers of 100 or more fail closed. The +registry is copied into the container and validated before inference. + +Within the same record/event: + +- The same bundle overwrites its earlier instances, including when TTA, deployment version, + timestamp or the predicted mask changes. A rerun that finds fewer regions leaves the higher + instances of the earlier run as they were. The same bundle with the same TTA setting finds + the same regions. +- A different bundle uses a different block, including different weights of the same + architecture. + +Install pr2mask's dictionary plus `model_version` in the receiving REDCap project. The import +of the one-instance release 20261027T083045Z has been confirmed; confirm instances 201+ and +`model_version` with a PACS-to-REDCap round trip. The importer must honor the supplied numeric +repeat instance and update existing values. The record/event must already identify the +intended case/scan; multiple scans routed to the same record/event and model intentionally +share the same instances. diff --git a/research/vestibular_schwannoma/deployment/pacs/pacs_inference.py b/research/vestibular_schwannoma/deployment/pacs/pacs_inference.py index 1065412..91f2d49 100755 --- a/research/vestibular_schwannoma/deployment/pacs/pacs_inference.py +++ b/research/vestibular_schwannoma/deployment/pacs/pacs_inference.py @@ -29,13 +29,14 @@ TTA_FLIP_AXES, ParallelEnsemblePatchInferenceEngine, ) -from redcap_output import write_redcap_output, model_repeat_instance +from redcap_output import model_instance_base, write_redcap_output SW_BATCH_SIZE = 1 THREADS_PER_ENSEMBLE_MODEL = 3 DEFAULT_WORK_DIR = Path("/output_tmp") DEFAULT_PR2MASK_DIR = Path("/pr2mask") +IMAGE_NAME = "fastmonai-vsseg" FINAL_OUTPUT_DIRS = ("fused", "fused_vote_map", "reports", "labels", "redcap") LOG_NAME = "pacs_command.log" @@ -83,6 +84,13 @@ def _deployment_label(deployment: dict) -> str: return "single model" if count == 1 else f"{count}-model ensemble" +def _model_version(deployment: dict, version: str) -> str: + """Model and container release for pr2mask's REDCap model_version field.""" + count = len(deployment["members"]) + members = "single model" if count == 1 else f"{count}-fold" + return f"{deployment['model_type']} {members}, {IMAGE_NAME}:{version}" + + def _create_inference_engine(deployment: dict): model_paths = deployment.get("model_paths") if model_paths and len(model_paths) > 1: @@ -286,7 +294,12 @@ def _run_postprocessing( check=True, ) _tag_pr2mask_labels(work_dir, deployment, use_tta=use_tta) - write_redcap_output(work_dir, input_dir, deployment) + write_redcap_output( + work_dir, + input_dir, + deployment, + model_version=_model_version(deployment, version), + ) _copy_final_outputs(work_dir, output_dir) _publish_log(log_path, output_dir) @@ -316,7 +329,7 @@ def run_inference( print("=" * 60) deployment = load_deployment(model_type) - model_repeat_instance(deployment["bundle_sha256"]) + model_instance_base(deployment["bundle_sha256"]) patch_config = deployment["patch_config"] label = _deployment_label(deployment) print(f"Deployment: {label}") diff --git a/research/vestibular_schwannoma/deployment/pacs/redcap_model_instances.json b/research/vestibular_schwannoma/deployment/pacs/redcap_model_instances.json index 6d5dbf0..82a6d06 100644 --- a/research/vestibular_schwannoma/deployment/pacs/redcap_model_instances.json +++ b/research/vestibular_schwannoma/deployment/pacs/redcap_model_instances.json @@ -1,3 +1,3 @@ { - "c5642793b0f57b49ea8f04a08d2b8c9deff1523813d87d89eec204f85846d6ec": 11 + "c5642793b0f57b49ea8f04a08d2b8c9deff1523813d87d89eec204f85846d6ec": 200 } diff --git a/research/vestibular_schwannoma/deployment/pacs/redcap_output.py b/research/vestibular_schwannoma/deployment/pacs/redcap_output.py index c8612bd..4b37341 100644 --- a/research/vestibular_schwannoma/deployment/pacs/redcap_output.py +++ b/research/vestibular_schwannoma/deployment/pacs/redcap_output.py @@ -1,4 +1,4 @@ -"""pr2mask REDCap EAV rows, routed to the model's repeat instance of the pr2mask instrument.""" +"""pr2mask REDCap EAV rows, one repeat instance of the pr2mask instrument per region.""" from __future__ import annotations @@ -14,12 +14,16 @@ MODEL_INSTANCES_PATH = Path(__file__).with_name("redcap_model_instances.json") PR2MASK_DICTIONARY_PATH = Path(__file__).with_name("pr2mask_dictionary.csv") DICTIONARY_REQUIRED = ["Variable / Field Name", "Form Name", "Field Type", "Field Label"] -# Instances 1-9 of the pr2mask instrument are left to the original (manual) mask -# and pr2mask's own region numbering, so model instances start at 10. -FIRST_MODEL_INSTANCE = 10 +INSTRUMENT = "pr2mask" +MODEL_VERSION_FIELD = "model_version" +# Each bundle owns a block of 100 instances: pr2mask's region k goes to base + k. +# Instances below 100 are left to the original (manual) mask, pr2mask's own region +# numbering and earlier single-instance models. +REGION_BLOCK = 100 +MAX_INSTANCE = 2147483647 -def model_repeat_instance(bundle_sha256): +def model_instance_base(bundle_sha256): """Explicit immutable allocation: never truncate a hash into a REDCap integer.""" registry = json.loads(MODEL_INSTANCES_PATH.read_text()) if ( @@ -29,7 +33,10 @@ def model_repeat_instance(bundle_sha256): for key in registry ) or any( - type(n) is not int or not FIRST_MODEL_INSTANCE <= n <= 2147483647 + type(n) is not int + or n < REGION_BLOCK + or n % REGION_BLOCK + or n + REGION_BLOCK - 1 > MAX_INSTANCE for n in registry.values() ) or len(set(registry.values())) != len(registry) @@ -37,9 +44,10 @@ def model_repeat_instance(bundle_sha256): raise ValueError("Invalid REDCap model-instance registry") if bundle_sha256 not in registry: raise ValueError( - "Unregistered model bundle: allocate a permanent REDCap instance in redcap_model_instances.json" + "Unregistered model bundle: allocate a permanent REDCap instance block in " + "redcap_model_instances.json" ) - return str(registry[bundle_sha256]) + return registry[bundle_sha256] def _json(value): @@ -60,36 +68,45 @@ def pr2mask_dictionary_rows(): def _dictionary_columns(path): - """pr2mask's column names; an empty mask gets them from the shipped dictionary.""" + """pr2mask's column names plus model_version, also written to its dictionary.""" with zipfile.ZipFile(path) as archive: contents = {name: archive.read(name) for name in archive.namelist()} names, rows = _read_instrument(contents["instrument.csv"].decode("utf-8-sig")) if not rows: # pr2mask lists its columns from the first region, so an empty mask has none. rows = [{k: row.get(k, "") for k in names} for row in pr2mask_dictionary_rows()] - buffer = io.StringIO() - writer = csv.DictWriter(buffer, fieldnames=names) - writer.writeheader() - writer.writerows(rows) - contents["instrument.csv"] = buffer.getvalue().encode() - with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_DEFLATED) as archive: - for name, data in contents.items(): - archive.writestr(name, data) + if all(row["Variable / Field Name"] != MODEL_VERSION_FIELD for row in rows): + rows.append( + dict.fromkeys(names, "") + | { + "Variable / Field Name": MODEL_VERSION_FIELD, + "Form Name": INSTRUMENT, + "Field Type": "text", + "Field Label": MODEL_VERSION_FIELD, + } + ) + buffer = io.StringIO() + writer = csv.DictWriter(buffer, fieldnames=names) + writer.writeheader() + writer.writerows(rows) + contents["instrument.csv"] = buffer.getvalue().encode() + with zipfile.ZipFile(path, "w", compression=zipfile.ZIP_DEFLATED) as archive: + for name, data in contents.items(): + archive.writestr(name, data) return [row["Variable / Field Name"] for row in rows] -def _largest_region(rows): - """Values of the region with the most voxels; ties keep the lowest region number.""" +def _regions(rows): + """pr2mask's values per region; it numbers regions from 1 as repeat instances.""" regions = {} for row in rows: - regions.setdefault(int(row["redcap_repeat_instance"]), {})[ - row["field_name"] - ] = row["value"] - return max( - (regions[n] for n in sorted(regions)), - key=lambda r: float(r["number_of_pixel"]), - default={}, - ) + number = int(row["redcap_repeat_instance"]) + if not 1 <= number < REGION_BLOCK: + raise ValueError( + f"pr2mask region {number} is outside the {REGION_BLOCK - 1} model instances" + ) + regions.setdefault(number, {})[row["field_name"]] = row["value"] + return regions def _source_dataset(input_dir): @@ -99,9 +116,11 @@ def _source_dataset(input_dir): raise ValueError("No DICOM source images found") -def write_redcap_output(work_dir, input_dir, deployment): - """Store pr2mask's columns for the largest region in the model's permanent instance.""" - instance = model_repeat_instance(deployment["bundle_sha256"]) +def write_redcap_output(work_dir, input_dir, deployment, *, model_version): + """Store each region's pr2mask columns in its own instance of the model's block.""" + if not model_version: + raise ValueError("model_version is required") + base = model_instance_base(deployment["bundle_sha256"]) paths = list((Path(work_dir) / "redcap").glob("*/output.json")) if len(paths) != 1: raise ValueError("Expected exactly one pr2mask REDCap output.json") @@ -112,7 +131,7 @@ def write_redcap_output(work_dir, input_dir, deployment): keys = ("record_id", "redcap_event_name", "redcap_repeat_instrument") if rows: destination = {k: rows[0][k] for k in keys} - if destination["redcap_repeat_instrument"] != "pr2mask": + if destination["redcap_repeat_instrument"] != INSTRUMENT: raise ValueError("Unexpected REDCap instrument") if any(row[k] != destination[k] for row in rows for k in keys): raise ValueError("Mixed REDCap destinations") @@ -124,14 +143,22 @@ def write_redcap_output(work_dir, input_dir, deployment): redcap_event_name=str(source.get("ReferringPhysicianName", "")).removeprefix( "EventName:" ), - redcap_repeat_instrument="pr2mask", + redcap_repeat_instrument=INSTRUMENT, ) - # Region instances would collide with the model instances, so pr2mask's columns - # hold the largest region and are blanked when there is none. - destination["redcap_repeat_instance"] = instance columns = _dictionary_columns(path.with_name("output_data_dictionary.zip")) - largest = _largest_region(rows) - rows = [ - dict(destination, field_name=name, value=largest.get(name, "")) for name in columns - ] + # An empty mask still fills the first instance, with blank columns, to record the run. + regions = _regions(rows) or {1: {}} + rows = [] + for number, region in sorted(regions.items()): + values = region | {MODEL_VERSION_FIELD: model_version} + instance = str(base + number) + rows += [ + dict( + destination, + redcap_repeat_instance=instance, + field_name=name, + value=values.get(name, ""), + ) + for name in columns + ] path.write_text(_json(rows) + "\n") diff --git a/research/vestibular_schwannoma/tests/deployment/test_pacs_inference.py b/research/vestibular_schwannoma/tests/deployment/test_pacs_inference.py index baef860..8df858b 100644 --- a/research/vestibular_schwannoma/tests/deployment/test_pacs_inference.py +++ b/research/vestibular_schwannoma/tests/deployment/test_pacs_inference.py @@ -348,7 +348,12 @@ def run(command, **kwargs): ) tag_mask.assert_called_once_with(labels_series, deployment, use_tta=True) - export_redcap.assert_called_once_with(work_dir, input_dir, deployment) + export_redcap.assert_called_once_with( + work_dir, + input_dir, + deployment, + model_version="dynunet 5-fold, fastmonai-vsseg:20260817T120000Z", + ) self.assertEqual(len(calls), 3) identity = "20260817T120000Z_dynunet_tta" self.assertLessEqual(len(identity), 64) diff --git a/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py b/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py index 31f1f07..af54ca3 100644 --- a/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py +++ b/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py @@ -12,7 +12,7 @@ PACS_DIR = Path(__file__).resolve().parents[2] / "deployment" / "pacs" sys.path.insert(0, str(PACS_DIR)) -from redcap_output import write_redcap_output, model_repeat_instance # noqa: E402 +from redcap_output import write_redcap_output, model_instance_base # noqa: E402 import redcap_output # noqa: E402 from vestibular_schwannoma.tests.deployment.test_dicom_output import _write_test_image # noqa: E402 @@ -23,6 +23,8 @@ "physical_size,pr2mask,text,Physical size\n" ) PR2MASK_COLUMNS = ["mask_base64_gzip_compressed", "number_of_pixel", "physical_size"] +COLUMNS = PR2MASK_COLUMNS + ["model_version"] +MODEL_VERSION = "dynunet 5-fold, fastmonai-vsseg:20261110T000000Z" ROUTING = ("record_id", "redcap_event_name", "redcap_repeat_instrument", "redcap_repeat_instance") DEPLOYMENT = { "model_type": "dynunet", @@ -72,6 +74,19 @@ def dictionary_fields(report): return z.read("OriginID.txt"), [r["Variable / Field Name"] for r in rows] +def export(work, source, deployment=DEPLOYMENT): + write_redcap_output(work, source, deployment, model_version=MODEL_VERSION) + + +def instances(report): + """The written rows, and their values per repeat instance.""" + rows = json.loads((report / "output.json").read_text()) + values = {} + for row in rows: + values.setdefault(row["redcap_repeat_instance"], {})[row["field_name"]] = row["value"] + return rows, values + + class Pr2maskDictionaryTests(unittest.TestCase): def test_shipped_dictionary_lists_pr2mask_columns(self): names = [row["Variable / Field Name"] for row in redcap_output.pr2mask_dictionary_rows()] @@ -91,37 +106,76 @@ def setUp(self): patcher.start() self.addCleanup(patcher.stop) - def test_largest_region_fills_pr2mask_columns_in_model_instance(self): + def test_each_region_fills_its_own_instance(self): with tempfile.TemporaryDirectory() as tmp: source, work, report = fixture(Path(tmp)) - original = (report / "output_data_dictionary.zip").read_bytes() - write_redcap_output(work, source, DEPLOYMENT) - rows = json.loads((report / "output.json").read_text()) - self.assertEqual([r["field_name"] for r in rows], PR2MASK_COLUMNS) + export(work, source) + rows, values = instances(report) + self.assertEqual([r["field_name"] for r in rows], COLUMNS * 3) self.assertEqual( {tuple(r[k] for k in ROUTING) for r in rows}, - {("SYNTHETIC", "test_arm_1", "pr2mask", "11")}, - ) - values = {r["field_name"]: r["value"] for r in rows} - # Regions 2 and 3 tie on size: the lower region number fills the columns. - self.assertEqual( - [values[name] for name in PR2MASK_COLUMNS], ["m2", "50", "50.2"] + {("SYNTHETIC", "test_arm_1", "pr2mask", str(n)) for n in (201, 202, 203)}, ) - # pr2mask's dictionary already lists its columns and is left as written. - self.assertEqual((report / "output_data_dictionary.zip").read_bytes(), original) - - def test_empty_mask_sends_blank_columns(self): + for region, size in [(1, "20"), (2, "50"), (3, "50")]: + self.assertEqual( + values[str(200 + region)], + { + "mask_base64_gzip_compressed": f"m{region}", + "number_of_pixel": size, + "physical_size": f"{size}.{region}", + "model_version": MODEL_VERSION, + }, + ) + self.assertEqual(dictionary_fields(report), (b"PR2MASK", COLUMNS)) + + def test_empty_mask_fills_first_instance_with_blank_columns(self): with tempfile.TemporaryDirectory() as tmp: source, work, report = fixture(Path(tmp), empty=True) - write_redcap_output(work, source, DEPLOYMENT) - rows = json.loads((report / "output.json").read_text()) - self.assertEqual([r["field_name"] for r in rows], PR2MASK_COLUMNS) + export(work, source) + rows, values = instances(report) + self.assertEqual([r["field_name"] for r in rows], COLUMNS) self.assertEqual( {tuple(r[k] for k in ROUTING) for r in rows}, - {("SYNTHETIC", "test_arm_1", "pr2mask", "11")}, + {("SYNTHETIC", "test_arm_1", "pr2mask", "201")}, ) - self.assertEqual({r["value"] for r in rows}, {""}) - self.assertEqual(dictionary_fields(report), (b"PR2MASK", PR2MASK_COLUMNS)) + self.assertEqual( + values["201"], + dict.fromkeys(PR2MASK_COLUMNS, "") | {"model_version": MODEL_VERSION}, + ) + self.assertEqual(dictionary_fields(report), (b"PR2MASK", COLUMNS)) + + def test_model_version_from_pr2mask_is_replaced_not_duplicated(self): + with tempfile.TemporaryDirectory() as tmp: + source, work, report = fixture(Path(tmp)) + rows = json.loads((report / "output.json").read_text()) + rows.append(dict(rows[0], field_name="model_version", value="pr2mask")) + (report / "output.json").write_text(json.dumps(rows)) + with zipfile.ZipFile(report / "output_data_dictionary.zip", "w") as z: + z.writestr("OriginID.txt", "PR2MASK") + z.writestr( + "instrument.csv", + DICTIONARY_HEADER + DICTIONARY_ROWS + "model_version,pr2mask,text,Model\n", + ) + export(work, source) + rows, values = instances(report) + self.assertEqual([r["field_name"] for r in rows], COLUMNS * 3) + self.assertEqual({v["model_version"] for v in values.values()}, {MODEL_VERSION}) + self.assertEqual(dictionary_fields(report), (b"PR2MASK", COLUMNS)) + + def test_region_outside_the_block_is_rejected(self): + with tempfile.TemporaryDirectory() as tmp: + source, work, report = fixture(Path(tmp)) + rows = json.loads((report / "output.json").read_text()) + rows[-1]["redcap_repeat_instance"] = "100" + (report / "output.json").write_text(json.dumps(rows)) + with self.assertRaisesRegex(ValueError, "outside"): + export(work, source) + + def test_model_version_is_required(self): + with tempfile.TemporaryDirectory() as tmp: + source, work, _ = fixture(Path(tmp)) + with self.assertRaisesRegex(ValueError, "model_version"): + write_redcap_output(work, source, DEPLOYMENT, model_version="") def test_mixed_destinations_rejected(self): with tempfile.TemporaryDirectory() as tmp: @@ -130,46 +184,55 @@ def test_mixed_destinations_rejected(self): rows[-1]["record_id"] = "OTHER" (report / "output.json").write_text(json.dumps(rows)) with self.assertRaisesRegex(ValueError, "Mixed REDCap destinations"): - write_redcap_output(work, source, DEPLOYMENT) + export(work, source) def test_live_registry_holds_only_the_dynunet_bundle(self): self.assertEqual( json.loads(redcap_output.MODEL_INSTANCES_PATH.read_text()), - {DEPLOYMENT["bundle_sha256"]: 11}, + {DEPLOYMENT["bundle_sha256"]: 200}, ) - def test_models_have_distinct_destinations_and_reruns_replace_complete_result(self): + def test_models_have_separate_blocks_and_reruns_replace_their_instances(self): stored = {} other = dict(DEPLOYMENT, bundle_sha256="d" * 64) with tempfile.TemporaryDirectory() as tmp: registry = Path(tmp) / "registry.json" - registry.write_text(json.dumps({DEPLOYMENT["bundle_sha256"]: 10, "d" * 64: 11})) + registry.write_text(json.dumps({DEPLOYMENT["bundle_sha256"]: 100, "d" * 64: 200})) with patch.object(redcap_output, "MODEL_INSTANCES_PATH", registry): for deployment, empty in [(DEPLOYMENT, False), (other, False), (DEPLOYMENT, True)]: with tempfile.TemporaryDirectory() as tmp: source, work, report = fixture(Path(tmp), empty=empty) - write_redcap_output(work, source, deployment) + export(work, source, deployment) for row in json.loads((report / "output.json").read_text()): key = tuple(row[k] for k in ROUTING) + (row["field_name"],) stored[key] = row["value"] by_instance = {} for key, value in stored.items(): by_instance.setdefault(key[3], {})[key[4]] = value - self.assertEqual(set(by_instance), {"10", "11"}) - self.assertEqual(by_instance["10"], dict.fromkeys(PR2MASK_COLUMNS, "")) - self.assertEqual(by_instance["11"]["physical_size"], "50.2") + # A rerun with fewer regions leaves the higher instances as they were. + self.assertEqual(set(by_instance), {"101", "102", "103", "201", "202", "203"}) + self.assertEqual( + by_instance["101"], + dict.fromkeys(PR2MASK_COLUMNS, "") | {"model_version": MODEL_VERSION}, + ) + self.assertEqual(by_instance["202"]["physical_size"], "50.2") - def test_unknown_and_colliding_registry_entries_are_rejected(self): + def test_unknown_and_invalid_registry_entries_are_rejected(self): with self.assertRaisesRegex(ValueError, "Unregistered"): - model_repeat_instance("f" * 64) + model_instance_base("f" * 64) with tempfile.TemporaryDirectory() as tmp: path = Path(tmp) / "registry.json" - for registry in ({"a" * 64: 10, "b" * 64: 10}, {"a" * 64: 9}): + for registry in ( + {"a" * 64: 200, "b" * 64: 200}, + {"a" * 64: 11}, + {"a" * 64: 250}, + {"a" * 64: True}, + ): with self.subTest(registry=registry): path.write_text(json.dumps(registry)) with patch.object(redcap_output, "MODEL_INSTANCES_PATH", path): with self.assertRaisesRegex(ValueError, "Invalid"): - model_repeat_instance("a" * 64) + model_instance_base("a" * 64) if __name__ == "__main__": From 72841617d4dc32198f43a2cae2818018665ec38d Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Mon, 9 Nov 2026 09:54:58 +0100 Subject: [PATCH 3/8] feat(vs-pacs): send redcap values over 65,535 bytes blank REDCap cuts longer values. pr2mask's mask text covers the whole scan (about 105,000 characters for a 512x448x176 MPRAGE), so a cut value is undecodable. Send it blank and log it; the full mask is in the PACS labels series. --- .../deployment/pacs/README.md | 6 ++++-- .../deployment/pacs/redcap_output.py | 6 ++++++ .../tests/deployment/test_redcap_output.py | 18 ++++++++++++++++++ 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/research/vestibular_schwannoma/deployment/pacs/README.md b/research/vestibular_schwannoma/deployment/pacs/README.md index 9c57e4e..f3e84a4 100644 --- a/research/vestibular_schwannoma/deployment/pacs/README.md +++ b/research/vestibular_schwannoma/deployment/pacs/README.md @@ -135,8 +135,10 @@ and the TTA setting. pr2mask stores each region's mask in seven `mask_*` fields. `mask_base64_gzip_compressed` covers the whole scan, so its length depends on the scan size, not the region: about 84,000 characters for a 512×512×120 scan. REDCap cuts values longer than 65,535 characters, so a -region's mask only decodes for scans under about 24.5 million voxels (about 93 slices at -512×512). The full mask is in the PACS `labels` series. +region's mask only fits for scans under about 24.5 million voxels (about 93 slices at +512×512). A cut mask can't be decoded, so the exporter sends any value over 65,535 bytes +blank and says so in the container log. The other fields are unaffected, and the full mask +is always in the PACS `labels` series. `pr2mask_dictionary.csv` holds the `instrument.csv` rows of a non-empty run of the packaged pr2mask. An empty mask, where pr2mask lists no columns, takes them from there. When the Fiona diff --git a/research/vestibular_schwannoma/deployment/pacs/redcap_output.py b/research/vestibular_schwannoma/deployment/pacs/redcap_output.py index 4b37341..8306a3a 100644 --- a/research/vestibular_schwannoma/deployment/pacs/redcap_output.py +++ b/research/vestibular_schwannoma/deployment/pacs/redcap_output.py @@ -21,6 +21,8 @@ # numbering and earlier single-instance models. REGION_BLOCK = 100 MAX_INSTANCE = 2147483647 +# REDCap cuts longer values, which leaves pr2mask's mask text undecodable. +MAX_VALUE_BYTES = 65535 def model_instance_base(bundle_sha256): @@ -151,6 +153,10 @@ def write_redcap_output(work_dir, input_dir, deployment, *, model_version): rows = [] for number, region in sorted(regions.items()): values = region | {MODEL_VERSION_FIELD: model_version} + for name, value in values.items(): + if len(value.encode("utf-8")) > MAX_VALUE_BYTES: + print(f"REDCap: {name} of region {number} is over {MAX_VALUE_BYTES} bytes, sent blank") + values[name] = "" instance = str(base + number) rows += [ dict( diff --git a/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py b/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py index af54ca3..bc723f7 100644 --- a/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py +++ b/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py @@ -162,6 +162,24 @@ def test_model_version_from_pr2mask_is_replaced_not_duplicated(self): self.assertEqual({v["model_version"] for v in values.values()}, {MODEL_VERSION}) self.assertEqual(dictionary_fields(report), (b"PR2MASK", COLUMNS)) + def test_values_redcap_would_cut_are_sent_blank(self): + with tempfile.TemporaryDirectory() as tmp: + source, work, report = fixture(Path(tmp)) + rows = json.loads((report / "output.json").read_text()) + limit = redcap_output.MAX_VALUE_BYTES + for row in rows: + if row["field_name"] == "mask_base64_gzip_compressed": + fits = row["redcap_repeat_instance"] == "1" + row["value"] = "m" * (limit if fits else limit + 1) + (report / "output.json").write_text(json.dumps(rows)) + export(work, source) + _, values = instances(report) + self.assertEqual( + [values[n]["mask_base64_gzip_compressed"] for n in ("201", "202", "203")], + ["m" * limit, "", ""], + ) + self.assertEqual(values["202"]["number_of_pixel"], "50") + def test_region_outside_the_block_is_rejected(self): with tempfile.TemporaryDirectory() as tmp: source, work, report = fixture(Path(tmp)) From d9c839057346a047eaba57a52f419dd0bca14bc5 Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Mon, 9 Nov 2026 10:20:35 +0100 Subject: [PATCH 4/8] fix(vs-pacs): send long redcap values as pr2mask wrote them Reverts the blanking of values over 65,535 bytes. It isn't known whether the receiving REDCap cuts them, and blanking would throw away a mask that REDCap might store whole. The previous release sent the full value and its import works. --- .../deployment/pacs/README.md | 11 +++++------ .../deployment/pacs/redcap_output.py | 6 ------ .../tests/deployment/test_redcap_output.py | 18 ------------------ 3 files changed, 5 insertions(+), 30 deletions(-) diff --git a/research/vestibular_schwannoma/deployment/pacs/README.md b/research/vestibular_schwannoma/deployment/pacs/README.md index f3e84a4..066df41 100644 --- a/research/vestibular_schwannoma/deployment/pacs/README.md +++ b/research/vestibular_schwannoma/deployment/pacs/README.md @@ -133,12 +133,11 @@ goes to the PACS as the `labels` series, whose DICOM header also records the ful and the TTA setting. pr2mask stores each region's mask in seven `mask_*` fields. `mask_base64_gzip_compressed` -covers the whole scan, so its length depends on the scan size, not the region: about 84,000 -characters for a 512×512×120 scan. REDCap cuts values longer than 65,535 characters, so a -region's mask only fits for scans under about 24.5 million voxels (about 93 slices at -512×512). A cut mask can't be decoded, so the exporter sends any value over 65,535 bytes -blank and says so in the container log. The other fields are unaffected, and the full mask -is always in the PACS `labels` series. +covers the whole scan at 16 bits per voxel, so its length depends on the scan size, not the +region: about 84,000 characters for a 512×512×120 scan and 105,000 for 512×448×176. REDCap may +cut values longer than 65,535 characters; this has not been checked in the receiving project. +The exporter sends the value as pr2mask wrote it, and the full mask is always in the PACS +`labels` series. `pr2mask_dictionary.csv` holds the `instrument.csv` rows of a non-empty run of the packaged pr2mask. An empty mask, where pr2mask lists no columns, takes them from there. When the Fiona diff --git a/research/vestibular_schwannoma/deployment/pacs/redcap_output.py b/research/vestibular_schwannoma/deployment/pacs/redcap_output.py index 8306a3a..4b37341 100644 --- a/research/vestibular_schwannoma/deployment/pacs/redcap_output.py +++ b/research/vestibular_schwannoma/deployment/pacs/redcap_output.py @@ -21,8 +21,6 @@ # numbering and earlier single-instance models. REGION_BLOCK = 100 MAX_INSTANCE = 2147483647 -# REDCap cuts longer values, which leaves pr2mask's mask text undecodable. -MAX_VALUE_BYTES = 65535 def model_instance_base(bundle_sha256): @@ -153,10 +151,6 @@ def write_redcap_output(work_dir, input_dir, deployment, *, model_version): rows = [] for number, region in sorted(regions.items()): values = region | {MODEL_VERSION_FIELD: model_version} - for name, value in values.items(): - if len(value.encode("utf-8")) > MAX_VALUE_BYTES: - print(f"REDCap: {name} of region {number} is over {MAX_VALUE_BYTES} bytes, sent blank") - values[name] = "" instance = str(base + number) rows += [ dict( diff --git a/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py b/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py index bc723f7..af54ca3 100644 --- a/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py +++ b/research/vestibular_schwannoma/tests/deployment/test_redcap_output.py @@ -162,24 +162,6 @@ def test_model_version_from_pr2mask_is_replaced_not_duplicated(self): self.assertEqual({v["model_version"] for v in values.values()}, {MODEL_VERSION}) self.assertEqual(dictionary_fields(report), (b"PR2MASK", COLUMNS)) - def test_values_redcap_would_cut_are_sent_blank(self): - with tempfile.TemporaryDirectory() as tmp: - source, work, report = fixture(Path(tmp)) - rows = json.loads((report / "output.json").read_text()) - limit = redcap_output.MAX_VALUE_BYTES - for row in rows: - if row["field_name"] == "mask_base64_gzip_compressed": - fits = row["redcap_repeat_instance"] == "1" - row["value"] = "m" * (limit if fits else limit + 1) - (report / "output.json").write_text(json.dumps(rows)) - export(work, source) - _, values = instances(report) - self.assertEqual( - [values[n]["mask_base64_gzip_compressed"] for n in ("201", "202", "203")], - ["m" * limit, "", ""], - ) - self.assertEqual(values["202"]["number_of_pixel"], "50") - def test_region_outside_the_block_is_rejected(self): with tempfile.TemporaryDirectory() as tmp: source, work, report = fixture(Path(tmp)) From 3b3d045d8fbe0bdebd95a0df21e278e148e2f274 Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Mon, 9 Nov 2026 10:43:47 +0100 Subject: [PATCH 5/8] docs(vs-pacs): say old redcap instances are no longer updated --- research/vestibular_schwannoma/deployment/pacs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/research/vestibular_schwannoma/deployment/pacs/README.md b/research/vestibular_schwannoma/deployment/pacs/README.md index 066df41..0732822 100644 --- a/research/vestibular_schwannoma/deployment/pacs/README.md +++ b/research/vestibular_schwannoma/deployment/pacs/README.md @@ -155,7 +155,7 @@ has block 200 (instances 201-299). Instances below 100 belong to the original (m at 1, pr2mask's own region numbering and the earlier one-instance-per-model scheme: the nnU-Net large container in the separate nnU-Net repository still writes its largest region to 10. DynUNet wrote its largest region to 11 up to 20261027T083045Z (3 before 2026-10-21; the -retired UNet used 2); those results are removed in REDCap. Block 100 is planned for nnU-Net. +retired UNet used 2); those results are no longer updated. Block 100 is planned for nnU-Net. Keep blocks unique across both projects, and don't renumber or reuse them after REDCap imports. Register each new bundle with an unused block before building a release. Unknown bundles, duplicate blocks, invalid bases and region numbers of 100 or more fail closed. The From 778f3703e4128af74da329ada2dda4121dee57b4 Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Mon, 9 Nov 2026 10:59:20 +0100 Subject: [PATCH 6/8] docs(pacs): stop writing archive checksum files into release folders --- research/RESEARCH_PACS_DEPLOYMENT.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/research/RESEARCH_PACS_DEPLOYMENT.md b/research/RESEARCH_PACS_DEPLOYMENT.md index 74cb5c2..5c4ecbf 100644 --- a/research/RESEARCH_PACS_DEPLOYMENT.md +++ b/research/RESEARCH_PACS_DEPLOYMENT.md @@ -204,13 +204,10 @@ docker tag ":$BUILD_VERSION" ":latest" ARCHIVE="-$BUILD_VERSION.tar.gz" docker save ":$BUILD_VERSION" ":latest" | gzip > "$ARCHIVE" - -sha256sum "$ARCHIVE" > "$ARCHIVE.sha256" -sha256sum -c "$ARCHIVE.sha256" ``` -The SHA-256 detects archive corruption during transfer; it does not test the image. After -`docker load`, either tag can be called. The dated tag is the stable release and rollback +Don't write a `.sha256` file into the release folder. After `docker load`, the image ID +confirms the right image arrived, and either tag can be called. The dated tag is the stable release and rollback identity; `latest` follows whichever archive was loaded or tagged last. ## 9. Research PACS Deployment @@ -219,7 +216,7 @@ Register the qualified image and its DICOM selection criteria with the Research Provide the Research PACS administrator with: -- the dated tag, `latest` alias, image ID or registry digest, and archive checksum; +- the dated tag, `latest` alias, and image ID or registry digest; - the Git commit, ROR executable identity, and Fiona base image ID or digest; - the model-bundle names and hashes; - the bundle's DICOM schema; From b64858c80904f3482785962af1675d424bc4d377 Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Tue, 10 Nov 2026 10:57:43 +0100 Subject: [PATCH 7/8] build(vs-pacs): pin fastmonai 0.13.1 for the grid restore fix 0.13.1 resamples the predicted mask onto the scan's own grid. With 0.12.0, masks and vote maps could sit up to half a slice off on clinical grids. Built as fastmonai-vsseg:20261110T082650Z. --- .../deployment/pacs/requirements.yml | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/research/vestibular_schwannoma/deployment/pacs/requirements.yml b/research/vestibular_schwannoma/deployment/pacs/requirements.yml index be5a9fd..47f49ea 100644 --- a/research/vestibular_schwannoma/deployment/pacs/requirements.yml +++ b/research/vestibular_schwannoma/deployment/pacs/requirements.yml @@ -14,10 +14,11 @@ dependencies: - torchvision==0.27.1+cpu # fastMONAI 0.11.0 adds TTA with a chosen subset of flips, used for the # original + AP+IS pair. 0.12.0 restores the mask to the scan grid with - # shape-based interpolation instead of copying the nearest model slice. Keep - # the deployment stack pinned; requirements plus the built image digest are - # the runtime contract. - - fastMONAI==0.12.0 + # shape-based interpolation instead of copying the nearest model slice. + # 0.13.1 resamples it onto the scan's own grid, so it no longer sits up to + # half a slice off. Keep the deployment stack pinned; requirements plus the + # built image digest are the runtime contract. + - fastMONAI==0.13.1 - fastai==2.8.7 # Keep the fastai/fastMONAI dispatch stack on the known-good versions from # the previously qualified PACS image. plum-dispatch 2.10.0 makes the fastai From db6ac8f0f50a06835f3298551b777f29855b4a04 Mon Sep 17 00:00:00 2001 From: Sathiesh Date: Tue, 10 Nov 2026 11:04:51 +0100 Subject: [PATCH 8/8] docs: shorten the readmes and agents guide Cuts repeated setup steps and long prose from the research READMEs and AGENTS.md. AGENTS.md now also covers nbdev_clean, the generated root README, how to run the VS tests, release folders and writing style. The VS README keeps the note on merging folds from different machines as a bullet, and the PACS README's model_version example names the current image. --- AGENTS.md | 46 ++-- research/README.md | 21 +- research/endometrial_cancer/README.md | 103 +++----- research/spine/README.md | 38 ++- research/vestibular_schwannoma/README.md | 161 ++++------- research/vestibular_schwannoma/data/README.md | 236 ++++------------- .../deployment/pacs/README.md | 249 +++++++----------- 7 files changed, 284 insertions(+), 570 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9ebf303..5107612 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,28 +1,38 @@ # Repository Guidelines -## nbdev workflow +fastMONAI is an nbdev library: `nbs/` is exported to `fastMONAI/`. `research/` holds +application projects written as plain Python and notebooks; nbdev does not export them. -- Treat `nbs/*.ipynb` as the source of truth for nbdev-exported package code. -- Do not edit generated `fastMONAI/*.py` modules directly. -- After changing library notebooks, run `nbdev_prepare` from the repository root to regenerate modules and run checks. +## Library (nbdev) + +- `nbs/*.ipynb` is the source of truth. Don't edit the generated `fastMONAI/*.py` modules. +- After changing library notebooks, run `nbdev_prepare` from the repository root. +- Clean only the notebooks you edited (`nbdev_clean --fname `). A repo-wide clean + rewrites the tutorials' stored outputs. +- `README.md` is generated from `nbs/index.ipynb`; edit the notebook, not the README. + +## Research projects + +- Each project has its own README; see `research/README.md`. +- CI runs `python -m pytest -q research/vestibular_schwannoma/tests` from the repository root. + Without pytest, run the deployment tests from `research/`: + `python -m unittest discover -s vestibular_schwannoma/tests/deployment -t .` +- Container release folders hold only the image archive, a short README and + `ror_select_entry.txt`. Don't write `.sha256` files. + +## Writing + +- Keep READMEs, comments and commit messages short and plain. Don't use em dashes. +- Comments say what the code does. Which bug a change fixes belongs in the commit message. ## Commit messages - Use Conventional Commits: `(): `, for example `fix(vs-pacs): publish pr2mask mask series`. -- Types: - - `feat`: new functionality - - `fix`: bug fix - - `docs`: documentation only - - `test`: tests only - - `refactor`: code change without a behaviour change - - `perf`: performance improvement - - `build`: dependencies, packaging or Docker images - - `ci`: CI configuration - - `chore`: maintenance that fits none of the above, such as `.gitignore` -- There is no `bug` or `minor` type: use `fix` for bugs, and `chore`, `docs` or - `refactor` for small changes. +- Types: `feat`, `fix`, `docs`, `test`, `refactor` (no behaviour change), `perf`, `build` + (dependencies, packaging, Docker images), `ci` and `chore` (anything else, such as + `.gitignore`). There is no `bug` or `minor` type. - Mark breaking changes with `!` after the type (`feat!: ...`) and explain them in the body. -- Write the summary in the imperative, lower case, without a trailing period and - under about 72 characters. Use the body to explain why. +- Write the summary in the imperative, lower case, without a trailing period and under about + 72 characters. Use the body to explain why. - Pull requests to `main` are squash-merged, so give the pull request title the same format. diff --git a/research/README.md b/research/README.md index 5293459..f939b8f 100644 --- a/research/README.md +++ b/research/README.md @@ -1,15 +1,12 @@ -# Research +# Research -## Shared deployment guidance - -Projects targeting the Research PACS should follow -[Research PACS deployment with ROR](RESEARCH_PACS_DEPLOYMENT.md). +Applications built with fastMONAI. Each project has its own README. +| Project | Contents | +| --- | --- | +| [spine](spine/README.md) | Spine segmentation from T2w MRI: training, evaluation and inference. | +| [endometrial_cancer](endometrial_cancer/README.md) | Endometrial cancer segmentation from VIBE, or from T2w, VIBE and ADC together. | +| [vestibular_schwannoma](vestibular_schwannoma/README.md) | Vestibular schwannoma segmentation from CE-T1w MRI: five-fold cross-validation, inference and a Research PACS container. | -## Table of contents -| Notebook | 1-Click Notebook | -|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [00-spine-segmentation-training-evaluation](https://nbviewer.org/github/MMIV-ML/fastMONAI/blob/main/research/spine/00-spine-segmentation-training-evaluation.ipynb)
Notebook to reproduce our test results and facilitate training on your own data. | [![Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/MMIV-ML/fastMONAI/blob/main/research/spine/00-spine-segmentation-training-evaluation.ipynb) | -| [01-spine-segmentation-inference](https://nbviewer.org/github/MMIV-ML/fastMONAI/blob/main/research/spine/01-spine-segmentation-inference.ipynb)
Notebook that shows how to run inference on your own data with our trained models. | [![Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/MMIV-ML/fastMONAI/blob/main/research/spine/01-spine-segmentation-inference.ipynb) | -| [02_ec_inference](https://nbviewer.org/github/MMIV-ML/fastMONAI/blob/main/research/endometrial_cancer/nbs/02_ec_inference.ipynb)
Notebook that shows how to run inference on your own data with our trained model. | [![Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/MMIV-ML/fastMONAI/blob/main/research/endometrial_cancer/nbs/02_ec_inference.ipynb) | -| [vestibular_schwannoma](vestibular_schwannoma/README.md)
Five-fold cross-validation, inference and Research PACS deployment for vestibular schwannoma segmentation. | Local setup, see the project README | +Projects that run in the Research PACS follow +[Research PACS deployment with ROR](RESEARCH_PACS_DEPLOYMENT.md). diff --git a/research/endometrial_cancer/README.md b/research/endometrial_cancer/README.md index bcea226..fe9b0ab 100644 --- a/research/endometrial_cancer/README.md +++ b/research/endometrial_cancer/README.md @@ -1,60 +1,35 @@ # Endometrial cancer segmentation -The primary objective of this repository is to reproduce the results reported in [Automated segmentation of endometrial cancer on MR images using deep learning](https://link.springer.com/content/pdf/10.1038/s41598-020-80068-9.pdf) and to integrate this model into research PACS (see [Results for VIBE](#results-for-vibe) section). -In addition, we have looked at improving the segmentation performance using multi-sequence MR images (T2w, VIBE, and ADC) (see [Results for multi-sequence (T2, VIBE, and ADC)](#results-for-multi-sequence-t2-vibe-and-adc) section). - -The trained weights and exported learner are available on [Hugging Face](https://huggingface.co/skaliy/endometrial_cancer_segmentation). -A Gradio app for VIBE is included in `src/app.py`. To run it on your local system, follow the instructions in the [Getting started](#getting-started) section. - -## Structure overview -``` -├── figs -│ -├── nbs # Jupyter notebooks for training and inference -│ ├── 01_ec_training.ipynb -│ ├── 02_ec_inference.ipynb -│ ├── 03_ec_training_multi.ipynb -│ ├── 04_ec_inference_multi.ipynb -│ -├── src -│ ├── app.py # Gradio app -│ ├── inference_script.py # Inference script for single-sequence -│ └── inference_script_multi.py # Inference for multi-sequence -│ -└── README.md -``` - -## Getting started -1. Clone this repository: -```bash -git clone https://github.com/MMIV-ML/fastMONAI -``` -2. Install fastMONAI by following the instructions provided [here](https://github.com/MMIV-ML/fastMONAI). -3. (Optional) Run the `01_ec_training.ipynb` `03_ec_training_multi.ipynb` notebook to train your own model. -4. Run the `02_ec_inference.ipynb`, `inference_script.py` or `app.py` to perform inference with the trained model for VIBE images. -5. Run the `04_ec_inference_multi.ipynb` or `inference_script_multi.py` to perform inference with the trained model for T2w, VIBE, and ADC images. - -If you choose to use `inference_script.py` or `inference_script_multi.py`, please follow these steps: - -- Make the script executable using the following command in the terminal: -```bash -chmod +x inference_script.py -``` -- Run the script by executing the following command in the terminal: -```bash -python inference_script.py IMG_PATH -``` - -If you choose to use `app.py`, install `gradio` and `opencv-python` (they are not fastMONAI dependencies) and execute the following command in the terminal: -```bash -gradio app.py -``` -The predicted mask will be stored in folder `ec_pred` in the `src` folder. - -## Results for VIBE -Note that our results are not directly comparable with the results reported in [study](https://link.springer.com/content/pdf/10.1038/s41598-020-80068-9.pdf), as we opted to use the test set for validation to allocate more data to training. Unlike the approach detailed in the study, we refrained from post-processing steps, such as retaining only the largest object. Predictions from new test cases indicate that this method could occasionally eliminate the tumor. Below is the box plot showcasing predictions on the validation set: + +Reproduces [Automated segmentation of endometrial cancer on MR images using deep learning](https://link.springer.com/content/pdf/10.1038/s41598-020-80068-9.pdf) +on VIBE images, and tests whether T2w, VIBE and ADC together improve the segmentation. The +VIBE model runs in the research PACS. Trained weights and the exported learner are on +[Hugging Face](https://huggingface.co/skaliy/endometrial_cancer_segmentation). + +## Contents + +- `nbs/01_ec_training.ipynb`, `nbs/03_ec_training_multi.ipynb`: train the VIBE and the + multi-sequence model. +- `nbs/02_ec_inference.ipynb` ([Colab](https://colab.research.google.com/github/MMIV-ML/fastMONAI/blob/main/research/endometrial_cancer/nbs/02_ec_inference.ipynb)), + `nbs/04_ec_inference_multi.ipynb`: run the trained models on your own data. +- `src/inference_script.py IMG_PATH`, `src/inference_script_multi.py`: command-line inference. +- `src/app.py`: Gradio app for VIBE. Install `gradio` and `opencv-python` (not fastMONAI + dependencies) and run `gradio app.py`; predictions are written to `src/ec_pred`. + +Install fastMONAI as described in the [main README](../../README.md). + +## Results + +The test set was used for validation to leave more data for training, so the results are not +directly comparable with the study. No post-processing is applied: keeping only the largest +object sometimes removed the tumour in new cases. DSC columns: inter-rater, model vs rater 1, +model vs rater 2. + +**VIBE**, median DSC 0.8946, 0.8212, 0.779: + ![](figs/vibe_boxplot.png) -The results from the validation set are also presented in the table below: +
+Per-case results (VIBE) | | subject_id | tumor_vol | inter_rater | r1_ml | r2_ml | n_components | |---:|-------------:|------------:|--------------:|---------:|-----------:|---------------:| @@ -90,16 +65,18 @@ The results from the validation set are also presented in the table below: | 29 | 531 | 22.08 | 0.89225 | 0.555066 | 0.505109 | 1 | | 30 | 540 | 8.35 | 0.923702 | 0.855009 | 0.840958 | 1 | -Median DSC: 0.8946, 0.8212, 0.779 +
+ +Prediction on a new subject in the research PACS: -Prediction on a new subject in the research PACS: ![](figs/research_pacs_predicition.png) -## Results for multi-sequence (T2, VIBE, and ADC) -The box plot of the predictions on the validation set: +**T2w, VIBE and ADC**, median DSC 0.8946, 0.8521, 0.8137: + ![](figs/t2_vibe_adc_boxplot.png) -The results from the validation set are also presented in the table below: +
+Per-case results (T2w, VIBE and ADC) | | subject_id | tumor_vol | inter_rater | r1_ml | r2_ml | n_components | |---:|-------------:|------------:|--------------:|---------:|----------:|---------------:| @@ -135,7 +112,9 @@ The results from the validation set are also presented in the table below: | 29 | 531 | 22.08 | 0.89225 | 0.349648 | 0.319533 | 1 | | 30 | 540 | 8.35 | 0.923702 | 0.890343 | 0.88052 | 1 | -Median DSC: 0.8946, 0.8521, 0.8137 +
+ +## Support -## Support and Contribution -For any issues related to the model or the source code, please open an issue in the corresponding GitHub repository. Contributions to the code or the model are welcome and should be proposed through a pull request. +Open an issue for problems with the model or the code. Contributions are welcome as pull +requests. diff --git a/research/spine/README.md b/research/spine/README.md index 0de2185..9aa49d4 100644 --- a/research/spine/README.md +++ b/research/spine/README.md @@ -1,30 +1,24 @@ -# Multi-Center CNN-based spine segmentation from T2w MRI using small amounts of data -The source code, designed to reproduce our test results and facilitate training and running inference on your own data. -The trained weights and exported learners are available on [Hugging Face](https://huggingface.co/skaliy/spine-segmentation). +# Multi-center CNN-based spine segmentation from T2w MRI using small amounts of data -## Getting started -### Quick Start -You can run the notebooks directly in Google Colab. +Code to reproduce our test results, train on your own data and run inference. Trained +weights and exported learners are on [Hugging Face](https://huggingface.co/skaliy/spine-segmentation). -### Local Setup (Optional) -1. Clone this repository: -```bash -git clone https://github.com/MMIV-ML/fastMONAI -``` -2. Install fastMONAI by following the instructions provided [here](https://github.com/MMIV-ML/fastMONAI/tree/main). -3. (Optional) Run the `00-spine-segmentation-training-evaluation.ipynb` notebook to train your own model or to reproduce the test results reported in our paper. -4. Run the `01-spine-segmentation-inference.ipynb` or `inference_script.py` to perform inference with the trained model on own data +| Notebook | Colab | +| --- | --- | +| [00-spine-segmentation-training-evaluation](https://nbviewer.org/github/MMIV-ML/fastMONAI/blob/main/research/spine/00-spine-segmentation-training-evaluation.ipynb): reproduce the test results or train your own model | [Open](https://colab.research.google.com/github/MMIV-ML/fastMONAI/blob/main/research/spine/00-spine-segmentation-training-evaluation.ipynb) | +| [01-spine-segmentation-inference](https://nbviewer.org/github/MMIV-ML/fastMONAI/blob/main/research/spine/01-spine-segmentation-inference.ipynb): run the trained model on your own data | [Open](https://colab.research.google.com/github/MMIV-ML/fastMONAI/blob/main/research/spine/01-spine-segmentation-inference.ipynb) | + +## Local setup + +Clone this repository and [install fastMONAI](../../README.md). Then run the notebooks, or +run inference from the command line: -If you choose to use `inference_script.py` -- Make the script executable using the following command in the terminal: -```bash -chmod +x inference_script.py -``` -- Run the script by executing the following command in the terminal: ```bash python inference_script.py IMG_PATH ``` -## Research PACS -Our spine segmentation application deployed in the research PACS system at our hospital: +## Research PACS + +The model runs in the research PACS at our hospital: + ![](figs/research_pacs.png) diff --git a/research/vestibular_schwannoma/README.md b/research/vestibular_schwannoma/README.md index f8baf17..0491e26 100644 --- a/research/vestibular_schwannoma/README.md +++ b/research/vestibular_schwannoma/README.md @@ -1,139 +1,76 @@ -# Vestibular Schwannoma Segmentation (CE-T1w) +# Vestibular schwannoma segmentation (CE-T1w) -Patch-based 3D segmentation of vestibular schwannoma in contrast-enhanced T1-weighted MRI, -with five-fold cross-validation, optional all-data training, inference, and PACS deployment. +Patch-based 3D segmentation of vestibular schwannoma in contrast-enhanced T1-weighted MRI: +five-fold cross-validation, inference and a Research PACS container. -## Contents +| Path | Contents | +| --- | --- | +| `data/` | Case index with fixed folds, data preparation ([data/README.md](data/README.md)) | +| `scripts/train_5fold.py` | Five-fold training and evaluation | +| `scripts/merge_parallel_folds.py` | Combine folds trained in parallel | +| `notebooks/` | Walkthroughs of cross-validation and inference | +| `workflow/` | Shared configuration, training, evaluation and inference code | +| `deployment/pacs/` | Model bundle builder and PACS container ([README](deployment/pacs/README.md)) | +| `tests/` | Workflow and deployment tests | -- `scripts/train_5fold.py`: command-line five-fold training and evaluation. -- `scripts/merge_parallel_folds.py`: validate and combine parallel fold subsets for inference. -- `notebooks/`: educational walkthroughs of cross-validation and inference. -- `workflow/`: shared configuration, training, evaluation, and artifact handling. -- `data/ml_dataset.csv`: public case index and fixed fold assignments. -- `scripts/prepare_data.py`: download, prepare, and verify the exact study dataset. -- `deployment/pacs/`: Safetensors bundle builder and ROR/PACS container. -- `tests/`: workflow and deployment tests. +## Setup -## Setup and data - -Use fastMONAI 0.12.0 or a matching development checkout. From the fastMONAI repository root: +From the repository root, install this checkout and prepare the 344-case dataset +(see [data/README.md](data/README.md)): ```bash pip install -e '.[dev]' +cd research/vestibular_schwannoma +python scripts/prepare_data.py download +python scripts/prepare_data.py prepare ``` -Download and prepare the 344-case dataset with `scripts/prepare_data.py`; see -[data/README.md](data/README.md) for scripted acquisition, corrected labels, and -reference verification. The default prepared layout is `../nii_data/`, matching -the shared CSV. UNet and DynUNet use MONAI. SegMamba is optional; the install -commands are in `SEGMAMBA_INSTALL_HINT` in `workflow/models.py`, and training prints -them when SegMamba is requested but not installed. +UNet and DynUNet use MONAI. SegMamba is optional; training prints the install commands +(`SEGMAMBA_INSTALL_HINT` in `workflow/models.py`) when it is requested but missing. ## Training -Use the CLI for unattended training. Run the commands below from -`research/vestibular_schwannoma/`; relative data and output paths resolve from that -project directory: +Run from `research/vestibular_schwannoma/`. The default trains UNet, DynUNet and SegMamba on +all five folds for 500 epochs; see `--help` for all options. ```bash -python scripts/train_5fold.py --models unet --folds 1 --epochs 5 --no-compile -python scripts/train_5fold.py --models unet # One model, all five folds -python scripts/train_5fold.py --models dynunet_small dynunet_xs +python scripts/train_5fold.py --models unet --folds 1 --epochs 5 --no-compile # quick test +python scripts/train_5fold.py --models dynunet python scripts/train_5fold.py --skip-unavailable ``` -The default is three models, five folds, and 500 epochs. Models and folds run sequentially -within a launcher. Run `python scripts/train_5fold.py --help` for all options. - -DynUNet variants use the same architecture, deep supervision, patch size, and training -settings. They differ only in channel widths: the existing `dynunet` uses -`[32, 64, 128, 256, 320]`, `dynunet_small` uses `[16, 32, 64, 128, 160]`, and -`dynunet_xs` uses `[8, 16, 32, 64, 80]`. +`dynunet_small` and `dynunet_xs` are DynUNet with half and a quarter of the channels. -For one process per GPU, assign one visible GPU and a new `--results-root` to each process: +To use several GPUs, run one fold per process, each with its own `--results-root`, then merge: ```bash CUDA_VISIBLE_DEVICES=0 python scripts/train_5fold.py --models unet --folds 1 --results-root cv_results/unet_fold_1 & -CUDA_VISIBLE_DEVICES=1 python scripts/train_5fold.py --models unet --folds 2 --results-root cv_results/unet_fold_2 & -CUDA_VISIBLE_DEVICES=2 python scripts/train_5fold.py --models unet --folds 3 --results-root cv_results/unet_fold_3 & -CUDA_VISIBLE_DEVICES=3 python scripts/train_5fold.py --models unet --folds 4 --results-root cv_results/unet_fold_4 & -CUDA_VISIBLE_DEVICES=4 python scripts/train_5fold.py --models unet --folds 5 --results-root cv_results/unet_fold_5 & +# ... folds 2-5 on GPUs 1-4 wait +python scripts/merge_parallel_folds.py cv_results/unet_fold_{1..5}/completed_run_ids.json \ + --model unet --output-root cv_results/unet_5fold_merged ``` -Populate `preprocessed/` with one process before launching parallel jobs; concurrent initial -cache creation is unsupported. Each process sees its assigned GPU as CUDA device 0. - -All jobs must share one MLflow tracking store. Jobs from the same checkout do this -automatically when `MLFLOW_TRACKING_URI` is unset. For multiple machines, set the same remote -URI in every shell: - -```bash -export MLFLOW_TRACKING_URI=http://mlflow.example:5000 -``` - -Never merge run IDs from different tracking stores. - -Folds trained on different machines, or in notebook 01 and with the script, can be merged -if every machine prepared the data with `scripts/prepare_data.py`, `verify` passed, and all -jobs use the same commit, conda environment and training settings, including -`--queue-workers` and `--queue-length`. - -## Merge parallel folds - -Each launcher updates `completed_run_ids.json` after every successful fold. A subset run is not -inference-ready. Merge disjoint registries into a new results root; the merger requires folds -1-5, a matching training contract, distinct run IDs, no overlaps, and a new output root: - -```bash -python scripts/merge_parallel_folds.py \ - cv_results/unet_fold_1/completed_run_ids.json \ - cv_results/unet_fold_2/completed_run_ids.json \ - cv_results/unet_fold_3/completed_run_ids.json \ - cv_results/unet_fold_4/completed_run_ids.json \ - cv_results/unet_fold_5/completed_run_ids.json \ - --model unet \ - --output-root cv_results/unet_5fold_merged -``` - -Training and merging fail if their output directory already exists; they never overwrite -weights or manifests. For example, replace an existing `cv_results/unet_fold_1` like this: - -```bash -python scripts/train_5fold.py \ - --models unet \ - --folds 1 \ - --results-root cv_results/unet_fold_1_retrained_20260902 - -python scripts/merge_parallel_folds.py \ - cv_results/unet_fold_1_retrained_20260902/completed_run_ids.json \ - cv_results/unet_fold_2/completed_run_ids.json \ - cv_results/unet_fold_3/completed_run_ids.json \ - cv_results/unet_fold_4/completed_run_ids.json \ - cv_results/unet_fold_5/completed_run_ids.json \ - --model unet \ - --output-root cv_results/unet_5fold_merged_20260902 -``` - -Contracts must match. The original directory and MLflow run remain unchanged. - -## Inference and artifacts - -- Cross-validation uses fixed folds. Optional all-data training keeps every case in training; - its duplicated validation case is only an internal fastai monitor. -- Inference loads declared models from `cv_results//inference_run_ids.json` and - enforces their preprocessing and output contracts. -- Evaluation and inference use TTA by default and preserve every predicted region. Predictions - require clinical review. -- Fold checkpoints live under `//fold_/checkpoints/`. They support - warm-starting, not exact resume. MLflow stores the final/best artifacts; inference and - deployment strict-load `.safetensors` files. -- Generated data, results, tracking stores, checkpoints, and bundles are excluded from Git. - -For container preparation and execution, see -[deployment/pacs/README.md](deployment/pacs/README.md). +- Fill `preprocessed/` with a single process first; parallel jobs can't build the cache. +- All jobs must use the same MLflow tracking store. From one checkout this is automatic; across + machines set the same `MLFLOW_TRACKING_URI` everywhere. +- Folds from different machines, or from notebook 01 and the script, merge if every machine + prepared and verified the data with `scripts/prepare_data.py` and all jobs use the same + commit, conda environment and training settings, including `--queue-workers` and + `--queue-length`. +- Training and merging never overwrite an existing output folder. To replace a fold, train it + into a new folder and merge again. + +## Inference and results + +- Inference loads the models listed in `cv_results//inference_run_ids.json` and applies + their preprocessing. TTA is on by default and every predicted region is kept. Predictions + need clinical review. +- Fold checkpoints are in `//fold_/checkpoints/` (warm start only). + MLflow stores the final and best weights; inference and deployment load `.safetensors`. +- Data, results, tracking stores, checkpoints and bundles are not in Git. ## Citation and license -This project is part of fastMONAI. Use the parent repository's `CITATION.cff` and Apache-2.0 -license, and cite the originating datasets under their applicable terms. +Part of fastMONAI: use its `CITATION.cff` and Apache-2.0 license, and cite the datasets under +their own terms. diff --git a/research/vestibular_schwannoma/data/README.md b/research/vestibular_schwannoma/data/README.md index 96b11b2..a016b07 100644 --- a/research/vestibular_schwannoma/data/README.md +++ b/research/vestibular_schwannoma/data/README.md @@ -1,132 +1,53 @@ # Reproduce the study dataset -The shared `ml_dataset.csv` defines **344 native-space CE-T1 scans and tumour -masks**, their order, volumes, quartile labels, and fixed cross-validation folds. -`reference_dataset.json` fingerprints the current study data. Preparation must -match those voxel values and geometry before publishing an output directory. +`ml_dataset.csv` defines the **344 native-space CE-T1 scans and tumour masks**: their order, +volumes, quartile labels and fixed cross-validation folds. `reference_dataset.json` +fingerprints the data; preparation must match it exactly. | Source | Public cases | Study cases | Selection | | --- | ---: | ---: | --- | -| Queen Square / TCIA Vestibular-Schwannoma-SEG, version 2 | 242 | 239 | Exclude `vs_gk_8`, `vs_gk_73`, and `vs_gk_131` | +| Queen Square / TCIA Vestibular-Schwannoma-SEG, version 2 | 242 | 239 | Excludes `vs_gk_73` (meningioma), and `vs_gk_8` and `vs_gk_131` (radiologist review, commit `d8bd2de`) | | crossMoDA 2022 / Tilburg | 105 | 105 | `training_source`, `center == Tilburg`, CE-T1 only | -`vs_gk_73` is a meningioma, as recorded in the historical combined-data notebook. -Cases 8 and 131 were removed following radiologist review in fastMONAI commit -`d8bd2de`; the specific findings are not recorded there. The eight other gaps in -Queen Square numbering are already absent from the public source. crossMoDA -London cases overlap Queen Square and are omitted. Target/validation images, -T2 images, cochlea labels, backup masks, HUS data, and other collections are -not included in the prepared study directory. +crossMoDA London cases overlap Queen Square and are left out, as are T2, target/validation +images, cochlea labels and HUS data. ## Setup -Run these commands from `research/vestibular_schwannoma/`. Python 3.10+ and a CPU -are sufficient; preparation does not import fastMONAI or PyTorch or require a GPU. +Run from `research/vestibular_schwannoma/`. Python 3.10+ and a CPU are enough; fastMONAI and +PyTorch are not needed. Allow 80-100 GB of free disk. ```bash pip install -r data/requirements.txt -python scripts/prepare_data.py --help -``` - -Allow roughly 80–100 GB free for downloads, extraction, and prepared copies. -Downloaded sources remain untouched. Existing output directories are refused; -use a new `--output` directory for another run. - -## 1. Download by script - -The default DICOM downloader is the pip package -[`idc-index`](https://github.com/ImagingDataCommons/idc-index), installed by -`data/requirements.txt`. The same TCIA collection is available in -[NCI Imaging Data Commons (IDC)](https://portal.imaging.datacommons.cancer.gov/collections/vestibular_schwannoma_seg/). -No Data Retriever installation, browser download, account, or cloud credentials -are needed. The tested package version is pinned to `0.12.5` with IDC v24 metadata. - -```bash -python scripts/prepare_data.py download -``` - -This downloads and checksum-verifies the approximately 7.8 GB -[crossMoDA 2022 training archive](https://zenodo.org/records/6504722), extracts it, -downloads the official TCIA manifest and support files, and downloads the **239 -Queen Square T1 MR series** from IDC (29,400 DICOM instances, approximately 15.5 GB). -The shared CSV selects cases; T2, radiation-therapy series, and the three excluded -cases are omitted from IDC downloads. Missing or ambiguous T1 series abort before -transferring data. Re-running resumes/reuses cached downloads. IDC's sync compares -existing files with remote sizes; preparation then checks exact voxel hashes and -geometry. `queen_square/idc_manifest.json` records the CSV checksum, IDC and package -versions, selected series UIDs, expected instance counts, and completion status. - -Sources can be acquired separately: - -```bash -python scripts/prepare_data.py download --dataset tilburg -python scripts/prepare_data.py download --dataset queen-square -``` - -Expected layout: - -```text -data/raw/ -├── queen_square/ -│ ├── assets/ # manifest, contours, matrices, mapping -│ ├── idc_manifest.json # selected T1 series and download provenance -│ └── dicom/VS-SEG-001//MR_/... -├── crossmoda2022/ -│ ├── crossmoda2022_training.zip -│ └── crossmoda2022_training/ -│ ├── infos_source_training.csv -│ ├── training_source/ -│ └── training_target/ # downloaded but not used -└── corrected_labels/ # extracted from the bundled 17-mask ZIP +python scripts/prepare_data.py download # or --dataset tilburg | queen-square | labels +python scripts/prepare_data.py prepare +python scripts/prepare_data.py verify ``` -The 17 replacement masks are included in this repository as -[`corrected_labels/vs_corrected_labels_v1.zip`](corrected_labels/vs_corrected_labels_v1.zip). -`download` checks and extracts this archive automatically; no separate hosting -account or download is needed. Use `download --dataset labels` to extract just -the masks. The ZIP contains its own case mapping and provenance manifest; -preparation verifies every replacement mask against `reference_dataset.json`. +## Download -## Alternatives: TCIA Data Retriever CLI or graphical app +`download` fetches the 7.8 GB [crossMoDA 2022 training archive](https://zenodo.org/records/6504722) +and the 239 Queen Square T1 series (about 15.5 GB) from +[NCI Imaging Data Commons](https://portal.imaging.datacommons.cancer.gov/collections/vestibular_schwannoma_seg/) +with `idc-index` 0.12.5 (IDC v24). No account or credentials are needed. Downloads are +checksum-verified and resume when rerun. Files go to `data/raw/`; existing sources are never +changed. -To use the current [TCIA Data Retriever](https://github.com/TCIA/data-retriever) -binary instead of IDC: +To use the [TCIA Data Retriever](https://github.com/TCIA/data-retriever) instead: ```bash python scripts/prepare_data.py download --dataset queen-square \ --tcia-backend retriever --retriever /path/to/TCIA_Data_Retriever ``` -This invokes `--cli`, `-i`, `-o`, `--skip-existing`, -`--directory-mode descriptive`, and `--accept-data-policy` to download the full -collection manifest. Providing `--retriever` also selects this backend when -`--tcia-backend` is omitted. The older Java NBIA Data Retriever uses different CLI -flags and is not interchangeable with this executable. - -Download DICOM using the collection manifest and Data Retriever from the -[TCIA collection page](https://www.cancerimagingarchive.net/collection/vestibular-schwannoma-seg/). -Classic and descriptive directory names are supported. Download support files: - -```bash -python scripts/prepare_data.py download --dataset queen-square --assets-only -``` - -Supply the DICOM collection with `prepare --queen-square -/path/to/Vestibular-Schwannoma-SEG`. DICOM metadata selects the T1 MR series; -radiation-therapy files are not used as MR images. - -## 2. Prepare and verify - -For a fresh checkout without an existing `../nii_data/` directory: +For a manual TCIA download, fetch the support files with +`download --dataset queen-square --assets-only` and pass the collection to +`prepare --queen-square /path/to/Vestibular-Schwannoma-SEG`. -```bash -python scripts/prepare_data.py prepare -python scripts/prepare_data.py verify -``` +## Prepare and verify -Default output is `../nii_data/`, matching the shared CSV and training notebook. -Alternate output writes its own CSV with updated paths and unchanged case order, -folds, volumes, and quartiles: +`prepare` writes to `../nii_data/` by default, matching the CSV. Another `--output` gets its +own CSV with the new paths and the same folds: ```bash python scripts/prepare_data.py prepare --output data/prepared @@ -134,23 +55,11 @@ python scripts/prepare_data.py verify --data-root data/prepared python scripts/train_5fold.py --data-csv data/prepared/ml_dataset.csv --models unet ``` -CLI paths resolve relative to this project, even when invoked from elsewhere. -The bundled masks are used automatically if they have not already been extracted. -`--workers` controls CPU parallelism; default 2. To reuse the existing VS_Seg data: - -```bash -python scripts/prepare_data.py prepare \ - --from-existing /path/to/VS_Seg \ - --output data/prepared -``` +`prepare --from-existing /path/to/VS_Seg` reuses an existing VS_Seg checkout, and `--workers` +sets the CPU parallelism (default 2). Existing output folders are refused. -The shortcut uses that checkout's `nii_data/queen_square_data`, `crossmoda_data`, -and `errors_fixed_nl`. Individual `--queen-square-nifti`, `--crossmoda`, and -`--corrected-labels` options accept equivalent data elsewhere. - -Preparation selects only shared CSV cases, converts native Queen Square T1 DICOM, -applies 17 NL masks, retains Tilburg tumour label 1, and removes only these -verified six-neighbour components: +Preparation converts the Queen Square T1 DICOM, applies the 17 corrected masks, keeps +Tilburg tumour label 1, and removes only these verified stray components: | Case | Kept voxels | Removed voxels | | --- | ---: | ---: | @@ -163,84 +72,39 @@ verified six-neighbour components: | `crossmoda2022_etz_38` | 3616 | 1 | | `crossmoda2022_etz_67` | 3568 | 1 | -Unexpected signatures fail. Other masks retain every tumour region. The final -inventory is **344 images + 344 masks**, a manifest, and a generated CSV. Each -pair must match reference voxel fingerprints, geometry, and CSV volume. Missing -corrections or mismatched data abort and remove staged partial output. - -Queen Square rasterization preserves the historical SimpleITK conversion, -rounded contour coordinates, tumour-name selection, and polygon filling used -in this study. It is not a general-purpose converter. Substituting Slicer or -changing rasterization requires a new dataset version and reference. Acquisition -does not register or resample; training preprocessing remains separate. - -Fingerprints hash C-order uint16 images and uint8 integer masks. Compression and -mask storage dtype do not affect them; affine and spacing are checked separately. -The historical encoding decodes label 1 as `0.9999999997671694` in 103 Tilburg -masks. Integer rounding within `1e-6` accounts for this encoding; actual fractional -labels are rejected. - -## Sharing the 17 replacement masks - -Correction cases: **4, 5, 29, 35, 43, 44, 45, 47, 55, 56, 61, 63, 71, 74, 76, -96, 119**. These are new NL masks drawn for scans whose original tumour -annotations contained errors. +Every other mask keeps all its regions. The result is 344 images, 344 masks, a manifest and a +CSV, each pair checked against the reference voxel hashes, geometry and volume. Any mismatch +aborts and removes the partial output. -Contributors: **Njål Lura** and **Satheshkumar Kaliyugarasan**. -The release uses **CC BY 4.0** and asks readers to cite the accompanying study -paper when available. +The Queen Square conversion reproduces the study's historical SimpleITK rasterization; it is +not a general converter, and changing it needs a new dataset version. Fingerprints hash +C-order uint16 images and uint8 masks, so compression and storage type don't matter. -The [`corrected_labels/`](corrected_labels/) folder contains one ZIP, -a [short README](corrected_labels/README.md), and a [license notice](corrected_labels/LICENSE.txt). -These small masks are versioned with the preparation code on GitHub. Model -weights are not part of this release. Use a tagged repository -release to identify the code, case index, and masks used for a study. - -Maintainer packaging command: - -```bash -python scripts/prepare_data.py bundle-labels --data-root ../nii_data -``` +## Corrected masks -The ZIP contains 17 masks, a TCIA case mapping, provenance manifest, README, -and license. Fresh NIfTI headers omit annotation-editor text and extensions -while preserving geometry and tumour voxels. Existing bundles are refused; -package an explicitly reviewed future release under a new filename. +`corrected_labels/vs_corrected_labels_v1.zip` holds 17 new masks for Queen Square cases 4, 5, +29, 35, 43, 44, 45, 47, 55, 56, 61, 63, 71, 74, 76, 96 and 119, whose original annotations +contained errors. Drawn by Njål Lura and Satheshkumar Kaliyugarasan, licensed CC BY 4.0 +([README](corrected_labels/README.md)). `download` extracts the ZIP automatically. -Original Queen Square data are -[CC BY 4.0](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=70229053); -crossMoDA data are [CC BY-NC-SA 4.0](https://zenodo.org/records/6504722). Their -licenses are separate from the Apache-2.0 software license. This release contains -new tumour masks for Queen Square images. +The ZIP is rebuilt with `python scripts/prepare_data.py bundle-labels --data-root ../nii_data`; +an existing bundle is never overwritten. Queen Square data are +[CC BY 4.0](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=70229053) and +crossMoDA data [CC BY-NC-SA 4.0](https://zenodo.org/records/6504722), separate from the +Apache-2.0 code license. -## Maintaining the reference +## Reference -The reference was frozen from the current dataset. Do not regenerate it to make -mismatched preparation pass. Fingerprint an explicitly reviewed future version -using a new path: +Don't regenerate the reference to make a mismatch pass. Fingerprint a reviewed new version to a +new file: ```bash python scripts/prepare_data.py freeze-reference \ --data-root /path/to/reviewed/data --output data/reference_dataset_v2.json ``` -Audit indexed pairs in mixed legacy storage with `verify --data-root /path/to/data ---allow-extra-files`. Normal preparation/verification enforces exact inventory. - -Download helpers were adapted from `VS_Seg/preprocess_data`; conversion reproduces -`VS_Seg/preprocessing/simple_data_conversion.py` (Apache-2.0). -Native preparation from the original local TCIA/crossMoDA downloads was checked -against all 344 reference pairs, including the 17-label ZIP and all eight -component cleanups. Standalone verification also checked the exact 688-file -inventory and the generated CSV's unchanged folds and metadata. - -The pip-installed IDC route (`idc-index==0.12.5`, IDC v24) was tested with live -downloads of cases 1, 4, and 12: 360 DICOM instances, approximately 190 MB. All -three converted images and final masks matched the frozen reference's voxel -hashes, geometry, and CSV volumes, including case 4's human correction and case -12's 79-voxel cleanup. Removing a downloaded test slice and rerunning restored -it with the same SHA-256. IDC metadata covered all 239 study T1 series; the full -15.5 GB cohort was not redownloaded for this check. +All 344 pairs were verified from the original downloads. The IDC route was tested on cases 1, +4 and 12. Tests: ```bash python -m pytest -q tests/workflow/test_data_preparation.py \ diff --git a/research/vestibular_schwannoma/deployment/pacs/README.md b/research/vestibular_schwannoma/deployment/pacs/README.md index 0732822..f09dc7c 100644 --- a/research/vestibular_schwannoma/deployment/pacs/README.md +++ b/research/vestibular_schwannoma/deployment/pacs/README.md @@ -1,76 +1,47 @@ -# Vestibular Schwannoma ROR/PACS Container - -Build the CE-T1w vestibular schwannoma container with the five-fold DynUNet ensemble. The -UNet bundle was retired in October 2026 because it had more false positives than DynUNet. -Multiple Safetensors members run concurrently in separate CPU workers with three -threads per model. Five-fold deployment therefore requires at least 15 available CPU -threads; the qualified target has 16. TTA is enabled by default and averages two passes, the -original and one copy flipped anterior-posterior and inferior-superior together (`TTA_FLIP_AXES` -in `parallel_inference.py`), as in the nnU-Net container. On the 344 held-out cases this gave -about the same false positives as all 8 flips at a quarter of the cost. - -See [Research PACS deployment with ROR](../../../RESEARCH_PACS_DEPLOYMENT.md) -for the general build, qualification, export, and handoff workflow. - -## Prepare model bundles - -Use the environment pinned in `requirements.yml` to prepare bundles. Strict model -metadata checks also depend on the PyTorch version; matching fastMONAI alone is -insufficient. Provide complete MLflow run IDs. - -Five-fold ensemble: - -```bash -python prepare_model_bundle.py \ - --model-type dynunet \ - --run "fold_1=${FOLD_1_RUN_ID}" \ - --run "fold_2=${FOLD_2_RUN_ID}" \ - --run "fold_3=${FOLD_3_RUN_ID}" \ - --run "fold_4=${FOLD_4_RUN_ID}" \ - --run "fold_5=${FOLD_5_RUN_ID}" \ - --artifact-role best -``` +# Vestibular schwannoma PACS container -Repeat `--run MEMBER=RUN_ID` for other ensemble sizes, or use -`--artifact MEMBER=/path/model.safetensors` for a local artifact. The builder validates and -strict-loads each member into the ignored `model_bundles//` directory. +ROR/PACS container that runs the five-fold DynUNet ensemble on a CE-T1w series. CPU only: +each of the five models runs in its own worker with three threads, so it needs at least 15 +CPU threads. TTA is on by default and averages the original and one copy flipped +anterior-posterior and inferior-superior (`TTA_FLIP_AXES` in `parallel_inference.py`), as in +the nnU-Net container. For the general workflow see +[Research PACS deployment with ROR](../../../RESEARCH_PACS_DEPLOYMENT.md). -The intermediate `model_mask` and `vote_map` series use deterministic `2.25` UIDs. -Published series keep pr2mask's UIDs. +## Build a release -## Test and build +1. Prepare the model bundle in the environment pinned in `requirements.yml` (the strict + metadata check depends on the PyTorch version), with full MLflow run IDs: -Run the deployment tests first (the test discovery runs from `research/`): + ```bash + python prepare_model_bundle.py --model-type dynunet --artifact-role best \ + --run "fold_1=$RUN_1" --run "fold_2=$RUN_2" --run "fold_3=$RUN_3" \ + --run "fold_4=$RUN_4" --run "fold_5=$RUN_5" + ``` -```bash -conda activate fastmonai -(cd ../../.. && python -m unittest discover -s vestibular_schwannoma/tests/deployment -t .) -bash -n entrypoint.sh -``` + The bundle goes to `model_bundles/dynunet/` (not in Git). `--artifact MEMBER=/path.safetensors` + adds a local file instead of a run. A new bundle must be registered in + `redcap_model_instances.json` (see below) or the container refuses to run. -Build dated and `latest` tags for the same image: +2. Test (from `research/`) and build: -```bash -BUILD_VERSION="$(date -u +%Y%m%dT%H%M%SZ)" + ```bash + (cd ../../.. && python -m unittest discover -s vestibular_schwannoma/tests/deployment -t .) + BUILD_VERSION="$(date -u +%Y%m%dT%H%M%SZ)" + docker build --pull --build-arg VERSION="$BUILD_VERSION" -f .ror/virt/Dockerfile \ + -t "fastmonai-vsseg:$BUILD_VERSION" -t fastmonai-vsseg:latest . + ``` -docker build --pull \ - --build-arg VERSION="$BUILD_VERSION" \ - -f .ror/virt/Dockerfile \ - -t "fastmonai-vsseg:$BUILD_VERSION" \ - -t "fastmonai-vsseg:latest" \ - . -``` + Qualify on test data with `ror trigger -cont "fastmonai-vsseg:$BUILD_VERSION" -each -keep`. -Qualify and deliver the dated tag: +3. Package both tags in `release_packages/$BUILD_VERSION/` with a short README and + `ror_select_entry.txt`. Don't add a `.sha256`; the image ID after `docker load` is the check. -```bash -ror trigger -cont "fastmonai-vsseg:$BUILD_VERSION" -each -keep \ - -envs '{"model-type":"dynunet","tta":true}' -``` + ```bash + docker save "fastmonai-vsseg:$BUILD_VERSION" fastmonai-vsseg:latest \ + | gzip > "release_packages/$BUILD_VERSION/fastmonai-vsseg-$BUILD_VERSION.tar.gz" + ``` -As a ror workflow, add this entry to the select statements. Release packages also -include it as `ror_select_entry.txt`. Replace `ENTRY_NAME` with the workflow `name` -in the PACS `config.json` and fill in the series selector: +The PACS entry uses `fastmonai-vsseg:latest`, so loading a new archive is enough: ```json "ENTRY_NAME": { @@ -80,99 +51,61 @@ in the PACS `config.json` and fill in the series selector: }, ``` -The entry uses `latest`, so a new release needs no config change. The release archive -carries both tags, and loading it moves `latest` to the new image. The dated tag shows -which release `latest` is. - -`"{}"` runs the defaults. Use `"{\"tta\":false}"` to turn TTA off. Don't write `"\"\""` -here: that passes two quote characters and is rejected. - -## Runtime contract - -`ROR_CONT_OPTIONS` accepts: - -- `model-type`: `dynunet` (the default and only model). -- `tta`: JSON Boolean, default `true`. - -Unknown keys and invalid values fail before inference. Header preflight rejects -inconsistent Study, Series, SOP, modality, or geometry data; nonstandard source UIDs and -missing optional Frame of Reference metadata produce aggregated warnings. - -The runtime writes intermediate `model_mask` and `vote_map` DICOM series. Fiona's `pr2mask` -reads them and creates `fused`, `fused_vote_map`, `reports`, and its own mask series in -`labels/`, using pr2mask's folder layout. fastMONAI provenance (SeriesDescription, -SoftwareVersions, DerivationDescription, and a `MASK` ImageType value) is added to the -`labels` series before it is published. Its study identity, window, and UIDs stay as pr2mask -wrote them, so the PACS files the mask with the other pr2mask outputs. `model_mask` and -`vote_map` are not published. Vote-map probabilities are `round(probability x 65535)`. -pr2mask draws values from 32768 up in blue, so values inside the mask are raised to at -least 32768 and values outside it lowered to at most 32767: the blue region is exactly the -mask, and yellow marks voxels with some foreground probability outside it. Existing `fused`, `fused_vote_map`, `reports`, `labels`, and `redcap` directories are rejected -to prevent overwrites. - -### REDCap output - -The final output includes pr2mask's `redcap//output.json` and -`output_data_dictionary.zip`, in pr2mask's EAV row format and instrument. `redcap` is an -owned output directory and must not already exist. There is no direct REDCap uploader. - -pr2mask splits the mask into connected regions (26-connected) and numbers them 1, 2, 3, ... -in scan order. The exporter writes each region to its own repeat instance of the `pr2mask` -instrument: region k goes to the model's block base + k. DynUNet's base is 200, so a mask -with one region fills instance 201, and a mask with two regions fills 201 and 202. Instance -200 + k always holds pr2mask's "label k of N" (`mask_label_info`). Only instances that have a -region are written. An empty mask fills instance 201 with blank pr2mask columns, which records -that the model ran and found nothing. It is routed with the source PatientID and -ReferringPhysicianName (`EventName:` prefix removed), matching pr2mask routing. - -Every written instance has all of pr2mask's columns plus `model_version`, for example -`dynunet 5-fold, fastmonai-vsseg:20261110T081500Z`. pr2mask itself has no such field. It is on -the REDCap form, and the exporter adds it to `output_data_dictionary.zip` as a text field. If -pr2mask ever writes its own `model_version`, the exporter's value replaces it. The mask itself -goes to the PACS as the `labels` series, whose DICOM header also records the full bundle hash -and the TTA setting. - -pr2mask stores each region's mask in seven `mask_*` fields. `mask_base64_gzip_compressed` -covers the whole scan at 16 bits per voxel, so its length depends on the scan size, not the -region: about 84,000 characters for a 512×512×120 scan and 105,000 for 512×448×176. REDCap may -cut values longer than 65,535 characters; this has not been checked in the receiving project. -The exporter sends the value as pr2mask wrote it, and the full mask is always in the PACS -`labels` series. - -`pr2mask_dictionary.csv` holds the `instrument.csv` rows of a non-empty run of the packaged -pr2mask. An empty mask, where pr2mask lists no columns, takes them from there. When the Fiona -base changes, regenerate it from a smoke test and check that a non-empty run lists the same -columns. - -Up to 20261026T134845Z the exporter also sent eight `vs_*` fields, among them the mask as -`vs_mask_json`. They are no longer sent. - -### Stable model destinations and overwrites - -`redcap_model_instances.json` gives each full bundle SHA-256 a permanent block of 100 -instances, stored as the block's base (a multiple of 100, at least 100). The DynUNet bundle -has block 200 (instances 201-299). Instances below 100 belong to the original (manual) mask -at 1, pr2mask's own region numbering and the earlier one-instance-per-model scheme: the nnU-Net -large container in the separate nnU-Net repository still writes its largest region to 10. -DynUNet wrote its largest region to 11 up to 20261027T083045Z (3 before 2026-10-21; the -retired UNet used 2); those results are no longer updated. Block 100 is planned for nnU-Net. -Keep blocks unique across both projects, and don't renumber or reuse them after REDCap -imports. Register each new bundle with an unused block before building a release. Unknown -bundles, duplicate blocks, invalid bases and region numbers of 100 or more fail closed. The -registry is copied into the container and validated before inference. - -Within the same record/event: - -- The same bundle overwrites its earlier instances, including when TTA, deployment version, - timestamp or the predicted mask changes. A rerun that finds fewer regions leaves the higher - instances of the earlier run as they were. The same bundle with the same TTA setting finds - the same regions. -- A different bundle uses a different block, including different weights of the same - architecture. - -Install pr2mask's dictionary plus `model_version` in the receiving REDCap project. The import -of the one-instance release 20261027T083045Z has been confirmed; confirm instances 201+ and -`model_version` with a PACS-to-REDCap round trip. The importer must honor the supplied numeric -repeat instance and update existing values. The record/event must already identify the -intended case/scan; multiple scans routed to the same record/event and model intentionally -share the same instances. +## Options + +`ROR_CONT_OPTIONS` accepts `model-type` (`dynunet`, the default and only model) and `tta` +(JSON Boolean, default `true`). `"{}"` runs the defaults and `"{\"tta\":false}"` turns TTA +off. Unknown keys and invalid values fail before inference, as do inconsistent study, series, +SOP, modality or geometry headers. + +## Outputs + +The container writes two intermediate series, `model_mask` and `vote_map`, which pr2mask +turns into the published outputs: + +- `fused`: the mask drawn on the MR. +- `fused_vote_map`: the vote map on the MR. Blue is exactly the mask; yellow marks voxels + outside it with some foreground probability. +- `reports`: pr2mask's report, with measurements per region. +- `labels`: the mask series, in pr2mask's study, window and UIDs, plus our provenance + (SeriesDescription, SoftwareVersions, DerivationDescription with the bundle hash and TTA, and a + `MASK` ImageType value). +- `redcap`: pr2mask's REDCap rows, see below. + +These folders must not exist in the output folder beforehand. + +## REDCap + +`redcap//output.json` holds pr2mask's EAV rows for the `pr2mask` instrument. +There is no direct uploader; the PACS imports the file. + +- **One instance per region.** pr2mask numbers the mask's connected regions 1, 2, 3, ... + (26-connected, scan order). Region k goes to repeat instance base + k, where the base comes + from `redcap_model_instances.json`. DynUNet's base is 200, so its regions fill 201, 202, ... + Only instances with a region are written. +- **Empty mask:** instance 201 with blank columns, so REDCap shows the model ran. It is routed + by the source PatientID and ReferringPhysicianName (`EventName:` removed), like pr2mask. +- **`model_version`** is added to every instance, for example + `dynunet 5-fold, fastmonai-vsseg:20261110T082650Z`. The field is on the REDCap form, not in + pr2mask; the exporter also adds it to `output_data_dictionary.zip`. +- **Mask text.** pr2mask's `mask_base64_gzip_compressed` covers the whole scan at 16 bits per + voxel: about 84,000 characters at 512x512x120 and 105,000 at 512x448x176. REDCap may cut + values over 65,535 characters; this hasn't been checked in the receiving project. The value + is sent as pr2mask wrote it, and the full mask is always in `labels`. +- **Empty pr2mask dictionary.** For an empty mask pr2mask lists no columns, so the exporter + takes them from `pr2mask_dictionary.csv`. Regenerate that file from a non-empty run when + the Fiona base changes pr2mask's columns. + +### Instance registry + +`redcap_model_instances.json` gives each bundle hash a permanent block of 100 instances, +stored as the base (a multiple of 100). DynUNet `c5642793...` has 200. Instance 1 is the manual +mask; nnU-Net large still writes its largest region to 10, and block 100 is planned for it. +Older DynUNet results at 11 (and 2/3 before that) are no longer updated. + +- A rerun of the same bundle overwrites its instances. If it finds fewer regions, the higher + instances of the earlier run stay. +- New weights mean a new bundle hash. Give it an unused block, or replace the old entry to + overwrite the old model's results. +- Never renumber blocks after REDCap imports. Unknown bundles, duplicate or invalid bases and + region numbers of 100 or more fail before any output is published.