diff --git a/.claude/skills/mblt-vision/SKILL.md b/.claude/skills/mblt-vision/SKILL.md index 4c1974b..d326291 100644 --- a/.claude/skills/mblt-vision/SKILL.md +++ b/.claude/skills/mblt-vision/SKILL.md @@ -74,10 +74,13 @@ description: >- the shared instance attribute). - YOLOX and DAMO-YOLO dispatch on `post_cfg.head` (`yolox` / `damoyolo`) and reuse the anchorless filter and NMS; only their decode is new (no half-cell offset; DAMO `reg_max: 16` - is 17 bins; DAMO head layout resolved jointly). Both letterbox top-left - (`LetterBox.center: false`, padding 114 / 0), take unscaled input (no Normalize), and YOLOX - reads BGR (`Reader.color_mode`). Shape-only geometry must follow `letterbox_center(pre_cfg)`. - Configs follow mblt-model-ops `jm/temp` `pipeline.yaml`; YAMLs are `local_artifact_only`. + is 17 bins; DAMO head layout resolved jointly). Upstream's test transform is the source of + truth for `pre_cfg`: both letterbox top-left (`LetterBox.center: false`, padding 114 / 0) + with upstream's truncated `int(w * r)` size (`size_rounding: floor`) and take unscaled input + (no Normalize); YOLOX reads cv2 BGR (`Reader.color_mode`) and restores boxes by `/ r`; + DAMO-YOLO reads PIL RGB (`Reader.style: pil`, which COCO evaluation honours through + `CustomCOCODataset(decoder=)`) and restores each axis separately (`per_axis_ratio: true`). Shape-only geometry must follow `letterbox_layout(pre_cfg)`. + `post_cfg` follows mblt-model-ops `jm/temp` `pipeline.yaml`; YAMLs are `local_artifact_only`. - Rank WiderFace evaluation by Hard-set AP. Expose Medium-set then Easy-set AP as secondary metrics, and do not compute mean AP across difficulty splits. - Treat face_detection as a single-class WiderFace task. Each YOLO head family gets a thin diff --git a/AGENTS.md b/AGENTS.md index 813ada5..f22e4c7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -107,15 +107,24 @@ The current ownership boundary is deliberate: bins -- decoded as the softmax expectation times stride. Neither adds Ultralytics' half-cell offset. A 640 input's stride-8 DAMO class map is 80x80x80, so the head set's layout (NCHW or NHWC) is resolved jointly from the unambiguous distribution maps and a mixed set fails. -- Their `pre_cfg` follows mblt-model-ops' `models//pipeline.yaml` on branch - `jm/temp` (commit `0368367e8`): YOLOX is BGR (`Reader.color_mode: BGR`), top-left - letterboxed with 114; DAMO-YOLO is RGB, top-left with zeros (the December 2022 checkpoints' - geometry, not upstream HEAD's stretch). Neither declares `Normalize`: both take the - unscaled 0-255 image, and the ONNX path casts the byte tensor to the graph's float dtype. - `LetterBox.center: false` anchors top-left; every place that derives geometry from shapes - alone (`PostBase.ratio_pads_for`, `Results` plotting) must use the model's own - `letterbox_center(pre_cfg)` rather than the centered default. No Hub repository exists - yet, so their YAMLs keep `file_cfg.local_artifact_only: true`. +- Their `pre_cfg` is upstream's own test transform, which is the source of truth (their + `post_cfg` follows mblt-model-ops' `models//pipeline.yaml` on branch `jm/temp`, + commit `0368367e8`). YOLOX is `yolox/data/data_augment.py:preproc`: a cv2 BGR image + (`Reader.color_mode: BGR`), resized to `int(w * r) x int(h * r)`, top-left on a 114 canvas, + boxes restored by `/ r`. DAMO-YOLO is the December 2022 release's `Resize` and + `to_image_list` (tinyvision/DAMO-YOLO `55ae14f`; not upstream HEAD's stretch): a PIL RGB + decode (`Reader.style: pil`), the same truncated resize, top-left with zeros, and boxes + restored per axis by `BoxList.resize`. COCO evaluation decodes with the same library as the + model's `Reader` (`CustomCOCODataset(decoder="pil")` for `style: pil`), because the loader + decodes before preprocessing. Hence `LetterBox.size_rounding: floor` on both and + `per_axis_ratio: true` on DAMO-YOLO; tests compare both pipelines pixel-for-pixel with + inlined copies of the upstream code. Neither declares `Normalize`: both take the unscaled + 0-255 image, and the ONNX path casts the byte tensor to the graph's float dtype. Every place + that derives geometry from shapes alone (`PostBase.ratio_pads_for`, `Results` plotting, + dense crops, semantic targets) must use the model's own `letterbox_layout(pre_cfg)` + rather than Ultralytics' default. Instance segmentation rejects `size_rounding: floor` and + `per_axis_ratio`, and OBB rejects `per_axis_ratio`, because their restorations cannot honour + them. No Hub repository exists yet, so their YAMLs keep `file_cfg.local_artifact_only: true`. - For WiderFace evaluation, rank results by Hard-set AP and retain Medium-set then Easy-set AP as secondary metrics. Do not compute a mean across splits. - The YOLOv5/YOLOv7 face repositories do not yet publish project-pinned immutable revisions diff --git a/mblt_vision/README.md b/mblt_vision/README.md index 45638f1..c006702 100644 --- a/mblt_vision/README.md +++ b/mblt_vision/README.md @@ -369,8 +369,12 @@ validation split, following Ultralytics' depth-validation convention. - YOLOX and DAMO-YOLO have no Hub artifact yet, so pass a local `model_path`. Their GPU scores are the ONNX exports of the upstream checkpoints, measured through this package (upstream publishes 40.5 for YOLOX-s and 41.8 for DAMO-YOLO-T). Neither preprocesses like - Ultralytics: both letterbox top-left and feed the unscaled 0-255 image, and YOLOX reads - BGR. DAMO-YOLO follows its December 2022 checkpoints, the only ones still downloadable. + Ultralytics: both reproduce upstream's own test transform, which letterboxes top-left with + a truncated `int(w * r)` size (`LetterBox.size_rounding: floor`) and feeds the unscaled + 0-255 image. YOLOX reads cv2 BGR and restores boxes by the one resize ratio. DAMO-YOLO + follows its December 2022 checkpoints, the only ones still downloadable: it decodes with + PIL (`Reader.style: pil`) and restores each axis by its own ratio + (`LetterBox.per_axis_ratio: true`). diff --git a/mblt_vision/__init__.py b/mblt_vision/__init__.py index a8bf94f..e56c12a 100644 --- a/mblt_vision/__init__.py +++ b/mblt_vision/__init__.py @@ -19,7 +19,7 @@ from ._api import list_tasks as list_tasks from .wrapper import MBLT_Engine as MBLT_Engine -__version__ = "0.0.5" +__version__ = "0.0.6" _TASK_MODULES = ( face_detection, diff --git a/mblt_vision/models/DAMO-YOLO-M-distill.yaml b/mblt_vision/models/DAMO-YOLO-M-distill.yaml index 1ae4fb5..223ece6 100644 --- a/mblt_vision/models/DAMO-YOLO-M-distill.yaml +++ b/mblt_vision/models/DAMO-YOLO-M-distill.yaml @@ -1,9 +1,13 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' DAMO-YOLO-M-distill/pipeline.yaml (branch jm/temp, 0368367e8): - # the downloadable checkpoints are the December 2022 release, whose test - # transform letterboxes RGB with zeros anchored top-left (41.79 mAP50-95 on - # COCO val2017, against 39.72 through upstream HEAD's stretch) and feeds the - # unscaled image -- hence no Normalize. reg_max 16 means 17 distance bins. + # pre_cfg is the test transform of upstream DAMO-YOLO's December 2022 release + # (tinyvision/DAMO-YOLO 55ae14f), the only checkpoints still downloadable: a + # PIL RGB decode, cv2 resized to int(w * r) x int(h * r) -- truncated, hence + # size_rounding: floor -- zero-padded into the top-left corner, and left + # unscaled, hence no Normalize. BoxList.resize restores each axis by its own + # original-over-resized ratio, hence per_axis_ratio. (41.79 mAP50-95 on COCO + # val2017, against 39.72 through upstream HEAD's stretch.) post_cfg follows + # mblt-model-ops' DAMO-YOLO-M-distill/pipeline.yaml (branch jm/temp, 0368367e8); + # reg_max 16 means 17 distance bins. # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/DAMO-YOLO-M-distill @@ -12,14 +16,15 @@ DEFAULT: local_artifact_only: true pre_cfg: Reader: - style: numpy - color_mode: RGB + style: pil LetterBox: img_size: - 640 - 640 center: false padding_value: 0 + size_rounding: floor + per_axis_ratio: true SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/DAMO-YOLO-M.yaml b/mblt_vision/models/DAMO-YOLO-M.yaml index b106497..9665fb1 100644 --- a/mblt_vision/models/DAMO-YOLO-M.yaml +++ b/mblt_vision/models/DAMO-YOLO-M.yaml @@ -1,9 +1,13 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' DAMO-YOLO-M/pipeline.yaml (branch jm/temp, 0368367e8): - # the downloadable checkpoints are the December 2022 release, whose test - # transform letterboxes RGB with zeros anchored top-left (41.79 mAP50-95 on - # COCO val2017, against 39.72 through upstream HEAD's stretch) and feeds the - # unscaled image -- hence no Normalize. reg_max 16 means 17 distance bins. + # pre_cfg is the test transform of upstream DAMO-YOLO's December 2022 release + # (tinyvision/DAMO-YOLO 55ae14f), the only checkpoints still downloadable: a + # PIL RGB decode, cv2 resized to int(w * r) x int(h * r) -- truncated, hence + # size_rounding: floor -- zero-padded into the top-left corner, and left + # unscaled, hence no Normalize. BoxList.resize restores each axis by its own + # original-over-resized ratio, hence per_axis_ratio. (41.79 mAP50-95 on COCO + # val2017, against 39.72 through upstream HEAD's stretch.) post_cfg follows + # mblt-model-ops' DAMO-YOLO-M/pipeline.yaml (branch jm/temp, 0368367e8); + # reg_max 16 means 17 distance bins. # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/DAMO-YOLO-M @@ -12,14 +16,15 @@ DEFAULT: local_artifact_only: true pre_cfg: Reader: - style: numpy - color_mode: RGB + style: pil LetterBox: img_size: - 640 - 640 center: false padding_value: 0 + size_rounding: floor + per_axis_ratio: true SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/DAMO-YOLO-S-distill.yaml b/mblt_vision/models/DAMO-YOLO-S-distill.yaml index 52f535e..3d986b9 100644 --- a/mblt_vision/models/DAMO-YOLO-S-distill.yaml +++ b/mblt_vision/models/DAMO-YOLO-S-distill.yaml @@ -1,9 +1,13 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' DAMO-YOLO-S-distill/pipeline.yaml (branch jm/temp, 0368367e8): - # the downloadable checkpoints are the December 2022 release, whose test - # transform letterboxes RGB with zeros anchored top-left (41.79 mAP50-95 on - # COCO val2017, against 39.72 through upstream HEAD's stretch) and feeds the - # unscaled image -- hence no Normalize. reg_max 16 means 17 distance bins. + # pre_cfg is the test transform of upstream DAMO-YOLO's December 2022 release + # (tinyvision/DAMO-YOLO 55ae14f), the only checkpoints still downloadable: a + # PIL RGB decode, cv2 resized to int(w * r) x int(h * r) -- truncated, hence + # size_rounding: floor -- zero-padded into the top-left corner, and left + # unscaled, hence no Normalize. BoxList.resize restores each axis by its own + # original-over-resized ratio, hence per_axis_ratio. (41.79 mAP50-95 on COCO + # val2017, against 39.72 through upstream HEAD's stretch.) post_cfg follows + # mblt-model-ops' DAMO-YOLO-S-distill/pipeline.yaml (branch jm/temp, 0368367e8); + # reg_max 16 means 17 distance bins. # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/DAMO-YOLO-S-distill @@ -12,14 +16,15 @@ DEFAULT: local_artifact_only: true pre_cfg: Reader: - style: numpy - color_mode: RGB + style: pil LetterBox: img_size: - 640 - 640 center: false padding_value: 0 + size_rounding: floor + per_axis_ratio: true SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/DAMO-YOLO-S.yaml b/mblt_vision/models/DAMO-YOLO-S.yaml index 8b27477..244e7c9 100644 --- a/mblt_vision/models/DAMO-YOLO-S.yaml +++ b/mblt_vision/models/DAMO-YOLO-S.yaml @@ -1,9 +1,13 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' DAMO-YOLO-S/pipeline.yaml (branch jm/temp, 0368367e8): - # the downloadable checkpoints are the December 2022 release, whose test - # transform letterboxes RGB with zeros anchored top-left (41.79 mAP50-95 on - # COCO val2017, against 39.72 through upstream HEAD's stretch) and feeds the - # unscaled image -- hence no Normalize. reg_max 16 means 17 distance bins. + # pre_cfg is the test transform of upstream DAMO-YOLO's December 2022 release + # (tinyvision/DAMO-YOLO 55ae14f), the only checkpoints still downloadable: a + # PIL RGB decode, cv2 resized to int(w * r) x int(h * r) -- truncated, hence + # size_rounding: floor -- zero-padded into the top-left corner, and left + # unscaled, hence no Normalize. BoxList.resize restores each axis by its own + # original-over-resized ratio, hence per_axis_ratio. (41.79 mAP50-95 on COCO + # val2017, against 39.72 through upstream HEAD's stretch.) post_cfg follows + # mblt-model-ops' DAMO-YOLO-S/pipeline.yaml (branch jm/temp, 0368367e8); + # reg_max 16 means 17 distance bins. # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/DAMO-YOLO-S @@ -12,14 +16,15 @@ DEFAULT: local_artifact_only: true pre_cfg: Reader: - style: numpy - color_mode: RGB + style: pil LetterBox: img_size: - 640 - 640 center: false padding_value: 0 + size_rounding: floor + per_axis_ratio: true SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/DAMO-YOLO-T-distill.yaml b/mblt_vision/models/DAMO-YOLO-T-distill.yaml index f3d5e44..82ffe73 100644 --- a/mblt_vision/models/DAMO-YOLO-T-distill.yaml +++ b/mblt_vision/models/DAMO-YOLO-T-distill.yaml @@ -1,9 +1,13 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' DAMO-YOLO-T-distill/pipeline.yaml (branch jm/temp, 0368367e8): - # the downloadable checkpoints are the December 2022 release, whose test - # transform letterboxes RGB with zeros anchored top-left (41.79 mAP50-95 on - # COCO val2017, against 39.72 through upstream HEAD's stretch) and feeds the - # unscaled image -- hence no Normalize. reg_max 16 means 17 distance bins. + # pre_cfg is the test transform of upstream DAMO-YOLO's December 2022 release + # (tinyvision/DAMO-YOLO 55ae14f), the only checkpoints still downloadable: a + # PIL RGB decode, cv2 resized to int(w * r) x int(h * r) -- truncated, hence + # size_rounding: floor -- zero-padded into the top-left corner, and left + # unscaled, hence no Normalize. BoxList.resize restores each axis by its own + # original-over-resized ratio, hence per_axis_ratio. (41.79 mAP50-95 on COCO + # val2017, against 39.72 through upstream HEAD's stretch.) post_cfg follows + # mblt-model-ops' DAMO-YOLO-T-distill/pipeline.yaml (branch jm/temp, 0368367e8); + # reg_max 16 means 17 distance bins. # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/DAMO-YOLO-T-distill @@ -12,14 +16,15 @@ DEFAULT: local_artifact_only: true pre_cfg: Reader: - style: numpy - color_mode: RGB + style: pil LetterBox: img_size: - 640 - 640 center: false padding_value: 0 + size_rounding: floor + per_axis_ratio: true SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/DAMO-YOLO-T.yaml b/mblt_vision/models/DAMO-YOLO-T.yaml index 392b143..7332802 100644 --- a/mblt_vision/models/DAMO-YOLO-T.yaml +++ b/mblt_vision/models/DAMO-YOLO-T.yaml @@ -1,9 +1,13 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' DAMO-YOLO-T/pipeline.yaml (branch jm/temp, 0368367e8): - # the downloadable checkpoints are the December 2022 release, whose test - # transform letterboxes RGB with zeros anchored top-left (41.79 mAP50-95 on - # COCO val2017, against 39.72 through upstream HEAD's stretch) and feeds the - # unscaled image -- hence no Normalize. reg_max 16 means 17 distance bins. + # pre_cfg is the test transform of upstream DAMO-YOLO's December 2022 release + # (tinyvision/DAMO-YOLO 55ae14f), the only checkpoints still downloadable: a + # PIL RGB decode, cv2 resized to int(w * r) x int(h * r) -- truncated, hence + # size_rounding: floor -- zero-padded into the top-left corner, and left + # unscaled, hence no Normalize. BoxList.resize restores each axis by its own + # original-over-resized ratio, hence per_axis_ratio. (41.79 mAP50-95 on COCO + # val2017, against 39.72 through upstream HEAD's stretch.) post_cfg follows + # mblt-model-ops' DAMO-YOLO-T/pipeline.yaml (branch jm/temp, 0368367e8); + # reg_max 16 means 17 distance bins. # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/DAMO-YOLO-T @@ -12,14 +16,15 @@ DEFAULT: local_artifact_only: true pre_cfg: Reader: - style: numpy - color_mode: RGB + style: pil LetterBox: img_size: - 640 - 640 center: false padding_value: 0 + size_rounding: floor + per_axis_ratio: true SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/YOLOX-Darknet53.yaml b/mblt_vision/models/YOLOX-Darknet53.yaml index 3c36862..392b883 100644 --- a/mblt_vision/models/YOLOX-Darknet53.yaml +++ b/mblt_vision/models/YOLOX-Darknet53.yaml @@ -1,7 +1,10 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' YOLOX-Darknet53/pipeline.yaml (branch jm/temp, 0368367e8): - # upstream's ValTransform(legacy=False) feeds a cv2 BGR image, resized onto a - # 114-filled canvas anchored top-left, and left unscaled -- hence no Normalize. + # pre_cfg is upstream YOLOX's own test transform (yolox/data/data_augment.py + # preproc via ValTransform(legacy=False)): a cv2 BGR image resized to + # int(w * r) x int(h * r) -- truncated, hence size_rounding: floor -- onto a + # 114-filled canvas anchored top-left, and left unscaled, hence no Normalize. + # Boxes are restored by dividing by r. post_cfg follows mblt-model-ops' + # YOLOX-Darknet53/pipeline.yaml (branch jm/temp, 0368367e8). # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/YOLOX-Darknet53 @@ -18,6 +21,7 @@ DEFAULT: - 640 center: false padding_value: 114 + size_rounding: floor SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/YOLOX-Nano.yaml b/mblt_vision/models/YOLOX-Nano.yaml index bb8d84f..0e5ecb8 100644 --- a/mblt_vision/models/YOLOX-Nano.yaml +++ b/mblt_vision/models/YOLOX-Nano.yaml @@ -1,7 +1,10 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' YOLOX-Nano/pipeline.yaml (branch jm/temp, 0368367e8): - # upstream's ValTransform(legacy=False) feeds a cv2 BGR image, resized onto a - # 114-filled canvas anchored top-left, and left unscaled -- hence no Normalize. + # pre_cfg is upstream YOLOX's own test transform (yolox/data/data_augment.py + # preproc via ValTransform(legacy=False)): a cv2 BGR image resized to + # int(w * r) x int(h * r) -- truncated, hence size_rounding: floor -- onto a + # 114-filled canvas anchored top-left, and left unscaled, hence no Normalize. + # Boxes are restored by dividing by r. post_cfg follows mblt-model-ops' + # YOLOX-Nano/pipeline.yaml (branch jm/temp, 0368367e8). # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/YOLOX-Nano @@ -18,6 +21,7 @@ DEFAULT: - 416 center: false padding_value: 114 + size_rounding: floor SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/YOLOX-Tiny.yaml b/mblt_vision/models/YOLOX-Tiny.yaml index 5239662..6a8c61d 100644 --- a/mblt_vision/models/YOLOX-Tiny.yaml +++ b/mblt_vision/models/YOLOX-Tiny.yaml @@ -1,7 +1,10 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' YOLOX-Tiny/pipeline.yaml (branch jm/temp, 0368367e8): - # upstream's ValTransform(legacy=False) feeds a cv2 BGR image, resized onto a - # 114-filled canvas anchored top-left, and left unscaled -- hence no Normalize. + # pre_cfg is upstream YOLOX's own test transform (yolox/data/data_augment.py + # preproc via ValTransform(legacy=False)): a cv2 BGR image resized to + # int(w * r) x int(h * r) -- truncated, hence size_rounding: floor -- onto a + # 114-filled canvas anchored top-left, and left unscaled, hence no Normalize. + # Boxes are restored by dividing by r. post_cfg follows mblt-model-ops' + # YOLOX-Tiny/pipeline.yaml (branch jm/temp, 0368367e8). # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/YOLOX-Tiny @@ -18,6 +21,7 @@ DEFAULT: - 416 center: false padding_value: 114 + size_rounding: floor SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/YOLOX-l.yaml b/mblt_vision/models/YOLOX-l.yaml index 607e7ce..6f8542b 100644 --- a/mblt_vision/models/YOLOX-l.yaml +++ b/mblt_vision/models/YOLOX-l.yaml @@ -1,7 +1,10 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' YOLOX-l/pipeline.yaml (branch jm/temp, 0368367e8): - # upstream's ValTransform(legacy=False) feeds a cv2 BGR image, resized onto a - # 114-filled canvas anchored top-left, and left unscaled -- hence no Normalize. + # pre_cfg is upstream YOLOX's own test transform (yolox/data/data_augment.py + # preproc via ValTransform(legacy=False)): a cv2 BGR image resized to + # int(w * r) x int(h * r) -- truncated, hence size_rounding: floor -- onto a + # 114-filled canvas anchored top-left, and left unscaled, hence no Normalize. + # Boxes are restored by dividing by r. post_cfg follows mblt-model-ops' + # YOLOX-l/pipeline.yaml (branch jm/temp, 0368367e8). # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/YOLOX-l @@ -18,6 +21,7 @@ DEFAULT: - 640 center: false padding_value: 114 + size_rounding: floor SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/YOLOX-m.yaml b/mblt_vision/models/YOLOX-m.yaml index 5b801b0..2567222 100644 --- a/mblt_vision/models/YOLOX-m.yaml +++ b/mblt_vision/models/YOLOX-m.yaml @@ -1,7 +1,10 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' YOLOX-m/pipeline.yaml (branch jm/temp, 0368367e8): - # upstream's ValTransform(legacy=False) feeds a cv2 BGR image, resized onto a - # 114-filled canvas anchored top-left, and left unscaled -- hence no Normalize. + # pre_cfg is upstream YOLOX's own test transform (yolox/data/data_augment.py + # preproc via ValTransform(legacy=False)): a cv2 BGR image resized to + # int(w * r) x int(h * r) -- truncated, hence size_rounding: floor -- onto a + # 114-filled canvas anchored top-left, and left unscaled, hence no Normalize. + # Boxes are restored by dividing by r. post_cfg follows mblt-model-ops' + # YOLOX-m/pipeline.yaml (branch jm/temp, 0368367e8). # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/YOLOX-m @@ -18,6 +21,7 @@ DEFAULT: - 640 center: false padding_value: 114 + size_rounding: floor SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/YOLOX-s.yaml b/mblt_vision/models/YOLOX-s.yaml index 777ee74..ec2d373 100644 --- a/mblt_vision/models/YOLOX-s.yaml +++ b/mblt_vision/models/YOLOX-s.yaml @@ -1,7 +1,10 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' YOLOX-s/pipeline.yaml (branch jm/temp, 0368367e8): - # upstream's ValTransform(legacy=False) feeds a cv2 BGR image, resized onto a - # 114-filled canvas anchored top-left, and left unscaled -- hence no Normalize. + # pre_cfg is upstream YOLOX's own test transform (yolox/data/data_augment.py + # preproc via ValTransform(legacy=False)): a cv2 BGR image resized to + # int(w * r) x int(h * r) -- truncated, hence size_rounding: floor -- onto a + # 114-filled canvas anchored top-left, and left unscaled, hence no Normalize. + # Boxes are restored by dividing by r. post_cfg follows mblt-model-ops' + # YOLOX-s/pipeline.yaml (branch jm/temp, 0368367e8). # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/YOLOX-s @@ -18,6 +21,7 @@ DEFAULT: - 640 center: false padding_value: 114 + size_rounding: floor SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/models/YOLOX-x.yaml b/mblt_vision/models/YOLOX-x.yaml index 9164604..7f23903 100644 --- a/mblt_vision/models/YOLOX-x.yaml +++ b/mblt_vision/models/YOLOX-x.yaml @@ -1,7 +1,10 @@ DEFAULT: - # pre_cfg/post_cfg follow mblt-model-ops' YOLOX-x/pipeline.yaml (branch jm/temp, 0368367e8): - # upstream's ValTransform(legacy=False) feeds a cv2 BGR image, resized onto a - # 114-filled canvas anchored top-left, and left unscaled -- hence no Normalize. + # pre_cfg is upstream YOLOX's own test transform (yolox/data/data_augment.py + # preproc via ValTransform(legacy=False)): a cv2 BGR image resized to + # int(w * r) x int(h * r) -- truncated, hence size_rounding: floor -- onto a + # 114-filled canvas anchored top-left, and left unscaled, hence no Normalize. + # Boxes are restored by dividing by r. post_cfg follows mblt-model-ops' + # YOLOX-x/pipeline.yaml (branch jm/temp, 0368367e8). # No Hub artifact is published yet: pass a local model_path. file_cfg: repo_id: mobilint/YOLOX-x @@ -18,6 +21,7 @@ DEFAULT: - 640 center: false padding_value: 114 + size_rounding: floor SetOrder: shape: HWC post_cfg: diff --git a/mblt_vision/utils/datasets/dataloader.py b/mblt_vision/utils/datasets/dataloader.py index decea74..58875ac 100644 --- a/mblt_vision/utils/datasets/dataloader.py +++ b/mblt_vision/utils/datasets/dataloader.py @@ -15,6 +15,7 @@ from faster_coco_eval import COCO from PIL import Image +from mblt_vision.utils.letterbox import LetterBoxLayout, deprecated_center_argument from mblt_vision.utils.preprocess.letterbox import letterbox_semantic_mask from .cityscapes import CITYSCAPES_SOURCE_TO_TRAIN_ID @@ -49,7 +50,11 @@ class CustomCOCODataset(torch.utils.data.Dataset[tuple[np.ndarray, int, int, int """ def __init__( - self, root: str, annFile: str, min_keypoints: int | None = None + self, + root: str, + annFile: str, + min_keypoints: int | None = None, + decoder: str = "cv2", ) -> None: """Initialize the custom COCO dataset. @@ -58,7 +63,19 @@ def __init__( annFile (str): Path to the COCO annotation JSON file. min_keypoints: If set, keep only images with at least one annotation whose ``num_keypoints`` is greater than this value. + decoder: ``"cv2"`` (the default) or ``"pil"``, the library that decodes + each image to RGB. Pass the model's ``Reader.style`` decoder: PIL + and OpenCV can decode the same JPEG to different pixels, and + DAMO-YOLO's upstream evaluation decodes with PIL. + + Raises: + ValueError: If ``decoder`` is unsupported or the annotation is unreadable. """ + if decoder not in {"cv2", "pil"}: + raise ValueError( + f"Unsupported COCO image decoder {decoder!r}; expected 'cv2' or 'pil'." + ) + self.decoder = decoder self.root = root try: raw_annotation = json.loads(Path(annFile).read_text(encoding="utf-8")) @@ -103,6 +120,11 @@ def _load_image(self, image_id: int) -> np.ndarray: raise ValueError( f"COCO image ID {image_id} resolves outside the image root: {file_name!r}." ) from exc + if self.decoder == "pil": + # Upstream DAMO-YOLO's np.asarray(Image.open(path).convert("RGB")), + # copied so the array is writable like cv2's. + with Image.open(image_path) as pil_image: + return np.array(pil_image.convert("RGB")) image = cv2.imread(str(image_path)) # Load image (BGR format) if image is None: @@ -420,7 +442,9 @@ def get_ade20k_loader( batch_size: int, preprocess_fn: Callable, image_size: tuple[int, int], - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> torch.utils.data.DataLoader: """Create an ADE20K loader that applies matching letterbox geometry to masks. @@ -429,12 +453,16 @@ def get_ade20k_loader( batch_size: Number of samples per batch. preprocess_fn: Image preprocessing function that returns letterbox metadata. image_size: Configured model input size as ``(height, width)``. - center: The model's ``LetterBox.center``, so targets are anchored like images. + layout: The model's ``letterbox_layout(pre_cfg)``, so targets share the + images' geometry. + center: Deprecated spelling of a boolean ``layout``. Returns: Configured ADE20K validation loader. """ + layout = deprecated_center_argument(layout, center, "get_ade20k_loader") + def loader( batch: list[Any], ) -> tuple[ @@ -463,7 +491,7 @@ def loader( processed_target, target_ratio_pad = letterbox_semantic_mask( target, [input_height, input_width], - center=center, + layout=layout, ) if target_ratio_pad != ratio_pad: raise ValueError( @@ -589,13 +617,18 @@ def get_cityscapes_loader( batch_size: int, preprocess_fn: Callable, image_size: tuple[int, int], - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> torch.utils.data.DataLoader: """Create a Cityscapes loader with image-matching letterbox geometry. - ``center`` is the model's ``LetterBox.center``, so targets are anchored like images. + ``layout`` is the model's ``letterbox_layout(pre_cfg)``, so targets share the + images' geometry; ``center`` is its deprecated boolean spelling. """ + layout = deprecated_center_argument(layout, center, "get_cityscapes_loader") + def loader( batch: list[Any], ) -> tuple[ @@ -624,7 +657,7 @@ def loader( processed_target, target_ratio_pad = letterbox_semantic_mask( target, [input_height, input_width], - center=center, + layout=layout, ) if target_ratio_pad != ratio_pad: raise ValueError( diff --git a/mblt_vision/utils/evaluation/eval_ade20k.py b/mblt_vision/utils/evaluation/eval_ade20k.py index 020afe1..ee286ef 100644 --- a/mblt_vision/utils/evaluation/eval_ade20k.py +++ b/mblt_vision/utils/evaluation/eval_ade20k.py @@ -14,7 +14,7 @@ get_ade20k_loader, get_cityscapes_loader, ) -from ..letterbox import letterbox_center +from ..letterbox import letterbox_layout from ._pipeline import map_batched_inference if TYPE_CHECKING: @@ -281,7 +281,7 @@ def eval_semantic_segmentation( batch_size, model.preprocess_with_metadata, image_size=image_size_tuple, - center=letterbox_center(model.pre_cfg), + layout=letterbox_layout(model.pre_cfg), ) default_nc = 150 description = "Evaluating ADE20K" @@ -292,7 +292,7 @@ def eval_semantic_segmentation( batch_size, model.preprocess_with_metadata, image_size=image_size_tuple, - center=letterbox_center(model.pre_cfg), + layout=letterbox_layout(model.pre_cfg), ) default_nc = 19 description = "Evaluating Cityscapes" diff --git a/mblt_vision/utils/evaluation/eval_coco.py b/mblt_vision/utils/evaluation/eval_coco.py index 9082a89..f3d562b 100644 --- a/mblt_vision/utils/evaluation/eval_coco.py +++ b/mblt_vision/utils/evaluation/eval_coco.py @@ -318,6 +318,18 @@ def format_coco_results( return results +def _coco_image_decoder(pre_cfg: dict[str, Any]) -> str: + """Return the decoder matching the model's ``Reader``: PIL for ``style: pil``. + + The COCO loader decodes images itself, so a model whose reader decodes files + with PIL (DAMO-YOLO) would otherwise be fed OpenCV's pixels. + """ + + reader_cfg = pre_cfg.get("Reader") + style = reader_cfg.get("style") if isinstance(reader_cfg, dict) else None + return "pil" if isinstance(style, str) and style.lower() == "pil" else "cv2" + + def eval_coco( model: MBLT_Engine, data_path: str, @@ -361,15 +373,18 @@ def eval_coco_metrics( "COCO evaluation requires model post_cfg.dataset to be 'coco', " f"got {dataset_name!r}." ) + decoder = _coco_image_decoder(model.pre_cfg) if task in {"object_detection", "instance_segmentation"}: dataset = CustomCOCODataset( os.path.join(data_path, "val2017"), os.path.join(data_path, "instances_val2017.json"), + decoder=decoder, ) else: dataset = CustomCOCODataset( os.path.join(data_path, "val2017"), os.path.join(data_path, "person_keypoints_val2017.json"), + decoder=decoder, ) _validate_coco_dataset_taxonomy(dataset, task) diff --git a/mblt_vision/utils/evaluation/eval_dota.py b/mblt_vision/utils/evaluation/eval_dota.py index 740875c..b229061 100644 --- a/mblt_vision/utils/evaluation/eval_dota.py +++ b/mblt_vision/utils/evaluation/eval_dota.py @@ -30,7 +30,13 @@ _canonicalize_quadrilateral, _polygon_has_positive_image_overlap, ) -from ..letterbox import RatioPad, letterbox_center, resolve_ratio_pad +from ..letterbox import ( + LetterBoxLayout, + RatioPad, + deprecated_center_argument, + letterbox_layout, + resolve_ratio_pad, +) from ._pipeline import map_batched_inference if TYPE_CHECKING: @@ -521,19 +527,26 @@ def _ratio_pad_for_shape( input_shape: tuple[int, ...], org_shape: tuple[int, int], ratio_pad: RatioPad | None, - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> tuple[float, tuple[float, float]]: """Return letterbox gain and padding for an image. - ``center`` is the model's ``LetterBox.center``, used only when no metadata - was recorded. + ``layout`` is the model's ``letterbox_layout(pre_cfg)``, used only when no + metadata was recorded; ``center`` is its deprecated boolean spelling. """ + layout = deprecated_center_argument(layout, center, "_ratio_pad_for_shape") if len(input_shape) < 2: raise ValueError(f"Expected at least 2 input dimensions, got {input_shape}.") ratio, pad = resolve_ratio_pad( - (input_shape[0], input_shape[1]), org_shape, ratio_pad, center + (input_shape[0], input_shape[1]), org_shape, ratio_pad, layout ) + if ratio[0] != ratio[1]: + raise ValueError( + f"Rotated boxes need one uniform letterbox ratio, got {tuple(ratio)}." + ) return float(ratio[0]), (float(pad[0]), float(pad[1])) @@ -542,10 +555,16 @@ def _ground_truth_to_input_space( input_shape: tuple[int, ...], org_shape: tuple[int, int], ratio_pad: RatioPad | None, - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> dict[str, torch.Tensor]: - """Transform original-image DOTAv1 polygons to letterboxed ``xywhr`` boxes.""" - gain, pad = _ratio_pad_for_shape(input_shape, org_shape, ratio_pad, center) + """Transform original-image DOTAv1 polygons to letterboxed ``xywhr`` boxes. + + ``center`` is the deprecated boolean spelling of ``layout``. + """ + layout = deprecated_center_argument(layout, center, "_ground_truth_to_input_space") + gain, pad = _ratio_pad_for_shape(input_shape, org_shape, ratio_pad, layout) def transform(polygons: torch.Tensor | None, boxes: torch.Tensor) -> torch.Tensor: if polygons is None: @@ -849,7 +868,7 @@ def eval_dota( dataloader = get_dota_loader(dataset, batch_size, model.preprocess_with_metadata) model.set_postprocess_thresholds(conf_thres=conf_thres, iou_thres=iou_thres) ground_truths = _load_ground_truths(data_path, dataset) - center = letterbox_center(model.pre_cfg) + layout = letterbox_layout(model.pre_cfg) iouv = torch.linspace(0.5, 0.95, 10) stats = _empty_stats() @@ -892,7 +911,7 @@ def decode(batch: Any, out_npu: Any) -> tuple[list[Any], list[Any]]: input_shape, (int(image_shape[0]), int(image_shape[1])), image_ratio_pad, - center, + layout, ) image_stats.append( _process_image_stats(_nms_output_to_predictions(nms_out), target, iouv) diff --git a/mblt_vision/utils/letterbox.py b/mblt_vision/utils/letterbox.py index 8808a36..285008f 100644 --- a/mblt_vision/utils/letterbox.py +++ b/mblt_vision/utils/letterbox.py @@ -2,11 +2,97 @@ from __future__ import annotations +import warnings from collections.abc import Mapping from dataclasses import dataclass -from typing import Any, TypeAlias +from typing import Any, Literal, TypeAlias RatioPad: TypeAlias = tuple[tuple[float, float], tuple[float, float]] +SizeRounding: TypeAlias = Literal["round", "floor"] +_SIZE_ROUNDINGS = ("round", "floor") + + +@dataclass(frozen=True) +class LetterBoxLayout: + """How a model's letterbox sizes, places, and later inverts its resize. + + The defaults are Ultralytics': Python ``round`` for the resized size, the + padding split around the image, and one uniform ratio for restoration. + Upstream YOLOX (``data_augment.preproc``) and DAMO-YOLO (``Resize`` of the + December 2022 release) truncate the size with ``int(w * r)`` and anchor the + image top-left. YOLOX restores boxes by dividing by ``r``; DAMO-YOLO's + ``BoxList.resize`` scales each axis by the original over the truncated size. + + Attributes: + center: Split the padding around the image, or anchor it top-left. + size_rounding: ``"round"`` (half to even) or ``"floor"`` (upstream ``int()``). + per_axis_ratio: Restore x and y by the resized-to-original ratio of each + axis instead of the one aspect-preserving ratio. + """ + + center: bool = True + size_rounding: SizeRounding = "round" + per_axis_ratio: bool = False + + def __post_init__(self) -> None: + if not isinstance(self.center, bool): + raise TypeError( + f"LetterBox center must be a boolean, got {type(self.center).__name__}." + ) + if self.size_rounding not in _SIZE_ROUNDINGS: + raise ValueError( + f"LetterBox size_rounding must be 'round' or 'floor', got {self.size_rounding!r}." + ) + if not isinstance(self.per_axis_ratio, bool): + raise TypeError( + "LetterBox per_axis_ratio must be a boolean, " + f"got {type(self.per_axis_ratio).__name__}." + ) + + @property + def is_default(self) -> bool: + """Whether this is the Ultralytics layout the shape-only helpers assume.""" + + return self == LetterBoxLayout() + + +def deprecated_center_argument( + layout: LetterBoxLayout | bool | None, center: bool | None, owner: str +) -> LetterBoxLayout | bool: + """Resolve the ``center=`` keyword that ``layout=`` replaced. + + Args: + layout: The ``layout`` argument as passed, ``None`` when omitted, so an + explicit ``layout=True`` still counts as given. + center: The deprecated ``center`` keyword, ``None`` when omitted. + owner: Function name used in the warning and error. + + Returns: + ``center`` when it was given, else ``layout``, else Ultralytics' default + ``True``. + + Raises: + TypeError: If both ``layout`` and ``center`` were given. + """ + + if center is None: + return True if layout is None else layout + if layout is not None: + raise TypeError(f"{owner}() takes layout or the deprecated center, not both.") + warnings.warn( + f"{owner}(center=...) is deprecated; pass layout=LetterBoxLayout(center=...).", + DeprecationWarning, + stacklevel=3, + ) + return center + + +def _as_layout(layout: LetterBoxLayout | bool) -> LetterBoxLayout: + """Accept the older ``center`` boolean wherever a layout is expected.""" + + if isinstance(layout, LetterBoxLayout): + return layout + return LetterBoxLayout(center=layout) @dataclass(frozen=True) @@ -18,33 +104,44 @@ class LetterBoxGeometry: ratio: float resized_shape: tuple[int, int] pad: tuple[int, int] + per_axis_ratio: bool = False @classmethod def from_shapes( cls, input_shape: tuple[int, int], original_shape: tuple[int, int], - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> LetterBoxGeometry: - """Calculate letterbox geometry, centered by default as Ultralytics does. + """Calculate letterbox geometry, Ultralytics' by default. Args: input_shape: Target shape as ``(height, width)``. original_shape: Source shape as ``(height, width)``. - center: Split the padding evenly around the resized image. ``False`` - anchors the image at the top-left corner and pads only the - bottom and right, as YOLOX and DAMO-YOLO do. + layout: The model's ``LetterBoxLayout``. A boolean is read as + ``LetterBoxLayout(center=layout)``: ``False`` anchors the image at + the top-left corner and pads only the bottom and right. + center: Deprecated spelling of a boolean ``layout``. Returns: Calculated resize ratio, resized shape, and top-left padding. """ + layout = _as_layout( + deprecated_center_argument(layout, center, "LetterBoxGeometry.from_shapes") + ) input_height, input_width = input_shape original_height, original_width = original_shape ratio = min(input_height / original_height, input_width / original_width) - resized_height = int(round(original_height * ratio)) - resized_width = int(round(original_width * ratio)) - if center: + if layout.size_rounding == "floor": + resized_height = int(original_height * ratio) + resized_width = int(original_width * ratio) + else: + resized_height = int(round(original_height * ratio)) + resized_width = int(round(original_width * ratio)) + if layout.center: left = int(round((input_width - resized_width) / 2 - 0.1)) top = int(round((input_height - resized_height) / 2 - 0.1)) else: @@ -55,12 +152,24 @@ def from_shapes( ratio=ratio, resized_shape=(resized_height, resized_width), pad=(left, top), + per_axis_ratio=layout.per_axis_ratio, ) @property def ratio_pad(self) -> RatioPad: - """Return metadata consumed by inverse letterbox operations.""" + """Return metadata consumed by inverse letterbox operations. + + The ratios are ``(x, y)``: both the aspect-preserving ratio, or with + ``per_axis_ratio`` each axis' resized-over-original size. + """ + if self.per_axis_ratio: + resized_height, resized_width = self.resized_shape + original_height, original_width = self.original_shape + return ( + (resized_width / original_width, resized_height / original_height), + self.pad, + ) return ((self.ratio, self.ratio), self.pad) @property @@ -109,7 +218,9 @@ def resolve_ratio_pad( input_shape: tuple[int, int], original_shape: tuple[int, int], ratio_pad: RatioPad | None = None, - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> RatioPad: """Return supplied letterbox metadata or derive it from image shapes. @@ -117,16 +228,18 @@ def resolve_ratio_pad( input_shape: Letterboxed shape as ``(height, width)``. original_shape: Source shape as ``(height, width)``. ratio_pad: Optional metadata recorded during preprocessing. - center: Anchoring used to derive missing metadata; see + layout: Layout used to derive missing metadata; see ``LetterBoxGeometry.from_shapes``. + center: Deprecated spelling of a boolean ``layout``. Returns: Resize ratios and top-left padding as ``((ratio_x, ratio_y), (pad_x, pad_y))``. """ + layout = deprecated_center_argument(layout, center, "resolve_ratio_pad") if ratio_pad is not None: return ratio_pad - return LetterBoxGeometry.from_shapes(input_shape, original_shape, center).ratio_pad + return LetterBoxGeometry.from_shapes(input_shape, original_shape, layout).ratio_pad def letterbox_center(pre_cfg: Mapping[str, Any]) -> bool: @@ -146,12 +259,32 @@ def letterbox_center(pre_cfg: Mapping[str, Any]) -> bool: TypeError: If ``center`` is not a boolean. """ + return letterbox_layout(pre_cfg).center + + +def letterbox_layout(pre_cfg: Mapping[str, Any]) -> LetterBoxLayout: + """Return a model's configured ``LetterBoxLayout``. + + Args: + pre_cfg: Model preprocessing configuration. + + Returns: + ``pre_cfg.LetterBox``'s ``center``, ``size_rounding`` and + ``per_axis_ratio``, each defaulting to Ultralytics' behaviour. + + Raises: + TypeError: If ``center`` or ``per_axis_ratio`` is not a boolean. + ValueError: If ``size_rounding`` is not ``"round"`` or ``"floor"``. + """ + letterbox_cfg = pre_cfg.get("LetterBox") if not isinstance(letterbox_cfg, Mapping): - return True - center = letterbox_cfg.get("center", True) - if not isinstance(center, bool): - raise TypeError( - f"pre_cfg.LetterBox.center must be a boolean, got {type(center).__name__}." + return LetterBoxLayout() + try: + return LetterBoxLayout( + center=letterbox_cfg.get("center", True), + size_rounding=letterbox_cfg.get("size_rounding", "round"), + per_axis_ratio=letterbox_cfg.get("per_axis_ratio", False), ) - return center + except (TypeError, ValueError) as error: + raise type(error)(f"pre_cfg.{error}") from error diff --git a/mblt_vision/utils/postprocess/_letterbox.py b/mblt_vision/utils/postprocess/_letterbox.py index 40d4fa4..67bc9bb 100644 --- a/mblt_vision/utils/postprocess/_letterbox.py +++ b/mblt_vision/utils/postprocess/_letterbox.py @@ -7,7 +7,13 @@ import torch -from ..letterbox import LetterBoxGeometry, RatioPad, resolve_ratio_pad +from ..letterbox import ( + LetterBoxGeometry, + LetterBoxLayout, + RatioPad, + deprecated_center_argument, + resolve_ratio_pad, +) from .common import normalize_ratio_pads @@ -48,7 +54,9 @@ def resolve_ratio_pads( batch_size: int, shapes: Sequence[tuple[int, int]], input_shape: tuple[int, int], - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> list[RatioPad]: """Normalize letterbox metadata and derive values missing from a dense task batch. @@ -57,8 +65,9 @@ def resolve_ratio_pads( batch_size: Number of images in the output batch. shapes: Original image shapes. input_shape: Configured model input shape. - center: The model's letterbox anchoring (``letterbox_center(pre_cfg)``), - used for metadata the caller did not record. + layout: The model's ``letterbox_layout(pre_cfg)`` (or the older + ``center`` boolean), used for metadata the caller did not record. + center: Deprecated spelling of a boolean ``layout``. Returns: One resolved ratio/padding pair per batch item. @@ -67,9 +76,10 @@ def resolve_ratio_pads( ValueError: If ratio/padding metadata is invalid for the batch. """ + layout = deprecated_center_argument(layout, center, "resolve_ratio_pads") pads = normalize_ratio_pads(ratio_pad, batch_size) return [ - resolve_ratio_pad(input_shape, shape, pad, center) + resolve_ratio_pad(input_shape, shape, pad, layout) for pad, shape in zip(pads, shapes) ] @@ -80,6 +90,7 @@ def crop_letterbox( ratio_pad: RatioPad, input_shape: tuple[int, int], task_name: str, + layout: LetterBoxLayout | bool = True, ) -> torch.Tensor: """Crop letterbox padding from a dense two-dimensional output. @@ -89,6 +100,7 @@ def crop_letterbox( ratio_pad: Resize ratio and padding applied during preprocessing. input_shape: Configured model input height and width. task_name: Task label used in validation errors. + layout: The model's letterbox layout, which sizes the resized extent. Returns: Output with letterbox padding removed. @@ -97,7 +109,7 @@ def crop_letterbox( ValueError: If inverse letterboxing produces an empty crop. """ - geometry = LetterBoxGeometry.from_shapes(input_shape, shape) + geometry = LetterBoxGeometry.from_shapes(input_shape, shape, layout) output_shape = (int(output.shape[0]), int(output.shape[1])) top, bottom, left, right = geometry.crop_bounds(output_shape, pad=ratio_pad[1]) cropped = output[top:bottom, left:right] diff --git a/mblt_vision/utils/postprocess/base.py b/mblt_vision/utils/postprocess/base.py index e63377a..b738164 100644 --- a/mblt_vision/utils/postprocess/base.py +++ b/mblt_vision/utils/postprocess/base.py @@ -8,7 +8,7 @@ import torch from ..._tasks import normalize_vision_task -from ..letterbox import RatioPad, letterbox_center, resolve_ratio_pad +from ..letterbox import RatioPad, letterbox_layout, resolve_ratio_pad from ..preprocess._validation import normalize_image_size from ..types import ListTensorLike, TensorLike from .common import ( @@ -101,11 +101,25 @@ def __init__( self.imh, self.imw = normalize_image_size( img_size, name="pre_cfg.LetterBox.img_size" ) - self.letterbox_center = letterbox_center(pre_cfg) + self.letterbox_layout = letterbox_layout(pre_cfg) + self.letterbox_center = self.letterbox_layout.center task = post_cfg.get("task") if task is None: raise ValueError("task should be provided in post_cfg") self.task = normalize_vision_task(task) + if self.task == "instance_segmentation" and ( + self.letterbox_layout.size_rounding != "round" + or self.letterbox_layout.per_axis_ratio + ): + # ``scale_masks`` crops prototypes with Ultralytics' rounded extent and + # one ratio, so it cannot restore either option faithfully. + raise ValueError( + "instance_segmentation supports only pre_cfg.LetterBox.size_rounding " + "'round' without per_axis_ratio." + ) + if self.task == "obb" and self.letterbox_layout.per_axis_ratio: + # A rotated box's width and height do not lie along x and y. + raise ValueError("obb does not support pre_cfg.LetterBox.per_axis_ratio.") task_key = self.task dataset = post_cfg.get("dataset") self.dataset = dataset.lower() if isinstance(dataset, str) else None @@ -310,12 +324,13 @@ def ratio_pads_for( img0_shape: tuple[int, int] | list[tuple[int, int]], ratio_pad: RatioPad | list[RatioPad | None] | None, ) -> RatioPad | list[RatioPad | None] | None: - """Fill letterbox metadata the caller did not record with this model's anchoring. + """Fill letterbox metadata the caller did not record with this model's layout. - The shared inverse helpers derive missing metadata as a centered letterbox. - That is right for Ultralytics models, so their metadata passes through - untouched; a top-left model (``pre_cfg.LetterBox.center: false``) gets its - own zero padding instead of a shift by half the border. + The shared inverse helpers derive missing metadata as Ultralytics' centered, + rounded letterbox. That is right for Ultralytics models, so their metadata + passes through untouched; any other ``LetterBoxLayout`` (a top-left + ``center: false``, a ``size_rounding: floor``, or ``per_axis_ratio``) gets + its own geometry instead. The result keeps the caller's cardinality. One shared image shape with shared (or no) metadata yields one shared pad, which the helpers broadcast @@ -331,7 +346,7 @@ def ratio_pads_for( Metadata the shared helpers can use as-is. """ - if self.letterbox_center: + if self.letterbox_layout.is_default: return ratio_pad per_image_pads = ratio_pad is not None and not _is_ratio_pad(ratio_pad) shared_shape = len(img0_shape) == 2 and isinstance(img0_shape[0], int) @@ -341,7 +356,7 @@ def ratio_pads_for( img1_shape, (int(shape[0]), int(shape[1])), cast(RatioPad | None, ratio_pad), - center=False, + self.letterbox_layout, ) if per_image_pads: pads = list(cast(Sequence[RatioPad | None], ratio_pad)) @@ -350,7 +365,7 @@ def ratio_pads_for( shapes = normalize_image_shapes(img0_shape) pads = normalize_ratio_pads(ratio_pad, len(shapes)) return [ - resolve_ratio_pad(img1_shape, shape, pad, center=False) + resolve_ratio_pad(img1_shape, shape, pad, self.letterbox_layout) for pad, shape in zip(pads, shapes) ] diff --git a/mblt_vision/utils/postprocess/common.py b/mblt_vision/utils/postprocess/common.py index 573590b..7daedc9 100644 --- a/mblt_vision/utils/postprocess/common.py +++ b/mblt_vision/utils/postprocess/common.py @@ -824,17 +824,13 @@ def scale_boxes( np.ndarray | torch.Tensor: The scaled bounding boxes, in the format of (x1, y1, x2, y2) """ ratio, pad = resolve_ratio_pad(img1_shape, img0_shape, ratio_pad) - gain = ratio[0] - if isinstance(boxes, np.ndarray): - if padding: - boxes[..., [0, 2]] -= pad[0] # x padding - boxes[..., [1, 3]] -= pad[1] # y padding - boxes[..., :4] /= gain - return clip_boxes(boxes, img0_shape) + # Ratios are (x, y). Every uniform letterbox records the same value twice; + # DAMO-YOLO's per-axis restoration records each axis' own. if padding: boxes[..., [0, 2]] -= pad[0] # x padding boxes[..., [1, 3]] -= pad[1] # y padding - boxes[..., :4] /= gain + boxes[..., [0, 2]] /= ratio[0] + boxes[..., [1, 3]] /= ratio[1] return clip_boxes(boxes, img0_shape) @@ -880,17 +876,11 @@ def scale_coords( np.ndarray | torch.Tensor: The scaled coordinates, in the format of (x, y) """ ratio, pad = resolve_ratio_pad(img1_shape, img0_shape, ratio_pad) - gain = ratio[0] - if isinstance(coords, np.ndarray): - if padding: - coords[..., 0] -= pad[0] # x padding - coords[..., 1] -= pad[1] # y padding - coords[..., :2] /= gain - return clip_coords(coords, img0_shape) if padding: coords[..., 0] -= pad[0] # x padding coords[..., 1] -= pad[1] # y padding - coords[..., :2] /= gain + coords[..., 0] /= ratio[0] + coords[..., 1] /= ratio[1] return clip_coords(coords, img0_shape) @@ -914,6 +904,12 @@ def scale_rboxes( Rescaled rotated boxes in ``xywhr`` format. """ ratio, pad = resolve_ratio_pad(img1_shape, img0_shape, ratio_pad) + if ratio[0] != ratio[1]: + # A rotated box's width and height do not lie along x and y, so a + # per-axis ratio has no faithful inverse here. + raise ValueError( + f"Rotated boxes need one uniform letterbox ratio, got {tuple(ratio)}." + ) gain = ratio[0] scaled = rboxes.clone() if padding: diff --git a/mblt_vision/utils/postprocess/depth_post.py b/mblt_vision/utils/postprocess/depth_post.py index 4ce3c8b..3d2824b 100644 --- a/mblt_vision/utils/postprocess/depth_post.py +++ b/mblt_vision/utils/postprocess/depth_post.py @@ -8,7 +8,7 @@ import torch import torch.nn.functional as functional -from ..letterbox import RatioPad, letterbox_center +from ..letterbox import RatioPad, letterbox_layout from ..types import ListTensorLike, TensorLike from ._letterbox import crop_letterbox, get_letterbox_input_shape, resolve_ratio_pads from .base import PostBase @@ -26,7 +26,8 @@ def __init__(self, pre_cfg: dict[str, Any], post_cfg: dict[str, Any]) -> None: self.input_shape = get_letterbox_input_shape( pre_cfg, "Depth estimation", "Depth" ) - self.letterbox_center = letterbox_center(pre_cfg) + self.letterbox_layout = letterbox_layout(pre_cfg) + self.letterbox_center = self.letterbox_layout.center def __call__( self, @@ -51,7 +52,7 @@ def __call__( depth.shape[0], shapes, self.input_shape, - self.letterbox_center, + self.letterbox_layout, ) restored = [ self._restore(depth[index], shapes[index], pads[index]) @@ -107,7 +108,9 @@ def _restore( ) -> torch.Tensor: """Crop padded depth pixels and bilinearly resize to an original image shape.""" - cropped = crop_letterbox(depth, shape, ratio_pad, self.input_shape, "Depth") + cropped = crop_letterbox( + depth, shape, ratio_pad, self.input_shape, "Depth", self.letterbox_layout + ) return functional.interpolate( cropped[None, None], size=shape, mode="bilinear", align_corners=False )[0, 0] diff --git a/mblt_vision/utils/postprocess/semantic_seg_post.py b/mblt_vision/utils/postprocess/semantic_seg_post.py index 6b52582..502f014 100644 --- a/mblt_vision/utils/postprocess/semantic_seg_post.py +++ b/mblt_vision/utils/postprocess/semantic_seg_post.py @@ -8,7 +8,7 @@ import torch import torch.nn.functional as functional -from ..letterbox import RatioPad, letterbox_center +from ..letterbox import RatioPad, letterbox_layout from ..types import ListTensorLike, TensorLike from ._letterbox import crop_letterbox, get_letterbox_input_shape, resolve_ratio_pads from .base import PostBase @@ -30,7 +30,8 @@ def __init__(self, pre_cfg: dict[str, Any], post_cfg: dict[str, Any]) -> None: self.input_shape = get_letterbox_input_shape( pre_cfg, "Semantic segmentation", "Semantic" ) - self.letterbox_center = letterbox_center(pre_cfg) + self.letterbox_layout = letterbox_layout(pre_cfg) + self.letterbox_center = self.letterbox_layout.center dataset = post_cfg.get("dataset") if not isinstance(dataset, str): raise ValueError( @@ -75,7 +76,7 @@ def __call__( output.shape[0], shapes, self.input_shape, - self.letterbox_center, + self.letterbox_layout, ) restored = [ self._restore(output[index], is_logits, shapes[index], pads[index]) @@ -175,7 +176,14 @@ def _restore( align_corners=False, )[0] channels = [ - crop_letterbox(channel, shape, ratio_pad, self.input_shape, "Semantic") + crop_letterbox( + channel, + shape, + ratio_pad, + self.input_shape, + "Semantic", + self.letterbox_layout, + ) for channel in output ] cropped_logits = torch.stack(channels) @@ -193,7 +201,14 @@ def _restore( size=self.input_shape, mode="nearest", )[0, 0].to(dtype=torch.int64) - cropped = crop_letterbox(output, shape, ratio_pad, self.input_shape, "Semantic") + cropped = crop_letterbox( + output, + shape, + ratio_pad, + self.input_shape, + "Semantic", + self.letterbox_layout, + ) return functional.interpolate( cropped[None, None].float(), size=shape, mode="nearest" )[0, 0].to(torch.int64) diff --git a/mblt_vision/utils/preprocess/base.py b/mblt_vision/utils/preprocess/base.py index 7d2b775..2f6c0d8 100644 --- a/mblt_vision/utils/preprocess/base.py +++ b/mblt_vision/utils/preprocess/base.py @@ -4,11 +4,14 @@ from typing import Any import torch +from PIL import Image def _spatial_shape(value: Any) -> tuple[int, int] | None: """Return the height and width of image-like preprocessing input.""" + if isinstance(value, Image.Image): + return value.height, value.width shape = getattr(value, "shape", None) if shape is None or len(shape) < 2: return None diff --git a/mblt_vision/utils/preprocess/letterbox.py b/mblt_vision/utils/preprocess/letterbox.py index ea1645f..ca546ad 100644 --- a/mblt_vision/utils/preprocess/letterbox.py +++ b/mblt_vision/utils/preprocess/letterbox.py @@ -3,8 +3,15 @@ import cv2 import numpy as np import torch - -from ..letterbox import LetterBoxGeometry, RatioPad +from PIL import Image + +from ..letterbox import ( + LetterBoxGeometry, + LetterBoxLayout, + RatioPad, + SizeRounding, + deprecated_center_argument, +) from ..types import TensorLike from ._validation import normalize_image_size, normalize_uint8_rgb_array from .base import PreOps @@ -15,7 +22,9 @@ def _apply_letterbox( img_size: list[int], interpolation: int, padding_value: int | tuple[int, int, int], - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> tuple[np.ndarray, RatioPad]: """Resize and pad an array while preserving its aspect ratio. @@ -24,15 +33,17 @@ def _apply_letterbox( img_size: Target size as ``[height, width]``. interpolation: OpenCV interpolation mode. padding_value: Constant border value. - center: Center the resized array, or anchor it top-left. + layout: The model's ``LetterBoxLayout``, or the older ``center`` boolean. + center: Deprecated spelling of a boolean ``layout``. Returns: The letterboxed array and its resize/padding metadata. """ + layout = deprecated_center_argument(layout, center, "_apply_letterbox") input_shape = (int(img_size[0]), int(img_size[1])) original_shape = (int(image.shape[0]), int(image.shape[1])) - geometry = LetterBoxGeometry.from_shapes(input_shape, original_shape, center) + geometry = LetterBoxGeometry.from_shapes(input_shape, original_shape, layout) resized_height, resized_width = geometry.resized_shape if image.shape[:2] != geometry.resized_shape: image = cv2.resize( @@ -66,7 +77,9 @@ def letterbox_semantic_mask( mask: np.ndarray, img_size: list[int], ignore_label: int = 255, - center: bool = True, + layout: LetterBoxLayout | bool | None = None, + *, + center: bool | None = None, ) -> tuple[np.ndarray, RatioPad]: """Letterbox a semantic mask without interpolating class IDs. @@ -74,8 +87,9 @@ def letterbox_semantic_mask( mask: Two-dimensional semantic class map. img_size: Target size as ``[height, width]``. ignore_label: Class value used for padded pixels. - center: Center the mask, or anchor it top-left. Pass the model's - ``LetterBox.center`` so the target matches its image's geometry. + layout: The model's ``letterbox_layout(pre_cfg)`` (or the older + ``center`` boolean), so the target matches its image's geometry. + center: Deprecated spelling of a boolean ``layout``. Returns: The letterboxed mask and its resize/padding metadata. @@ -88,7 +102,8 @@ def letterbox_semantic_mask( raise ValueError( f"Semantic masks must be two-dimensional, got shape {mask.shape}." ) - return _apply_letterbox(mask, img_size, cv2.INTER_NEAREST, ignore_label, center) + layout = deprecated_center_argument(layout, center, "letterbox_semantic_mask") + return _apply_letterbox(mask, img_size, cv2.INTER_NEAREST, ignore_label, layout) class LetterBox(PreOps): @@ -97,15 +112,23 @@ class LetterBox(PreOps): Resizes the image while maintaining aspect ratio, adding padding to meet target dimensions. Floating-point RGB inputs in ``[0, 1]`` are scaled to byte RGB; other floating-point values must be finite and in ``[0, 255]``. - The defaults are Ultralytics' letterbox: centered, padded with 114. YOLOX - anchors the same resize top-left with 114, and DAMO-YOLO's December 2022 - checkpoints anchor it top-left with zeros. + The defaults are Ultralytics' letterbox: centered, padded with 114, the + resized size rounded. Upstream YOLOX truncates the size with ``int()`` and + anchors it top-left with 114; DAMO-YOLO's December 2022 checkpoints do the + same with zeros and restore each axis by its own ratio. PIL images, as + ``Reader(style="pil")`` returns them, are converted with ``np.asarray`` the + way DAMO-YOLO's dataset does. Ref: https://github.com/ultralytics/ultralytics/blob/main/ultralytics/data/augment.py#L1535 """ def __init__( - self, img_size: list[int], center: bool = True, padding_value: int = 114 + self, + img_size: list[int], + center: bool = True, + padding_value: int = 114, + size_rounding: SizeRounding = "round", + per_axis_ratio: bool = False, ) -> None: """Initializes LetterBox with target image size. @@ -113,17 +136,20 @@ def __init__( img_size (list[int]): Target image size [h, w]. center: Split the padding around the image, or anchor it top-left. padding_value: Byte value filling every channel of the padding. + size_rounding: ``"round"`` the resized size, or ``"floor"`` it as + upstream YOLOX and DAMO-YOLO do with ``int(w * r)``. + per_axis_ratio: Record each axis' resized-over-original ratio for + restoration instead of the aspect-preserving one. Raises: - TypeError: If ``center`` is not a boolean or ``padding_value`` not an integer. - ValueError: If ``padding_value`` is outside ``[0, 255]``. + TypeError: If ``center`` or ``per_axis_ratio`` is not a boolean, or + ``padding_value`` not an integer. + ValueError: If ``padding_value`` is outside ``[0, 255]`` or + ``size_rounding`` is unsupported. """ super().__init__() self.img_size = normalize_image_size(img_size, name="img_size") - if not isinstance(center, bool): - raise TypeError( - f"LetterBox center must be a boolean, got {type(center).__name__}." - ) + self.layout = LetterBoxLayout(center, size_rounding, per_axis_ratio) if isinstance(padding_value, bool) or not isinstance(padding_value, int): raise TypeError( "LetterBox padding_value must be an integer, " @@ -134,10 +160,12 @@ def __init__( f"LetterBox padding_value must be in [0, 255], got {padding_value}." ) self.center = center + self.size_rounding = size_rounding + self.per_axis_ratio = per_axis_ratio self.padding_value = padding_value self.ratio_pad: tuple[tuple[float, float], tuple[float, float]] | None = None - def __call__(self, x: TensorLike) -> torch.Tensor: + def __call__(self, x: TensorLike | Image.Image) -> torch.Tensor: """Executes YOLO preprocessing (letterbox resizing). The call's geometry is also left in ``self.ratio_pad`` for compatibility. @@ -145,7 +173,7 @@ def __call__(self, x: TensorLike) -> torch.Tensor: callers must use ``with_ratio_pad`` instead. Args: - x (TensorLike): Input image. + x (TensorLike | Image.Image): Input image. Returns: torch.Tensor: Preprocessed image in HWC format on the selected device. @@ -153,20 +181,25 @@ def __call__(self, x: TensorLike) -> torch.Tensor: img, self.ratio_pad = self.with_ratio_pad(x) return img - def with_ratio_pad(self, x: TensorLike) -> tuple[torch.Tensor, RatioPad]: + def with_ratio_pad( + self, x: TensorLike | Image.Image + ) -> tuple[torch.Tensor, RatioPad]: """Letterbox an image and return its geometry without touching instance state. Args: - x (TensorLike): Input image. + x (TensorLike | Image.Image): Input image. Returns: The preprocessed HWC image on the selected device and its ``ratio_pad``. """ if isinstance(x, torch.Tensor): x = x.detach().cpu().numpy() + elif isinstance(x, Image.Image): + x = np.asarray(x) elif not isinstance(x, np.ndarray): raise TypeError( - f"LetterBox expects a NumPy array or tensor, got {type(x).__name__}." + "LetterBox expects a NumPy array, tensor or PIL image, " + f"got {type(x).__name__}." ) if x.ndim != 3: raise ValueError(f"LetterBox expects an HWC image, got shape {x.shape}.") @@ -176,6 +209,6 @@ def with_ratio_pad(self, x: TensorLike) -> tuple[torch.Tensor, RatioPad]: self.img_size, cv2.INTER_LINEAR, (self.padding_value,) * 3, - self.center, + self.layout, ) return torch.from_numpy(img).to(self.device).byte(), ratio_pad diff --git a/mblt_vision/utils/results.py b/mblt_vision/utils/results.py index d36a4bd..8bbed17 100644 --- a/mblt_vision/utils/results.py +++ b/mblt_vision/utils/results.py @@ -30,6 +30,7 @@ LetterBoxGeometry, RatioPad, letterbox_center, + letterbox_layout, resolve_ratio_pad, ) from mblt_vision.utils.postprocess.common import ( @@ -414,15 +415,17 @@ def _shape_ratio_pad( ) -> RatioPad | None: """Return letterbox metadata for a plot, which only has the image shapes. - ``None`` lets the shared helpers derive their centered geometry, which is - right for Ultralytics models; a top-left model (``LetterBox.center: false``) - needs its own zero padding spelled out. + ``None`` lets the shared helpers derive their centered, rounded geometry, + which is right for Ultralytics models; any other layout (a top-left + ``LetterBox.center: false``, ``size_rounding: floor``, ``per_axis_ratio``) + needs its own geometry spelled out. """ - if letterbox_center(self.pre_cfg): + layout = letterbox_layout(self.pre_cfg) + if layout.is_default: return None return resolve_ratio_pad( - (int(img1_shape[0]), int(img1_shape[1])), img0_shape, center=False + (int(img1_shape[0]), int(img1_shape[1])), img0_shape, layout=layout ) def _restore_dense_map( @@ -443,7 +446,7 @@ def _restore_dense_map( geometry = LetterBoxGeometry.from_shapes( (int(input_shape[0]), int(input_shape[1])), image_shape, - letterbox_center(self.pre_cfg), + letterbox_layout(self.pre_cfg), ) output_shape = (int(output.shape[0]), int(output.shape[1])) top, bottom, left, right = geometry.crop_bounds(output_shape) diff --git a/tests/test_eval_coco.py b/tests/test_eval_coco.py index 6613f44..63cc503 100644 --- a/tests/test_eval_coco.py +++ b/tests/test_eval_coco.py @@ -46,6 +46,7 @@ def __len__(self) -> int: return len(self.ids) class _Model: + pre_cfg: dict[str, Any] = {} post_cfg = {"task": "pose_estimation", "dataset": "coco"} def set_postprocess_thresholds( @@ -85,7 +86,9 @@ def _evaluate( def test_coco_evaluation_rejects_non_coco_model_taxonomy() -> None: """Do not evaluate another taxonomy using the hard-coded COCO ID mapping.""" - model = SimpleNamespace(post_cfg={"task": "object_detection", "dataset": "dotav1"}) + model = SimpleNamespace( + pre_cfg={}, post_cfg={"task": "object_detection", "dataset": "dotav1"} + ) with pytest.raises(ValueError, match="post_cfg.dataset to be 'coco'"): eval_coco_module.eval_coco_metrics(model, "/dataset", batch_size=1) @@ -97,11 +100,13 @@ def test_coco_evaluation_rejects_noncanonical_artifact_categories( """Do not score direct COCO artifacts with an incompatible category taxonomy.""" dataset = SimpleNamespace(coco=SimpleNamespace(cats={999: {}}, anns={})) - monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_: dataset) + monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_, **__: dataset) with pytest.raises(ValueError, match=r"unsupported category IDs: \[999\]"): eval_coco_module.eval_coco_metrics( - SimpleNamespace(post_cfg={"task": "object_detection", "dataset": "coco"}), + SimpleNamespace( + pre_cfg={}, post_cfg={"task": "object_detection", "dataset": "coco"} + ), "/dataset", batch_size=1, ) @@ -132,11 +137,13 @@ def test_coco_evaluation_rejects_duplicate_raw_annotation_ids( anns={1: annotation}, ), ) - monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_: dataset) + monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_, **__: dataset) with pytest.raises(ValueError, match="duplicate or invalid raw IDs"): eval_coco_module.eval_coco_metrics( - SimpleNamespace(post_cfg={"task": "object_detection", "dataset": "coco"}), + SimpleNamespace( + pre_cfg={}, post_cfg={"task": "object_detection", "dataset": "coco"} + ), "/dataset", batch_size=1, ) @@ -150,11 +157,13 @@ def test_coco_evaluation_rejects_empty_ground_truth( dataset = SimpleNamespace( coco=SimpleNamespace(cats={1: {}}, imgs={1: {"height": 1, "width": 1}}, anns={}) ) - monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_: dataset) + monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_, **__: dataset) with pytest.raises(ValueError, match="at least one annotation"): eval_coco_module.eval_coco_metrics( - SimpleNamespace(post_cfg={"task": "object_detection", "dataset": "coco"}), + SimpleNamespace( + pre_cfg={}, post_cfg={"task": "object_detection", "dataset": "coco"} + ), "/dataset", batch_size=1, ) @@ -172,11 +181,13 @@ def test_coco_evaluation_rejects_undeclared_annotation_categories( imgs={1: {"height": 1, "width": 1}}, ) ) - monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_: dataset) + monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_, **__: dataset) with pytest.raises(ValueError, match="invalid task-specific annotations"): eval_coco_module.eval_coco_metrics( - SimpleNamespace(post_cfg={"task": "object_detection", "dataset": "coco"}), + SimpleNamespace( + pre_cfg={}, post_cfg={"task": "object_detection", "dataset": "coco"} + ), "/dataset", batch_size=1, ) @@ -204,12 +215,13 @@ def test_coco_evaluation_rejects_invalid_task_payloads( }, ) ) - monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_: dataset) + monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_, **__: dataset) with pytest.raises(ValueError, match="invalid task-specific annotations"): eval_coco_module.eval_coco_metrics( SimpleNamespace( - post_cfg={"task": "instance_segmentation", "dataset": "coco"} + pre_cfg={}, + post_cfg={"task": "instance_segmentation", "dataset": "coco"}, ), "/dataset", batch_size=1, @@ -238,12 +250,13 @@ def test_coco_evaluation_rejects_polygons_outside_image_bounds( }, ) ) - monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_: dataset) + monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_, **__: dataset) with pytest.raises(ValueError, match="invalid task-specific annotations"): eval_coco_module.eval_coco_metrics( SimpleNamespace( - post_cfg={"task": "instance_segmentation", "dataset": "coco"} + pre_cfg={}, + post_cfg={"task": "instance_segmentation", "dataset": "coco"}, ), "/dataset", batch_size=1, @@ -336,11 +349,13 @@ def test_coco_evaluation_rejects_labeled_keypoints_outside_images( }, ) ) - monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_: dataset) + monkeypatch.setattr(eval_coco_module, "CustomCOCODataset", lambda *_, **__: dataset) with pytest.raises(ValueError, match="invalid task-specific annotations"): eval_coco_module.eval_coco_metrics( - SimpleNamespace(post_cfg={"task": "pose_estimation", "dataset": "coco"}), + SimpleNamespace( + pre_cfg={}, post_cfg={"task": "pose_estimation", "dataset": "coco"} + ), "/dataset", batch_size=1, ) diff --git a/tests/test_yolox_damoyolo.py b/tests/test_yolox_damoyolo.py index 2417b60..c808b64 100644 --- a/tests/test_yolox_damoyolo.py +++ b/tests/test_yolox_damoyolo.py @@ -2,6 +2,7 @@ from __future__ import annotations +from pathlib import Path from types import SimpleNamespace import cv2 @@ -9,7 +10,12 @@ import pytest import torch -from mblt_vision.utils.letterbox import LetterBoxGeometry, letterbox_center +from mblt_vision.utils.letterbox import ( + LetterBoxGeometry, + LetterBoxLayout, + letterbox_center, + letterbox_layout, +) from mblt_vision.utils.postprocess import build_postprocess from mblt_vision.utils.postprocess.damoyolo_post import DAMOYOLODetectionPost from mblt_vision.utils.postprocess.yolox_post import YOLOXDetectionPost @@ -82,7 +88,7 @@ def _damo_heads(batch: int = 1, seed: int = 0) -> tuple[list, list]: def test_top_left_geometry_pads_only_bottom_and_right() -> None: - geometry = LetterBoxGeometry.from_shapes((640, 640), (480, 640), center=False) + geometry = LetterBoxGeometry.from_shapes((640, 640), (480, 640), layout=False) assert geometry.pad == (0, 0) assert geometry.borders == (0, 160, 0, 0) @@ -536,7 +542,7 @@ def test_semantic_targets_follow_a_top_left_letterbox(loader_name: str) -> None: 1, preprocess.with_metadata, image_size=(64, 64), - center=False, + layout=LetterBoxLayout(center=False), ) _, targets, _, ratio_pads, _ = next(iter(loader)) @@ -572,7 +578,7 @@ def fake_loader(*args: object, **kwargs: object) -> object: with pytest.raises(_Stop): eval_module.eval_semantic_segmentation(model, "/ade20k", 1) - assert seen["center"] is False + assert seen["layout"] == LetterBoxLayout(center=False) def test_dota_ground_truth_fallback_follows_the_models_anchoring() -> None: @@ -582,7 +588,7 @@ def test_dota_ground_truth_fallback_follows_the_models_anchoring() -> None: centered = eval_dota._ratio_pad_for_shape((640, 640), (480, 640), None) top_left = eval_dota._ratio_pad_for_shape( - (640, 640), (480, 640), None, center=False + (640, 640), (480, 640), None, LetterBoxLayout(center=False) ) assert centered == (1.0, (0.0, 80.0)) @@ -613,3 +619,416 @@ def test_top_left_mask_crop_without_metadata_matches_the_explicit_pads( assert bool((derived > 0.5).all()) torch.testing.assert_close(derived, explicit, rtol=0, atol=0) + + +# --- Upstream preprocessing parity ----------------------------------------------- +# +# The references below copy upstream's own test transforms, which are the source of +# truth for these families: YOLOX ``yolox/data/data_augment.py:preproc`` and +# DAMO-YOLO's December 2022 ``Resize`` + ``to_image_list`` (tinyvision/DAMO-YOLO +# 55ae14f), whose checkpoints are the only ones still downloadable. + +# Shapes where truncation and half-to-even rounding differ (335 * 1.28 = 428.8), plus +# exact and tiny ones where they agree. +UPSTREAM_SHAPES = [(335, 500), (427, 640), (480, 640), (500, 333), (1, 7), (123, 457)] + + +def _model_pre_cfg(name: str) -> dict: + import yaml + + import mblt_vision + + path = Path(mblt_vision.__file__).parent / "models" / f"{name}.yaml" + return yaml.safe_load(path.read_text())["DEFAULT"]["pre_cfg"] + + +def _upstream_yolox_preproc(bgr: np.ndarray, input_size: tuple[int, int]) -> np.ndarray: + padded_img = np.ones((input_size[0], input_size[1], 3), dtype=np.uint8) * 114 + r = min(input_size[0] / bgr.shape[0], input_size[1] / bgr.shape[1]) + resized_img = cv2.resize( + bgr, + (int(bgr.shape[1] * r), int(bgr.shape[0] * r)), + interpolation=cv2.INTER_LINEAR, + ).astype(np.uint8) + padded_img[: int(bgr.shape[0] * r), : int(bgr.shape[1] * r)] = resized_img + return padded_img + + +def _upstream_damo_transform(rgb: np.ndarray, size: int) -> np.ndarray: + h, w = rgb.shape[:2] + r = min(size / w, size / h) + resized = cv2.resize( + rgb, (int(w * r), int(h * r)), interpolation=cv2.INTER_LINEAR + ).astype(np.uint8) + canvas = np.zeros((size, size, 3), dtype=np.uint8) + canvas[: resized.shape[0], : resized.shape[1]] = resized + return canvas + + +def _random_rgb(shape: tuple[int, int], seed: int) -> np.ndarray: + return np.random.default_rng(seed).integers(0, 256, (*shape, 3), dtype=np.uint8) + + +def test_floor_rounding_truncates_the_resized_size_as_upstream_does() -> None: + floor = LetterBoxGeometry.from_shapes( + (640, 640), (335, 500), LetterBoxLayout(center=False, size_rounding="floor") + ) + rounded = LetterBoxGeometry.from_shapes((640, 640), (335, 500), False) + + # 335 * 1.28 = 428.8: upstream's int() keeps 428 where round() gives 429. + assert floor.resized_shape == (428, 640) + assert rounded.resized_shape == (429, 640) + assert floor.ratio_pad == ((1.28, 1.28), (0, 0)) + + +def test_per_axis_ratio_records_each_axis_resized_over_original() -> None: + geometry = LetterBoxGeometry.from_shapes( + (640, 640), + (335, 500), + LetterBoxLayout(center=False, size_rounding="floor", per_axis_ratio=True), + ) + + assert geometry.ratio_pad == ((640 / 500, 428 / 335), (0, 0)) + + +@pytest.mark.parametrize("model", ["YOLOX-s", "YOLOX-Nano"]) +@pytest.mark.parametrize("shape", UPSTREAM_SHAPES) +def test_yolox_preprocessing_matches_upstream_preproc(model: str, shape) -> None: + pre_cfg = _model_pre_cfg(model) + size = tuple(pre_cfg["LetterBox"]["img_size"]) + rgb = _random_rgb(shape, seed=shape[0]) + + image, metadata = build_preprocess(pre_cfg).with_metadata(rgb) + + expected = _upstream_yolox_preproc(np.ascontiguousarray(rgb[..., ::-1]), size) + assert np.array_equal(image.numpy(), expected) + # Upstream's COCOEvaluator restores boxes by dividing by r alone. + r = min(size[0] / shape[0], size[1] / shape[1]) + assert metadata["ratio_pad"] == ((r, r), (0, 0)) + assert metadata["img0_shape"] == shape + + +@pytest.mark.parametrize("shape", UPSTREAM_SHAPES) +def test_damoyolo_preprocessing_matches_the_december_2022_transform( + tmp_path, shape +) -> None: + from PIL import Image + + pre_cfg = _model_pre_cfg("DAMO-YOLO-T") + path = tmp_path / "sample.jpg" + Image.fromarray(_random_rgb(shape, seed=shape[1])).save(path, quality=90) + + image, metadata = build_preprocess(pre_cfg).with_metadata(path) + + # Upstream decodes with PIL, not cv2, whose JPEG decode can differ. + decoded = np.asarray(Image.open(path).convert("RGB")) + assert np.array_equal(image.numpy(), _upstream_damo_transform(decoded, 640)) + h, w = shape + r = min(640 / w, 640 / h) + assert metadata["ratio_pad"] == ((int(w * r) / w, int(h * r) / h), (0, 0)) + assert metadata["img0_shape"] == shape + + +def test_damoyolo_restores_boxes_per_axis_like_boxlist_resize() -> None: + pre_cfg = _model_pre_cfg("DAMO-YOLO-T") + post = build_postprocess(pre_cfg, DAMO_POST) + shape = (335, 500) + boxes = torch.tensor([[10.0, 20.0, 300.0, 400.0, 0.9, 0.0]]) + + _, (restored,), _ = post.nmsout2eval([boxes], (640, 640), [shape]) + + # BoxList.resize: x by w / resized_w, y by h / resized_h (428 rows, not 429). + ratio_w, ratio_h = 500 / 640, 335 / 428 + x1, y1, x2, y2 = 10 * ratio_w, 20 * ratio_h, 300 * ratio_w, 400 * ratio_h + assert restored[0] == pytest.approx([x1, y1, x2 - x1, min(y2, 335) - y1], abs=1e-3) + + +def test_yolox_restores_boxes_by_dividing_by_r() -> None: + post = build_postprocess(_model_pre_cfg("YOLOX-s"), YOLOX_POST) + shape = (335, 500) + boxes = torch.tensor([[10.0, 20.0, 300.0, 400.0, 0.9, 0.0]]) + + _, (restored,), _ = post.nmsout2eval([boxes], (640, 640), [shape]) + + r = 1.28 + assert restored[0] == pytest.approx( + [10 / r, 20 / r, 300 / r - 10 / r, min(400 / r, 335) - 20 / r], abs=1e-3 + ) + + +def test_letterbox_layout_reads_and_validates_model_config() -> None: + assert letterbox_layout({}) == LetterBoxLayout() + assert letterbox_layout(_model_pre_cfg("DAMO-YOLO-T")) == LetterBoxLayout( + center=False, size_rounding="floor", per_axis_ratio=True + ) + with pytest.raises(ValueError, match="pre_cfg.LetterBox size_rounding"): + letterbox_layout({"LetterBox": {"size_rounding": "ceil"}}) + with pytest.raises(TypeError, match="per_axis_ratio"): + letterbox_layout({"LetterBox": {"per_axis_ratio": 1}}) + + +@pytest.mark.parametrize( + ("task", "option"), + [ + ("instance_segmentation", {"size_rounding": "floor"}), + ("instance_segmentation", {"per_axis_ratio": True}), + ("obb", {"per_axis_ratio": True}), + ], +) +def test_tasks_that_cannot_restore_a_layout_option_reject_it(task, option) -> None: + pre_cfg = {"LetterBox": {"img_size": [640, 640], "center": False, **option}} + post_cfg = { + "task": task, + "dataset": "dotav1" if task == "obb" else "coco", + "nl": 3, + } + + with pytest.raises(ValueError, match="LetterBox"): + build_postprocess(pre_cfg, post_cfg) + + +def test_letterbox_accepts_the_pil_reader_output_and_reports_its_shape() -> None: + from PIL import Image + + rgb = _random_rgb((48, 64), seed=1) + pre_cfg = { + "Reader": {"style": "pil"}, + "LetterBox": {"img_size": [64, 64], "center": False, "padding_value": 0}, + } + + image, metadata = build_preprocess(pre_cfg).with_metadata(Image.fromarray(rgb)) + + assert metadata["img0_shape"] == (48, 64) + assert np.array_equal(image.numpy()[:48], rgb) + assert not image.numpy()[48:].any() + + +# --- COCO decoding and the deprecated ``center=`` keyword ------------------------ + + +def _write_coco(tmp_path, shape: tuple[int, int]) -> tuple[Path, Path]: + import json + + from PIL import Image + + images = tmp_path / "val2017" + images.mkdir() + Image.fromarray(_random_rgb(shape, seed=7)).save(images / "1.jpg", quality=85) + annotation = tmp_path / "instances_val2017.json" + annotation.write_text( + json.dumps( + { + "images": [ + { + "id": 1, + "file_name": "1.jpg", + "height": shape[0], + "width": shape[1], + } + ], + "annotations": [], + "categories": [{"id": 1, "name": "person"}], + } + ) + ) + return images, annotation + + +@pytest.mark.parametrize("decoder", ["cv2", "pil"]) +def test_coco_dataset_decodes_with_the_requested_library(tmp_path, decoder) -> None: + from PIL import Image + + from mblt_vision.utils.datasets import CustomCOCODataset + + images, annotation = _write_coco(tmp_path, (37, 53)) + image, _, height, width = CustomCOCODataset( + str(images), str(annotation), decoder=decoder + )[0] + + path = images / "1.jpg" + expected = ( + np.asarray(Image.open(path).convert("RGB")) + if decoder == "pil" + else cv2.cvtColor(cv2.imread(str(path)), cv2.COLOR_BGR2RGB) + ) + assert np.array_equal(image, expected) + assert image.flags.writeable + assert (height, width) == (37, 53) + + +def test_coco_dataset_rejects_an_unknown_decoder(tmp_path) -> None: + from mblt_vision.utils.datasets import CustomCOCODataset + + images, annotation = _write_coco(tmp_path, (8, 8)) + with pytest.raises(ValueError, match="decoder"): + CustomCOCODataset(str(images), str(annotation), decoder="skimage") + + +@pytest.mark.parametrize( + ("model", "decoder"), [("DAMO-YOLO-T", "pil"), ("YOLOX-s", "cv2")] +) +def test_coco_evaluation_decodes_like_the_models_reader( + monkeypatch: pytest.MonkeyPatch, model: str, decoder: str +) -> None: + import importlib + + eval_coco = importlib.import_module("mblt_vision.utils.evaluation.eval_coco") + seen: dict[str, object] = {} + + class _Stop(Exception): + pass + + def fake_dataset(*args: object, **kwargs: object) -> object: + seen.update(kwargs) + raise _Stop + + monkeypatch.setattr(eval_coco, "CustomCOCODataset", fake_dataset) + engine = SimpleNamespace( + pre_cfg=_model_pre_cfg(model), + post_cfg={"task": "object_detection", "dataset": "coco"}, + ) + + with pytest.raises(_Stop): + eval_coco.eval_coco_metrics(engine, "/coco", 1) + + assert seen["decoder"] == decoder + + +def test_deprecated_center_keyword_still_works_with_a_warning() -> None: + from mblt_vision.utils.letterbox import resolve_ratio_pad + from mblt_vision.utils.preprocess import letterbox_semantic_mask + + mask = np.zeros((48, 64), dtype=np.uint8) + with pytest.warns(DeprecationWarning, match="center"): + legacy = letterbox_semantic_mask(mask, [64, 64], center=False) + current = letterbox_semantic_mask(mask, [64, 64], layout=False) + assert np.array_equal(legacy[0], current[0]) and legacy[1] == current[1] + + with pytest.warns(DeprecationWarning, match="center"): + geometry = LetterBoxGeometry.from_shapes((640, 640), (480, 640), center=False) + assert geometry.pad == (0, 0) + with pytest.warns(DeprecationWarning, match="center"): + assert resolve_ratio_pad((640, 640), (480, 640), center=False)[1] == (0, 0) + + with pytest.raises(TypeError, match="not both"): + letterbox_semantic_mask( + mask, [64, 64], layout=LetterBoxLayout(center=False), center=False + ) + + +@pytest.mark.parametrize("loader_name", ["get_ade20k_loader", "get_cityscapes_loader"]) +def test_semantic_loaders_accept_the_deprecated_center_keyword(loader_name) -> None: + from mblt_vision.utils.datasets import dataloader + + preprocess = build_preprocess( + {"LetterBox": {"img_size": [64, 64], "center": False}} + ) + image = np.full((48, 64, 3), 100, dtype=np.uint8) + target = np.full((48, 64), 3, dtype=np.uint8) + with pytest.warns(DeprecationWarning, match="center"): + loader = getattr(dataloader, loader_name)( + [(image, target, "sample")], + 1, + preprocess.with_metadata, + image_size=(64, 64), + center=False, + ) + + _, _, _, ratio_pads, _ = next(iter(loader)) + assert ratio_pads == [((1.0, 1.0), (0, 0))] + + +def _renamed_center_callables() -> list[tuple[str, object]]: + """Every callable whose ``center`` parameter became ``layout``, with a top-left call.""" + + import importlib + + from mblt_vision.utils.datasets import dataloader + from mblt_vision.utils.letterbox import resolve_ratio_pad + from mblt_vision.utils.postprocess._letterbox import resolve_ratio_pads + from mblt_vision.utils.preprocess import letterbox_semantic_mask + from mblt_vision.utils.preprocess.letterbox import _apply_letterbox + + eval_dota = importlib.import_module("mblt_vision.utils.evaluation.eval_dota") + mask = np.zeros((48, 64), dtype=np.uint8) + preprocess = build_preprocess( + {"LetterBox": {"img_size": [64, 64], "center": False}} + ) + sample = [(np.zeros((48, 64, 3), np.uint8), np.zeros((48, 64), np.uint8), "s")] + # One 10x10 square, so the result shows whether the 80-row centered pad was added. + square = torch.tensor([[[0.0, 0.0], [10.0, 0.0], [10.0, 10.0], [0.0, 10.0]]]) + gt = {"cls": torch.zeros(1), "bboxes": torch.zeros((1, 5)), "polygons": square} + + def loader(name: str): + return lambda **kw: next( + iter( + getattr(dataloader, name)( + sample, 1, preprocess.with_metadata, image_size=(64, 64), **kw + ) + ) + )[3] + + return [ + ( + "from_shapes", + lambda **kw: LetterBoxGeometry.from_shapes( + (640, 640), (480, 640), **kw + ).pad, + ), + ( + "resolve_ratio_pad", + lambda **kw: resolve_ratio_pad((640, 640), (480, 640), **kw), + ), + ( + "resolve_ratio_pads", + lambda **kw: resolve_ratio_pads(None, 1, [(480, 640)], (640, 640), **kw), + ), + ( + "letterbox_semantic_mask", + lambda **kw: letterbox_semantic_mask(mask, [64, 64], **kw)[1], + ), + ( + "_apply_letterbox", + lambda **kw: _apply_letterbox(mask, [64, 64], cv2.INTER_NEAREST, 0, **kw)[ + 1 + ], + ), + ("get_ade20k_loader", loader("get_ade20k_loader")), + ("get_cityscapes_loader", loader("get_cityscapes_loader")), + ( + "_ratio_pad_for_shape", + lambda **kw: eval_dota._ratio_pad_for_shape( + (640, 640), (480, 640), None, **kw + ), + ), + ( + "_ground_truth_to_input_space", + lambda **kw: eval_dota._ground_truth_to_input_space( + gt, (640, 640), (480, 640), None, **kw + )["bboxes"].tolist(), + ), + ] + + +@pytest.mark.parametrize("name", [name for name, _ in _renamed_center_callables()]) +def test_every_renamed_callable_accepts_the_deprecated_center_keyword(name) -> None: + call = dict(_renamed_center_callables())[name] + + with pytest.warns(DeprecationWarning, match="center"): + legacy = call(center=False) + assert legacy == call(layout=LetterBoxLayout(center=False)) + # An explicit layout=True is given, not omitted, even though it equals the default. + for layout in (LetterBoxLayout(center=False), True, False): + with pytest.raises(TypeError, match="not both"): + call(layout=layout, center=False) + + +def test_deprecated_center_argument_tells_an_omitted_layout_from_layout_true() -> None: + from mblt_vision.utils.letterbox import deprecated_center_argument + + assert deprecated_center_argument(None, None, "f") is True + assert deprecated_center_argument(True, None, "f") is True + with pytest.warns(DeprecationWarning): + assert deprecated_center_argument(None, False, "f") is False + with pytest.raises(TypeError, match="not both"): + deprecated_center_argument(True, False, "f")