Segment Anything Annotator & Reviewer

A PyQt desktop application for pixel-level image annotation with Meta's Segment Anything Model (SAM). Generate masks from point, box, or circle prompts; refine them as polygons; and save annotations in Labelme-compatible JSON files.

This project uses the original SAM models (vit_b, vit_l, and vit_h), not SAM 2.

Features

• SAM-assisted segmentation using positive/negative points, boxes, and circles
• Manual polygon creation and vertex editing
• Multiple mask proposals
• Category and object ID annotation
• Merge and subtract polygon operations
• Polygon simplification
• Hide/show controls for all polygons
• Brightness, contrast, and colour-inversion preview controls
• Jump-to-image dialog and keyboard navigation
• Labelme-compatible JSON output
• Resumable annotation review sessions with timing and audit metadata
• Original/reviewed JSON, polygon geometry, and raster-mask comparison
• Duplicate-safe CSV review export
• Optional STCN-based video annotation workflow
• CUDA, Apple MPS, and CPU device selection

Quick start

1. Create an environment

Python 3.8 or newer is required. A virtual environment is recommended.

Windows PowerShell:

python -m venv annotator_env
.\annotator_env\Scripts\Activate.ps1

Linux or macOS:

python3 -m venv annotator_env
source annotator_env/bin/activate

2. Install PyTorch

Install the build appropriate for your operating system and accelerator using the official PyTorch installer.

For a CPU-only installation:

python -m pip install torch torchvision

3. Install the application dependencies

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

4. Start the image annotator

python annotator.py --app_resolution 1000,1600 --model_type vit_b --keep_input_size True --max_size 720

The --app_resolution value is height,width.

Argument	Description
--model_type	SAM checkpoint type: vit_b, vit_l, or vit_h
--keep_input_size	Keep the original image size for SAM when True
--max_size	Maximum input dimension when resizing is enabled

SAM checkpoints

Click Load SAM to download the selected checkpoint automatically. The checkpoint is saved in the repository root as vit_b.pth, vit_l.pth, or vit_h.pth.

You can instead download a checkpoint manually from the official SAM repository and place it in the repository root using the corresponding filename.

vit_b is the smallest and fastest option. vit_l and vit_h require more memory but use larger model backbones.

Image annotation workflow

1. Click Category File and select a text file containing one category per line, such as categories.txt.
2. Click Image Directory and select a folder containing .jpg or .png images.
3. Click Save Directory and select the output folder.
4. Click Load SAM.
5. Select a prompt mode and annotate the image.
6. Choose one of the mask proposals and click Accept.
7. Edit the resulting polygon if needed, then save the annotation.

The application writes one Labelme-compatible JSON file per image.

Prompt and editing controls

• Point Prompt: left-click for positive points and right-click for negative points.
• Box Prompt: draw a bounding box around the object.
• Circle Prompt: draw a circular prompt around the object.
• Manual Polygons: click along the object boundary to create a polygon.
• Edit Polygons: select and move polygon vertices. Arrow keys nudge the selected polygon.
• Merge Polygons: select a source polygon, activate Merge, then select the target.
• Subtract Polygons: select the polygon to subtract, activate Subtract, then select the target.

Preserving annotation JSON

Loading and saving an existing annotation retains its version, flags, descriptions, unknown metadata, image references, and ordered shape coordinates, including repeated vertices. Missing standard image fields are filled when saving; JSON whitespace and number formatting may change. Empty shape records are retained in the file without being drawn. Newly created annotations use the usual embedded image data and default fields.

Labels such as "0", "01", and "-1" are treated as literal text by default. For older annotations that intentionally store nonnegative category indices, launch with --legacy-label-indices to explicitly map those labels through the current categories.txt order. Out-of-range and negative values stay literal. Saving then writes the displayed labels; use a copied dataset or separate review output when converting legacy files. An unedited OK decision still copies the original JSON and does not convert labels by itself.

This preserves data that exists in the loaded file; it cannot reconstruct metadata or vertices already removed by earlier saves. Geometry-changing tools such as Reduce Points, merge, and subtract are tracked separately in #36.

Image display controls

Show labels replaces the old Class On/Off button. Uncheck it to hide polygon names and group/object-ID text while keeping outlines and vertex handles visible and editable. This preference stays active when moving between frames and does not change saved annotations. Accepting a SAM proposal always opens the label/group dialog, whether labels are shown or hidden.

Press a digit 0–9 to toggle every polygon with that group/object ID (not its class label or category-list position). For example, H hides all polygons, then 3 shows object 3; another 3 hides it again. If part of an object is visible, its digit hides the whole object. Digits with no matching object do nothing. On image navigation, polygons start visible as usual. These shortcuts work in annotation and review mode, and leave numeric typing in text fields and dialogs alone. While SAM proposals are active, 1–3 select Proposal 1–3, corresponding to SAM results 0–2, instead of toggling objects. 0 and 4–9 still toggle objects. Accepting/rejecting proposals or changing frames restores all digits to object visibility. There are three proposal buttons labelled 1, 2, and 3, disabled when no proposals are active. Previews preserve the source image aspect ratio, including after window resizing, with space reserved for each button's digit.

Use Invert colours below Reset B/C to toggle a colour-negative preview, for example when inspecting difficult endolaser frames. Inversion applies after brightness and contrast and stays active when navigating between frames. Click it again to restore the normal colours. Reset B/C resets the two sliders only and leaves the inversion toggle unchanged.

These controls are available in annotation and review mode. They affect the preview only; source images, saved image data, polygons, and SAM input stay unchanged.

Annotation review mode

Review mode reuses the image annotator and polygon canvas to inspect existing Labelme JSON annotations. Original annotations are treated as read-only input; all reviewed JSON files, session state, and audit exports are written to a separate output directory.

The application uses the original SAM models for interactive segmentation. The SAM2 propagated frame option described below is provenance metadata for annotations generated by a separate SAM2 propagation workflow.

Prepare the review directories

Create three separate directories:

dataset/
|-- images/
|   |-- sequence-1/
|   |   |-- 000001.jpg
|   |   |-- 000002.jpg
|   |   `-- 000003.jpg
|-- original-json/
|   `-- sequence-1/
|       |-- 000001.json
|       `-- 000003.json
`-- reviewed-json/

The review output directory must already exist and must not be the original annotation directory or a directory inside it. Using a separate output directory per reviewer is recommended when reviewers work concurrently.

Images and annotations are discovered recursively. Supported images are .jpg, .jpeg, and .png. They are matched by their case-insensitive relative path without the extension; for example, images/sequence-1/000001.jpg matches original-json/sequence-1/000001.json.

Images without matching JSON files are reported as warnings and remain available as view-only temporal context. Orphan JSON files, duplicate relative keys, invalid JSON, malformed polygon points, unreadable images, and dimension mismatches are also reported during validation. Errors block the session; warnings can be reviewed and accepted.

Start a review session

python annotator.py --review --app_resolution 1000,1600 --model_type vit_b --keep_input_size True --max_size 720

The startup dialog requests:

• reviewer ID (required);
• reviewer role (optional and remembered for the next session);
• default provenance for newly created review targets;
• image directory;
• original annotation directory;
• reviewed annotation output directory.

The fields can be prefilled from the command line:

python annotator.py --review \
  --reviewer-id "reviewer-01" \
  --reviewer-role "grader" \
  --review-images "/path/to/images" \
  --review-annotations "/path/to/original-json" \
  --review-output "/path/to/reviewed-json"

Use quoted Windows paths in PowerShell or Git Bash when a directory contains spaces.

Review argument	Description
--review	Open the review-session configuration dialog
--reviewer-id	Prefill the required reviewer identifier
--reviewer-role	Prefill the optional reviewer role
--review-images	Prefill the image directory
--review-annotations	Prefill the read-only original JSON directory
--review-output	Prefill the reviewed JSON and session-output directory

Review workflow

1. Validate the directories and accept any understood warnings.
2. Inspect the image and its loaded annotation. A previously reviewed JSON in the output directory takes precedence when a session is resumed.
3. Optionally edit polygons manually or load SAM for prompt-assisted changes.
4. Select the annotation provenance.
5. Add an optional problem status or reviewer note.
6. Record one decision:
   • No change: discard unsaved edits and copy the original JSON exactly;
   • Minor correction: save the current annotation as a reviewed JSON;
   • Major correction: save the current annotation as a reviewed JSON;
   • Unable to review: record the outcome without requiring a new JSON.
7. The app saves the audit record and advances to the next unfinished review target.
8. Use Finish review session only when the session should no longer resume.

Closing the application does not complete a session. Notes, provenance, timer state, decisions, and current progress are persisted in SQLite. Reopening the same reviewer and directory configuration resumes the active session at the first in-progress or unreviewed target. A completed session is never resumed; the same configuration starts a new session instead.

Timing and navigation

The active timer starts automatically on an annotated review target. Pause and Resume exclude interruptions, while Reset timer resets only the current target after confirmation. Context images pause the timer automatically and disable review decisions.

Decision shortcuts submit the displayed decision, save or copy the annotation, and advance. Navigation shortcuts never complete a target. Shortcuts are suppressed while typing in notes or another editable field.

Control	Review-mode action
Page Up / Last Image	Previous image, including context images
Page Down / Next Image	Next image, including context images
J / Jump	Select any image by index or filename
K	No change / OK; save and advance
M	Minor correction; save and advance
L	Major correction; save and advance
O	Unable to review; record and advance
Space	Pause or resume the active review timer
Alt+Left / Previous target	Previous matched target; navigation only
Alt+Right / Next target	Next matched target; navigation only
Export review CSV	Rewrite the current session CSV from SQLite

The progress display counts annotation targets only, not view-only context images. Reviewed includes the four explicit decisions, Opened but undecided means in_progress, and Not yet opened means unreviewed. After at least five completed targets, the approximate remaining time uses the median completed target time so long interruptions do not dominate the estimate.

Provenance categories

Category	Meaning
Unknown / legacy	Source provenance is unavailable
Manual keyframe	Annotation was created manually on a selected keyframe
SAM2 propagated frame	Annotation came from an external SAM2 propagation step
Reviewed propagated frame	A propagated annotation was reviewed or corrected

Provenance is stored per review target and can be changed when revisiting an item. The configuration-dialog default is applied only to new targets; resumed targets preserve their stored per-item values.

Change detection and audit files

Every completed decision records:

• decision status and active review time;
• reviewer ID, role, notes, and problem status;
• original and reviewed JSON SHA-256 hashes;
• whether the exact JSON file changed;
• whether polygon geometry changed;
• whether the rasterised foreground mask changed;
• annotation provenance;
• start, completion, creation, and update timestamps;
• original, reviewed, and image paths.

Geometry comparison ignores polygon-list order, polygon start-point rotation, and reversed polygon direction. The exact JSON hash still detects formatting, metadata, label, or path changes. Raster comparison detects changes to the combined foreground pixels independently of JSON formatting.

Review mode creates these files in the review output directory:

reviewed-json/
|-- review_session.sqlite3
|-- review_session_<session-uuid>.csv
`-- sequence-1/
    |-- 000001.json
    `-- 000003.json

SQLite is the authoritative resumable record. The session-specific CSV is rewritten atomically at session startup, after each decision, on manual export, and when the session is completed. Repeated exports update existing rows rather than appending duplicates. The CSV also retains inactive history if the dataset changes during an active session.

Close the CSV in Excel while reviewing. Windows may lock an open CSV and block replacement; the app will warn about the failed export while preserving the SQLite record.

Review data safety

• Never use the original JSON directory as the review output.
• Back up both the reviewed JSON files and review_session.sqlite3.
• Do not edit the SQLite database manually while the annotator is running.
• Keep reviewer output directories separate unless shared reviewed JSON files are explicitly intended.
• Use Finish review session deliberately; merely closing the app preserves the active session for resumption.
• Finishing a partial session leaves in_progress and unreviewed statuses unchanged but prevents that session from resuming automatically.

Image annotator shortcuts

Shortcut	Action
Shift+I	Choose image directory
Shift+L	Choose save directory
Page Up	Previous image
Page Down	Next image
J	Jump to an image
N or Ctrl+W	Manual polygon mode
P	Point prompt mode
B	Box prompt mode
C	Circle prompt mode
A	Accept the selected proposal
R	Reject proposals and clear prompts
1–3	Select Proposal 1–3 while SAM proposals are active
0–9	Hide/show the matching group/object ID (1–3 select active SAM proposals first)
E	Edit polygon mode
Arrow keys	Nudge a selected polygon
D	Delete selected polygons
H	Hide/show all polygons
Shift+M	Merge polygons
Shift+S	Subtract polygons
Shift+R	Reduce polygon points
U	Undo the last point
Ctrl+U	Undo the last shape edit
S	Save
Alt+S	Save as
Ctrl + mouse wheel	Zoom
Esc	Cancel an active merge or subtract operation

Video annotation

Video annotation additionally requires STCN and its stcn.pth checkpoint:

segment-anything-annotator-v2/
├── STCN/
├── stcn.pth
├── annotator_video.py
└── ...

Organize extracted video frames as:

video_folder/
├── video_1/
│   ├── 00000.jpg
│   ├── 00001.jpg
│   └── ...
└── video_2/
    └── ...

Start the video annotator with:

python annotator_video.py --app_resolution 1000,1600 --model_type vit_b --keep_input_size True --max_size 720 --max_size_STCN 600

Annotate the first frame with SAM, select objects with Ctrl + left-click, add them to memory, and use Propagate on subsequent frames.

Shortcut	Video action
N	Next frame
B	Previous frame
E	Edit mode
A	Accept proposal
R	Reject proposal
D	Delete
S	Save
Space	Propagate
1–3	Select a mask proposal

Platform notes

Windows

• Use a current graphics driver and the matching PyTorch build for CUDA.
• PowerShell may require permission to activate a virtual environment. See Microsoft's documentation for Set-ExecutionPolicy if activation is blocked.
• The application also runs on CPU when CUDA is unavailable.

macOS

• Apple Silicon systems use PyTorch's MPS backend when it is available.
• The application falls back to CPU if MPS is unavailable.

Linux: Qt xcb plugin error

This project uses opencv-python-headless because the GUI is provided by PyQt5. The headless OpenCV package avoids a conflict between OpenCV's bundled Qt plugins and PyQt5's xcb plugin.

Older versions of this project also installed metaseg, which depends on the GUI-enabled OpenCV package. When updating an existing environment, remove that unused dependency and both OpenCV variants before reinstalling:

python -m pip uninstall -y metaseg opencv-python opencv-python-headless
python -m pip install -r requirements.txt

As a temporary workaround without reinstalling, point Qt to PyQt5's plugins:

export QT_PLUGIN_PATH="$(python -c 'import PyQt5, pathlib; print(pathlib.Path(PyQt5.__file__).parent / "Qt5" / "plugins")')"

If xcb still fails after using the headless OpenCV package, install the Qt/XCB system libraries supplied by your Linux distribution.

Tests

Run the complete automated test suite from the repository root:

python -m unittest discover -s tests -v

The tests cover review path validation, resumable SQLite storage, JSON and geometry comparison, raster-mask comparison, provenance, duplicate-safe CSV export, application icon assets, and the complete review audit workflow.

Citation

If you use Segment Anything Annotator & Reviewer in your research, please cite the specific software version used. The citation for version 1.0.0 is:

@software{namour_2026_21726098,
  author       = {Namour, Adriana and
                  Bergeles, Christos},
  title        = {Segment Anything Annotator \& Reviewer},
  month        = jul,
  year         = 2026,
  publisher    = {Zenodo},
  version      = {v1.0.0},
  doi          = {10.5281/zenodo.21726098},
  url          = {https://doi.org/10.5281/zenodo.21726098},
}

For other releases, use the repository's Cite this repository link or the Zenodo record for all versions.

Project history and acknowledgement

This repository is based on haochenheheda/segment-anything-annotator. It builds on Meta Segment Anything, Labelme, and STCN.

See LICENSE for licensing information.