From 8af59b7d29ec4d3b2eb110717361056438b92328 Mon Sep 17 00:00:00 2001 From: Viet-Anh Nguyen Date: Sun, 30 Aug 2026 11:01:00 +0700 Subject: [PATCH] docs: add current implementation agent guide --- AGENT.md | 246 +++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 279 ++---------------------------------------------------- README.md | 4 +- 3 files changed, 256 insertions(+), 273 deletions(-) create mode 100644 AGENT.md diff --git a/AGENT.md b/AGENT.md new file mode 100644 index 0000000..7e61958 --- /dev/null +++ b/AGENT.md @@ -0,0 +1,246 @@ +# AnyLabeling agent guide + +This file describes the current repository architecture and the checks an +automated coding agent must perform. Keep it aligned with `pyproject.toml`, +`anylabeling/app_info.py`, and `.github/workflows/` when those files change. + +## Project summary + +AnyLabeling is a Python 3.11+ desktop image-annotation application built with +PyQt6. Its auto-labeling service runs YOLOv5/v8, SAM/MobileSAM, SAM 2/2.1, and +SAM 3 models through ONNX Runtime; SAM 2 also has a native CoreML path on +macOS. The repository produces: + +- `anylabeling`, the default CPU PyPI distribution; +- `anylabeling-gpu`, the Linux/Windows CUDA PyPI distribution; and +- six standalone CPU/accelerated binaries for Linux, Windows, and Apple + Silicon macOS. + +The application entry point is `anylabeling.app:main`. The version and default +build device are defined in `anylabeling/app_info.py`. + +## Non-negotiable working rules + +- Use a dedicated AnyLabeling virtual or Conda environment. Do not install + project dependencies into a shared machine or user environment. +- Preserve unrelated changes in the working tree. +- Fix bugs with a focused regression test. Reproduce the failure first when + practical, then verify both the focused test and the full suite. +- Set `QT_QPA_PLATFORM=offscreen` for automated Qt tests and smoke tests. +- Use PyQt6 APIs. PySide6 is a development-only resource compiler dependency, + not the runtime UI toolkit. +- Keep UI operations on the Qt main thread. Background model workers must + always report errors and release their worker/thread references. +- Do not grow `label_widget.py` for standalone functionality that belongs in a + focused widget, service, or utility. +- Treat label files and model files as user data: close handles + deterministically, preserve unknown JSON fields, and avoid destructive + migration behavior. +- Never claim accelerator support from provider selection alone. Validate the + provider in an isolated environment and run real inference on matching + hardware when the runtime path changes. + +## Architecture map + +The main UI ownership chain is: + +```text +anylabeling/app.py +└── views/mainwindow.py: MainWindow + └── views/labeling/label_wrapper.py: LabelingWrapper + └── views/labeling/label_widget.py: LabelingWidget + ├── widgets/canvas.py: shapes, selection, grouping, undo state + ├── widgets/auto_labeling/auto_labeling.py: model-facing UI + ├── label_file.py: AnyLabeling JSON serialization + └── dialogs and supporting widgets +``` + +`LabelingWidget` owns file navigation, canvas state, labels, actions, and most +save/dirty-state behavior. Canvas mutations that affect persisted annotations +must store an undo snapshot and emit the signal that marks the document dirty. +Grouping and ungrouping are examples covered by `tests/test_canvas_grouping.py`. + +The auto-labeling path is: + +```text +services/auto_labeling/ +├── registry.py model type registry +├── model.py base QObject and image loading +├── model_manager.py downloads, lifecycle, threaded prediction +├── runtime.py ONNX provider selection/session creation +├── types.py AutoLabelingResult and prompt modes +├── segment_anything.py SAM family detection/dispatch +├── sam_onnx.py SAM 1 and MobileSAM +├── sam2_onnx.py SAM 2 ONNX +├── sam2_coreml.py SAM 2 native CoreML +├── sam3_onnx.py SAM 3 +└── yolov5.py/yolov8.py detection models +``` + +Concrete model classes register at import time with +`@ModelRegistry.register("type")`. When adding a model, import its module from +`services/auto_labeling/__init__.py` and add a matching entry to +`configs/auto_labeling/models.yaml`. Downloaded weights live in +`~/anylabeling_data/models//`. + +Qt resources are declared in `anylabeling/resources/resources.qrc` and +compiled into `resources.py`. Translations are under +`anylabeling/resources/translations/`. After changing icons, `.qrc`, or `.ts` +files, run `python scripts/compile_languages.py`; do not hand-edit generated +resource output. + +## Accelerator and package behavior + +`services/auto_labeling/runtime.py` is the single provider-selection layer. +`ANYLABELING_DEVICE` overrides the build default. Supported names include +`CPU`, `GPU`, `AUTO`, `CUDA`, `COREML`, `DIRECTML`, `ROCM`, `MIGRAPHX`, +`OPENVINO`, `TENSORRT`, `CANN`, `QNN`, `VITISAI`, `WEBGPU`, and the documented +NPU aliases. Selected accelerators retain CPU fallback where available. + +Provider packages must be isolated because ONNX Runtime variants conflict: + +- CPU: `onnxruntime`; +- NVIDIA: `onnxruntime-gpu[cuda,cudnn]` (currently `<1.27` for CUDA 12 driver + compatibility); +- Intel: `onnxruntime-openvino`; +- Windows DirectML: `onnxruntime-directml`; +- other NPUs: the appropriate vendor runtime. + +The GPU publish workflow rewrites authoritative PEP 621 metadata in +`pyproject.toml` to produce `anylabeling-gpu` and replace the CPU runtime +dependency. There is no `setup.py`; do not reintroduce packaging logic there. +macOS intentionally excludes pip-installed PyQt6 and uses a separate +`[macos]` extra for CoreML. + +Use `scripts/check_accelerator.py` to inspect provider availability and +selection. A provider appearing in `get_available_providers()` does not prove +that its native libraries, device, model operators, or inference path work. + +## Dedicated development environment + +Create one environment per runtime variant. For the normal CPU development +path on Linux or Windows: + +```bash +python -m venv .venv +.venv/bin/python -m pip install --upgrade pip +.venv/bin/python -m pip install -e ".[dev]" "ruff==0.15.2" +``` + +On Windows, use `.venv\\Scripts\\python.exe`. On macOS, use a dedicated Conda +environment, install `pyqt=6` from conda-forge, then install `.[macos,dev]`. +Never install CPU and accelerator ONNX Runtime wheels into the same test +environment. + +Run the application from the environment with either: + +```bash +.venv/bin/python anylabeling/app.py +.venv/bin/anylabeling +``` + +## Required validation + +The baseline for every code change is: + +```bash +QT_QPA_PLATFORM=offscreen .venv/bin/python -m unittest discover -s tests -v +.venv/bin/ruff check anylabeling --exclude anylabeling/resources/resources.py +.venv/bin/ruff format --check anylabeling --exclude anylabeling/resources/resources.py +``` + +Run a focused module during iteration, for example: + +```bash +QT_QPA_PLATFORM=offscreen .venv/bin/python -m unittest tests.test_label_file -v +``` + +The suite is standard-library `unittest`. Tests must be deterministic, clean up +temporary files and Qt objects, and avoid network access unless they are +explicit optional integration tests. `tests/test_real_inference.py` skips +models not present under `~/anylabeling_data/models/`; skipped inference tests +are not evidence that a changed model path works. + +Add proportional manual validation after automated tests: + +- UI or canvas change: launch the app, perform the exact interaction, save, + reopen, and exercise undo/redo where relevant. +- File I/O change: repeatedly save, overwrite, reload, rename/delete, and run + with `ResourceWarning` promoted to an error on Windows when handles matter. +- Model lifecycle change: test successful load, malformed config, missing or + corrupt download, cancellation/retry, and a subsequent valid load. +- Provider change: use a fresh environment, verify the selected provider, run + representative real inference, and confirm CPU fallback. +- Packaging change: build from a clean checkout, inspect the archive/wheel, + install it into a second empty environment, and smoke-test the installed + entry point. + +CI runs `.github/workflows/tests.yml` on Ubuntu, Windows, and macOS with Python +3.11, 3.12, and 3.13. All nine jobs must pass. For operating-system-specific +bugs, also test on the affected physical OS when access is available. + +## Build and release + +Build Python distributions and validate their metadata with: + +```bash +.venv/bin/python -m build --sdist --wheel --outdir dist/ . +.venv/bin/python -m twine check dist/* +``` + +Build a local standalone binary with: + +```bash +.venv/bin/python -m pip install pyinstaller +bash scripts/build_executable.sh +``` + +Tags matching `v*.*.*` start three gated workflows: CPU PyPI publishing, GPU +PyPI publishing, and GitHub Release binary builds. Before tagging: + +1. update `__version__` in `anylabeling/app_info.py`; +2. run the full fresh-environment test and lint checks; +3. build and install the wheel in a clean environment; +4. run real model/accelerator checks for affected inference paths; and +5. ensure the working tree and release notes describe exactly what will ship. + +After tagging, wait for every workflow. Confirm both PyPI projects, all six +named release assets, archive integrity, file sizes/checksums, and native-runner +launch smoke tests. The expected assets are: + +```text +AnyLabeling-Linux-CPU-x64 +AnyLabeling-Linux-GPU-x64 +AnyLabeling-Windows-CPU-x64.exe +AnyLabeling-Windows-GPU-x64.exe +AnyLabeling-macOS-CPU.zip +AnyLabeling-macOS-GPU.zip +``` + +Never move a tag after publishing. Correct a bad release with a new patch +version. Keep the README's “Latest Release” section and the documentation +download page in sync with the latest stable tag. + +## Current regression invariants + +- Use scoped Qt enum members (for example, + `QFileDialog.FileMode.ExistingFile`) so PyQt6 dialog subclasses do not depend + on inherited enum aliases. +- Every `io_open()` caller must leave its file closed after the context exits; + Windows must be able to overwrite and delete a label immediately afterward. +- Group/ungroup mutations must persist, mark the document dirty, and remain + undoable. +- Invalid images in SAM preload must be skipped without killing the preload + worker. +- Model download/load failures must clean up worker state so another model can + be selected without restarting the app. +- Provider choice must be deterministic and append CPU fallback when + available; requested unavailable providers must fail over cleanly. + +## Change hygiene + +Keep changes small and explain user-visible behavior in the PR. Reference the +issue, include the reproduction and regression test, report automated and +manual validation, and call out OS or hardware limitations honestly. Do not +mix generated files, caches, local environments, model weights, or build +outputs into commits. diff --git a/CLAUDE.md b/CLAUDE.md index c0ae429..afdd48b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,275 +1,12 @@ # CLAUDE.md -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +@AGENT.md -AnyLabeling is a desktop image-annotation app built on PyQt6, with an -auto-labeling backend that runs ONNX models (YOLOv5/v8, SAM1/MobileSAM, -SAM2, SAM3) and a CoreML path for SAM2 on macOS. PyPI ships two parallel -packages from the same source tree: `anylabeling` (CPU, default) and -`anylabeling-gpu` (Linux/Windows, swaps in `onnxruntime-gpu`). +Claude Code must follow the shared repository guidance in `AGENT.md`. In +particular, work in a dedicated environment, add a regression test for every +bug fix, run the full headless test suite before handing off changes, and do +not tag a release until all cross-platform release checks are green. -## Common commands - -```bash -# Run the app from source (no install needed for dev) -python anylabeling/app.py - -# Run the installed CLI -anylabeling - -# Editable install for development (CPU) -pip install -e ".[dev]" -# NVIDIA dev (dedicated environment): -# pip install -e ".[dev]" -# pip uninstall -y onnxruntime -# pip install "onnxruntime-gpu[cuda,cudnn]>=1.20.0,<1.27" -# macOS dev: pip install -e ".[macos,dev]" # plus conda install -c conda-forge pyqt=6 - -# Lint + format (ruff config is in pyproject.toml) -ruff check . -ruff format . - -# Run all tests -python -m unittest discover -s tests -v - -# Run one test file -python -m unittest tests.test_label_colormap -v - -# Run one test method -python -m unittest tests.test_label_colormap.TestLabelColormapMutability.test_copy_is_always_writable - -# Build a wheel + sdist (CPU). For GPU, sed __preferred_device__ to "GPU" first. -python -m build --sdist --wheel --outdir dist/ . - -# Build a standalone executable -bash build_executable.sh # delegates to PyInstaller via anylabeling.spec -``` - -App-level CLI flags: `--reset-config`, `--logger-level {debug,info,warning,error,fatal}`, -`--config `, `--output / -O / -o`, `--nodata`, `--autosave`, `--nosortlabels`, -`--flags`, plus a positional `filename` (image or label file). Default user -config lives at `~/.anylabelingrc`. - -## High-level architecture - -### Entry point and UI tree - -`anylabeling/app.py` sets `MKL/NUMEXPR/OMP_NUM_THREADS=1` (workaround for a -macOS-M1 bus error in `np.linalg.solve`) before any heavy imports, then -constructs a `QApplication` and a `MainWindow`. The UI tree is intentionally -shallow: - -``` -MainWindow (anylabeling/views/mainwindow.py) -└── LabelingWrapper (anylabeling/views/labeling/label_wrapper.py) - └── LabelingWidget (anylabeling/views/labeling/label_widget.py, ~3.2k LOC) - ├── Canvas (anylabeling/views/labeling/widgets/canvas.py) - ├── AutoLabelingWidget (drives ModelManager from the UI side) - ├── LabelDialog / Brightness / FileDialogPreview / ZoomWidget … - └── ExportDialog -``` - -`LabelingWidget` is the "god widget" — it owns the file list, the canvas, the -toolbars, the shape list, the label list, file I/O, undo/redo, and most -keybindings. When in doubt, that file is where things live. - -### Auto-labeling pipeline - -``` -anylabeling/services/auto_labeling/ -├── registry.py # @ModelRegistry.register("yolov8") decorator → singleton dict -├── model.py # abstract Model(QObject); predict_shapes() returns AutoLabelingResult -├── model_manager.py # ModelManager(QObject): loads models.yaml, downloads weights, -│ # dispatches predict_shapes_threading() -├── types.py # AutoLabelingResult, AutoLabelingMode (point/rectangle, ADD/REMOVE) -├── lru_cache.py # image-embedding cache for SAM-family models -├── segment_anything.py # variant detector — picks SAM1/SAM2/SAM3 from ONNX inputs/config -├── sam_onnx.py # SAM1 / MobileSAM ONNX runner -├── sam2_onnx.py # SAM2 ONNX runner -├── sam3_onnx.py # SAM3 ONNX runner (text + geometric prompts) -├── sam2_coreml.py # macOS CoreML path for SAM2.1 -└── yolov5.py / yolov8.py -``` - -Two registry-relevant facts: - -- Concrete models register themselves via `@ModelRegistry.register("type-name")` - at import time. `anylabeling/services/auto_labeling/__init__.py` imports - every module so the side-effects fire — adding a new model means importing - it here too. -- `models.yaml` (`anylabeling/configs/auto_labeling/models.yaml`) is the - catalog the UI reads. Each entry has `name`, `display_name`, `type` - (matches a registry key), `download_url`, plus model-specific fields like - `encoder_model_path`, `decoder_model_path`, `input_size`. New model = add - an entry here *and* a registered class. - -Weights live under `~/anylabeling_data/models//` after first download. - -### CPU / GPU / macOS packaging - -Static metadata is in `pyproject.toml`. `setup.py` is a small shim that -reads `__preferred_device__` from `anylabeling/app_info.py` and, when set -to `"GPU"` on non-Darwin, overrides the package name to `anylabeling-gpu` -and swaps `onnxruntime` for `onnxruntime-gpu`. The publish workflows -(`.github/workflows/python-publish-{cpu,gpu}.yml`) `sed` that constant -just before building, so both wheels come out of the same source tree. - -`pyproject.toml` excludes `PyQt6` on Darwin -(`PyQt6>=...; platform_system != 'Darwin'`). macOS users install PyQt -through conda. The macOS extra is `[macos]` (currently `coremltools>=9.0` so -Python 3.13 receives a native Apple Silicon wheel). - -### Qt resources and translations - -- `anylabeling/resources/resources.qrc` (XML) compiles to `resources.py`. -- `anylabeling/resources/translations/{en_US,vi_VN,zh_CN}.{ts,qm}`. -- `scripts/compile_languages.py` rebuilds `.qm` files and `resources.py` - from existing `.ts` files. Use after editing translations or icons. -- `scripts/generate_languages.py` does the full extract: runs `pyuic6` on - `.ui` files, `pylupdate6` to refresh `.ts`, then the compile step. -- Both shell out to `pyside6-rcc` and `pyside6-lrelease` (PyQt6 dropped - the standalone `pyrcc` in Qt6) and post-rewrite `from PySide6` to - `from PyQt6` so the generated module uses the runtime's Qt binding. - `PySide6-Essentials` is in the `[dev]` extras for this purpose only. - -### Tests - -`tests/` is plain `unittest`. Notable files: - -- `tests/test_label_colormap.py` — regression test for issue #227 - (`imgviz.label_colormap()` returns read-only on imgviz>=2.0; the call - site needs `.copy()`). -- `tests/test_real_inference.py` — end-to-end ONNX inference for - SAM1/SAM2/SAM3/YOLOv8. Each class skips itself if its model files are - not under `~/anylabeling_data/models/`. The SAM3 text-prompt tests look - for `../samexporter/images/truck.jpg` (sibling-repo path) and silently - fall back to `sample_images/evan-foley-...jpg` (no truck), which makes - three SAM3 tests fail — see step 3 of the playbook below. - -## Pre-publish local experiments - -Run these **before tagging a release** (`vX.Y.Z`). The CI matrix in -`.github/workflows/tests.yml` already gates publish on every tag push, but -running locally first is faster and catches obvious dep-resolution -failures before burning CI minutes. - -### 1. Fresh-venv install with latest deps - -The point of a *fresh* venv is to let pip resolve every dependency to the -newest version compatible with `pyproject.toml` — this is what end users -get on `pip install anylabeling[-gpu]`, and it is exactly the path that -produced the `imgviz>=2.0` read-only crash in #227. - -```bash -python -m venv /tmp/anylabeling-check -/tmp/anylabeling-check/bin/pip install --upgrade pip -/tmp/anylabeling-check/bin/pip install . -``` - -Watch for: any wheel that fails to build, any dep that pip cannot resolve. - -### 2. Run the full unittest suite - -```bash -/tmp/anylabeling-check/bin/python -m unittest discover -s tests -v -``` - -Expected: all tests pass; `test_real_inference` cases skip cleanly when -model files are not on disk — that is fine. Step 3 below covers running -those tests with real models. - -### 3. (Recommended) Real-model inference - -`tests/test_real_inference.py` exercises ONNX inference end-to-end for -SAM1 / SAM2 / SAM3 / YOLOv8. Each test class skips itself when its model -files are missing, so download whichever you can validate on the local -machine. Models live under `~/anylabeling_data/models/`. - -```bash -mkdir -p ~/anylabeling_data/models && cd ~/anylabeling_data/models - -# YOLOv8n (~13 MB) -curl -sL -o /tmp/yolov8n.zip https://github.com/vietanhdev/anylabeling-assets/releases/download/v0.4.0/yolov8n-r20230415.zip -mkdir -p yolov8n-r20230415 && unzip -q -o /tmp/yolov8n.zip -d yolov8n-r20230415 - -# MobileSAM (~37 MB) -curl -sL -o /tmp/msam.zip https://huggingface.co/vietanhdev/segment-anything-onnx-models/resolve/main/mobile_sam_20230629.zip -mkdir -p mobile_sam_20230629 && unzip -q -o /tmp/msam.zip -d mobile_sam_20230629 - -# SAM2 hiera-tiny (~155 MB) -curl -sL -o /tmp/sam2.zip https://huggingface.co/vietanhdev/segment-anything-2-onnx-models/resolve/main/sam2_hiera_tiny.zip -mkdir -p sam2_hiera_tiny_20240803 && unzip -q -o /tmp/sam2.zip -d sam2_hiera_tiny_20240803 - -# SAM3 ViT-H (~3.4 GB — only needed when SAM3 code paths changed) -curl -sL -o /tmp/sam3.zip https://huggingface.co/vietanhdev/segment-anything-3-onnx-models/resolve/main/sam3_vit_h.zip -mkdir -p sam3_vit_h_20260220 && unzip -q -o /tmp/sam3.zip -d sam3_vit_h_20260220 -``` - -The SAM3 text-prompt tests need a truck image at the sibling-repo path: - -```bash -mkdir -p ../samexporter/images -curl -sL -o ../samexporter/images/truck.jpg \ - https://raw.githubusercontent.com/vietanhdev/samexporter/main/images/truck.jpg -``` - -Then re-run the inference tests: - -```bash -/tmp/anylabeling-check/bin/python -m unittest tests.test_real_inference -v -``` - -Source of truth for model URLs is -`anylabeling/configs/auto_labeling/models.yaml`. - -### 4. Smoke-test the import chain that users hit at startup - -This is the *exact* path that crashed in #227. If it imports clean against -freshly resolved deps, the package will at least start. - -```bash -QT_QPA_PLATFORM=offscreen /tmp/anylabeling-check/bin/python -c " -from anylabeling.views.labeling import label_widget -from anylabeling import app -print('startup imports OK') -" -``` - -### 5. Repeat against every supported Python (3.11, 3.12, 3.13) - -PyPI ships one wheel that has to work on every Python listed in -`pyproject.toml` classifiers. Use `uv` to spin them up quickly: - -```bash -uv python install 3.11 3.12 3.13 -for v in 3.11 3.12 3.13; do - PY=$(uv python find $v) - VENV=/tmp/al-py${v//./} - rm -rf $VENV && $PY -m venv $VENV - $VENV/bin/pip install --upgrade pip --quiet - $VENV/bin/pip install . --quiet - $VENV/bin/python -m unittest discover -s tests -done -``` - -### 6. Then push and let CI confirm cross-platform - -The matrix in `.github/workflows/tests.yml` runs steps 1, 2, 4 on -Ubuntu + Windows + macOS × Python 3.11/3.12/3.13. The publish workflows -(`python-publish-cpu.yml`, `python-publish-gpu.yml`, `release.yml`) all -declare `needs: test`, so a red matrix blocks the PyPI upload and the -GitHub release binary builds. Step 3 (real-model inference) is *not* -automated in CI because the SAM3 model alone is 3.4 GB — run it locally -when touching ONNX inference, model loading, or preprocessing code. - -## Why this gate exists - -`anylabeling-gpu==0.4.30` shipped to PyPI broken because no automated test -ran `pip install .` against current dep floors before publish. The fix in -`label_widget.py:45` (call `.copy()` on `imgviz.label_colormap()`) had a -regression test in `tests/test_label_colormap.py`, but nothing executed it -on the publish path. The workflows in `.github/workflows/` now do. - -When adding a new dependency or raising a floor, **assume it can break -import-time code paths** — read-only numpy arrays, removed deprecated -APIs, changed default dtypes — and rely on the steps above to catch it. +When a change touches Qt behavior, frozen binaries, model loading, or an ONNX +Runtime provider, also complete the relevant manual checks described in +`AGENT.md`; unit tests alone are not sufficient for those paths. diff --git a/README.md b/README.md index ff59eda..f57a0e2 100644 --- a/README.md +++ b/README.md @@ -59,9 +59,9 @@ Required model weights are downloaded automatically on first use. ## Latest Release -[AnyLabeling v0.4.42](https://github.com/vietanhdev/anylabeling/releases/tag/v0.4.42) is the current stable release. It includes cross-platform accelerator selection, packaged CUDA/CoreML support, stability fixes for the file dialog and canvas, 16-bit TIFF editing, SAM 3 frozen-build support, and corrected Linux/macOS packaging. +[AnyLabeling v0.4.43](https://github.com/vietanhdev/anylabeling/releases/tag/v0.4.43) is the current stable release. It fixes recovery after model download/load failures, skips invalid images during SAM preload, persists grouped shapes with undo support, and closes label files reliably after saving and loading. -All six v0.4.42 CPU and accelerated artifacts were checksum-verified and launch-tested on Linux, Windows, and Apple Silicon macOS. Avoid the superseded v0.4.40 macOS and Linux artifacts. +The release workflow tests Python 3.11–3.13 on Linux, Windows, and macOS, then builds and launch-smoke-tests all six CPU and accelerated artifacts on their native runners. Use the [Download page](https://anylabeling.nrl.ai/download) for direct platform links, or see [all GitHub releases](https://github.com/vietanhdev/anylabeling/releases).