Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
aeb51b3
Keep the beta channel config on beta after merging 0.8.0
oleost Oct 10, 2026
e18d905
Docs: one home per fact, a test that keeps them true to the code
oleost Oct 10, 2026
073b401
Merge pull request #43 from oleost/docs/lean-structure
oleost Oct 10, 2026
31a9600
Docs: the maintainer commits straight to beta, contributors open a PR
oleost Oct 10, 2026
202e8de
Docs: research and plan for face recognition on object sensors
oleost Oct 10, 2026
5ae9334
Docs: face recognition plan reads the AI Act with the Commission's gu…
oleost Oct 10, 2026
b7b2dfa
0.8.1b1: larger AI models, a model comparison and tunable teaching li…
oleost Oct 11, 2026
f197e1c
0.8.1b2: switching AI models gives the memory back
oleost Oct 11, 2026
caa1cf0
Docs: try changes on a real Home Assistant as a local app
oleost Oct 11, 2026
8250396
0.8.1b3: recognise people by their face, shared by every camera
oleost Oct 11, 2026
4f09e1b
0.8.1b4: teach a face from photos on the phone
oleost Oct 11, 2026
039b7e9
0.8.1b5: boxes on the Live tab always match the picture
oleost Oct 11, 2026
be179e4
0.8.1b5: let the face model go when it was turned off while loading
oleost Oct 11, 2026
8e472b8
Docs: tighter README, new screenshots and a demo set of sensors
oleost Oct 11, 2026
2e34622
seed_demo.py: say clearly when an app cannot teach faces from photos
oleost Oct 11, 2026
7d90e81
Docs: the demo sensors and README screenshots in the release checklis…
oleost Oct 11, 2026
2e66ddf
Docs: faces and the larger models in the 0.9.0 roadmap; AI Act guidel…
oleost Oct 12, 2026
411e7a1
Log only ONNX Runtime errors: SFace's file made 174 warnings per load
oleost Oct 12, 2026
af03d0b
0.9.0b1
oleost Oct 12, 2026
f0130a8
test_docs: a module name counts if any module of that name defines it
oleost Oct 12, 2026
c63613d
Faces: even out a backlit face's light, ask sooner and in a burst, sa…
oleost Oct 12, 2026
5c90637
0.9.0b2
oleost Oct 12, 2026
f6393a7
Faces: look for a face while a person is there; History shows what wa…
oleost Oct 12, 2026
27dd4a3
0.9.0b3
oleost Oct 12, 2026
2bbf5f4
Faces: "<name> seen by" and "<name> last seen"; a confirmed frame is …
oleost Oct 12, 2026
bf9c0ca
0.9.0b4
oleost Oct 12, 2026
58bb821
Faces: build the index of taught faces one at a time, store each embe…
oleost Oct 12, 2026
f764d96
0.9.0b5
oleost Oct 12, 2026
c22a961
0.9.0
oleost Oct 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Bug report
description: Something does not work as described in the documentation
labels: [bug]
body:
- type: markdown
attributes:
value: |
Thanks for reporting! Remove camera addresses and passwords from anything you paste.
- type: input
id: version
attributes:
label: VisionState version
description: Settings β†’ Apps β†’ VisionState (or VisionState (beta))
placeholder: "0.8.0"
validations:
required: true
- type: dropdown
id: kind
attributes:
label: Sensor kind
options:
- State sensor
- Object sensor
- Reading sensor
- Not about one sensor
validations:
required: true
- type: input
id: system
attributes:
label: Home Assistant version and hardware
placeholder: "2026.10.0, Raspberry Pi 4 / Intel NUC / Proxmox VM"
- type: textarea
id: what
attributes:
label: What happened, and what did you expect?
description: Steps to see it again help a lot. A screenshot is welcome.
validations:
required: true
- type: textarea
id: log
attributes:
label: Log
description: The app's Log tab, around the time it happened.
render: text
8 changes: 8 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
blank_issues_enabled: true
contact_links:
- name: Questions, ideas and shared meter readings
url: https://github.com/oleost/VisionState/discussions
about: Ask, suggest a feature, or share an export of readings (Quality tab β†’ Export readings) to help improve reading.
- name: Security vulnerability
url: https://github.com/oleost/VisionState/security/advisories/new
about: Report it privately, not as an issue.
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ jobs:
cache: pip
cache-dependency-path: visionstate/backend/requirements*.txt
- run: pip install -r requirements-dev.txt
- name: Cache AI model
- name: Cache AI models (all of them, for tests/test_models.py)
uses: actions/cache@v6
with:
path: visionstate/backend/models
key: models-${{ hashFiles('visionstate/backend/visionstate/backbones.json', 'visionstate/backend/visionstate/detectors.json', 'visionstate/backend/visionstate/readers.json') }}
- run: python -m visionstate.backbones models
key: models-all-${{ hashFiles('visionstate/backend/visionstate/backbones.json', 'visionstate/backend/visionstate/detectors.json', 'visionstate/backend/visionstate/readers.json') }}
- run: python -m visionstate.backbones models --all
- run: ruff check visionstate tests
- run: ruff format --check visionstate tests
- name: Types (pyright)
Expand Down
363 changes: 158 additions & 205 deletions CLAUDE.md

Large diffs are not rendered by default.

252 changes: 69 additions & 183 deletions README.md

Large diffs are not rendered by default.

11 changes: 11 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Security

Please report a vulnerability privately: **Security β†’ Report a vulnerability** on this repository
(GitHub's private vulnerability reporting), not as a public issue. You get an answer within a few
days, and a fix goes out as a beta first, then in the next stable release.

Supported: the latest stable release and the current beta.

What VisionState relies on, so you know what counts as a vulnerability: it is only reachable
through Home Assistant Ingress (other requests are refused); camera credentials are never written to
logs, errors or exports; nothing is sent anywhere but to your own Home Assistant and MQTT broker.
62 changes: 50 additions & 12 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,28 @@ A map of the code: what runs where, how one check flows through it, and which fi
for common changes. What VisionState does and why is in [`SCOPE.md`](SCOPE.md); how to work on
it is in [`CLAUDE.md`](../CLAUDE.md).

## Repository

```
repository.yaml, README.md, CLAUDE.md, SECURITY.md
visionstate/ the Home Assistant app: config.yaml, Dockerfile, DOCS.md, CHANGELOG.md, icon/logo
backend/ Python package `visionstate` + tests (the Dockerfile builds from visionstate/)
frontend/ Svelte app, Playwright tests in e2e/ (and promo.mjs: the README screenshots)
scripts/ channel.py (stable/beta config), fake_camera.py (test camera), seed_demo.py
(the demo sensors), CI helpers:
check_options.py, cpu_probe.py, upgrade_test.sh, discovery_ids.py
tools/wheelreader/ training code of the wheel reader model (not part of the app image)
docs/ SCOPE.md, ARCHITECTURE.md, RELEASING.md, WHEEL_READER_PLAN.md,
FACE_RECOGNITION_PLAN.md, promo/ (README images)
.github/ CI workflow, Dependabot, issue forms
```

**Stack:** Python 3.14, FastAPI, uvicorn, SQLAlchemy (SQLite), asyncio; ONNX Runtime (CPU only),
NumPy, Pillow, PyAV (bundled FFmpeg), scikit-learn; aiomqtt, websockets, httpx. Frontend: Svelte 5 +
Vite + TypeScript, plain CSS with design tokens, a hash router (Ingress-safe), bundled fonts. Docker
image `python:3.14-slim`, built by GitHub Actions and published on GHCR. Quality: pytest (with a fake
camera, Home Assistant and MQTT), ruff, pyright, svelte-check, Playwright, an image smoke test.

## The process

One Python process (`python -m visionstate`) serves the API and the built UI through Home
Expand All @@ -30,12 +52,13 @@ API as `request.app.state.runtime`.
| `db.py` | SQLAlchemy tables and the schema migrations (`MIGRATIONS`, `SCHEMA_VERSION`) |
| `storage.py` | Image files on disk: training samples, history frames, thumbnails |
| `engine/` | The runtime β€” see below |
| `api/` | The REST API (`/api/v1/…`), one file per topic: `sensors.py` (CRUD, quality, export), `history.py` (the history: one filter for the list, its count and facets; history frames), `samples.py` (training images), `teach.py` (taught boxes), `cameras.py` (camera and entity lists, the light, previews), `review.py` (review queue and answers), `imports.py` (import), `system.py` (UI config, status, AI models, storage, review rules, review reminder), `common.py` (input models and views shared by them) |
| `api/` | The REST API (`/api/v1/…`), one file per topic: `sensors.py` (CRUD, quality, export), `history.py` (the history: one filter for the list, its count and facets; history frames), `samples.py` (training images), `teach.py` (taught boxes; what is taught from a history frame is written into its row: `remember_taught`), `faces.py` (teaching a face on a sensor; the people and their faces, shared by every sensor), `cameras.py` (camera and entity lists, the light, previews), `review.py` (review queue and answers), `imports.py` (import), `system.py` (UI config, status, AI models and deleting downloaded ones, storage, review rules, review reminder, teaching and face limits), `common.py` (input models and views shared by them) |
| `sources.py` | Camera sources: Home Assistant camera, HTTP snapshot, RTSP; address checks and size limits; the Home Assistant REST client (states, services, notify services, the app's page) |
| `mqtt.py` | MQTT topics, Home Assistant discovery messages, the bridge (connect, publish, commands) |
| `ha_events.py` | Home Assistant WebSocket: state changes of trigger entities, entity registry |
| `lights.py` | A sensor's light: switched on for checks and while someone looks, shared by both |
| `backbones.py`, `detectors.py`, `readers.py` | The three AI models and their registries (`*.json`): embedding backbone (state sensors), object detector, number reader (and the wheel reader for mechanical counters, `readers.WheelReader`; trained with `tools/wheelreader`) |
| `faces.py` | Faces in person boxes (registry `faces.json`, never bundled): finding them (YuNet), the quality gate, aligning, evening out the light (`even_light`), embedding, telling who it is |
| `classifier.py` | The small per-sensor classifier ("head") on top of the backbone |
| `teach.py` | Object sensors: comparing detected boxes with the boxes the user taught |
| `imaging.py` | Decoding, regions (ROI), crops, change signatures, drawing boxes |
Expand All @@ -49,7 +72,7 @@ API as `request.app.state.runtime`.
parts it uses β€” its class line says what it depends on:

```
Runtime ── ChecksMixin ──┬─ ObjectChecksMixin ── TeachingMixin ── ModelsMixin ── TrainingMixin
Runtime ── ChecksMixin ──┬─ ObjectChecksMixin ── FacesMixin, TeachingMixin ── ModelsMixin ── TrainingMixin
β”‚ β”‚ └─ PublishingMixin
β”‚ └─ ReadingChecksMixin ── ModelsMixin, PublishingMixin
β”œβ”€ HistoryMixin ── PublishingMixin (all of them on RuntimeBase)
Expand All @@ -61,11 +84,12 @@ Runtime ── ChecksMixin ──┬─ ObjectChecksMixin ── TeachingMixin
| `base.py` | `RuntimeBase`: the shared state (`__init__`), loading a sensor, its live state, waking it |
| `runtime.py` | Lifecycle (start/stop), sensors coming and going, waking a sensor, triggers |
| `state.py` | `SensorConfig` (a snapshot of a sensor row) and `LiveState` (everything a sensor knows between checks) |
| `logic.py` | Pure decisions, no I/O: debouncing object tracks, review, triggers, when to check next |
| `logic.py` | Pure decisions, no I/O: debouncing object tracks, review, triggers, when to check next, looking for a face while a person is there (`steer_face_burst`) |
| `checks.py` | Each sensor's loop, change detection (probe), the light, `run_once` and the state sensors' check |
| `objects.py`, `reading.py` | The check of an object sensor and of a reading sensor |
| `objects.py`, `reading.py` | The check of an object sensor (with its questions in Review, and "last seen" read back from the history at start: `restore_last_seen`) and of a reading sensor |
| `models.py` | Loading and switching the three AI models; `detect_objects`, `read_number` |
| `teaching.py` | Taught boxes of object sensors (index, embeddings) |
| `faces.py` | Faces on object sensors: the face model (downloaded and loaded in the background, let go when unused), the index of the people's taught faces (shared by every sensor), the face step of a check (`apply_faces`), each person's own entities (where they are seen, when they were last seen) |
| `training.py` | Training the state sensors' heads; embeddings of samples |
| `publishing.py` | Discovery and values to Home Assistant, the review queue entity, commands from it |
| `history.py` | History clean-up by age and size, disk use |
Expand All @@ -82,7 +106,7 @@ sensor loop (checks._sensor_loop) woken by: its interval, a trigger ent
β”œβ”€ decode (imaging.decode)
└─ by kind:
state sensor β†’ checks._run_states backbone embedding β†’ head β†’ probabilities
object sensor β†’ objects._run_objects detector β†’ taught boxes (teaching) β†’ tracks (logic)
object sensor β†’ objects._run_objects detector β†’ taught boxes (teaching) β†’ faces β†’ tracks (logic)
reading sensor β†’ reading._run_reading reader β†’ parse β†’ plausibility β†’ debounce
β”‚
β”œβ”€ publish (publishing._send β†’ mqtt.MqttBridge.publish) state, attributes, image
Expand All @@ -96,9 +120,16 @@ error is kept in `LiveState.error` and shown in the UI.

- **SQLite** (`<data>/visionstate.db`): `sensor` (+ `state`), `sample` (+ `sample_label`,
`embedding` cache), `prediction` (the history and the review queue), `reading_stat`,
`model_info` (per sensor), `setting` (global settings as JSON).
- **Files** (`<media>/`): `samples/<sensor>/`, `history/<sensor>/`, thumbnails; `<data>/heads/`
(trained heads), `<data>/models/` (downloaded models).
`model_info` (per sensor), `setting` (global settings as JSON), `person` (+ `face`,
`face_embedding` cache: the people taught by face, shared by every sensor).
- **Faces** (`<data>/faces/`): crops of the people's taught faces β€” in the app's own storage, not
the media folder (see `storage.Storage.face_path`).
- **Files** (`<media>/`): `samples/<sensor>/` (training images; for object sensors the crops of
taught boxes), `history/<sensor>/`, `thumbs/`; `<data>/heads/` (trained heads), `<data>/models/`
(downloaded models, left out of backups).
- **Object sensors' taught boxes** are `sample` rows with `object_label`, `detected`, `box` and
`score`; own labels live in `sensor.objects["custom"]`. Readings and object events are
`prediction` rows (see `db.py`).
- **Settings of a sensor** that have many fields (`triggers`, `objects`, `reading`, `review`) are
stored as JSON and always read through `settings.merge_*`, so a field added later gets its
default for existing sensors without a migration.
Expand All @@ -113,9 +144,10 @@ error is kept in `LiveState.error` and shown in the UI.
| `lib/router.svelte.ts` | Hash routes (`paths`), a page's query (`route.query`, `setQuery`) |
| `lib/history.ts` | The history filter: from and to the page URL, and as the API takes it |
| `lib/ui.ts` | UI constants (tabs per sensor kind, wizard texts); `tokens.css` colours, type, spacing |
| `pages/` | Dashboard, the new sensor wizard (`NewSensor.svelte`; its Detect step per kind in `pages/wizard/`), Review, History, Settings, and `SensorPage` with one file per tab in `pages/sensor/` (its History tab is `HistoryView`) |
| `lib/objects.ts`, `lib/reading.ts`, `lib/roi.ts`, `lib/format.ts` | Helpers: names and colours of objects and own labels, reading values and units, region geometry, formatting |
| `pages/` | Dashboard, the new sensor wizard (`NewSensor.svelte`; its Detect step per kind in `pages/wizard/`), Review, History, People, Settings, and `SensorPage` with one file per tab in `pages/sensor/` (its History tab is `HistoryView`) |
| `lib/components/history/` | `HistoryView` (filters, list, paging, new rows) shared by the History page and the History tabs, and one row component per sensor kind |
| `lib/components/` | Building blocks (region editor, frames with boxes, editors for triggers, reading, states…) |
| `lib/components/` | Building blocks (region editor, frames with boxes, editors for triggers, reading, states, limits that inherit (`LimitsEditor`)…) |

## Recipes

Expand All @@ -139,8 +171,14 @@ a test that an old database is upgraded (see `tests/test_entity_ids.py`).
**A new API endpoint**: in the `api/` file of its topic, input as a pydantic model, its function
in `lib/api.ts`, its type in `lib/types.ts`.

**A new AI model**: a new entry in the registry JSON (never change a released one), checked
against real inputs in the tests.
**A new history event**: its name in `settings.HISTORY_EVENTS`, which rows it matches in
`EVENT_ROWS` (`api/history.py`; every filter becomes SQL in `history_query`, nowhere else), its
label in `HISTORY_EVENTS` in `lib/ui.ts`.

**A new AI model**: a new entry in the registry JSON (never change a released one's file or
inputs), with its measured footprint (`ram_mb`, `time_ms`, `accuracy`; see `backbones.Footprint`).
`tests/test_models.py` checks every entry against real inputs once it is downloaded
(`python -m visionstate.backbones models --all`, as CI does).

**A new page or tab**: the route in `router.svelte.ts` (tab: `TABS_BY_KIND` in `ui.ts`), the page
in `pages/`, and an e2e test in `e2e/ui.spec.ts` on all three devices.
Loading
Loading