diff --git a/CLAUDE.md b/CLAUDE.md index 2b599ce..5c401bf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -106,7 +106,8 @@ itself (`__main__.py`). `/set?state=open|closed|partial`, `/set?night=1`), `/photo/.jpg` (the CC0 photos from `backend/tests/assets`, for object sensors), `/display.jpg?text=12:05&style=lcd|led` (a drawn seven-segment display) and `/counter.jpg?value=89939.5&digits=7&decimals=3` (a drawn mechanical - counter; `.5` = the last wheel half way to the next digit). + counter; `.5` = the last wheel half way to the next digit; with `&camera=a`, + `/set?wider=400&camera=a` makes it come back wider, like a camera that changed its size). - **The Home Assistant side:** any local MQTT broker (e.g. Mosquitto on 1883) and `VISIONSTATE_MQTT_HOST=127.0.0.1`: discovery configs appear under `homeassistant/…`, values under `visionstate//…`. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7cb8281..2594cb4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -58,6 +58,7 @@ API as `request.app.state.runtime`. | `ha_events.py` | Home Assistant WebSocket: state changes of trigger entities, entity registry | | `lights.py` | A sensor's light: switched on for checks and while someone looks, shared by both | | `backbones.py`, `detectors.py`, `readers.py` | The three AI models and their registries (`*.json`): embedding backbone (state sensors), object detector, number reader (and the wheel reader for mechanical counters, `readers.WheelReader`; trained with `tools/wheelreader`) | +| `change_limit.py` | Counters: how much the value may go up since the last accepted reading — what it learns, time not seen (pure logic; the engine keeps a `Watch` per sensor) | | `faces.py` | Faces in person boxes (registry `faces.json`, never bundled): finding them (YuNet), the quality gate, aligning, evening out the light (`even_light`), embedding, telling who it is | | `classifier.py` | The small per-sensor classifier ("head") on top of the backbone | | `teach.py` | Object sensors: comparing detected boxes with the boxes the user taught | @@ -107,7 +108,7 @@ sensor loop (checks._sensor_loop) woken by: its interval, a trigger ent └─ by kind: state sensor → checks._run_states backbone embedding → head → probabilities object sensor → objects._run_objects detector → taught boxes (teaching) → faces → tracks (logic) - reading sensor → reading._run_reading reader → parse → plausibility → debounce + reading sensor → reading._run_reading reader → parse → plausibility (change_limit) → debounce │ ├─ publish (publishing._send → mqtt.MqttBridge.publish) state, attributes, image └─ history: a Prediction row + frame, when the value changed or the frame goes to review @@ -120,13 +121,18 @@ error is kept in `LiveState.error` and shown in the UI. - **SQLite** (`/visionstate.db`): `sensor` (+ `state`), `sample` (+ `sample_label`, `embedding` cache), `prediction` (the history and the review queue), `reading_stat`, - `model_info` (per sensor), `setting` (global settings as JSON), `person` (+ `face`, + `reading_watch` (a counter's limit between readings), `model_info` (per sensor), `setting` (global settings as JSON), `person` (+ `face`, `face_embedding` cache: the people taught by face, shared by every sensor). - **Faces** (`/faces/`): crops of the people's taught faces — in the app's own storage, not the media folder (see `storage.Storage.face_path`). -- **Files** (`/`): `samples//` (training images; for object sensors the crops of - taught boxes), `history//`, `thumbs/`; `/heads/` (trained heads), `/models/` - (downloaded models, left out of backups). +- **Files**: `/samples//` (training images; for object sensors the crops of taught + boxes — next to the database that lists them, so a backup holds both), `/thumbs/sample/`, + `/heads/` (trained heads), `/models/` (downloaded models; models and thumbnails are + left out of backups); `/history//` and `/thumbs/history/`. Versions before + the move kept training images in `/samples/`: they are read there until + `engine.HistoryMixin.move_samples` has copied them (`settings.SAMPLES_MOVE`). A file is removed + only after the row that lists it (`db.Database.after_commit`), and an image that is gone anyway + is left out of training and of the taught boxes instead of stopping the sensor. - **Object sensors' taught boxes** are `sample` rows with `object_label`, `detected`, `box` and `score`; own labels live in `sensor.objects["custom"]`. Readings and object events are `prediction` rows (see `db.py`). diff --git a/docs/FACE_RECOGNITION_PLAN.md b/docs/FACE_RECOGNITION_PLAN.md index 4751bd3..960141f 100644 --- a/docs/FACE_RECOGNITION_PLAN.md +++ b/docs/FACE_RECOGNITION_PLAN.md @@ -177,10 +177,10 @@ Design rules that follow: added to the person's faces by itself (that drifts: one wrong match teaches the next one). Unknown faces are not stored as templates; they only live in history frames, under the normal history limits (`history_days`, `history_max_gb`). -3. **Faces are kept where only the app can read them.** Training images live in `/media`, which - Home Assistant's media browser shows to every user of the house; taught face crops go to the - app's own `/data` instead. That puts them in Home Assistant backups — say so in DOCS.md, and that - backups should be encrypted. +3. **Faces are kept where only the app can read them.** Training images lived in `/media` then, + which Home Assistant's media browser shows to every user of the house; taught face crops go to + the app's own `/data` instead (training images followed later, see SCOPE.md §11). That puts + them in Home Assistant backups — say so in DOCS.md, and that backups should be encrypted. 4. **Embeddings are biometric data too.** Published attacks rebuild a recognisable face from an ArcFace-style template, so the cached embeddings are deleted with the person and never exported (an import computes them again). diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 7b3c323..e3e43f8 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -76,7 +76,7 @@ all of it and tell the maintainer what was checked and what was found. keep working, their entities keep their IDs, the log has no warnings or errors; - restart Home Assistant, reboot the host and restart the MQTT broker: the app and its entities come back by themselves; - - make a backup of the app and restore it: the sensors are back; + - make a backup of the app and restore it: the sensors are back, with their training images; - **soak for 1 hour** with the sensors checking: the app's memory (Supervisor app stats) and its data on disk level off instead of growing, the log stays quiet; - open the app in Home Assistant at desktop width and on a phone (Ingress); diff --git a/docs/SCOPE.md b/docs/SCOPE.md index 0ac105a..40f61fd 100644 --- a/docs/SCOPE.md +++ b/docs/SCOPE.md @@ -271,14 +271,77 @@ it read 297 of 329 right, tiny 304 and small 316; it took 9 for 3 or 8 on seven- (the class indices are stored in the registry, so the dictionary file is not needed). 3. Parse: only digits count, the configured `decimals` place the decimal point; `time_left` reads `h:mm` → minutes. Reject when empty, below the threshold (default 70 %), a counter - going down, a mechanical counter read with another number of digits than it has wheels, or a - change above `max_step`; otherwise publish after `debounce` equal reads. A counter read exactly + going down, a mechanical counter read with another number of digits than it has wheels, a + counter going up faster than its limit (below), or another value changing more than + `max_step`; otherwise publish after `debounce` equal reads. A counter read exactly one step of its last digit below the value (`readers.settling`) is the last wheel turning: the value stays, but it is no rejection (no history row, no review item, counted as accepted) — seen in a user's export, where one too-high value made every right reading after it a "went down" rejection. A right value typed in the review (`readers.right_value`) is taken as written with a point or comma, and digits only are placed with the configured decimals. - The last published value is restored from the history after a restart. + The last published value is restored from the history after a restart (or from + `reading_watch`, when that is newer: a counter's, or a value confirmed in Review). + +**Counter limit** (`change_limit.py`, `settings.COUNTER_LIMIT`): how much a counter may go up +since a reading last confirmed its published value is a rate times the time it was *not seen*, +plus two steps of the last digit. (Confirmed, not merely accepted: a reading still waiting for +its debounce leaves the published value older than it, and during a long shower that is minutes.) A fixed `max_step` between two readings (what counters had until then) was both +too loose and too tight: issue #40's water meter (0.04 m³ per one-minute reading = 2.4 m³/h, four +times its busiest shower) rejected right readings after a 35-minute update. +- *Not seen* is time the app was down (a pause longer than `gap_checks` intervals), and time the + readings were unsure. Once a reading was read with confidence over the limit (`held`), normal + pauses stop counting: otherwise a wheel misread over and over would get through after a while, + as the allowed rise grows. This is the reason the limit is not simply "rate × time since the + last accepted value". +- *Learn it* (`auto`, the default for new counters): per local day `reading_stat` keeps the + largest rise over `window_s` (600 s) and the day's rise, from the accepted readings (each with + the lower of its value and the next one's, so one read too high adds nothing; published values + would put a whole shower into one step) — only over stretches where the counter was read all + the time, so a rise over a pause is never learned. The limit takes the + second-largest day of the last `learn_days`, so one misread that slipped through does not count, + times `safety`; for a long pause it is capped by that many busy days. It needs `min_days` days + (a whole week: wash day and weekend). On issue #40's data, ×2 left room above every right + reading (the largest was 1.1 × the learned rate). Replayed three times over with readings every + minute, 14 times away (35 min to two days), 15 stretches of unsure readings and six wheels read + a digit too high for 20–120 min: the fixed 0.04 per reading kept the value behind the meter for + 528 hours; learned, 32 hours (the debounce during long showers, as with a fixed rate), it + learned 0.82 m³/h for a meter that ran at 0.75 at most, and after its first week no wrong + value was published. A wheel read too high right after a long time away does get through — + nothing tells it from water used meanwhile — and then the "went down" lock below applies. + Until it learned, a rate applies: `max_rate`, or when that is 0 `standard_steps_per_min` steps + of the last digit a minute (issue #40's 0.04 m³ a minute, three decimals). It once set no limit + until then; on issue #40's meter, switched to *Learn it*, a wheel read two digits off went + through as +2.09 m³ in its first week. The UI shows the rate in use, and fills in the standard + one when *Fixed rate* or *Fixed per reading* is chosen without a value; 0 there stays "no + limit", as it was before. +- *Fixed per reading* (`step`): `max_step` for every reading, however long since the last one — + the limit counters had before the rate, which some users find easier to tune ("how many + decimals may move between two checks"). Counters from before keep it unchanged (migration 14, + `db.keep_change_step`); after a restart a real rise waits for Review. +- Readings over the limit are never accepted by themselves, however many agree: only an answer in + Review (below) moves the counter on. A beta accepted a jump once readings agreed for five + minutes, unless it looked like one misread wheel; the maintainer took it out again — + the limit is there so that nothing overrides it without a person, and a camera that moved can + misread the same wrong value over and over (issue #40). A real jump while the readings were + unsure, or a counter that goes faster than it ever did, waits for Review. +- Confirmed in Review (*read correctly* for an unsure or implausible reading, or a typed right + value) on a rejected reading newer than the last accepted one: that value becomes the last one + (`engine.reading.anchor_reading`). It is also the way out for a counter stuck above its real + value, the "went down" lock — never automatic, as Home Assistant takes a counter going down + for a reset. +- The limit's state (`reading_watch`) is stored with every reading, so a restart knows how long + the app was away. Counters made before the limit (schema v14, and importing an older export: + `db.keep_change_step`) keep theirs as `fixed`: the old `max_step` per check interval, as a rate. +- **Camera changed** (reading sensors with a region): the region is a share of the picture, so a + camera back from a restart with another size reads something else (issue #40: an ESP32 camera + whose picture moved, nobody near it). The first frame after the region or source changed sets + `db.Sensor.frame_size`; a frame of another size is not read (`engine.reading.CAMERA_CHANGED`): + the value stays, the frame is kept once in the history (not in Review: the card on the sensor + asks), and Home Assistant gets a notification (`READING["camera_notification_id"]`). It ends + when the size is back, or when the user says the region is right (`POST + /sensors/{id}/accept-picture`) or draws a new one (both clear the size). Held rather than + paused, so a camera that comes back at its old size needs nothing. A camera that moved without + changing size is not caught by this: that is C3 in `docs/WHEEL_READER_PLAN.md`. **Wheel reader** (default for mechanical counters, `counter_reader: "wheels"`, registry `wheel_readers` in `readers.json`, model `wheels-v1`, bundled, 2.2 MB, ~2 ms for 8 wheels on one @@ -427,6 +490,11 @@ the version Home Assistant ships) instead of the occupancy class's house icon. **Reading sensors** publish the value on `sensor.` with `unit_of_measurement`, `device_class` and `state_class` from the mode (counter → `total_increasing`, value → `measurement` except `monetary`, time left → `duration` in `min`), plus the confidence sensor. +Home Assistant's MQTT discovery refuses an entity whose unit its device class does not take (exact +match: `m3` with `water` made the value entity disappear without a word), so `mqtt.ha_unit` writes a +unit as Home Assistant does when that is clear (`m3` → `m³`, `kwh` → `kWh`) and otherwise leaves the +device class out (a warning in the log). The lists are `settings.HA_DEVICE_CLASS_UNITS`. Units +without a device class are never touched: those entities exist, with history in their unit. Attributes: `read_text`, `rejected`, `last_update`, `last_trigger`. The button is "Read now". Four more entities with `enabled_by_default: false` (nothing changes for users who do not turn them on): *Raw reading* (`raw`, diagnostic), *Problem* (`problem`, enum of @@ -548,9 +616,24 @@ loads. |---|---|---| | Settings, DB, embeddings, trained heads | `/data` (per app) | Yes | | Downloaded backbone models | `/data/models` | Excluded (re-downloadable) | -| Training images (and boxes taught to object sensors, as crops) | `/media/visionstate/samples//` (beta: `/media/visionstate_beta`) | With the media folder | -| The people's taught faces (crops, shared by every sensor) | `/data/faces/` — not in the media folder, which every Home Assistant user can browse | Yes | -| History frames | `/media/…/history/` | Limits: 7 days and 2 GB by default, whichever comes first (see below) | +| Training images (and boxes taught to object sensors, as crops) | `/data/samples//` | Yes | +| Thumbnails of training images | `/data/thumbs/sample/` | Excluded (made again when asked for) | +| The people's taught faces (crops, shared by every sensor) | `/data/faces/` | Yes | +| History frames | `/media/visionstate/history/` (beta: `/media/visionstate_beta`) | With the media folder. Limits: 7 days and 2 GB by default, whichever comes first (see below) | + +What the user taught lives in `/data`, next to the database that lists it: Home Assistant backs up +an app's `/data` (also by itself before every update) but not the media folder, so training +images in the media folder came back without their rows, or rows without their images, after a +restore (issue #45). The media folder is also shown to every Home Assistant user by the media +browser. Only the history stays there: large, removed by its limits, fine to lose. Training images are +what the user labelled or taught (a taught box is a small crop, a state sensor's image a whole +frame), usually tens of MB in all; Settings → Storage shows how much. + +Versions up to 0.9.1b2 kept training images in the media folder. They are read there until they +are copied (`engine.HistoryMixin.move_samples`, in the clean-up loop, so start-up is not held up); +the old copies stay for `SAMPLES_MOVE["keep_old_days"]` after the first copy, so restoring the +backup made before the update still gives the older version its images, then they are removed. +Files no sensor lists are left in the media folder. History frames are full camera frames (JPEG 90). They are removed by age (`history_days`; frames waiting for review get twice as long) and by total size (`history_max_gb`, 0 = no limit): @@ -568,17 +651,17 @@ automatically. their label, detected class, box and score, and the own labels). - Camera credentials are removed from exported URLs; the importer re-enters them. - **Readings to share** (`GET /sensors/{id}/reading-export`, Quality tab of a reading sensor), - so reading can be improved: the readings checked by hand (newest first), then the accepted ones - nobody checked (newest first, `"answer": "unchecked"`; never used as labels without a check, + so reading can be improved: the readings checked by hand (newest first), then the ones nobody + checked, accepted or rejected (newest first, `"answer": "unchecked"`; never used as labels without a check, see `docs/WHEEL_READER_PLAN.md`), whose frame is kept — each only the region it was read in plus `export_margin` (the region is kept with every stored reading since 0.7.1b2, `probs["roi"]`; older ones are cut with today's region, `"region": "current"`). One ZIP of at most `READING["export_max_mb"]` (GitHub takes 25 MB per file; about 1,000 readings); the - readings that did not fit are counted (`left_out`). `readings.json` (`"format": 2` since - 0.7.1b5: read text, value, rejection reason, confidence, answer, right value, day only and + readings that did not fit are counted (`left_out`). `readings.json` (`"format": 3`; 2 left out + the rejected readings nobody answered, so a sensor's checks could not be replayed: read text, value, rejection reason, confidence, answer, right value, day only and seconds since the oldest reading; per reading the reader id and app version (kept with every - stored reading since 0.7.1b5, null before), whether the picture was greyscale (IR) and whether - the sensor's light was on; for the wheel reader each wheel's position, probability and most + stored reading since 0.7.1b5, null before), whether the picture was greyscale (IR), whether + the sensor's light was on and the camera picture's size; for the wheel reader each wheel's position, probability and most likely positions, the common shift and the value with the last wheel's fraction), the reading settings without anything that tells where the sensor is (no source, name or region), the user's optional "What meter is this?" text, a README and a CC0 LICENSE — shared images may then @@ -621,6 +704,7 @@ is part of backups and needs no restart. | **Review reminder** ✅ | A notification in Home Assistant (optionally a push) when frames have waited for review a long time | 0.8.0 (beta 0.7.1b1) | | **Larger models, tunable teaching** ✅ | D-FINE M and L, DINOv2 base, a model comparison in Settings, teaching limits globally and per sensor | 0.9.0 (beta 0.8.1b1) | | **Recognising faces** ✅ | Faces in person boxes (YuNet), people taught by face and shared by every camera (SFace, or AuraFace for a fast PC), a People page, an entity per person saying where they are, unknown person, review questions, export opt-in — phases 1–3 of [`FACE_RECOGNITION_PLAN.md`](FACE_RECOGNITION_PLAN.md), and teaching a face from photos | 0.9.0 (betas 0.8.1b3–b5) | +| **Counter limits** ✅ | A counter's limit grows with the time it was away (learned, a fixed rate or per reading), Review moves a counter on, readings held when the camera's picture changes size, training images in the app's storage (backups hold them) | 0.9.1 (betas 0.9.1b1–b6) | | **Better counter readings** | Benchmark, the last value as an expectation, calibrated confidence, extra decimal, self-calibrating regions, wheels-v2, learning per sensor — status per phase in [`WHEEL_READER_PLAN.md`](WHEEL_READER_PLAN.md) | planned | **Open ideas** (not scheduled): more on the History page (export of a filtered selection, a diff --git a/docs/WHEEL_READER_PLAN.md b/docs/WHEEL_READER_PLAN.md index f7af4fb..a1cc093 100644 --- a/docs/WHEEL_READER_PLAN.md +++ b/docs/WHEEL_READER_PLAN.md @@ -19,6 +19,18 @@ training are in [`tools/wheelreader`](../tools/wheelreader/README.md). - *Only three real meter types tested.* Gas meters, IR or night images and angled cameras are untested. - *Confidence is only moderately informative.* A threshold that rejects 4 % of the reads catches about half of the wrong ones (Dryad test). + - *Wheels at rest sit off their digit, each its own way*, but the decoder allows only one common + shift plus ±`wheel_slack_bins`. On oleost1 two black wheels sat at 0.3–0.4 past their digit all + day while the others sat at 0.1–0.2. The decoder put them outside the slack, where their + probability dropped to 0.02–0.08 although the digit was plain. That one wheel pulled the mean + under 0.70, so 22 of the 29 rejected readings were right. A low threshold is not the fix: at 0.56, + 3–4 of that day's 7 misreads would have passed, one of them too high (585 instead of 565). It can + also flip the digit. In one sharp, well-lit frame the black 7 had 0.49 of its mass at 7.2–7.3 + and 0.21 at 0.2. Only 7.2 fell inside the slack, so the decoder read 032.776 instead of 732.776 + at confidence 0.84. Only "went down" caught it. See A2. + - *Red wheels half way while the water stands still.* All of oleost1's misreads came from the last + wheel stopped exactly half way (6½). The neighbouring red 9 or 6 was then read as 8, and the last + wheel as 2, 8 or 9. The same hard frame then comes back for as long as nobody draws water. - *Every frame is read on its own.* The last value is only used afterwards, by hard checks (`went down`, `max_step`, `readers.settling`). - *Geometry is assumed:* equal-width cells and a fronto-parallel view. @@ -51,6 +63,7 @@ repository is CC0 and becomes a test asset): | `pedro40` 0001–0017 | issue #40 (CC0) | held out (wheels-v1 was trained on it in the end: hold out the *next* export instead) | | `bol_cold`, `bol_warm` | bolausson (GPL-3.0) | test only, never training | | `dryad_test` | Dryad (CC0) | test, right way up | +| `oleost1`, … | the maintainer's water meter (CC0 exports, one per few days from 2026-10-14; ESP32-CAM 1600×1200 with flash, 8 wheels, 3 red) | held out: test only, a meter with wheels sitting off their digits at rest | | new shared exports | §5 | **split by meter**: one meter is either training or test, never both | **Metrics**, reported per set by `tools/wheelreader/evaluate.py`: @@ -104,7 +117,8 @@ In the decoder the expectation can be tested, explained, set per sensor and used - only debounced, published values become L; - N readings in a row that clearly disagree with L, *and agree with each other*, release the prior. They are treated as a new start, which also covers a replaced or reset meter. This is - shown in the History as an event and on the Live tab; + shown in the History as an event and on the Live tab. The counter limit decided otherwise for + now (readings that agree never override it, `docs/SCOPE.md`): a release would wait for Review; - after a restart, L is the last value from the history (as today), but its age counts: an old L gives a wide window. - **UI and history:** a new reason, `disagrees_with_last` (lower-case key for the "problem" entity), @@ -123,6 +137,14 @@ In the decoder the expectation can be tested, explained, set per sensor and used - **Reading-level confidence:** use the decoder's alternatives: P(best V) / Σ P(top-k V) (a normalised margin between the best value and its nearest rivals, typically ±1 on a wheel), not the mean per wheel. The k-best come from the DP (keep the top few per state) or from the A1 window. +- **A wheel at rest counts by its whole digit**, not by the one bin the common shift puts it at: its + probability is the mass within about ±0.45 of its digit. The **weakest wheel** also counts on its + own, not only through the mean. On oleost1 (§1), with only the exported top three per wheel, the + right readings came out at 0.94–0.99 with their weakest wheel mostly ≥ 0.8. The misreads had a + weakest wheel of 0.14–0.43. "Weakest wheel under about 0.5 → unsure" would have passed nearly all + right readings and caught every misread, including the one that was published (last wheel read + as 2 at 0.44). This is to be measured with the full 100-bin output on the whole benchmark, set + against the value margin above. - **Pick the default threshold** from the risk–coverage curve (§3). The UI text stays "at least 70 % sure", but 70 % then really means 70 %. - **Registry:** T belongs to the model, so a calibrated wheels-v1 is a new entry @@ -148,8 +170,9 @@ first plan (2026-10-10): the sensor's light on) instead of a flag in the dialog; - frames are **not** deduplicated in the app — a reading of the same frame is still a reading; the training tools remove duplicate images (§B4); -- the unchecked accepted readings are **always** included (no option, no count limit): the images - are what is scarce. Their value is the reader's guess, so the tools use them as labels only after +- the unchecked readings, accepted and rejected, are **always** included (no option, no count + limit): the images are what is scarce, and with the rejected ones a sensor's checks can be + replayed in order (`seconds`). Their value is the reader's guess, so the tools use them as labels only after a check (§B4); - no spot-check button; `spot_rate` is the way to get accepted readings checked; - one ZIP of at most 24 MB (GitHub's limit is 25 MB per file), checked readings first, then the @@ -186,6 +209,10 @@ user looks at the images before sharing (README.txt in the ZIP). sequence confirms them (a counter only goes up: a value between two confirmed neighbours that fits the rate). Until the tools do that check, they skip them. - A label check after every training run (§3). +- A last wheel stopped half way has two right answers. On oleost1 the same frame was marked right + as 596 on some readings and as 597 on others. The tools count either neighbour as right there and + keep such cells as interval labels, as the per-sensor labels in §8 do. +- Collect the half-way red-wheel cells from exports as their own case for wheels-v2 (§7). - `VisionStateLocal/README.md` lists the source and licence of every set, as it does now. ## 6. Phase C — geometry: let the app correct itself @@ -205,7 +232,9 @@ showing at the side and wheels that sit a little off each other at rest. - vertical offset ±10 %; - height scale ±10 %; - rotation ±3°; - - cell borders moved ±10 % one by one, or as a linear stretch (cells of unequal width). + - cell borders moved ±10 % one by one, or as a linear stretch (cells of unequal width); + - a rest offset per wheel: where each wheel that is not turning sits relative to its digit, found + over the K frames (on oleost1 it was +0.3 for two wheels and +0.1 for the rest, the same all day). - It keeps the change with the best total decode score over the K frames. Cost: K × changes × about 2 ms; a coarse-to-fine search keeps it to a few seconds on a Pi. - **The result** is stored with the sensor (`reading.calibration`, reset when the region or the @@ -225,6 +254,10 @@ the shift of the region against a reference frame by phase correlation (numpy FF OpenCV) and move the region along. AI-on-the-edge does this with two reference marks the user picks. Doing it automatically, on the area around the region, needs no user step. It must check the correlation peak and skip frames where it is weak (light changes, IR switching). +A first real case was oleost1: an ESP32-CAM mount moved on its first day, the region was redrawn +twice, and the digits still sat at different heights in crops of the same size. +A camera that comes back with another picture size is already caught without this (readings held, +"camera changed", [`SCOPE.md`](SCOPE.md) §5); C3 is for a picture that moved at the same size. ### C4. Perspective correction @@ -237,7 +270,7 @@ working), the corner order and orientation, and dragging corners on a phone. ## 7. Phase D — wheels-v2 - **Drawn cells:** whole counter rows drawn in perspective and then cut like the app does; - monochrome IR; flash reflections on glass; fog and drops; DIN-like narrow fonts (OFL); the failure + monochrome IR; flash reflections on glass; red wheels stopped half way; fog and drops; DIN-like narrow fonts (OFL); the failure types from phase B. - **Context:** the cell plus a strip of each neighbour (or a model that reads the whole row and gives one position per wheel), so carries are seen together. Measure against the per-wheel model. diff --git a/scripts/fake_camera.py b/scripts/fake_camera.py index eb7568e..1a2d4c0 100644 --- a/scripts/fake_camera.py +++ b/scripts/fake_camera.py @@ -9,6 +9,8 @@ GET /display.jpg?text=12.5&style=lcd|led a drawn seven-segment display (for reading sensors) GET /counter.jpg?value=89939.5&digits=7&decimals=3 a drawn mechanical counter with rolling digits (value = the number on the wheels; .5 = last wheel half way) +GET /set?wider=400&camera=a counter.jpg?…&camera=a comes back 400 px wider (a camera that changed + its picture size after a restart); wider=0 sets it back """ import io @@ -27,6 +29,7 @@ from displays import render as render_display # noqa: E402 STATE = {"state": "closed", "night": False} +WIDER: dict[str, int] = {} # counter.jpg's "camera" -> pixels added on the right DOOR_HEIGHT = {"open": 0.08, "closed": 1.0, "partial": 0.5} @@ -69,6 +72,8 @@ def do_GET(self): # noqa: N802 STATE["state"] = query["state"][0] if "night" in query: STATE["night"] = query["night"][0] == "1" + if "wider" in query: + WIDER[query.get("camera", [""])[0]] = int(query["wider"][0]) self._send(str(STATE).encode(), "text/plain") elif url.path == "/display.jpg": query = parse_qs(url.query) @@ -88,6 +93,10 @@ def number(name: str, default: float, low: float, high: float) -> float: digits = int(number("digits", 7, 1, 8)) image = render_counter(number("value", 0, 0, 10**digits), digits, int(number("decimals", 3, 0, digits))) + if wider := WIDER.get(query.get("camera", [""])[0], 0): + wide = Image.new("RGB", (image.width + wider, image.height), (40, 40, 40)) + wide.paste(image, (0, 0)) + image = wide buf = io.BytesIO() image.save(buf, format="JPEG", quality=90) self._send(buf.getvalue(), "image/jpeg") diff --git a/visionstate/CHANGELOG.md b/visionstate/CHANGELOG.md index 57d7d6f..48b5aba 100644 --- a/visionstate/CHANGELOG.md +++ b/visionstate/CHANGELOG.md @@ -1,5 +1,66 @@ # Changelog +## 0.9.1 + +### Better + +- **Counters accept the value after a restart or update.** With *Learn it* or *Fixed rate* (below), + how much a counter may go up grows with the time it was not seen — the app was away, or the + readings were unsure — so the first reading after an update is no longer rejected as *changed + too much*. A wheel read one digit too high over and over still stays out. +- **Choose how a counter's limit works** (Settings → Sensor output): + - **Learn it** — how fast it goes up on a busy day, learned from a week of its readings. Until + then a standard rate applies (for a water meter with three decimals, 0.04 m³ a minute), shown + there and yours to change. + - **Fixed rate** — a rise per hour that you set. + - **Fixed per reading** — the largest rise from one reading to the next, the same however long + the app was away. + - **No limit.** + + Counters you already have keep the limit you set, as *Fixed per reading*: switch to *Learn it* + or *Fixed rate* for the value after a restart to be accepted by itself. +- **Read correctly in Review moves the counter on**: answering it for the newest rejected reading + makes that the value — the way on after a real jump over the limit (say the light failed during + a shower), and the way out for a counter stuck on a value read too high. +- **Readings are held when the camera's picture changes size** (a camera that comes back from a + restart with another resolution shows something else in the region): *problem* becomes + `camera_changed`, Home Assistant gets a notification, and the sensor asks you to check the + region — **The region is right** or **Adjust the region**. It reads again by itself when the + size comes back. +- **Training images are kept in the app's own storage**, next to the database that lists them, + so a Home Assistant backup of the app holds both. They were in the media folder, which the + app's backups — also the one Home Assistant makes before every update — do not include: after a + restore, the database could list images that were not there. The media folder is also shown to + every Home Assistant user by the media browser. Your images are copied over by themselves after + the update; the old copies in `/media/visionstate/samples` are removed 30 days later, so going + back to the backup made before the update still works until then. Backups of the app get larger + by the size of your training images (Settings → Storage shows it). History frames stay in the + media folder. +- **A light waits 2 seconds before the frame by default** (was 1 second): an ESP32 camera at full + size needs about a second per picture, and with 1 second the frame read was often dark. Sensors + you already have keep their setting — raise *Wait before taking the frame* to 2 s if readings + come out dark. +- **Export readings** also holds the rejected readings nobody answered, and each reading's + picture size, so an export shows the sensor's checks in order. +- **Switching the face model shows its progress** on the sensors' cards, the Faces card, People and + Settings, and the taught faces are learned again right after the new model is loaded. + +### Fixed + +- **A reading sensor with the unit `m3` now shows its value in Home Assistant.** Home Assistant + only takes `m³` for water and gas, and dropped the whole value entity without a word — only the + rate and the diagnostics appeared. VisionState now sends the unit the way Home Assistant writes + it (`m3` → `m³`, `kwh` → `kWh`); a unit the chosen type does not take at all is sent without the + type, so the entity still appears (not in the Energy dashboard; the app's log says so). Sensors + whose entities already work are not changed. +- **An object sensor no longer stops when the image of a taught box is gone** (issue #45, thanks + to @terryhonn for the report and @wzd8888 for the first fix). It showed *No such file or + directory* on every check. A box, or a state sensor's training image, whose file is missing or + unreadable is now left out — the log names it — and the sensor keeps working. +- **Deleting a training image, a taught box or a face no longer shows an error** when the file + cannot be removed (in use, or no permission): it is gone from VisionState, and the log names the + file. + ## 0.9.0 ### New diff --git a/visionstate/DOCS.md b/visionstate/DOCS.md index 1973456..1417c60 100644 --- a/visionstate/DOCS.md +++ b/visionstate/DOCS.md @@ -196,7 +196,8 @@ everyone. (**Recognise people by their face** under the objects, once Person is picked). A sheet says what is stored and which face model is downloaded (once, from Hugging Face), and nothing happens until you confirm. While it downloads the sensor keeps working without faces; the card - shows the progress, and **Try again** if it failed. Without internet, put the file the message + shows the progress, and **Try again** if it failed. The sensor's card on the start page and its + page say *Downloading faces…*, *Loading faces…* or *Learning faces…* until faces are named. Without internet, put the file the message names in the app's `models` folder by hand. - **Does my camera see faces well enough?** The **Live** tab draws a small rounded box around each face it finds in a person, and says for each person what it saw: a name, *face too small*, @@ -263,9 +264,9 @@ everyone. stay. Deleting a sensor keeps the faces taught on it. Turned off on every sensor, the face model is let go from memory; **Settings → AI model → Compare and manage the models** can then delete its file. -- **Where faces are kept**: in the app's own storage, not in the media folder (which Home - Assistant's media browser shows to every user) — so they are in Home Assistant backups: keep - those encrypted. The numbers the AI computes from a face are kept with them and deleted with the +- **Where faces are kept**: in the app's own storage, like training images, not in the media + folder (which Home Assistant's media browser shows to every user) — so they are in Home Assistant + backups: keep those encrypted. The numbers the AI computes from a face are kept with them and deleted with the person; an export never contains them. ## Reading sensors @@ -299,6 +300,11 @@ First choose **what it looks like**: | **Value** | Prices and other numbers that go up and down | `measurement` (none for money, as Home Assistant requires) | | **Time left** | Countdowns like `1:25` on appliances | minutes (`1:25` = 85 min), device class *duration* | +- **Unit** and **Home Assistant type**: Home Assistant only takes certain units for a type + (water: m³, L, gal, ft³, CCF, MCF). VisionState sends `m3` as `m³` and `kwh` as `kWh`; a unit + the type does not take at all is sent without the type, so the entity still appears, only not + in the Energy dashboard (the app's log says so). + - **Digits after the decimal point** decide where the decimal point is. Dots and commas on the display are ignored, because a stray dot is the most common misread. When the test read shows a decimal point (for example `1234.5`), VisionState offers to set the matching number. @@ -308,8 +314,8 @@ First choose **what it looks like**: - **Mechanical counters**: the coloured wheels are usually the decimals — a water meter with five black and three red wheels has 8 digits and 3 digits after the decimal point. The last wheel is rounded to the nearest digit. A counter that reads lower than before is rejected, and - a limit on how much the value may change (Settings → Sensor output) catches a misread that is - too high. A reading exactly one step of the last digit below the value is the + a limit on how fast it may go up (see *Counter limit* below) catches a misread that is too + high. A reading exactly one step of the last digit below the value is the last wheel turning: the value stays, but it is not counted as rejected and does not go to the review queue (the Live tab says *Last wheel turning*). If the last wheel never stands still, you can also leave it out of the region and count one digit and one decimal less. @@ -318,10 +324,36 @@ First choose **what it looks like**: choose *Wheel reader* on their Settings tab (their Live tab suggests it). - **Safety net**: a reading is rejected — and the last value kept — when the reader is less sure than the minimum (default 70 %), finds no number, a counter reads lower than before, a - mechanical counter is read with the wrong number of digits, or the value changes more than the - limit you set. Every rejected reading is listed in the **History** tab with the reason and - waits in the **Review** queue. A new value is published after 2 equal readings in a row - (adjustable). + mechanical counter is read with the wrong number of digits, a counter goes up faster than its + limit, or another value changes more than the limit you set. Every rejected reading is listed + in the **History** tab with the reason and waits in the **Review** queue. A new value is + published after 2 equal readings in a row (adjustable). +- **Counter limit** (Settings → Sensor output): how fast a counter may go up. *Learn it* (new + counters) learns it from the counter's own readings: the fastest it went up over ten minutes + and in a whole day, on a busy day of the last two weeks, with room to spare. It needs a week of + readings first; until then a rate per hour applies, which you may change — it starts at a + standard one (for a water meter with three decimals 0.04 m³ a minute) — and once learned, the + Settings tab shows what it learned in its place. *Fixed rate* takes a rise per hour you choose; + *Fixed per reading* the largest rise from one reading to the next, the same however long the + app was away (after a restart a real rise waits for your answer in Review); *No limit* rejects only a counter that reads lower. Choosing *Fixed rate* or + *Fixed per reading* fills in the standard limit to start from. + With *Learn it* and *Fixed rate*, the longer the counter was not seen — the app restarted or + was updated, the camera was away, the readings were unsure — the more it may have gone up, so + a reading after a restart is accepted. Time in which the counter was read clearly but too high does not count: a wheel + that is read one digit too high over and over stays rejected. + A rise over the limit while the app was watching (say the light failed during a shower) is + never accepted by itself, however many readings agree: it waits in the Review queue. + Answering *Read correctly*, or *Misread* with the right value, on the newest rejected reading + makes that the counter's value, so it goes on from there. Counters set up before the limit + could be learned keep the limit they had: *Fixed per reading* with the same change, or no limit. +- **Camera changed:** the region is kept as a part of the picture, so a camera that comes back + from a restart with another picture size (some ESP32 cameras fall back to their default size, + and also show another part of what they see) would make the region cover something else. + VisionState notices it: the readings are held (the value stays, *problem* is + `camera_changed`), the sensor shows a card and Home Assistant a notification. It goes on by + itself when the size is back. Otherwise look at the region on the Live tab: **The region is + right** reads again at the new size, **Adjust the region** opens Settings, where a new region + is drawn on the picture as it is now. Only reading sensors with a region are checked. - **Quality** tab: the share of readings that were accepted today, in the last 7 and 30 days, why the others were rejected, and a chart per day. Below it the rejected readings with their frame: tell whether the reader read the meter right (*Read correctly*) or not (*Misread*, optionally with the value it showed — the @@ -334,7 +366,7 @@ First choose **what it looks like**: The reader does **not** learn from these answers: they show how reliable the reading is and which setting to change. - **Help improve reading:** **Export readings** (Quality tab) makes a ZIP of the readings you - checked and the accepted ones nobody checked (marked as unchecked) — only the region of each, + checked and the ones nobody checked, accepted or rejected (marked as unchecked) — only the region of each, not the whole picture — with what was read and what was right, which reader read it and (for the wheel reader) where it saw each wheel. Each image is cut with the region the reading was made in, so moving the region later does not spoil the readings from before. Write what the @@ -354,7 +386,7 @@ First choose **what it looks like**: | Entity | What it is | |---|---| | `sensor._raw_reading` | What the reader read last, also when it was rejected | - | `sensor._problem` | `ok`, or why the last reading was rejected (`went_down`, `unsure`, `nothing_read`, `wrong_digit_count`, `changed_too_much`) | + | `sensor._problem` | `ok`, or why the last reading was rejected (`went_down`, `unsure`, `nothing_read`, `wrong_digit_count`, `changed_too_much`, `camera_changed`) | | `sensor._accepted_24_h` | Share of the readings of the last 24 hours that were accepted, in % | | `sensor._rate` | Counters only: how fast it goes up, over the last 15 minutes (Settings → Sensor output). kWh gives **kW**, m³ gives **m³/h**, L gives **L/min**, other units *unit*/h — for example to spot a water leak | | `image._reader_image` | What the reader saw at the last reading: the region after display processing (*What the reader sees* on the Live tab) | @@ -430,7 +462,7 @@ For a camera in a dark place, such as a meter cabinet: pick a light, switch or h the camera (the first step of the wizard, or **Settings → General**). VisionState switches it on: - **for each check**: before the frame is taken, and off again afterwards. *Wait before taking the - frame* (default 1 s) gives the light and the camera time; meanwhile VisionState fetches a frame + frame* (default 2 s; longer if the frames still come out dark) gives the light and the camera time; meanwhile VisionState fetches a frame every second and throws it away. Many cameras (an ESP32 camera, for one) hand out a picture they took earlier, and only adjust their exposure between pictures: this way the frame that is read is a fresh one, taken in the light. It stays on during the checks that follow a trigger; @@ -478,10 +510,18 @@ How long history is kept is set under **Settings → Storage** (below). ## Backup, export and import -- Settings, the database and trained models live in the app's data folder and are part of - Home Assistant backups. -- Training images and history frames are stored in `/media/visionstate` (the beta app uses - `/media/visionstate_beta`). +- Settings, the database, trained models and **training images** (also the boxes taught to object + sensors and the taught faces) live in the app's own storage and are part of Home Assistant + backups of the app, so a restore brings back what you taught together with its images. +- **History frames** are stored in `/media/visionstate` (the beta app uses + `/media/visionstate_beta`); they are only in a backup that includes the media folder. +- Older versions kept training images in `/media/visionstate/samples`. They are copied to the + app's own storage by themselves after the update; the copies in the media folder are removed + 30 days later (until then, going back to the backup made before the update still finds them). + Images there that no sensor uses are left alone and can be deleted by hand. +- A training image that has gone missing (for example after restoring a backup made without it) + is left out of training and of what an object sensor was taught; the log names it, and the + sensor keeps working. - **Settings → Storage** shows how much space history frames and training images use and how much is free. History is kept for **7 days** but at most **2 GB** by default — whichever is reached first; the oldest frames are removed first, frames waiting for review last (they are @@ -529,7 +569,9 @@ The models, chosen under **Settings → AI model** for all sensors of a kind (sm times as slow per face), which tells faces apart better in poor light and at an angle. YuNet was trained on a research data set and SFace's paper names research data sets too (their terms are not commercial); AuraFace's makers say its data allows commercial use. Switching - compares every taught face again with the new model; the people are kept. Only loaded while a + computes every taught face again with the new model (*Learning faces… 12/40* on the sensors' + cards, the progress on the Faces card and the People page; the sensors find people without + naming them until it is done); the people are kept. Only loaded while a sensor recognises faces. - **Reading sensors:** PP-OCRv6 tiny (PaddleOCR) — included, a few milliseconds per reading. PP-OCRv6 small is larger and reads unusual fonts more reliably (downloaded on first use, diff --git a/visionstate/backend/tests/test_change_limit.py b/visionstate/backend/tests/test_change_limit.py new file mode 100644 index 0000000..771d7af --- /dev/null +++ b/visionstate/backend/tests/test_change_limit.py @@ -0,0 +1,92 @@ +"""How much a counter may go up: the limit over time, what it learns, agreeing readings.""" + +import pytest + +from visionstate import change_limit +from visionstate.change_limit import Learned, Watch +from visionstate.settings import COUNTER_LIMIT, merge_reading + +WATER = merge_reading({"mode": "counter", "display": "counter", "digits": 7, "decimals": 3}) +FIXED = {**WATER, "change_limit": "fixed", "max_rate": 2.4} # 0.04 m³ a minute +MARGIN = COUNTER_LIMIT["margin_steps"] * 0.001 +BUSY = Learned(per_hour=0.55, per_day=0.8, days=9) + + +def test_limit_grows_with_time(): + assert change_limit.limit(FIXED, None, 60) == pytest.approx(MARGIN + 0.04) + assert change_limit.limit(FIXED, None, 35 * 60) == pytest.approx(MARGIN + 1.4) + assert change_limit.limit({**FIXED, "max_rate": 0}, None, 60) is None + assert change_limit.limit({**WATER, "change_limit": "off"}, BUSY, 60) is None + # "auto", nothing learned yet: the rate set, or the standard one (0.04 m³ a minute here). + assert change_limit.limit(WATER, None, 60) == pytest.approx(MARGIN + 0.04) + assert change_limit.limit({**WATER, "max_rate": 1.2}, None, 60) == pytest.approx(MARGIN + 0.02) + assert change_limit.standard_rate({**WATER, "decimals": 1}) == pytest.approx(240) + # "step": the same for every reading, however long the counter was away; 0: no limit. + step = {**WATER, "change_limit": "step", "max_step": 0.04} + assert change_limit.limit(step, BUSY, 60) == change_limit.limit(step, BUSY, 86_400) == 0.04 + assert change_limit.limit({**step, "max_step": 0}, BUSY, 60) is None + safety = COUNTER_LIMIT["safety"] + assert change_limit.limit(WATER, BUSY, 60) == pytest.approx(MARGIN + safety * 0.55 / 60) + # Away for three days: no more than three busy days, not three days of showering. + assert change_limit.limit(WATER, BUSY, 3 * 86_400) == pytest.approx(MARGIN + safety * 3 * 0.8) + + +def test_learn_needs_days_and_ignores_the_busiest_one(): + week = [(0.05, 0.5)] * (COUNTER_LIMIT["min_days"] - 1) + assert change_limit.learn(week) is None + learned = change_limit.learn([*week, (0.09, 0.7), (4.0, 9.0)]) # one day with a misread + assert learned is not None and learned.days == COUNTER_LIMIT["min_days"] + 1 + assert learned.per_hour == pytest.approx(0.09 * 3600 / COUNTER_LIMIT["window_s"]) + assert learned.per_day == pytest.approx(0.7) + assert change_limit.learn([(0.0, 0.0)] * 10) is None # nothing used: nothing to learn + + +def test_only_unseen_time_counts(): + watch = Watch(seen_at=0.0, accepted_at=0.0) + change_limit.advance(watch, 60, 60) + assert watch.budget_s == 60 + # Read with confidence over the limit: the next minutes do not add to the limit. + watch.held = True + for minute in range(2, 30): + change_limit.advance(watch, minute * 60, 60) + assert watch.budget_s == 60 + # The app was down for an hour: that does count (beyond a normal pause). + change_limit.advance(watch, 29 * 60 + 3600, 60) + assert watch.budget_s == pytest.approx(60 + 3600 - change_limit.gap(60)) + + +def test_learns_the_rise_over_the_window(): + watch = Watch() + assert change_limit.observe(watch, 0, 100.0, 60) is None # the first one: only the start + for minute in range(1, 21): + rise = change_limit.observe(watch, minute * 60, 100.0 + 0.01 * minute, 60) + assert rise is not None and rise[1] == pytest.approx(0.01 if minute > 1 else 0.0) + assert rise is not None and rise[0] == pytest.approx(0.01 * COUNTER_LIMIT["window_s"] / 60) + # One reading too high (a misread that got through) counts with the next one's value. + assert change_limit.observe(watch, 21 * 60, 101.2, 60) == (pytest.approx(0.1), pytest.approx(0.01)) + assert change_limit.observe(watch, 22 * 60, 100.22, 60) == (pytest.approx(0.11), pytest.approx(0.02)) + # A pause (down, unsure, rejected): the rise over it is not learned. + assert change_limit.observe(watch, 3600, 101.0, 60) is None + assert change_limit.observe(watch, 3660, 101.0, 60) == (0.0, 0.0) + + +def test_accept_starts_the_limit_again(): + watch = Watch(accepted_at=0.0, budget_s=900.0, held=True) + change_limit.accept(watch, 1000) + assert (watch.accepted_at, watch.budget_s, watch.held) == (1000, 0.0, False) + + +def test_held_by_readings_over_the_limit(): + """However long readings over the limit go on, the limit does not grow: only time away counts.""" + watch = Watch(seen_at=0.0, accepted_at=0.0, learned=BUSY) + change_limit.hold(watch) + for minute in range(1, 301): + change_limit.advance(watch, minute * 60, 60) + assert watch.held and watch.budget_s == 0.0 + + +def test_learning_counter_keeps_out_issue_40s_misread(): + """Issue #40: switched to "auto", a wheel read two digits off (635.709 to 637.799) 32 minutes + after the last accepted reading went through, as nothing was learned yet.""" + allowed = change_limit.limit(WATER, None, 1920) + assert allowed is not None and allowed < 637.799 - 635.709 diff --git a/visionstate/backend/tests/test_faces.py b/visionstate/backend/tests/test_faces.py index 5c75c1b..405bc00 100644 --- a/visionstate/backend/tests/test_faces.py +++ b/visionstate/backend/tests/test_faces.py @@ -4,6 +4,7 @@ 2018, Butch Wilmore, the two of them together, and a crew of nine with both of them in it. """ +import asyncio import io import json import zipfile @@ -339,6 +340,33 @@ def teach(name): assert all(len(index.ids) == 2 for index in built) with rt.db.session() as s: assert s.scalar(select(func.count()).select_from(FaceEmbedding)) == 2 + # Embedding the taught faces again (a new model) shows how far it is, and nothing when done. + with rt.db.session() as s: + s.execute(delete(FaceEmbedding)) + progress, embed = [], rt._embed_taught + rt._embed_taught = lambda *args: progress.append(rt.face_status()["embedding"]) or embed(*args) + try: + assert len(rt._build_face_index(rt.face_model, 0).ids) == 2 + finally: + del rt._embed_taught + assert progress == [{"done": 0, "total": 2}, {"done": 1, "total": 2}] + assert rt.face_status()["embedding"] is None + # A newly loaded model embeds them before it is used: the checks go on without faces meanwhile. + with rt.db.session() as s: + s.execute(delete(FaceEmbedding)) + unused = [] + rt._embed_taught = lambda *args: unused.append(rt.face_model is None) or embed(*args) + + async def reload(): + rt.face_model, rt.face_index, rt._face_task = None, None, asyncio.current_task() + await rt._load_faces() + + try: + client.portal.call(reload) + finally: + del rt._embed_taught + assert unused == [True, True] and rt.face_model is not None and rt.face_index is not None + assert len(rt.face_index.ids) == 2 # Each person's own entity says where they are seen: on this sensor now. assert people["people"][0]["seen"]["now"] == "beach" and people["people"][0]["seen"]["sensor_id"] == sid image = client.get(f"/api/v1/faces/{face_id}/image") @@ -448,14 +476,16 @@ def offline(*_args, **_kwargs): with TestClient(create_app(settings)) as client: rt = client.app.state.runtime assert client.post("/api/v1/faces/retry").status_code == 200 - assert wait_for(lambda: rt.face_status()["error"], timeout=10) + # The error is set inside the loading task, which ends just after it. + assert wait_for(lambda: rt.face_status()["error"] and not rt.face_status()["loading"], timeout=10) error = rt.face_status()["error"] assert faces.DETECTOR.filename in error and "by hand" in error assert rt.face_model is None and not rt.face_status()["loading"] # No retry by itself: only when asked. assert rt.wanted_faces() is None and rt.face_status()["error"] == error assert client.post("/api/v1/faces/retry").status_code == 200 - assert wait_for(lambda: rt.face_status()["error"], timeout=10) + # The error is set inside the loading task, which ends just after it. + assert wait_for(lambda: rt.face_status()["error"] and not rt.face_status()["loading"], timeout=10) settings_view = client.get("/api/v1/settings").json() assert [m["id"] for m in settings_view["face_models"]] == list(faces.RECOGNISERS) assert all(not m["installed"] and m["data"] for m in settings_view["face_models"]) diff --git a/visionstate/backend/tests/test_publish.py b/visionstate/backend/tests/test_publish.py index 9402e11..ae83daa 100644 --- a/visionstate/backend/tests/test_publish.py +++ b/visionstate/backend/tests/test_publish.py @@ -44,7 +44,7 @@ def test_sensor_not_sent_to_home_assistant(tmp_path): "kind": "reading", "source_type": "http", "source": "x", - "reading": {"mode": "counter", "decimals": 1}, + "reading": {"mode": "counter", "decimals": 1, "change_limit": "off"}, "interval_s": 1, "debounce": 1, "publish": False, diff --git a/visionstate/backend/tests/test_reading.py b/visionstate/backend/tests/test_reading.py index 21dd126..8855111 100644 --- a/visionstate/backend/tests/test_reading.py +++ b/visionstate/backend/tests/test_reading.py @@ -13,11 +13,11 @@ from sqlalchemy import select from visionstate import readers -from visionstate.db import Database, Prediction, Sensor, keep_text_reader +from visionstate.db import Database, Prediction, Sensor, keep_change_step, keep_text_reader from visionstate.main import create_app from visionstate.mqtt import SensorDescriptor, discovery_messages from visionstate.readers import counter_line, decode, format_value, implausible, parse, wrong_digit_count -from visionstate.settings import KIND_READING, READING, VERSION, merge_reading +from visionstate.settings import COUNTER_LIMIT, KIND_READING, READING, VERSION, merge_reading from .conftest import ASSETS, MODEL_DIR, requires_reader from .displays import counter_box, counter_positions, render, render_counter @@ -51,13 +51,15 @@ def test_parse(text, settings, expected): def test_implausible_readings(): - counter = cfg(mode="counter", max_step=10) - assert implausible(100, None, counter) is None # nothing to compare with yet - assert implausible(99, 100, counter) == "went down" - assert implausible(111, 100, counter) == "changed too much" - assert implausible(105, 100, counter) is None + counter = cfg(mode="counter", max_step=10) # a counter's limit is change_limit's, not max_step + assert implausible(100, None, counter, 10) is None # nothing to compare with yet + assert implausible(99, 100, counter, 10) == "went down" + assert implausible(111, 100, counter, 10) == "changed too much" + assert implausible(105, 100, counter, 10) is None + assert implausible(1000, 100, counter) is None # no limit value = cfg(mode="value") assert implausible(1.2, 1.9, value) is None # values may go down, no step limit set + assert implausible(1.2, 1.9, cfg(mode="value", max_step=0.5)) == "changed too much" def test_counter_digit_count(): @@ -129,6 +131,32 @@ def entity(reading): assert "state_class" not in price # Home Assistant rejects measurement for money +def test_discovery_writes_the_unit_as_home_assistant_takes_it(): + """Home Assistant refuses a unit its device class does not take ("m3" for water): the entity + would be missing. Clear spellings are corrected, otherwise the device class is left out.""" + + def configs(reading): + sensor = SensorDescriptor("meter", "Meter", [], KIND_READING, [], merge_reading(reading)) + return {topic.split("/")[-2]: payload for topic, payload in discovery_messages("homeassistant", sensor)} + + water = configs({"mode": "counter", "unit": "m3", "device_class": "water"}) + assert (water["state"]["unit_of_measurement"], water["state"]["device_class"]) == ("m³", "water") + assert (water["rate"]["unit_of_measurement"], water["rate"]["device_class"]) == ("m³/h", "volume_flow_rate") + energy = configs({"mode": "counter", "unit": " kwh ", "device_class": "energy"})["state"] + assert (energy["unit_of_measurement"], energy["device_class"]) == ("kWh", "energy") + # "mwh" could be MWh or mWh, "dm³" is no water unit: sent without the device class. + for unit, device_class in (("mwh", "energy"), ("dm³", "water")): + state = configs({"mode": "counter", "unit": unit, "device_class": device_class})["state"] + assert state["unit_of_measurement"] == unit and "device_class" not in state + assert state["state_class"] == "total_increasing" + # Left alone: what Home Assistant already takes, so no entity changes its unit. + plain = configs({"mode": "counter", "unit": "m3"}) + assert plain["state"]["unit_of_measurement"] == "m3" and plain["rate"]["unit_of_measurement"] == "m3/h" + assert configs({"mode": "value", "unit": "kr", "device_class": "monetary"})["state"]["unit_of_measurement"] == "kr" + no_unit = configs({"mode": "counter", "device_class": "water"})["state"] + assert no_unit["device_class"] == "water" and "unit_of_measurement" not in no_unit + + def test_diagnostic_entities_of_reading_sensors(): """Raw reading, problem, accepted share for every reading sensor; a rate for counters only. @@ -348,6 +376,35 @@ def test_counters_made_before_the_wheel_reader_keep_the_text_reader(tmp_path): assert keep_text_reader(None) is None +def test_counters_keep_their_change_limit_as_a_rate(tmp_path): + path = tmp_path / "old.db" + Database(path).init() + con = sqlite3.connect(path) + row = ( + "INSERT INTO sensor (slug, name, kind, source_type, source, interval_s, threshold, debounce, enabled, " + "entity_prefix, publish, created_at, reading) VALUES (?, ?, 'reading', 'http', 'x', ?, 0.7, 2, 1, 0, 1, " + "'2026-10-01', ?)" + ) + con.execute(row, ("water", "Water", 60, json.dumps({"mode": "counter", "decimals": 3, "max_step": 0.04}))) + con.execute(row, ("gas", "Gas", 30, json.dumps({"mode": "counter", "max_step": 0}))) + con.execute(row, ("price", "Price", 60, json.dumps({"mode": "value", "max_step": 5}))) + con.execute("PRAGMA user_version = 13") # as a database of 0.9.0 + con.commit() + con.close() + db = Database(path) + db.init() + with db.session() as s: + kept = {r.slug: merge_reading(r.reading) for r in s.query(Sensor)} + # The change per reading stays exactly that; no limit stays no limit; values keep max_step. + assert (kept["water"]["change_limit"], kept["water"]["max_step"]) == ("step", 0.04) + assert kept["gas"]["change_limit"] == "off" + assert kept["price"]["max_step"] == 5 + assert keep_change_step({"mode": "counter", "change_limit": "auto"}) == { + "mode": "counter", + "change_limit": "auto", + } + + def test_darkest_turns_coloured_digits_as_dark_as_black_ones(): image = Image.new("RGB", (3, 1)) image.putpixel((0, 0), (200, 30, 35)) # red digit @@ -373,10 +430,16 @@ class CounterCamera: def __init__(self, value: float): self.value = value + self.wider = 0 # pixels added on the right: a camera that came back with another picture size async def grab(self, source_type, source): + image = render_counter(self.value) + if self.wider: + wide = Image.new("RGB", (image.width + self.wider, image.height), (40, 40, 40)) + wide.paste(image, (0, 0)) + image = wide buf = io.BytesIO() - render_counter(self.value).save(buf, format="JPEG", quality=92) + image.save(buf, format="JPEG", quality=92) return buf.getvalue() async def close(self): @@ -422,7 +485,13 @@ def test_reading_sensor_flow(settings): "kind": "reading", "source_type": "http", "source": "http://fake", - "reading": {"mode": "counter", "decimals": 1, "unit": "kWh", "device_class": "energy"}, + "reading": { + "mode": "counter", + "decimals": 1, + "unit": "kWh", + "device_class": "energy", + "change_limit": "off", + }, "interval_s": 1, "debounce": 1, } @@ -599,6 +668,79 @@ def test_restart_keeps_checking_against_the_last_value(settings): assert client.get(f"/api/v1/sensors/{sid}").json()["reading"]["value"] == "500" +@requires_reader +def test_counter_limit_over_time(settings): + """A counter's limit: a misread that repeats stays out, time the app was down lets the value + on, a jump while watching waits for Review however many readings agree, and a value confirmed + in Review is the new last value.""" + camera = DisplayCamera("00500") + reading = {"mode": "counter", "change_limit": "fixed", "max_rate": 360} # 0.1 a second + body = {"name": "Meter", "kind": "reading", "source_type": "http", "source": "x", "interval_s": 1, "debounce": 1} + body["reading"] = reading + sent: dict[str, object] = {} + + async def publish(topic, payload, retain=False): + sent[topic] = payload + + with TestClient(create_app(settings)) as client: + rt = client.app.state.runtime + rt.grabber = camera + rt.mqtt.publish = publish + sid = client.post("/api/v1/sensors", json=body).json()["id"] + + def view(): + return client.get(f"/api/v1/sensors/{sid}").json()["reading"] + + def last(): + return view()["last"] or {} + + assert wait_for(lambda: view()["value"] == "500", timeout=60) + assert view()["limit"]["allowed"] is not None and view()["limit"]["needed"] == COUNTER_LIMIT["min_days"] + + # The 100 wheel read a step on, again and again: rejected, and the limit does not grow. + camera.text = "00600" + assert wait_for(lambda: last().get("value") == "600" and last().get("reason") == "changed too much", 30) + held = rt.live[sid].watch.budget_s + assert wait_for(lambda: client.get("/api/v1/review").json()["total"] >= 4, timeout=30) + assert view()["value"] == "500" and view()["limit"]["held"] + assert rt.live[sid].watch.budget_s == held + + # The app was down for an hour: the counter went on, and that rise is accepted. + camera.text = "00650" + assert wait_for(lambda: last().get("value") == "650", timeout=30) + rt.live[sid].watch.seen_at -= 3600 + assert wait_for(lambda: view()["value"] == "650", timeout=30) + + # A real jump while it was watching: rejected however many readings agree, until Review. + camera.text = "00800" + assert wait_for(lambda: last().get("value") == "800", timeout=30) + assert wait_for(lambda: client.get("/api/v1/review").json()["items"][0]["probs"]["value"] == "800", 30) + sure = rt.live[sid].watch.seen_at + assert wait_for(lambda: rt.live[sid].watch.seen_at >= sure + 5, timeout=30) # several more readings + assert last().get("reason") == "changed too much" and view()["value"] == "650" + assert sent.get("visionstate/meter/state") == "650" + + # Read lower (650 < 700 < 800): confirmed in Review, it becomes the last value. + camera.text = "00700" + assert wait_for(lambda: last().get("value") == "700", timeout=30) + assert wait_for(lambda: client.get("/api/v1/review").json()["items"][0]["probs"]["value"] == "700", 30) + newest = client.get("/api/v1/review").json()["items"][0] + assert newest["probs"]["reason"] == "changed too much" + assert client.post(f"/api/v1/review/{newest['id']}", json={"action": "read_ok"}).status_code == 200 + assert wait_for(lambda: view()["value"] == "700", timeout=30) + assert wait_for(lambda: sent.get("visionstate/meter/state") == "700", timeout=10) + # After a restart it goes on from the confirmed value (which is not in the history as a value). + with TestClient(create_app(settings)) as client: + client.app.state.runtime.grabber = camera + + def first(): # the first reading after the restart (none yet: {}) + return client.get(f"/api/v1/sensors/{sid}").json()["reading"]["last"] or {} + + assert wait_for(lambda: "at" in first(), timeout=60) + assert first()["reason"] is None + assert client.get(f"/api/v1/sensors/{sid}").json()["reading"]["value"] == "700" + + @requires_reader def test_counter_sensor_flow(settings): """A counter read as text (OCR): a wrong number of digits is rejected, the export, the preview.""" @@ -610,6 +752,7 @@ def test_counter_sensor_flow(settings): "decimals": 3, "unit": "m³", "counter_reader": "ocr", + "change_limit": "off", # the counter jumps between checks seconds apart } body = { "name": "Water meter", @@ -690,11 +833,102 @@ def export_entry(): assert bad["wrong_digit_count"] is True +class NoticeHa: + """Home Assistant stand-in that records the notifications.""" + + enabled = True + + def __init__(self): + self.calls: list[tuple[str, dict]] = [] + + async def call_service(self, domain, service, data): + self.calls.append((service, data)) + + async def app_path(self): + return "/app/abc_visionstate" + + async def close(self): + pass + + +@requires_reader +def test_camera_changed_holds_readings(settings): + """A camera that comes back with another picture size holds the readings until it is checked.""" + camera = CounterCamera(89931) + reading = {"mode": "counter", "display": "counter", "digits": 7, "decimals": 3, "change_limit": "off"} + body = { + "name": "Water meter", + "kind": "reading", + "source_type": "http", + "source": "http://fake", + "roi": counter_box(7), + "reading": reading, + "interval_s": 3600, + "debounce": 1, + } + with TestClient(create_app(settings)) as client: + rt = client.app.state.runtime + rt.grabber, rt.ha = camera, NoticeHa() + sid = client.post("/api/v1/sensors", json=body).json()["id"] + + def view(): + return client.get(f"/api/v1/sensors/{sid}").json()["reading"] + + def check() -> None: + at = (view()["last"] or {}).get("at") + client.post(f"/api/v1/sensors/{sid}/classify") + assert wait_for(lambda: (view()["last"] or {}).get("at") != at, timeout=30) + + def history() -> list[dict]: + return client.get("/api/v1/history", params={"sensor": sid}).json()["items"] + + assert wait_for(lambda: view()["value"] == "89.931", timeout=60) + with rt.db.session() as s: + assert s.get(Sensor, sid).frame_size == [800, 360] + + # Another size: not read, the value stays, the frame is kept once, Home Assistant is told. + camera.wider, camera.value = 400, 89990 + rows = len(history()) + check() + check() + got = view() + assert got["value"] == "89.931" and got["last"]["reason"] == "camera changed" + assert got["camera_changed"]["from"] == [800, 360] and got["camera_changed"]["to"] == [1200, 360] + assert wait_for(lambda: len(history()) == rows + 1) + service, data = rt.ha.calls[-1] + assert service == "create" and "800×360 to 1200×360" in data["message"] + assert data["notification_id"] == READING["camera_notification_id"] + "water_meter" + quality = client.get(f"/api/v1/sensors/{sid}/reading-quality").json() + assert quality["verified"]["waiting"] == 0 # nothing to review: the card on the sensor asks + + # Back to its size: reading again by itself, the notification goes. + camera.wider = 0 + check() + assert view()["camera_changed"] is None and view()["value"] == "89.990" + assert rt.ha.calls[-1][0] == "dismiss" + + # Changed again, and the user says the region is still right: the new size is the one. + camera.wider, camera.value = 400, 89995 + check() + assert view()["camera_changed"] is not None + assert client.post(f"/api/v1/sensors/{sid}/accept-picture").status_code == 200 + assert wait_for(lambda: view()["camera_changed"] is None, timeout=30) + assert rt.ha.calls[-1][0] == "dismiss" + with rt.db.session() as s: + assert s.get(Sensor, sid).frame_size == [1200, 360] + + # A new region is drawn on the picture as it is now. + client.patch(f"/api/v1/sensors/{sid}", json={"roi": {"x": 0.1, "y": 0.1, "w": 0.5, "h": 0.5}}) + with rt.db.session() as s: + assert s.get(Sensor, sid).frame_size is None + + @requires_reader def test_wheel_counter_sensor_flow(settings): """A counter read with the wheel reader (the default): turning wheels, the preview, the export.""" camera = CounterCamera(89949.8) - reading = {"mode": "counter", "display": "counter", "digits": 7, "decimals": 3, "unit": "m³"} + # The counter jumps between checks seconds apart: no limit. + reading = {"mode": "counter", "display": "counter", "digits": 7, "decimals": 3, "unit": "m³", "change_limit": "off"} body = { "name": "Gas meter", "kind": "reading", @@ -754,7 +988,7 @@ def export(meter: str = "") -> dict: return manifest manifest = export() - assert manifest["format"] == 2 and manifest["meter"] is None and manifest["left_out"] == 0 + assert manifest["format"] == 3 and manifest["meter"] is None and manifest["left_out"] == 0 assert manifest["reader"] == readers.DEFAULT_WHEEL_READER assert manifest["settings"]["counter_reader"] == "wheels" # The checked reading first, then the accepted one nobody checked, marked as such. @@ -762,16 +996,25 @@ def export(meter: str = "") -> dict: assert (unchecked["answer"], unchecked["value"], unchecked["right_value"]) == ("unchecked", "90.100", None) assert entry["reader"] == readers.DEFAULT_WHEEL_READER and entry["app"] == VERSION assert entry["answer"] == "read correctly" and entry["seconds"] == 0 <= unchecked["seconds"] - assert entry["lamp"] is False and entry["greyscale"] is False + assert entry["lamp"] is False and entry["greyscale"] is False and entry["frame_size"] == [800, 360] assert round(entry["wheels"]["value"]) == 89950 and len(entry["wheels"]["per_wheel"]) == 7 assert export(meter=" Gas meter, white on black ")["meter"] == "Gas meter, white on black" verified = client.get(f"/api/v1/sensors/{sid}/reading-quality").json()["verified"] - assert verified["unchecked_accepted"] == 1 + assert verified["unchecked"] == 1 + + # A rejected reading nobody answered goes in too, newest first among the unchecked ones. + with client.app.state.runtime.db.session() as s: + frame = s.scalars(select(Prediction.frame).where(Prediction.sensor_id == sid)).first() + details = {"text": "0099999", "value": "99.999", "reason": "went down"} + s.add(Prediction(sensor_id=sid, state_key="reading", confidence=0.5, probs=details, frame=frame)) + rejected = export()["readings"][1] + assert (rejected["answer"], rejected["rejected"], rejected["value"]) == ("unchecked", "went down", "99.999") + assert client.get(f"/api/v1/sensors/{sid}/reading-quality").json()["verified"]["unchecked"] == 2 # The ZIP stays small enough for GitHub: what does not fit is left out (the oldest) and counted. with patch.dict(READING, {"export_max_mb": 0.0001}): # room for one image only small = export() - assert [r["answer"] for r in small["readings"]] == ["read correctly"] and small["left_out"] == 1 + assert [r["answer"] for r in small["readings"]] == ["read correctly"] and small["left_out"] == 2 # A sensor export keeps the choice; one made before the wheel reader keeps the text reader. exported = client.get(f"/api/v1/sensors/{sid}/export").content diff --git a/visionstate/backend/tests/test_storage.py b/visionstate/backend/tests/test_storage.py index 0cd992a..e286290 100644 --- a/visionstate/backend/tests/test_storage.py +++ b/visionstate/backend/tests/test_storage.py @@ -120,3 +120,220 @@ def test_age_limit_and_usage(settings): assert client.put("/api/v1/storage", json={"history_days": 0, "history_max_gb": 2}).status_code == 422 assert client.put("/api/v1/storage", json={"history_days": 7, "history_max_gb": -1}).status_code == 422 + + +# --- training images: kept in the app's storage, moved there from the media folder ------------ + + +@pytest.fixture +def runtime(settings): + from visionstate.db import Database + from visionstate.engine import Runtime + + settings.ensure_dirs() + db = Database(settings.db_path) + db.init() + yield Runtime(settings, db) + db.engine.dispose() + + +def state_sensor(rt) -> int: + from visionstate.db import Sensor, State + + with rt.db.session() as s: + row = Sensor(slug="door", name="Door", source_type="http", source="x", interval_s=5, threshold=0.6, + debounce=1, states=[State(key="open", name="Open", color="#fff"), + State(key="closed", name="Closed", color="#000")]) # fmt: skip + s.add(row) + s.flush() + return row.id + + +def old_sample(rt, sensor_id: int, seed: int) -> tuple[int, str]: + """A training image where versions before the move kept it (the media folder).""" + from visionstate.db import Sample + + filename = rt.storage.new_filename() + path = rt.storage.old_sample_path(sensor_id, filename) + path.parent.mkdir(parents=True, exist_ok=True) + noise(seed).save(path, format="JPEG") + with rt.db.session() as s: + row = Sample(sensor_id=sensor_id, filename=filename, origin="upload") + s.add(row) + s.flush() + return row.id, filename + + +def test_training_images_are_saved_in_the_apps_storage(runtime): + sid = state_sensor(runtime) + runtime.add_sample(sid, noise(1), "upload", None) + assert len(list((runtime.settings.data_dir / "samples" / str(sid)).iterdir())) == 1 + assert not runtime.settings.old_samples_dir.exists() # a new installation never makes the old folder + + +def test_training_images_move_from_the_media_folder(runtime): + from visionstate.settings import SAMPLES_MOVE + + rt = runtime + sid = state_sensor(rt) + (_, first), (second_id, second) = old_sample(rt, sid, 1), old_sample(rt, sid, 2) + stranger = rt.settings.old_samples_dir / str(sid) / "not-ours.jpg" # no row lists it + noise(3).save(stranger, format="JPEG") + old_thumb = rt.settings.thumbs_dir / "sample" / "1.jpg" + old_thumb.parent.mkdir(parents=True) + noise(4).save(old_thumb, format="JPEG") + # Before the move they are read where they are. + assert rt.storage.sample_path(sid, first) == rt.storage.old_sample_path(sid, first) + + rt.move_samples() + for name in (first, second): + new = rt.storage.new_sample_path(sid, name) + assert rt.storage.sample_path(sid, name) == new + assert new.read_bytes() == rt.storage.old_sample_path(sid, name).read_bytes() # the old copy stays + assert not list(rt.storage.new_sample_path(sid, "x").parent.glob("*.part")) + assert not (rt.settings.data_dir / "samples" / str(sid) / "not-ours.jpg").exists() + moved_at = rt.db.get_setting("samples_moved_at") + assert moved_at + + # A deleted image goes from both places; a later run copies nothing back. + rt.storage.delete_sample(sid, second_id, second) + assert not rt.storage.old_sample_path(sid, second).exists() + rt.move_samples() + assert not rt.storage.new_sample_path(sid, second).exists() + assert rt.db.get_setting("samples_moved_at") == moved_at + + # After the days the old copies are kept, they go; what no sensor knows stays. + long_ago = utcnow() - timedelta(days=SAMPLES_MOVE["keep_old_days"] + 1) + rt.db.set_setting("samples_moved_at", long_ago.isoformat()) + rt.move_samples() + assert not rt.storage.old_sample_path(sid, first).exists() + assert rt.storage.sample_path(sid, first).exists() and stranger.exists() + assert not old_thumb.exists() + stranger.unlink() + rt.move_samples() + assert not rt.settings.old_samples_dir.exists() # empty folders are removed + + +def test_a_full_disk_leaves_the_images_where_they_are(runtime, monkeypatch): + import shutil + + rt = runtime + sid = state_sensor(rt) + _, name = old_sample(rt, sid, 1) + + def full(*_args, **_kwargs): + raise OSError(28, "No space left on device") + + monkeypatch.setattr(shutil, "copy2", full) + rt.move_samples() + assert rt.storage.sample_path(sid, name) == rt.storage.old_sample_path(sid, name) + assert rt.db.get_setting("samples_moved_at") is None # tried again later + monkeypatch.undo() + rt.move_samples() + assert rt.storage.sample_path(sid, name) == rt.storage.new_sample_path(sid, name) + + +def test_training_leaves_out_images_that_are_gone(runtime): + """A state sensor trains on the images it still has (a backup restored without them).""" + from visionstate.db import Sample + + class FakeEmbedder: + class spec: # noqa: N801 - mimics BackboneSpec + id = "fake" + + def embed(self, images): + return np.ones((len(images), 4), dtype=np.float32) + + rt = runtime + rt.embedder = FakeEmbedder() + sid = state_sensor(rt) + ids = [rt.add_sample(sid, noise(i), "upload", None) for i in range(3)] + with rt.db.session() as s: + samples = [s.get(Sample, i) for i in ids] + rt.storage.sample_path(sid, samples[1].filename).unlink() + rt.storage.sample_path(sid, samples[2].filename).write_bytes(b"broken") + kept, vectors = rt.vectors_for(rt.load_sensor(sid), samples) + assert [x.id for x in kept] == [ids[0]] and vectors.shape == (1, 4) + + +def test_a_file_is_removed_only_when_its_row_is(runtime): + from visionstate.db import Sample + + rt = runtime + sid = state_sensor(rt) + sample_id = rt.add_sample(sid, noise(1), "upload", None) + with rt.db.session() as s: + filename = s.get(Sample, sample_id).filename + path = rt.storage.sample_path(sid, filename) + with pytest.raises(RuntimeError), rt.db.session() as s: + sample = s.get(Sample, sample_id) + rt.db.after_commit(s, lambda: rt.storage.delete_sample(sid, sample_id, filename)) + s.delete(sample) + raise RuntimeError("the commit never happens") + assert path.exists() + with rt.db.session() as s: + rt.db.after_commit(s, lambda: rt.storage.delete_sample(sid, sample_id, filename)) + s.delete(s.get(Sample, sample_id)) + assert not path.exists() + + +def test_a_media_folder_inside_the_apps_storage_keeps_its_images(tmp_path): + """VISIONSTATE_MEDIA set to the data folder: old and new place are the same files.""" + from visionstate.db import Database + from visionstate.engine import Runtime + from visionstate.settings import SAMPLES_MOVE, Settings + + same = Settings(data_dir=tmp_path, media_dir=tmp_path, frontend_dir=tmp_path / "x", bundled_models_dir=MODEL_DIR) + same.ensure_dirs() + db = Database(same.db_path) + db.init() + rt = Runtime(same, db) + sid = state_sensor(rt) + rt.add_sample(sid, noise(1), "upload", None) + rt.db.set_setting("samples_moved_at", (utcnow() - timedelta(days=SAMPLES_MOVE["keep_old_days"] + 1)).isoformat()) + rt.move_samples() + assert len(list((tmp_path / "samples" / str(sid)).iterdir())) == 1 + db.engine.dispose() + + +def test_an_image_deleted_while_it_is_moved_is_not_brought_back(runtime, monkeypatch): + import shutil + + rt = runtime + sid = state_sensor(rt) + (_, gone), (_, raced), (_, kept) = old_sample(rt, sid, 1), old_sample(rt, sid, 2), old_sample(rt, sid, 3) + copy = shutil.copy2 + + def deleted_meanwhile(src, dst): + name = src.name + if name == gone: # removed before the copy starts + src.unlink() + raise FileNotFoundError(src) + copy(src, dst) + if name == raced: # removed (Storage.delete_sample) while it was copied + src.unlink() + + monkeypatch.setattr(shutil, "copy2", deleted_meanwhile) + rt.move_samples() + assert not rt.storage.new_sample_path(sid, gone).exists() + assert not rt.storage.new_sample_path(sid, raced).exists() + assert rt.storage.new_sample_path(sid, kept).exists() # one gone does not stop the others + assert rt.db.get_setting("samples_moved_at") + + +def test_a_file_that_cannot_be_removed_does_not_fail_the_request(runtime, caplog): + from visionstate.db import Sample + + rt = runtime + sid = state_sensor(rt) + sample_id = rt.add_sample(sid, noise(1), "upload", None) + + def locked(): + raise PermissionError("in use") + + with rt.db.session() as s: + rt.db.after_commit(s, locked) + s.delete(s.get(Sample, sample_id)) + with rt.db.session() as s: + assert s.get(Sample, sample_id) is None + assert "Could not remove a file" in caplog.text diff --git a/visionstate/backend/tests/test_teach.py b/visionstate/backend/tests/test_teach.py index c5552a8..9308bd9 100644 --- a/visionstate/backend/tests/test_teach.py +++ b/visionstate/backend/tests/test_teach.py @@ -275,6 +275,46 @@ def test_an_index_built_before_an_own_label_existed_is_rebuilt(settings): # noq db.engine.dispose() +def test_taught_boxes_without_a_readable_image_are_left_out(settings): # noqa: F811 + """Issue #45: one taught box whose image is gone (a backup restored without the images) or + broken stopped every check of the sensor. The index is built from the boxes that are left.""" + import asyncio + + from PIL import Image + + from visionstate.db import Database, Embedding, Sample, Sensor + from visionstate.engine import Runtime + + settings.ensure_dirs() + db = Database(settings.db_path) + db.init() + rt = Runtime(settings, db) + rt.embedder = FakeEmbedder() + with db.session() as s: + row = Sensor(slug="drive", name="Drive", kind=KIND_OBJECTS, source_type="http", source="x", interval_s=5, + threshold=0.6, debounce=1, objects={"classes": ["dog"]}) # fmt: skip + s.add(row) + s.flush() + sid = row.id + box = [0.1, 0.1, 0.5, 0.5] + kept, gone, broken = ( + rt.add_object_example(sid, Image.new("RGB", (64, 64)), box, "dog", "dog", 0.9, "live") for _ in range(3) + ) + with db.session() as s: + rt.storage.sample_path(sid, s.get(Sample, gone).filename).unlink() + rt.storage.sample_path(sid, s.get(Sample, broken).filename).write_bytes(b"not a jpeg") + index = asyncio.run(rt.taught_index(rt.load_sensor(sid))) + assert index is not None and index.ids == [kept] + # Every box gone: nothing to compare with, and still no error. + with db.session() as s: + rt.storage.sample_path(sid, s.get(Sample, kept).filename).unlink() + rt.taught_changed(sid) + with db.session() as s: # its cached embedding would keep it in the index + s.execute(Embedding.__table__.delete()) + assert asyncio.run(rt.taught_index(rt.load_sensor(sid))) is None + db.engine.dispose() + + # --- the API, with the real detector and backbone on CC0 photos ---------------------------- diff --git a/visionstate/backend/tests/test_triggers.py b/visionstate/backend/tests/test_triggers.py index 312c7f2..c8c184d 100644 --- a/visionstate/backend/tests/test_triggers.py +++ b/visionstate/backend/tests/test_triggers.py @@ -495,6 +495,7 @@ def test_change_detection_with_a_light_reads_a_new_frame_in_the_light(tmp_path, "interval_s": 3600, "debounce": 1, "threshold": 0.5, # "88888" below is read at ~0.69 + "reading": {"mode": "counter", "change_limit": "off"}, # 500 to 88888 in seconds } sid = client.post("/api/v1/sensors", json={**body, "triggers": triggers}).json()["id"] diff --git a/visionstate/backend/visionstate/api/common.py b/visionstate/backend/visionstate/api/common.py index 98efc59..01ded36 100644 --- a/visionstate/backend/visionstate/api/common.py +++ b/visionstate/backend/visionstate/api/common.py @@ -11,11 +11,12 @@ from sqlalchemy import func, select from sqlalchemy.orm import Session -from .. import detectors, imaging, teach +from .. import change_limit, detectors, imaging, teach from ..db import Face, ModelInfo, Sample, SampleLabel, Sensor from ..engine import LiveState, ObjectTrack, Runtime, SensorConfig, with_people from ..mqtt import main_entities from ..settings import ( + COUNTER_LIMIT, COUNTER_READERS, FACE_DEFAULTS, FACE_LIMITS, @@ -28,6 +29,7 @@ OBJECT_DEFAULTS, OBJECT_LIMITS, OBJECT_MAX_CLASSES, + READING_CHANGE_LIMITS, READING_DEFAULTS, READING_DEVICE_CLASSES, READING_DISPLAYS, @@ -297,6 +299,8 @@ class ReadingIn(BaseModel): digits: int = Field(default=READING_DEFAULTS["digits"], ge=_dlo["digits"], le=_dhi["digits"]) counter_reader: str = READING_DEFAULTS["counter_reader"] max_step: float = Field(default=READING_DEFAULTS["max_step"], ge=_dlo["max_step"], le=_dhi["max_step"]) + change_limit: str = READING_DEFAULTS["change_limit"] + max_rate: float = Field(default=READING_DEFAULTS["max_rate"], ge=_dlo["max_rate"], le=_dhi["max_rate"]) spot_rate: float = Field(default=READING_DEFAULTS["spot_rate"], ge=_dlo["spot_rate"], le=_dhi["spot_rate"]) rate_window_min: float = Field( default=READING_DEFAULTS["rate_window_min"], ge=_dlo["rate_window_min"], le=_dhi["rate_window_min"] @@ -316,6 +320,13 @@ def _display(cls, value: str) -> str: raise ValueError(f"Unknown display {value!r}") return value + @field_validator("change_limit") + @classmethod + def _change_limit(cls, value: str) -> str: + if value not in READING_CHANGE_LIMITS: + raise ValueError(f"Unknown change limit {value!r}") + return value + @field_validator("counter_reader") @classmethod def _counter_reader(cls, value: str) -> str: @@ -468,11 +479,30 @@ def reading_view(sensor: Sensor, live: LiveState | None) -> dict | None: """A reading sensor's part of its view: its settings, the published value and the last read.""" if sensor.kind != KIND_READING: return None + settings = merge_reading(sensor.reading) return { - **merge_reading(sensor.reading), + **settings, "value": live.debouncer.published if live else None, "last": live.reading if live else None, "has_image": bool(live and live.reading_image), + "camera_changed": live.camera_changed if live else None, + "limit": limit_view(settings, live.watch) if live and settings["mode"] == "counter" else None, + } + + +def limit_view(settings: dict, watch: change_limit.Watch) -> dict: + """What a counter's limit is now: learned (per hour and day, from how many days; ``needed`` + days before it is; ``rate``: what "auto" allows per hour once learned) and how much the next + reading may go up (None: no limit).""" + learned = watch.learned + return { + "days": watch.days, + "needed": COUNTER_LIMIT["min_days"], + "per_hour": learned.per_hour if learned else None, + "per_day": learned.per_day if learned else None, + "rate": change_limit.learned_rate(learned), + "allowed": change_limit.limit(settings, learned, watch.budget_s), + "held": watch.held, } diff --git a/visionstate/backend/visionstate/api/faces.py b/visionstate/backend/visionstate/api/faces.py index 93d8979..58ccfb7 100644 --- a/visionstate/backend/visionstate/api/faces.py +++ b/visionstate/backend/visionstate/api/faces.py @@ -10,6 +10,7 @@ import itertools import tempfile from datetime import datetime +from functools import partial from pathlib import Path import numpy as np @@ -451,7 +452,7 @@ def _forget_people(rt: Runtime, key: str | None) -> None: persons = [person_by_key(s, key)] if key is not None else list(s.scalars(select(Person))) for person in persons: for face in s.scalars(select(Face).where(Face.person_id == person.id)): - rt.storage.delete_face(face.filename) + rt.db.after_commit(s, partial(rt.storage.delete_face, face.filename)) s.delete(person) # their faces and embeddings go with them (foreign keys) @@ -488,7 +489,7 @@ def forget_face(face_id: int, request: Request) -> Response: face = s.get(Face, face_id) if face is None: raise HTTPException(404, "Face not found") - rt.storage.delete_face(face.filename) + rt.db.after_commit(s, partial(rt.storage.delete_face, face.filename)) s.delete(face) # its cached embeddings go with it (foreign key) rt.faces_changed() return Response(status_code=204) diff --git a/visionstate/backend/visionstate/api/imports.py b/visionstate/backend/visionstate/api/imports.py index b81e20c..d2028f2 100644 --- a/visionstate/backend/visionstate/api/imports.py +++ b/visionstate/backend/visionstate/api/imports.py @@ -12,7 +12,7 @@ from sqlalchemy import select from .. import bundle, detectors, faces, imaging -from ..db import Person, keep_text_reader +from ..db import Person, keep_change_step, keep_text_reader from ..engine import Runtime from ..redact import redact from ..settings import ( @@ -70,6 +70,8 @@ def _import_sync(rt: Runtime, path: Path) -> tuple[int, int, bool]: manifest = bundle.read_manifest(path) data = dict(manifest["sensor"]) data["reading"] = keep_text_reader(data.get("reading")) # exported before the wheel reader + # exported before a counter had a change limit of its own + data["reading"] = keep_change_step(data["reading"]) # Validate exactly like a sensor created in the UI (kind, limits, source type, states, objects, triggers). try: spec = SensorIn.model_validate( diff --git a/visionstate/backend/visionstate/api/review.py b/visionstate/backend/visionstate/api/review.py index a21feb5..a73fc85 100644 --- a/visionstate/backend/visionstate/api/review.py +++ b/visionstate/backend/visionstate/api/review.py @@ -4,14 +4,17 @@ import asyncio import logging +from datetime import UTC, datetime +from functools import partial from typing import cast from fastapi import APIRouter, HTTPException, Request from pydantic import BaseModel, Field from sqlalchemy import CursorResult, delete, func, select, update +from sqlalchemy.orm import Session from .. import imaging, readers -from ..db import Face, Prediction, Sample, SampleLabel, Sensor +from ..db import Face, Prediction, ReadingWatch, Sample, SampleLabel, Sensor from ..engine import Runtime, with_people from ..settings import FACE_UNKNOWN_KEY, KIND_OBJECTS, KIND_READING, merge_objects, merge_reading from .common import ( @@ -101,6 +104,31 @@ async def review_dismiss_all(sensor_id: int, request: Request) -> dict: return {"dismissed": dismissed} +# Rejected readings whose value, confirmed in Review, becomes the last value (see _newest_rejected). +ANCHOR_REASONS = ("unsure", "went down", "changed too much") + + +def _newest_rejected(s: Session, row: Prediction) -> bool: + """Whether a reading is a rejected one read after the last accepted value (so the value it + confirms can take that value's place: engine.reading.anchor_reading).""" + if row.published_key is not None: + return False + watch = s.get(ReadingWatch, row.sensor_id) + read_at = (row.created_at if row.created_at.tzinfo else row.created_at.replace(tzinfo=UTC)).timestamp() + if watch is not None and (watch.accepted_at or 0.0) >= read_at: + return False + newer = s.scalar( + select(func.count()) + .select_from(Prediction) + .where( + Prediction.sensor_id == row.sensor_id, + Prediction.published_key.is_not(None), + Prediction.created_at > row.created_at, + ) + ) + return not newer + + @router.post("/review/{prediction_id}") async def review_answer(prediction_id: int, body: ReviewIn, request: Request) -> dict: rt = runtime(request) @@ -112,6 +140,7 @@ async def review_answer(prediction_id: int, body: ReviewIn, request: Request) -> kind = sensor.kind if kind == KIND_OBJECTS: return await _answer_object(rt, request, prediction_id, body) + anchor: tuple[int, str, datetime] | None = None with rt.db.session() as s: row = s.get(Prediction, prediction_id) if row is None: @@ -129,10 +158,18 @@ async def review_answer(prediction_id: int, body: ReviewIn, request: Request) -> if number is None: raise HTTPException(400, f"Not a number: {body.value!r}") row.correct_value = readers.format_value(number, settings) + # A reading with the wrong number of digits is no value, even when read as it is shown. + probs = row.probs or {} + kept = probs.get("reason") in ANCHOR_REASONS + right = row.correct_value if body.action == "misread" else probs.get("value") if kept else None + if right is not None and _newest_rejected(s, row): + anchor = (row.sensor_id, right, row.created_at) row.reviewed = True elif body.action not in ("confirm", "label", "skip"): raise HTTPException(400, "Answer confirm, label or skip") if sensor.kind == KIND_READING: + if anchor is not None: + await rt.anchor_reading(*anchor) await rt.publish_review_count() return {"ok": True} with rt.db.session() as s: @@ -148,7 +185,7 @@ async def review_answer(prediction_id: int, body: ReviewIn, request: Request) -> sample = s.get(Sample, row.sample_id) if row.sample_id is not None else None if sample is not None and sample.sensor_id == sensor_id: if state_id is None: - rt.storage.delete_sample(sensor_id, sample.id, sample.filename) + rt.db.after_commit(s, partial(rt.storage.delete_sample, sensor_id, sample.id, sample.filename)) s.delete(sample) row.sample_id = None else: @@ -196,10 +233,10 @@ async def _answer_object(rt: Runtime, request: Request, prediction_id: int, body row.reviewed = True row.sample_id = None if taught is not None and is_face and (face := s.get(Face, taught)) is not None: - rt.storage.delete_face(face.filename) # the face the first answer taught + rt.db.after_commit(s, partial(rt.storage.delete_face, face.filename)) # the face the first answer taught s.delete(face) elif taught is not None and not is_face and (sample := s.get(Sample, taught)) is not None: - rt.storage.delete_sample(sensor_id, sample.id, sample.filename) + rt.db.after_commit(s, partial(rt.storage.delete_sample, sensor_id, sample.id, sample.filename)) s.delete(sample) if taught is not None: if is_face: diff --git a/visionstate/backend/visionstate/api/samples.py b/visionstate/backend/visionstate/api/samples.py index bb44bff..7e5bc90 100644 --- a/visionstate/backend/visionstate/api/samples.py +++ b/visionstate/backend/visionstate/api/samples.py @@ -4,6 +4,7 @@ import asyncio import tempfile +from functools import partial from pathlib import Path from typing import IO @@ -222,7 +223,7 @@ def delete_samples(sensor_id: int, body: IdsIn, request: Request) -> dict: get_sensor(s, sensor_id) samples = s.scalars(select(Sample).where(Sample.sensor_id == sensor_id, Sample.id.in_(body.sample_ids))).all() for sample in samples: - rt.storage.delete_sample(sensor_id, sample.id, sample.filename) + rt.db.after_commit(s, partial(rt.storage.delete_sample, sensor_id, sample.id, sample.filename)) s.delete(sample) _retrain(request, sensor_id) return {"deleted": len(samples)} diff --git a/visionstate/backend/visionstate/api/sensors.py b/visionstate/backend/visionstate/api/sensors.py index 6bed45b..1dee9e6 100644 --- a/visionstate/backend/visionstate/api/sensors.py +++ b/visionstate/backend/visionstate/api/sensors.py @@ -210,7 +210,7 @@ async def update_sensor(sensor_id: int, body: SensorPatch, request: Request) -> raise HTTPException(400, "Only object sensors have objects") if sensor.kind != KIND_READING and body.reading is not None: raise HTTPException(400, "Only reading sensors have reading settings") - old_roi, old_keys = sensor.roi, [st.key for st in sensor.states] + old_roi, old_keys, old_source = sensor.roi, [st.key for st in sensor.states], sensor.source for field in ("name", "source_type", "source", "interval_s", "threshold", "debounce", "enabled", "publish"): value = getattr(body, field) if value is not None: @@ -233,6 +233,8 @@ async def update_sensor(sensor_id: int, body: SensorPatch, request: Request) -> check_faces(faces.merge_limits(rt.face_choice, rt.global_faces, merged["faces"])) if body.reading is not None: sensor.reading = body.reading.model_dump() + if sensor.roi != old_roi or sensor.source != old_source: + sensor.frame_size = None # the next frame's size is the one the region is drawn on s.flush() changed = sensor.roi != old_roi or [st.key for st in sensor.states] != old_keys retrain = changed and sensor.kind == KIND_STATES @@ -246,6 +248,8 @@ async def delete_sensor(sensor_id: int, request: Request) -> Response: cfg = await asyncio.to_thread(rt.load_sensor, sensor_id) if cfg is None: raise HTTPException(404, "Sensor not found") + if rt.live_state(sensor_id).camera_changed is not None: + await rt.camera_notice(cfg, None) with rt.db.session() as s: s.delete(get_sensor(s, sensor_id)) await rt.sensor_deleted(sensor_id, cfg.descriptor) @@ -303,6 +307,17 @@ async def classify_now(sensor_id: int, request: Request) -> dict: return {"ok": True} +@router.post("/{sensor_id}/accept-picture") +async def accept_picture(sensor_id: int, request: Request) -> dict: + """A reading sensor's camera changed its picture size and the region is still right: read + again, with the next frame's size as the one the region is drawn on.""" + rt = runtime(request) + with rt.db.session() as s: + get_sensor(s, sensor_id, KIND_READING).frame_size = None + rt.wake(sensor_id, force=True, paused_too=True) + return {"ok": True} + + @router.post("/{sensor_id}/retrain") async def retrain(sensor_id: int, request: Request) -> dict: rt = runtime(request) @@ -479,8 +494,8 @@ def count(*where) -> int: "misread_accepted": count(~rejected, Prediction.read_ok.is_(False)), "right_accepted": count(~rejected, Prediction.read_ok.is_(True)), "waiting": count(Prediction.reviewed.is_(False)), - # accepted, nobody checked, the frame kept: they go into the export too - "unchecked_accepted": count(~rejected, Prediction.read_ok.is_(None), Prediction.frame.is_not(None)), + # nobody checked, the frame kept: they go into the export too + "unchecked": count(Prediction.read_ok.is_(None), Prediction.frame.is_not(None)), } items = s.scalars( select(Prediction) diff --git a/visionstate/backend/visionstate/api/system.py b/visionstate/backend/visionstate/api/system.py index c297e69..f43adbb 100644 --- a/visionstate/backend/visionstate/api/system.py +++ b/visionstate/backend/visionstate/api/system.py @@ -17,6 +17,7 @@ from ..db import Person, Prediction, Sensor from ..redact import redact from ..settings import ( + COUNTER_LIMIT, COUNTER_READERS, FACE_DEFAULTS, FACE_LIMITS, @@ -126,6 +127,7 @@ def ui_config() -> dict: "reading_sensor_defaults": READING_SENSOR_DEFAULTS, "reading_defaults": READING_DEFAULTS, "reading_limits": READING_LIMITS, + "counter_standard_steps_per_min": COUNTER_LIMIT["standard_steps_per_min"], "reading_modes": READING_MODES, "reading_displays": READING_DISPLAYS, "counter_readers": COUNTER_READERS, diff --git a/visionstate/backend/visionstate/api/teach.py b/visionstate/backend/visionstate/api/teach.py index 57c41b8..a787798 100644 --- a/visionstate/backend/visionstate/api/teach.py +++ b/visionstate/backend/visionstate/api/teach.py @@ -4,6 +4,7 @@ import asyncio from datetime import UTC, timedelta +from functools import partial from fastapi import APIRouter, HTTPException, Request, Response from PIL import Image @@ -243,7 +244,7 @@ def delete_examples(rt: Runtime, s: Session, sensor_id: int, where: list[ColumnE select(Sample).where(Sample.sensor_id == sensor_id, Sample.object_label.is_not(None), *where) ).all() for row in rows: - rt.storage.delete_sample(sensor_id, row.id, row.filename) + rt.db.after_commit(s, partial(rt.storage.delete_sample, sensor_id, row.id, row.filename)) s.delete(row) return len(rows) diff --git a/visionstate/backend/visionstate/bundle.py b/visionstate/backend/visionstate/bundle.py index bc12625..79ff775 100644 --- a/visionstate/backend/visionstate/bundle.py +++ b/visionstate/backend/visionstate/bundle.py @@ -106,7 +106,8 @@ def export_sensor(db: Database, storage: Storage, sensor_id: int, target: Path, return f"visionstate-{sensor.slug}.zip" -READINGS_FORMAT = 2 # readings.json; 2 added per-reading reader details, "meter", unchecked readings +READINGS_FORMAT = 3 # readings.json; 2 added per-reading reader details, "meter", unchecked readings; +# 3: unchecked rejected readings too, the camera picture's size READINGS_README = """Readings from a VisionState reading sensor ========================================== @@ -115,16 +116,21 @@ def export_sensor(db: Database, storage: Storage, sensor_id: int, target: Path, around it) — not the whole camera picture. readings.json lists for each image what the reader read, the value it made of it, why it was rejected (if it was), and the answer given in VisionState: "read correctly", "misread" (with the right value when it was entered), or -"unchecked" (an accepted reading nobody checked: the reader's value, not confirmed by a person). -The checked readings come first, newest first, then the unchecked ones, newest first. +"unchecked" (nobody checked it: the reader's value, not confirmed by a person; "rejected" says +whether it was accepted). The checked readings come first, newest first, then the unchecked ones, +newest first. Per reading it also says which reader read it ("reader"; null for readings from before VisionState 0.8.0, beta 0.7.1b5), the app version ("app"), whether the whole picture was greyscale (an -infrared or night camera) and whether VisionState had switched the sensor's light on ("lamp"). +infrared or night camera), whether VisionState had switched the sensor's light on ("lamp") and +the size of the whole camera picture in pixels ("frame_size"; a camera that changes its size after +a restart moves what the region shows). For the wheel reader, "wheels" says where it saw each wheel (positions 0.0-9.9, as seen in the image), how sure it was, its most likely positions, the common shift of all wheels and the value with the last wheel's fraction. "seconds" is the time since the oldest reading of the -export (no time of day). +export (no time of day): sorted by it, the readings are in the order they were made. VisionState +keeps only the readings that changed the value, the rejected ones and spot checks, so a reading +that only repeated the last value is not in it. The file stays under 24 MB (GitHub takes files up to 25 MB): the readings that did not fit are counted in "left_out" — export again later for newer ones. @@ -155,8 +161,8 @@ def export_sensor(db: Database, storage: Storage, sensor_id: int, target: Path, def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path, reader: str, meter: str = "") -> str: """Writes a reading sensor's readings to share to ``target`` (one ZIP). - The readings checked by hand come first, then the accepted ones nobody checked (marked - "unchecked"), each newest first; every image is the region crop of the stored frame. The ZIP + The readings checked by hand come first, then the ones nobody checked (marked "unchecked", + accepted or rejected), each newest first; every image is the region crop of the stored frame. The ZIP stays under ``READING["export_max_mb"]``; the readings that did not fit are counted. ``meter``: what the user says the meter is. Returns the suggested download filename; raises LookupError when there is nothing to export. @@ -174,11 +180,7 @@ def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path newest = Prediction.created_at.desc() rows = [ *s.scalars(select(Prediction).where(*mine, Prediction.read_ok.is_not(None)).order_by(newest)), - *s.scalars( - select(Prediction) - .where(*mine, Prediction.read_ok.is_(None), Prediction.published_key.is_not(None)) - .order_by(newest) - ), + *s.scalars(select(Prediction).where(*mine, Prediction.read_ok.is_(None)).order_by(newest)), ] current = sensor.roi slug = sensor.slug @@ -221,6 +223,7 @@ def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path "reader": details.get("reader"), "app": details.get("app"), "greyscale": imaging.is_night(frame), + "frame_size": list(frame.size), "lamp": details.get("lamp"), **({"wheels": details["wheels"]} if "wheels" in details else {}), } @@ -235,7 +238,18 @@ def export_readings(db: Database, storage: Storage, sensor_id: int, target: Path "meter": meter.strip()[: READING["export_meter_max_chars"]] or None, # How the sensor reads; nothing about where it is (no camera, name or region position). "settings": { - k: settings[k] for k in ("mode", "display", "digits", "counter_reader", "decimals", "unit", "max_step") + k: settings[k] + for k in ( + "mode", + "display", + "digits", + "counter_reader", + "decimals", + "unit", + "max_step", + "change_limit", + "max_rate", + ) }, "left_out": left_out, "readings": items, diff --git a/visionstate/backend/visionstate/change_limit.py b/visionstate/backend/visionstate/change_limit.py new file mode 100644 index 0000000..4a1afeb --- /dev/null +++ b/visionstate/backend/visionstate/change_limit.py @@ -0,0 +1,153 @@ +"""How much a counter may go up since its last accepted reading (settings.COUNTER_LIMIT). + +The limit grows with the time the counter was not seen: the app down, the camera away, or the +readings unsure. Time in which the counter was read with confidence but went over the limit does +not count, so a wheel misread over and over never waits its way through. Such readings are never +accepted by themselves, however many agree: they wait for an answer in Review. + +"auto" learns the limit from the counter's own accepted readings: the largest rise over a few +minutes per day and the rise per day (db.ReadingStat ``peak`` and ``total``), only over stretches +in which it was read all the time; until then a rate (set, or the standard one) applies. "step" +is the older limit: the same change for every reading, however long since the last one. Pure logic, no I/O: +the engine (engine/reading.py) keeps a ``Watch`` per sensor and stores it (db.ReadingWatch). +""" + +from __future__ import annotations + +import math +from collections import deque +from dataclasses import dataclass, field + +from .settings import COUNTER_LIMIT + +# Readings that were read with confidence but do not fit the last value. +IMPLAUSIBLE = ("went down", "changed too much") + + +@dataclass +class Learned: + """What a counter does on a busy day (the second busiest of the last days).""" + + per_hour: float # the largest rise over COUNTER_LIMIT["window_s"], as a rate per hour + per_day: float + days: int # days with readings it was learned from + + +@dataclass +class Watch: + """A counter's limit between readings. The first four fields are stored (db.ReadingWatch).""" + + seen_at: float | None = None # the last reading, of any kind + accepted_at: float | None = None # the last accepted reading + budget_s: float = 0.0 # time since then that counts towards the limit + held: bool = False # read with confidence over the limit since then: its time no longer counts + # Learning: accepted readings over the last window (time, value), and the newest one, which + # counts once the next one says it was no misread (see observe). + window: deque[tuple[float, float]] = field(default_factory=deque) + sample: tuple[float, float] | None = None + learned: Learned | None = None + days: int = 0 # days with readings so far (learned from min_days on) + learned_at: float = 0.0 + learned_day: str = "" + + +def gap(interval_s: float) -> float: + """A pause between two readings longer than this is time the counter was not seen.""" + return max(COUNTER_LIMIT["gap_checks"] * interval_s, COUNTER_LIMIT["gap_min_s"]) + + +def margin(settings: dict) -> float: + return COUNTER_LIMIT["margin_steps"] * 10 ** -int(settings["decimals"]) + + +def standard_rate(settings: dict) -> float: + """The standard limit per hour: COUNTER_LIMIT["standard_steps_per_min"] steps of the last digit a minute.""" + return COUNTER_LIMIT["standard_steps_per_min"] * 10 ** -int(settings["decimals"]) * 60 + + +def learned_rate(learned: Learned | None) -> float | None: + """The rate per hour "auto" allows once it learned (with its room to spare), or None.""" + return COUNTER_LIMIT["safety"] * learned.per_hour if learned is not None else None + + +def limit(settings: dict, learned: Learned | None, seconds: float) -> float | None: + """The largest rise allowed after ``seconds`` of counting time, or None (no limit).""" + hours = max(0.0, seconds) / 3600 + mode = settings["change_limit"] + if mode == "step": # the same for every reading, as counters had before the rate + step = float(settings["max_step"]) + return step if step > 0 else None + if mode == "fixed": + rate = float(settings["max_rate"]) + return margin(settings) + rate * hours if rate > 0 else None + if mode == "auto" and learned is not None: + days = max(1, math.ceil(seconds / 86_400)) + rise = min(learned.per_hour * hours, learned.per_day * days) + return margin(settings) + COUNTER_LIMIT["safety"] * rise + if mode == "auto": # still learning: the rate set, or the standard one + rate = float(settings["max_rate"]) or standard_rate(settings) + return margin(settings) + rate * hours + return None + + +def learn(days: list[tuple[float, float]]) -> Learned | None: + """The limit learned from (largest rise over the window, total rise) of each day with readings.""" + if len(days) < COUNTER_LIMIT["min_days"]: + return None + peaks = sorted(peak for peak, _ in days) + totals = sorted(total for _, total in days) + per_hour = peaks[-2] / (COUNTER_LIMIT["window_s"] / 3600) + if per_hour <= 0: + return None # nothing used yet: nothing to learn from + return Learned(per_hour=per_hour, per_day=max(totals[-2], peaks[-2]), days=len(days)) + + +def advance(watch: Watch, now: float, interval_s: float) -> None: + """Before a reading is checked: the time since the last one counts towards the limit. + + Once the counter was read with confidence over the limit, only the part of a pause longer + than a normal one counts (the app was down).""" + if watch.seen_at is not None: + passed = max(0.0, now - watch.seen_at) + watch.budget_s += max(0.0, passed - gap(interval_s)) if watch.held else passed + watch.seen_at = now + + +def accept(watch: Watch, now: float) -> None: + """A reading confirmed the published value (or made it): the limit starts again.""" + watch.accepted_at, watch.budget_s, watch.held = now, 0.0, False + + +def observe(watch: Watch, now: float, value: float, interval_s: float) -> tuple[float, float] | None: + """An accepted reading, for learning. Returns what the one before it adds to the day — (rise + over the window up to it, rise since the one before) — or None. + + A reading counts with the lower of its value and the next one's, so one read too high adds + nothing. A pause longer than a normal one (down, unsure, rejected) starts again: a rise over + a pause says nothing about how fast the counter goes up.""" + before = watch.sample + watch.sample = (now, value) + if before is None or now - before[0] > gap(interval_s): + watch.window.clear() + return None + at, earlier = before[0], min(before[1], value) + if not watch.window: + watch.window.append((at, earlier)) + return 0.0, 0.0 + cut = at - COUNTER_LIMIT["window_s"] + base = watch.window[0][1] + for t, v in watch.window: + if t > cut: + break + base = v + rise = earlier - watch.window[-1][1] + watch.window.append((at, max(earlier, watch.window[-1][1]))) + while len(watch.window) >= 2 and watch.window[1][0] <= cut: + watch.window.popleft() + return max(0.0, earlier - base), max(0.0, rise) + + +def hold(watch: Watch) -> None: + """A reading read with confidence but over the limit: from now on, normal pauses no longer + count towards the limit (see advance), so it does not grow until the counter was away.""" + watch.held = True diff --git a/visionstate/backend/visionstate/db.py b/visionstate/backend/visionstate/db.py index 71799d9..ea9b2aa 100644 --- a/visionstate/backend/visionstate/db.py +++ b/visionstate/backend/visionstate/db.py @@ -3,6 +3,7 @@ from __future__ import annotations import json +import logging from collections.abc import Callable, Iterator from contextlib import contextmanager from datetime import UTC, datetime @@ -27,7 +28,9 @@ from sqlalchemy.engine import Connection from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column, relationship, sessionmaker -SCHEMA_VERSION = 13 +log = logging.getLogger(__name__) + +SCHEMA_VERSION = 15 # Schema upgrades for existing databases, keyed on the version they upgrade to. # Each step is a list of (table, column, SQL type) columns to add. @@ -50,6 +53,9 @@ ], 11: [("prediction", "sample_id", "INTEGER")], # 13: the people and their faces (new tables: person, face, face_embedding). + # 14: what a counter's limit learns per day (and a new table: reading_watch). + 14: [("reading_stat", "peak", "FLOAT"), ("reading_stat", "total", "FLOAT")], + 15: [("sensor", "frame_size", "JSON")], } @@ -74,10 +80,36 @@ def _keep_text_readers(conn: Connection) -> None: ) +def keep_change_step(reading: dict | None) -> dict | None: + """Reading settings of a counter made before change_limit, with the limit it had kept. + + Its ``max_step`` (the change allowed between two readings) stays that ("step"); a counter + without one gets no limit ("off"), as before. A counter that has ``change_limit`` and + anything else are returned as they are. + """ + if not isinstance(reading, dict) or reading.get("mode", "counter") != "counter" or "change_limit" in reading: + return reading + step = float(reading.get("max_step") or 0) + return {**reading, "change_limit": "step" if step > 0 else "off"} + + +def _keep_change_steps(conn: Connection) -> None: + rows = conn.execute(text("SELECT id, reading FROM sensor WHERE reading IS NOT NULL")).all() + for sensor_id, raw in rows: + reading = json.loads(raw) if isinstance(raw, str) else raw + kept = keep_change_step(reading) + if kept is not reading: + conn.execute( + text("UPDATE sensor SET reading = :r WHERE id = :id"), {"r": json.dumps(kept), "id": sensor_id} + ) + + # Changes to the data of existing databases, keyed on the version they upgrade to (after the columns). DATA_MIGRATIONS: dict[int, Callable[[Connection], None]] = { # Counters made before 0.7.1b3 keep the text reader; new ones get the wheel reader. 12: _keep_text_readers, + # Counters keep the change limit they had (see keep_change_step). + 14: _keep_change_steps, } @@ -119,6 +151,10 @@ class Sensor(Base): entity_prefix: Mapped[bool] = mapped_column(Boolean, default=False) # Send values to Home Assistant (settings.SENSOR_PUBLISH_DEFAULT). Off: the entities stay unavailable. publish: Mapped[bool] = mapped_column(Boolean, default=True) + # Reading sensors: the camera picture's size [width, height] the region was drawn on (taken + # from the first frame after the region or source changed). A frame of another size holds the + # readings (engine/reading.py, "camera changed"): the region would show something else. + frame_size: Mapped[list | None] = mapped_column(JSON, nullable=True) created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow) states: Mapped[list[State]] = relationship( @@ -227,6 +263,25 @@ class ReadingStat(Base): reads: Mapped[int] = mapped_column(Integer, default=0) accepted: Mapped[int] = mapped_column(Integer, default=0) rejected: Mapped[dict] = mapped_column(JSON, default=dict) # reason -> count + # Counters, for the limit they learn (change_limit.learn): the largest rise over + # COUNTER_LIMIT["window_s"] and the total rise that day, while the counter was seen. + peak: Mapped[float | None] = mapped_column(Float, nullable=True) + total: Mapped[float | None] = mapped_column(Float, nullable=True) + + +class ReadingWatch(Base): + """A counter's limit between readings (change_limit.Watch), so a restart knows how long it was + away, and the last accepted value of a reading sensor when it is newer than the history's + (a counter's, or a value confirmed in Review).""" + + __tablename__ = "reading_watch" + + sensor_id: Mapped[int] = mapped_column(ForeignKey("sensor.id", ondelete="CASCADE"), primary_key=True) + value: Mapped[str | None] = mapped_column(String(64), nullable=True) # accepted at accepted_at + seen_at: Mapped[float | None] = mapped_column(Float, nullable=True) # epoch seconds + accepted_at: Mapped[float | None] = mapped_column(Float, nullable=True) + budget_s: Mapped[float] = mapped_column(Float, default=0.0) + held: Mapped[bool] = mapped_column(Boolean, default=False) class ModelInfo(Base): @@ -324,6 +379,20 @@ def init(self) -> None: index.create(conn, checkfirst=True) conn.execute(text(f"PRAGMA user_version = {SCHEMA_VERSION}")) + @staticmethod + def after_commit(session: Session, action: Callable[[], None]) -> None: + """Run ``action`` once ``session`` is committed, never when it is rolled back: a file is + removed only together with the row that lists it. A file that cannot be removed then is + logged: the row is gone already, so the request must not fail.""" + + def run(_session: Session) -> None: + try: + action() + except OSError as err: + log.warning("Could not remove a file: %s", err) + + event.listen(session, "after_commit", run, once=True) + @contextmanager def session(self) -> Iterator[Session]: session = self._factory() diff --git a/visionstate/backend/visionstate/engine/base.py b/visionstate/backend/visionstate/engine/base.py index 60fc325..ed553ee 100644 --- a/visionstate/backend/visionstate/engine/base.py +++ b/visionstate/backend/visionstate/engine/base.py @@ -69,6 +69,7 @@ def __init__(self, settings: Settings, db: Database): self.face_choice = faces.DEFAULT_RECOGNISER self.face_error = "" self.face_progress: dict | None = None # {"done", "total"} bytes while it downloads + self.face_embedding: dict | None = None # {"done", "total"} taught faces while they are embedded self._face_lock = asyncio.Lock() self._face_task: asyncio.Task | None = None self.face_index: faces.FaceIndex | None = None # the people's taught faces; built on use diff --git a/visionstate/backend/visionstate/engine/faces.py b/visionstate/backend/visionstate/engine/faces.py index 42e4b7a..b00f6b8 100644 --- a/visionstate/backend/visionstate/engine/faces.py +++ b/visionstate/backend/visionstate/engine/faces.py @@ -56,13 +56,15 @@ class FacesMixin(ModelsMixin, PublishingMixin, RuntimeBase): # --- The model ------------------------------------------------------------------------------ def face_status(self) -> dict: - """The face model for the UI: chosen, loaded, downloading (bytes), or why it failed.""" + """The face model for the UI: chosen, loaded, downloading (bytes), embedding the taught faces + (count), or why it failed.""" spec = faces.recogniser_spec(self.face_choice) return { "model": spec.id, "loaded": self.face_model is not None and self.face_model.id == spec.id, "loading": self._face_task is not None and not self._face_task.done(), "progress": self.face_progress, + "embedding": self.face_embedding, "error": self.face_error, "downloaded": self.face_files_present(spec.id), } @@ -127,15 +129,26 @@ async def _load_faces(self) -> None: return finally: self.face_progress = None - replaced, self.face_model, self.face_error = self.face_model, model, "" - log.info("Face model %s loaded", spec.id) + replaced, self.face_model, self.face_error = self.face_model, None, "" if replaced is not None: await asyncio.to_thread(backbones.release_memory) sensor_ids = await asyncio.to_thread(self.face_sensor_ids) if not sensor_ids: # Turned off while it loaded (a check in flight asked for it): let it go again. - await self.faces_settled() + del model + await asyncio.to_thread(backbones.release_memory) return + # The taught faces are embedded before the model is used (the UI shows how far it is): the + # checks meanwhile go on without faces, as while it downloaded, instead of waiting for it. + try: + await self.face_index_for(model) + except Exception: # noqa: BLE001 - the first check builds it again + log.exception("Could not embed the taught faces with %s", spec.id) + self.face_model = model + log.info("Face model %s loaded", spec.id) + sensor_ids = await asyncio.to_thread(self.face_sensor_ids) # turned off meanwhile: let it go + if not sensor_ids: + await self.faces_settled() for sensor_id in sensor_ids: self.wake(sensor_id, force=True) @@ -206,24 +219,41 @@ def _build_face_index_locked(self, model: faces.FaceModel, generation: int) -> f ) } ids, labels, vectors = [], [], [] - for face_id, key, filename, face in examples: - vector = cached.get(face_id) - if vector is None: - path = self.storage.face_path(filename) - if not path.exists(): - continue - crop = imaging.load(path) - aligned = faces.align(crop, faces.face_from_dict(face), model.recogniser.spec.input_size) - vector = model.recogniser.embed([aligned])[0] - row = {"face_id": face_id, "model": cache_key, "vector": vector.tobytes()} - with self.db.session() as s: # stored once, whoever comes first - s.execute(insert(FaceEmbedding).values(**row).on_conflict_do_nothing()) - ids.append(face_id) - labels.append(key) - vectors.append(vector) + missing, done = sum(1 for e in examples if e[0] not in cached), 0 + if missing: + self.face_embedding = {"done": 0, "total": missing} # a new model: the UI shows the progress + try: + for face_id, key, filename, face in examples: + vector = cached.get(face_id) + if vector is None: + vector = self._embed_taught(model, cache_key, face_id, filename, face) + done += 1 + self.face_embedding = {"done": done, "total": missing} + if vector is None: + continue + ids.append(face_id) + labels.append(key) + vectors.append(vector) + finally: + self.face_embedding = None stacked = np.stack(vectors) if vectors else np.zeros((0, 0), dtype=np.float32) return faces.FaceIndex(model.id, ids, labels, stacked, generation) + def _embed_taught( + self, model: faces.FaceModel, cache_key: str, face_id: int, filename: str, face: dict + ) -> np.ndarray | None: + """A taught face's embedding, stored for the next index (None: its image is gone).""" + path = self.storage.face_path(filename) + if not path.exists(): + return None + crop = imaging.load(path) + aligned = faces.align(crop, faces.face_from_dict(face), model.recogniser.spec.input_size) + vector = model.recogniser.embed([aligned])[0] + row = {"face_id": face_id, "model": cache_key, "vector": vector.tobytes()} + with self.db.session() as s: # stored once, whoever comes first + s.execute(insert(FaceEmbedding).values(**row).on_conflict_do_nothing()) + return vector + def faces_changed(self) -> None: """Taught faces were added, moved or removed: every sensor that recognises faces compares with the new set from its next check on.""" diff --git a/visionstate/backend/visionstate/engine/history.py b/visionstate/backend/visionstate/engine/history.py index a610699..5be749b 100644 --- a/visionstate/backend/visionstate/engine/history.py +++ b/visionstate/backend/visionstate/engine/history.py @@ -4,12 +4,13 @@ import asyncio import logging -from datetime import timedelta +import shutil +from datetime import datetime, timedelta from sqlalchemy import func, select from ..db import Prediction, Sample, utcnow -from ..settings import RUNTIME, STORAGE_DEFAULTS +from ..settings import RUNTIME, SAMPLES_MOVE, STORAGE_DEFAULTS from .base import RuntimeBase from .publishing import PublishingMixin @@ -19,6 +20,10 @@ class HistoryMixin(PublishingMixin, RuntimeBase): async def _cleanup_loop(self) -> None: while True: + try: + await asyncio.to_thread(self.move_samples) + except Exception: # noqa: BLE001 + log.exception("Moving training images failed") try: await asyncio.to_thread(self.cleanup_history) await self.publish_review_count() @@ -26,6 +31,65 @@ async def _cleanup_loop(self) -> None: log.exception("Cleanup failed") await asyncio.sleep(RUNTIME["cleanup_interval_s"]) + def move_samples(self) -> None: + """Copy the training images older versions kept in the media folder to the app's own + storage (Settings.samples_dir), where a backup holds them together with the database. + + Until then they are read where they are (Storage.sample_path). The old copies stay for + SAMPLES_MOVE["keep_old_days"] after the first move, so the older version still finds its + images when the backup made before the update is restored; then they are removed. + """ + old_root = self.settings.old_samples_dir + if not old_root.exists() or old_root.resolve() == self.settings.samples_dir.resolve(): + return # nothing to move (or the media folder is set to the app's storage itself) + with self.db.session() as s: + rows = s.execute(select(Sample.sensor_id, Sample.filename)).all() + copied = 0 + for sensor_id, filename in rows: + old = self.storage.old_sample_path(sensor_id, filename) + new = self.storage.new_sample_path(sensor_id, filename) + if new.exists() or not old.exists(): + continue + part = new.with_name(f"{new.name}.part") + try: + new.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(old, part) + part.replace(new) # whole or not at all + except FileNotFoundError: # deleted meanwhile + part.unlink(missing_ok=True) + continue + except OSError as err: # disk full: the rest is read from the media folder and tried later + part.unlink(missing_ok=True) + log.warning("Could not move training images to the app's storage: %s", err) + return + if not old.exists(): # deleted while it was copied (Storage.delete_sample removes both) + new.unlink(missing_ok=True) + continue + copied += 1 + if copied: + log.info("Copied %d training images from %s to the app's storage", copied, old_root) + moved_at = self.db.get_setting("samples_moved_at") + if not isinstance(moved_at, str): + self.db.set_setting("samples_moved_at", utcnow().isoformat()) + return + if utcnow() - datetime.fromisoformat(moved_at) < timedelta(days=SAMPLES_MOVE["keep_old_days"]): + return + removed = 0 + for sensor_id, filename in rows: + old = self.storage.old_sample_path(sensor_id, filename) + if old.exists() and self.storage.new_sample_path(sensor_id, filename).exists(): + old.unlink() + removed += 1 + shutil.rmtree(self.settings.thumbs_dir / "sample", ignore_errors=True) # older versions' thumbnails + for folder in [*old_root.iterdir(), old_root]: + if folder.is_dir(): + try: + folder.rmdir() # only when empty: what no sensor of this app knows is left alone + except OSError: + pass + if removed: + log.info("Removed %d moved training images from %s", removed, old_root) + def storage_limits(self) -> dict: """Effective history limits (Settings → Storage), with the defaults for what is not set.""" return { @@ -106,21 +170,21 @@ def _trim_history(self, max_gb: float) -> int: def storage_usage(self) -> dict: """Disk use for the Settings page: history, training images and free space.""" - import shutil - with self.db.session() as s: frames = s.scalar(select(func.count()).select_from(Prediction).where(Prediction.frame.is_not(None))) or 0 oldest = s.scalar(select(func.min(Prediction.created_at)).where(Prediction.frame.is_not(None))) images = s.scalar(select(func.count()).select_from(Sample)) or 0 media = self.settings.media_dir media.mkdir(parents=True, exist_ok=True) + # History goes to the media folder, training images to the app's storage: the fuller one counts. + free = min(shutil.disk_usage(media).free, shutil.disk_usage(self.settings.data_dir).free) return { "history_bytes": self.storage.history_bytes(), "history_frames": frames, "oldest_history": oldest, "training_bytes": self.storage.samples_bytes(), "training_images": images, - "free_bytes": shutil.disk_usage(media).free, + "free_bytes": free, "limited_by_size": self.history_trimmed, **self.storage_limits(), } diff --git a/visionstate/backend/visionstate/engine/logic.py b/visionstate/backend/visionstate/engine/logic.py index 943de7b..f276d12 100644 --- a/visionstate/backend/visionstate/engine/logic.py +++ b/visionstate/backend/visionstate/engine/logic.py @@ -215,12 +215,18 @@ def _end_face_burst(burst: FaceBurst, spent: bool) -> str | None: def reading_verdict( - text: readers.Text, value: float | None, last: float | None, threshold: float, settings: dict + text: readers.Text, + value: float | None, + last: float | None, + threshold: float, + settings: dict, + limit: float | None = None, ) -> tuple[str | None, bool]: """Whether a reading is rejected, and why (None: accepted), and whether it is "settling". Rejected: nothing read, another number of digits than the counter has wheels, too unsure, - or implausible next to the ``last`` published value (see readers.implausible). Settling: one + or implausible next to the ``last`` published value (see readers.implausible; ``limit``: how + much a counter may go up, change_limit.limit). Settling: one step below the last value, the last wheel of a counter turning; the value stays, but that is no rejection. """ @@ -230,7 +236,7 @@ def reading_verdict( return "wrong digit count", False if text.score < threshold: return "unsure", False - reason = readers.implausible(value, last, settings) + reason = readers.implausible(value, last, settings, limit) if reason == "went down" and readers.settling(value, last, settings): return None, True return reason, False diff --git a/visionstate/backend/visionstate/engine/reading.py b/visionstate/backend/visionstate/engine/reading.py index 45690c2..3537494 100644 --- a/visionstate/backend/visionstate/engine/reading.py +++ b/visionstate/backend/visionstate/engine/reading.py @@ -6,16 +6,17 @@ import logging import random import time -from datetime import UTC, datetime +from datetime import UTC, date, datetime, timedelta from PIL import Image from sqlalchemy import select +from sqlalchemy.orm import Session -from .. import imaging, readers -from ..db import Prediction, ReadingStat +from .. import change_limit, imaging, readers +from ..db import Prediction, ReadingStat, ReadingWatch, Sensor from ..mqtt import rate_unit, topics from ..redact import redact -from ..settings import READING, VERSION +from ..settings import COUNTER_LIMIT, READING, REMINDER, VERSION from .base import RuntimeBase from .logic import reading_verdict from .models import ModelsMixin @@ -24,24 +25,32 @@ log = logging.getLogger(__name__) +CAMERA_CHANGED = "camera changed" # the rejection reason while a camera's picture has another size + class ReadingChecksMixin(ModelsMixin, PublishingMixin, RuntimeBase): async def _run_reading(self, cfg: SensorConfig, image: Image.Image, lit: bool = False) -> None: """One check of a reading sensor: read the number, check it and publish it. A value is published after ``debounce`` equal readings in a row. Unsure, empty and - implausible readings (a counter going down, a jump above ``max_step``, a mechanical - counter read with another number of digits than it has wheels) are rejected: - the last value stays and the rejected reading is kept in the history and sent to the - review queue, every one of them. Every reading is counted per day (Quality tab). + implausible readings (a counter going down or up faster than its limit, change_limit.py; + another value changing more than ``max_step``; a mechanical counter read with another + number of digits than it has wheels) are rejected: the last value stays and the rejected + reading is kept in the history and sent to the review queue, every one of them. A + counter's readings over its limit are accepted after all when enough of them agree. + Every reading is counted per day (Quality tab). ``lit``: the frame was taken with the sensor's light switched on (kept with the reading). """ live = self.live_state(cfg.id) settings = cfg.reading + counter = settings["mode"] == "counter" if not live.reading_restored: live.reading_restored = True + restored = await asyncio.to_thread(self._restore_reading, cfg.id, live.watch) if live.debouncer.published is None: - live.debouncer.published = await asyncio.to_thread(self._last_reading, cfg.id) + live.debouncer.published = restored + if await self._camera_changed(cfg, image, lit): + return try: text, used = await self.read_number(image, cfg.roi, settings) except Exception as err: # noqa: BLE001 - reader missing or failed to load @@ -50,7 +59,16 @@ async def _run_reading(self, cfg: SensorConfig, image: Image.Image, lit: bool = now = time.time() value = readers.parse(text.text, settings) last = float(live.debouncer.published) if live.debouncer.published is not None else None - reason, settling = reading_verdict(text, value, last, cfg.threshold, settings) + watch = live.watch + limit = None + if counter: + change_limit.advance(watch, now, cfg.interval_s) + await asyncio.to_thread(self._learn_limit, cfg.id, watch, now) + limit = change_limit.limit(settings, watch.learned, watch.budget_s) + reason, settling = reading_verdict(text, value, last, cfg.threshold, settings, limit) + # Over the limit stays rejected, however many readings agree: only an answer in Review moves it on. + if counter and reason in change_limit.IMPLAUSIBLE and value is not None and last is not None: + change_limit.hold(watch) shown = readers.format_value(value, settings) live.reading = { "text": text.text, @@ -66,10 +84,91 @@ async def _run_reading(self, cfg: SensorConfig, image: Image.Image, lit: bool = changed = False if reason is None and not settling and shown is not None: # never let a lower reading through changed = live.debouncer.update(shown, cfg.debounce) - if changed: - live.changes.append(now) + if changed: + live.changes.append(now) + # The limit starts again when a reading confirms the published value — not with one that + # still waits for its debounce: until then the published value is the older one. + learned = None + published = live.debouncer.published + if counter and reason is None and value is not None: + if published is not None and (changed or settling or shown == published): + change_limit.accept(watch, now) + learned = change_limit.observe(watch, now, value, cfg.interval_s) await self._publish_reading(cfg, image, reader_jpeg, text, shown, reason, now) - await self._keep_reading(cfg, image, text, shown, reason, changed, lit) + await self._keep_reading(cfg, image, text, shown, reason, changed, lit, learned) + + async def _camera_changed(self, cfg: SensorConfig, image: Image.Image, lit: bool) -> bool: + """Hold the readings while the camera's picture has another size than the region was drawn on. + + The region is stored as a share of the picture, so a camera that comes back from a restart + with another size (an ESP32 camera falling back to its default, which also crops another + part of its sensor) makes the region show something else, and the reader reads that. The + first frame after the region or source changed sets the size (db.Sensor.frame_size). A + frame of another size is not read: the last value stays, the problem is "camera_changed", + the frame is kept in the history once and Home Assistant gets a notification. It ends by + itself when the size is back, or when the user accepts the new picture or changes the + region (both clear the stored size). Returns whether the readings are held. + """ + live = self.live_state(cfg.id) + size = list(image.size) + known = cfg.frame_size + if known is None: + await asyncio.to_thread(self._set_frame_size, cfg.id, size) + if known is None or known == size or cfg.roi is None: # the whole picture: nothing moves + if live.camera_changed is not None: + log.info("%s: reading again, with the camera's picture at %dx%d", cfg.name, *size) + live.camera_changed = None + await self.camera_notice(cfg, None) + return False + now = time.time() + held = live.camera_changed + if held is None or held["to"] != size: + log.warning( + "%s: the camera's picture changed from %dx%d to %dx%d: readings are held until the region is checked", + cfg.name, + *known, + *size, + ) + live.camera_changed = held = {"from": known, "to": size, "since": now} + details = {"text": "", "value": None, "reason": CAMERA_CHANGED, "roi": cfg.roi, "app": VERSION, "lamp": lit} + await asyncio.to_thread(self._record_reading, cfg.id, image, None, 0.0, details, False) + await self.camera_notice(cfg, held) + live.reading = {"text": "", "score": 0.0, "value": None, "reason": CAMERA_CHANGED, "settling": False, "at": now} + crop = imaging.crop_box(image, imaging.region_box(cfg.roi)) + live.reading_image = await asyncio.to_thread(imaging.encode_jpeg, crop, 85) + await asyncio.to_thread(self._count_reading, cfg.id, CAMERA_CHANGED) + await self._publish_reading(cfg, image, live.reading_image, readers.Text("", 0.0), None, CAMERA_CHANGED, now) + return True + + def _set_frame_size(self, sensor_id: int, size: list[int]) -> None: + with self.db.session() as s: + sensor = s.get(Sensor, sensor_id) + if sensor is not None: + sensor.frame_size = size + + async def camera_notice(self, cfg: SensorConfig, held: dict | None) -> None: + """The notification in Home Assistant that a camera's picture changed (None: take it away).""" + if not self.ha.enabled: + return + notification_id = READING["camera_notification_id"] + cfg.slug + try: + if held is None: + await self.ha.call_service("persistent_notification", "dismiss", {"notification_id": notification_id}) + return + text = ( + f"{cfg.name}: the camera's picture changed from {held['from'][0]}×{held['from'][1]} to " + f"{held['to'][0]}×{held['to'][1]} pixels, so the region may show something else. " + "Readings are held until the size is back, or until you check the region in VisionState." + ) + try: + link = await self.ha.app_path() + except Exception: # noqa: BLE001 - a notification without a link is still one + link = None + message = f"{text}\n\n[Open VisionState]({link})" if link else text + data = {"title": REMINDER["title"], "message": message, "notification_id": notification_id} + await self.ha.call_service("persistent_notification", "create", data) + except Exception as err: # noqa: BLE001 - Home Assistant restarting: the problem entity still says it + log.warning("Could not send the notification about %s's camera: %s", cfg.name, redact(str(err))) async def _publish_reading( self, @@ -114,14 +213,17 @@ async def _keep_reading( reason: str | None, changed: bool, lit: bool = False, + learned: tuple[float, float] | None = None, ) -> None: """Count the reading for the day, and keep it in the history when the value changed, it was rejected (it goes to review) or it was picked for a spot check. With it go what the reader told about it (which reader, the wheels) and the app version, - so an export says how each reading was made.""" + so an export says how each reading was made. Counters: ``learned`` is what the reading + adds to the day's limit (change_limit.observe), and their Watch is stored.""" live = self.live_state(cfg.id) - await asyncio.to_thread(self._count_reading, cfg.id, reason) + watch = live.watch if cfg.reading["mode"] == "counter" else None + await asyncio.to_thread(self._count_reading, cfg.id, reason, learned, watch, live.debouncer.published) spot = reason is None and not changed and random.random() < float(cfg.reading["spot_rate"]) if changed or reason is not None or spot: published = live.debouncer.published if reason is None else None @@ -165,7 +267,14 @@ async def _send_reading_diagnostics( _, _, factor = rate_unit(settings) await self._send(cfg, t["rate"], f"{per_hour * factor:.4g}", retain=True) - def _count_reading(self, sensor_id: int, reason: str | None) -> None: + def _count_reading( + self, + sensor_id: int, + reason: str | None, + learned: tuple[float, float] | None = None, + watch: change_limit.Watch | None = None, + published: str | None = None, + ) -> None: day = datetime.now().date().isoformat() # local time: "today" as the user sees it with self.db.session() as s: row = s.get(ReadingStat, (sensor_id, day)) @@ -177,6 +286,74 @@ def _count_reading(self, sensor_id: int, reason: str | None) -> None: row.accepted += 1 else: row.rejected = {**row.rejected, reason: row.rejected.get(reason, 0) + 1} + if learned is not None: + row.peak = max(row.peak or 0.0, learned[0]) + row.total = (row.total or 0.0) + learned[1] + if watch is not None: + _store_watch(s, sensor_id, watch, published) + + def _learn_limit(self, sensor_id: int, watch: change_limit.Watch, now: float) -> None: + """Work out what a counter's limit learned from its days: once an hour, and on a new day.""" + today = date.today() + if now - watch.learned_at < COUNTER_LIMIT["relearn_s"] and watch.learned_day == today.isoformat(): + return + first = (today - timedelta(days=COUNTER_LIMIT["learn_days"] - 1)).isoformat() + with self.db.session() as s: + rows = s.scalars( + select(ReadingStat).where( + ReadingStat.sensor_id == sensor_id, ReadingStat.day >= first, ReadingStat.peak.is_not(None) + ) + ).all() + days = [(row.peak or 0.0, row.total or 0.0) for row in rows] + watch.learned, watch.days = change_limit.learn(days), len(days) + watch.learned_at, watch.learned_day = now, today.isoformat() + + def _restore_reading(self, sensor_id: int, watch: change_limit.Watch) -> str | None: + """After a restart: the last published value (so the checks go on from it) and a counter's + Watch as it was stored. + + A value stored with the Watch is newer than the history's when it was accepted later (the + history keeps changes only; a value confirmed in Review is only stored there). Without a + stored Watch (the first start since there is one), the time of the last published value + stands in for when the counter was last seen.""" + with self.db.session() as s: + last = s.execute( + select(Prediction.published_key, Prediction.created_at) + .where(Prediction.sensor_id == sensor_id, Prediction.published_key.is_not(None)) + .order_by(Prediction.created_at.desc()) + .limit(1) + ).first() + stored = s.get(ReadingWatch, sensor_id) + value, at = (last.published_key, _epoch(last.created_at)) if last is not None else (None, None) + if stored is not None: + watch.seen_at, watch.accepted_at = stored.seen_at, stored.accepted_at + watch.budget_s, watch.held = stored.budget_s, stored.held + if stored.value is not None and (at is None or (stored.accepted_at or 0.0) >= at): + value = stored.value + else: + watch.seen_at = watch.accepted_at = at + return value + + async def anchor_reading(self, sensor_id: int, value: str, at: datetime) -> None: + """Make ``value``, read at ``at``, the last accepted value: a rejected reading confirmed in Review. + + Later readings are checked against it (a counter stuck on a value read too high, or held + by its limit, goes on from there), and Home Assistant gets it.""" + live = self.live_state(sensor_id) + live.debouncer.published, live.debouncer.candidate, live.debouncer.count = value, None, 0 + now = time.time() + watch = live.watch + watch.accepted_at = _epoch(at) + watch.budget_s, watch.held = max(0.0, now - watch.accepted_at), False + watch.seen_at, watch.sample = now, None # nothing learned across it + await asyncio.to_thread(self._save_watch, sensor_id, watch, value) + cfg = self.load_sensor(sensor_id) + if cfg is not None: + await self._send(cfg, topics(cfg.slug)["state"], value, retain=True) + + def _save_watch(self, sensor_id: int, watch: change_limit.Watch, value: str) -> None: + with self.db.session() as s: + _store_watch(s, sensor_id, watch, value) def _record_reading( self, @@ -205,12 +382,16 @@ def _record_reading( ) ) - def _last_reading(self, sensor_id: int) -> str | None: - """The last published value of a reading sensor (so a restart keeps checking against it).""" - with self.db.session() as s: - return s.scalar( - select(Prediction.published_key) - .where(Prediction.sensor_id == sensor_id, Prediction.published_key.is_not(None)) - .order_by(Prediction.created_at.desc()) - .limit(1) - ) + +def _store_watch(s: Session, sensor_id: int, watch: change_limit.Watch, value: str | None) -> None: + row = s.get(ReadingWatch, sensor_id) + if row is None: + row = ReadingWatch(sensor_id=sensor_id) + s.add(row) + row.value = value + row.seen_at, row.accepted_at, row.budget_s, row.held = watch.seen_at, watch.accepted_at, watch.budget_s, watch.held + + +def _epoch(at: datetime) -> float: + """Epoch seconds of a time from the database (SQLite keeps no time zone: they are UTC).""" + return (at if at.tzinfo else at.replace(tzinfo=UTC)).timestamp() diff --git a/visionstate/backend/visionstate/engine/state.py b/visionstate/backend/visionstate/engine/state.py index 94dc0ed..2b62711 100644 --- a/visionstate/backend/visionstate/engine/state.py +++ b/visionstate/backend/visionstate/engine/state.py @@ -12,6 +12,7 @@ from sqlalchemy.orm import Session, object_session from .. import detectors +from ..change_limit import Watch from ..db import Person, Sensor from ..mqtt import SensorDescriptor from ..settings import ( @@ -52,6 +53,7 @@ class SensorConfig: reading: dict = field(default_factory=lambda: merge_reading(None)) # reading sensors only entity_prefix: bool = False # see db.Sensor.entity_prefix publish: bool = True # see db.Sensor.publish + frame_size: list | None = None # see db.Sensor.frame_size @property def state_keys(self) -> list[str]: @@ -122,6 +124,7 @@ def from_row(cls, row: Sensor) -> SensorConfig: reading=merge_reading(row.reading), entity_prefix=bool(row.entity_prefix), publish=bool(row.publish), + frame_size=row.frame_size, ) @@ -233,6 +236,11 @@ class LiveState: # and (time, value) of accepted readings of a counter for its rate (see readers.counter_rate). reads: deque = field(default_factory=deque) rate_samples: deque = field(default_factory=deque) + watch: Watch = field(default_factory=Watch) # counters: how much the value may go up next + # Reading sensors: the camera's picture has another size than the region was drawn on + # ({"from": [w, h], "to": [w, h], "since": epoch seconds}); readings are held until it is + # back or the user accepts it (engine/reading.py). + camera_changed: dict | None = None def remember(self, data: bytes) -> str: frame_id = f"{time.time_ns():x}" diff --git a/visionstate/backend/visionstate/engine/teaching.py b/visionstate/backend/visionstate/engine/teaching.py index b1c21dc..4647e69 100644 --- a/visionstate/backend/visionstate/engine/teaching.py +++ b/visionstate/backend/visionstate/engine/teaching.py @@ -48,7 +48,7 @@ def _build_taught(self, cfg: SensorConfig, generation: int = 0) -> teach.TaughtI select(Sample).where(Sample.sensor_id == cfg.id, Sample.object_label.is_not(None)).order_by(Sample.id) ) examples = [x for x in rows if x.object_label is not None and teach.usable(x.object_label, parents)] - vectors = self._taught_vectors(cfg, examples) + examples, vectors = self._taught_vectors(cfg, examples) return teach.build( embedder.spec.id, [x.id for x in examples], @@ -59,11 +59,15 @@ def _build_taught(self, cfg: SensorConfig, generation: int = 0) -> teach.TaughtI generation, ) - def _taught_vectors(self, cfg: SensorConfig, examples: list[Sample]) -> np.ndarray: - """Embeddings of taught boxes, one image at a time (see teach.embed), cached in the DB.""" + def _taught_vectors(self, cfg: SensorConfig, examples: list[Sample]) -> tuple[list[Sample], np.ndarray]: + """Embeddings of taught boxes, one image at a time (see teach.embed), cached in the DB. + + Returns the boxes that have one: a box whose image is gone or unreadable (issue #45: a + backup restored without the images) is left out instead of stopping the sensor. + """ embedder = self.embedder if embedder is None or not examples: - return np.zeros((0, 0), dtype=np.float32) + return [], np.zeros((0, 0), dtype=np.float32) backbone_id = embedder.spec.id with self.db.session() as s: rows = s.scalars( @@ -74,10 +78,15 @@ def _taught_vectors(self, cfg: SensorConfig, examples: list[Sample]) -> np.ndarr ) ).all() cached = {r.sample_id: np.frombuffer(r.vector, dtype=np.float32) for r in rows} + gone = [] for x in examples: if x.id in cached: continue - image = imaging.load(self.storage.sample_path(cfg.id, x.filename)) + try: + image = imaging.load(self.storage.sample_path(cfg.id, x.filename)) + except OSError: # missing, or not a whole image + gone.append(x.id) + continue vector = teach.embed(embedder, [image])[0] with self.db.session() as s: s.merge( @@ -86,7 +95,14 @@ def _taught_vectors(self, cfg: SensorConfig, examples: list[Sample]) -> np.ndarr ) ) cached[x.id] = vector - return np.stack([cached[x.id] for x in examples]) + if gone: + log.warning( + "Sensor %s: %d taught boxes have no readable image and are left out: %s", cfg.slug, len(gone), gone + ) + examples = [x for x in examples if x.id in cached] + if not examples: + return [], np.zeros((0, 0), dtype=np.float32) + return examples, np.stack([cached[x.id] for x in examples]) async def apply_taught( self, diff --git a/visionstate/backend/visionstate/engine/training.py b/visionstate/backend/visionstate/engine/training.py index a6ff66b..7990317 100644 --- a/visionstate/backend/visionstate/engine/training.py +++ b/visionstate/backend/visionstate/engine/training.py @@ -76,7 +76,10 @@ def _retrain_sync(self, sensor_id: int) -> None: cfg, samples, labels = self._labelled_samples(sensor_id) if cfg is None or cfg.kind != KIND_STATES: return - vectors = self.vectors_for(cfg, samples) + kept, vectors = self.vectors_for(cfg, samples) + if len(kept) < len(samples): # images gone: trained on the others + label_of = {x.id: label for x, label in zip(samples, labels, strict=True)} + samples, labels = kept, [label_of[x.id] for x in kept] with self.db.session() as s: info = s.get(ModelInfo, sensor_id) version = (info.version if info else 0) + 1 @@ -113,11 +116,15 @@ def _retrain_sync(self, sensor_id: int) -> None: f"{result.accuracy:.1%}" if result.accuracy is not None else "n/a", ) - def vectors_for(self, cfg: SensorConfig, samples: list[Sample]) -> np.ndarray: - """Embeddings for samples, computing and caching the missing ones.""" + def vectors_for(self, cfg: SensorConfig, samples: list[Sample]) -> tuple[list[Sample], np.ndarray]: + """Embeddings for samples, computing and caching the missing ones. + + Returns the samples that have one: a sample whose image is gone or unreadable (a backup + restored without the images) is left out instead of failing the training. + """ embedder = self.embedder if embedder is None or not samples: - return np.zeros((0, 0), dtype=np.float32) + return [], np.zeros((0, 0), dtype=np.float32) backbone_id = embedder.spec.id keys = {s.id: imaging.roi_key(cfg.roi if s.use_roi else None) for s in samples} with self.db.session() as s: @@ -128,12 +135,19 @@ def vectors_for(self, cfg: SensorConfig, samples: list[Sample]) -> np.ndarray: r.sample_id: np.frombuffer(r.vector, dtype=np.float32) for r in rows if r.roi_key == keys[r.sample_id] } missing = [sample for sample in samples if sample.id not in cached] + gone = [] for start in range(0, len(missing), EMBED_BATCH): - batch = missing[start : start + EMBED_BATCH] - images = [] - for sample in batch: - image = imaging.load(self.storage.sample_path(cfg.id, sample.filename)) + batch, images = [], [] + for sample in missing[start : start + EMBED_BATCH]: + try: + image = imaging.load(self.storage.sample_path(cfg.id, sample.filename)) + except OSError: # missing, or not a whole image + gone.append(sample.id) + continue + batch.append(sample) images.append(imaging.crop(image, cfg.roi) if sample.use_roi else image) + if not images: + continue vectors = embedder.embed(images) with self.db.session() as s: for sample, vector in zip(batch, vectors, strict=False): @@ -146,14 +160,23 @@ def vectors_for(self, cfg: SensorConfig, samples: list[Sample]) -> np.ndarray: ) ) cached[sample.id] = vector - return np.stack([cached[sample.id] for sample in samples]) + if gone: + log.warning( + "Sensor %s: %d training images are missing or unreadable and left out: %s", cfg.slug, len(gone), gone + ) + samples = [sample for sample in samples if sample.id in cached] + if not samples: + return [], np.zeros((0, 0), dtype=np.float32) + return samples, np.stack([cached[sample.id] for sample in samples]) def suggest(self, cfg: SensorConfig, samples: list[Sample]) -> dict[int, tuple[str, float]]: """Predicted state for (unlabelled) samples, used as upload suggestions.""" head = self.heads.get(cfg.id) if not samples or head is None or self.embedder is None or head.backbone != self.embedder.spec.id: return {} - vectors = self.vectors_for(cfg, samples) + samples, vectors = self.vectors_for(cfg, samples) + if not samples: + return {} result = {} for sample, probs in zip(samples, head.predict(vectors, cfg.state_keys), strict=False): key, p = max(probs.items(), key=lambda kv: kv[1]) diff --git a/visionstate/backend/visionstate/mqtt.py b/visionstate/backend/visionstate/mqtt.py index 67dd875..4066cad 100644 --- a/visionstate/backend/visionstate/mqtt.py +++ b/visionstate/backend/visionstate/mqtt.py @@ -14,6 +14,7 @@ from .settings import ( APP_SLUG, + HA_DEVICE_CLASS_UNITS, KIND_OBJECTS, KIND_READING, KIND_STATES, @@ -264,6 +265,14 @@ def _reading_entities(d: _Discovery) -> list[tuple[str, dict]]: **hidden, **d.with_camera, } + device_class = reading.get("device_class") or "" + if reading.get("mode") != "time_left" and device_class and not ha_unit(reading.get("unit") or "", device_class)[1]: + log.warning( + "%s: Home Assistant does not take the unit %r with the device class %r; sent without the device class", + d.sensor.name, + reading.get("unit") or "", + device_class, + ) entities = [ d.config("sensor", "state", value), d.confidence(), @@ -285,8 +294,11 @@ def reading_entity(reading: dict, t: dict[str, str]) -> dict: not have a measurement state class in Home Assistant, so it gets none. """ mode = reading.get("mode", "value") - unit = "min" if mode == "time_left" else reading.get("unit") or None - device_class = "duration" if mode == "time_left" else reading.get("device_class") or None + if mode == "time_left": + unit, device_class = "min", "duration" + else: + unit, device_class = ha_unit(reading.get("unit") or "", reading.get("device_class") or "") + unit, device_class = unit or None, device_class or None if mode == "counter": state_class = "total_increasing" elif device_class == "monetary": @@ -306,9 +318,29 @@ def reading_entity(reading: dict, t: dict[str, str]) -> dict: return {k: v for k, v in payload.items() if v is not None} +def ha_unit(unit: str, device_class: str) -> tuple[str, str]: + """(unit, device class) as Home Assistant takes them (see settings.HA_DEVICE_CLASS_UNITS). + + A unit its device class does not take is written as Home Assistant writes it when that is + clear ("m3" -> "m³", "kwh" -> "kWh"); otherwise the device class is left out, so the entity + still appears, only without what the class adds (the Energy dashboard). A unit without a + device class, or a class without a unit, is left as it is: Home Assistant takes those, and + keeps the entity's history in that unit. + """ + unit = unit.strip() + valid = HA_DEVICE_CLASS_UNITS.get(device_class) + if valid is None or not unit or unit in valid: + return unit, device_class + cubed = unit.replace("3", "³") + if cubed in valid: + return cubed, device_class + same = [u for u in valid if u.casefold() == cubed.casefold()] + return (same[0], device_class) if len(same) == 1 else (unit, "") + + def rate_unit(reading: dict) -> tuple[str | None, str | None, float]: """(unit, device class, factor from "per hour") of a counter's rate, from the counter's unit.""" - unit = (reading.get("unit") or "").strip() + unit, _ = ha_unit(reading.get("unit") or "", reading.get("device_class") or "") if unit in READING_RATE_UNITS: return READING_RATE_UNITS[unit] return (f"{unit}/h" if unit else None), None, 1.0 diff --git a/visionstate/backend/visionstate/readers.py b/visionstate/backend/visionstate/readers.py index 14df99d..a07f2e1 100644 --- a/visionstate/backend/visionstate/readers.py +++ b/visionstate/backend/visionstate/readers.py @@ -500,13 +500,19 @@ def wrong_digit_count(text: str, settings: dict) -> bool: return settings["display"] == "counter" and digit_count(text) != int(settings["digits"]) -def implausible(value: float, last: float | None, settings: dict) -> str | None: - """Why a new value can not be right compared with the last accepted one, or None.""" +def implausible(value: float, last: float | None, settings: dict, limit: float | None = None) -> str | None: + """Why a new value can not be right compared with the last accepted one, or None. + + A counter may go up by ``limit`` at most (change_limit.limit; None: no limit), any other + value change by ``max_step`` either way (0: no limit).""" if last is None: return None - if settings["mode"] == "counter" and value < last: - return "went down" - step = float(settings["max_step"]) + if settings["mode"] == "counter": + if value < last: + return "went down" + step = limit or 0.0 + else: + step = float(settings["max_step"]) if step and abs(value - last) > step: return "changed too much" return None diff --git a/visionstate/backend/visionstate/settings.py b/visionstate/backend/visionstate/settings.py index 0e28f4a..6071a38 100644 --- a/visionstate/backend/visionstate/settings.py +++ b/visionstate/backend/visionstate/settings.py @@ -84,6 +84,8 @@ COUNTER_READERS = ("wheels", "ocr") # Home Assistant device classes offered for readings ("" = none). READING_DEVICE_CLASSES = ("", "energy", "water", "gas", "volume", "monetary", "duration", "power", "temperature") +# Counters: what limits how fast the value may go up (see READING_DEFAULTS["change_limit"]). +READING_CHANGE_LIMITS = ("auto", "fixed", "step", "off") READING_DEFAULTS = { "mode": "counter", "decimals": 0, # digits after the decimal point; the reader's own dots and commas are ignored @@ -92,7 +94,15 @@ "display": "auto", "digits": 6, # "counter" display only: wheels inside the region; other digit counts are rejected "counter_reader": "wheels", # "counter" display only: see COUNTER_READERS - "max_step": 0.0, # largest plausible change between two readings (0 = no limit) + # Values and times left: the largest plausible change between two readings (0 = no limit). + # Counters use change_limit instead (see COUNTER_LIMIT). + "max_step": 0.0, + # Counters: how fast the counter may go up. "auto" learns it from the counter's own readings + # (max_rate until then; 0: COUNTER_LIMIT["standard_steps_per_min"]), "fixed" takes max_rate + # (per hour, in the counter's unit; 0: no limit), "step" takes max_step for every reading, + # however long since the last one (0: no limit), "off" sets no limit. + "change_limit": "auto", + "max_rate": 0.0, # Share of accepted readings that is also sent to the review queue, to find misreads that # passed every check. Rejected readings always go there. "spot_rate": 0.0, @@ -104,12 +114,43 @@ "decimals": (0, 4), "digits": (1, 12), "max_step": (0.0, 1e9), + "max_rate": (0.0, 1e9), "spot_rate": (0.0, 1.0), "rate_window_min": (1.0, 1440.0), } +# The limit on how fast a counter may go up (change_limit.py). The time since the last accepted +# reading allows more change, but only time the counter was not seen: down, or read unsure. +# Readings that were read with confidence but went over the limit hold that time still, so a +# misread that repeats can never wait its way through. +COUNTER_LIMIT = { + # "auto": the largest rise over this window per day, and the day's total rise, of the last + # learn_days days; the second largest of them (one misread that slipped through does not + # count). Learned after min_days days with readings: a whole week, wash day and weekend. + "window_s": 600, + "learn_days": 14, + "min_days": 7, + "safety": 2.0, # "auto": the learned rate and day times this + # "auto" until it learned, and what the UI fills in for "fixed" and "step": this many steps of + # the counter's last digit a minute (0.04 m³ a minute for a water meter with three decimals). + "standard_steps_per_min": 40, + "margin_steps": 2, # always allowed: this many steps of the last digit (a wheel half way) + # A gap between two readings longer than gap_checks intervals (at least gap_min_s) is time the + # counter was not seen (the app was down, the camera away). + "gap_checks": 3, + "gap_min_s": 300.0, + "relearn_s": 3600.0, # how often the learned limit is worked out again +} # Diagnostic entities of a reading sensor (off by default in Home Assistant). "problem" is "ok" or # why the last reading was rejected, as a lower-case key. -READING_PROBLEMS = ("ok", "nothing_read", "unsure", "wrong_digit_count", "went_down", "changed_too_much") +READING_PROBLEMS = ( + "ok", + "nothing_read", + "unsure", + "wrong_digit_count", + "went_down", + "changed_too_much", + "camera_changed", +) # Unit of a counter -> (unit of its rate, Home Assistant device class, factor from "per hour"). # Other units get "/h" without a device class. READING_RATE_UNITS = { @@ -118,6 +159,18 @@ "m³": ("m³/h", "volume_flow_rate", 1.0), "L": ("L/min", "volume_flow_rate", 1 / 60), } +# The units Home Assistant accepts per device class of READING_DEVICE_CLASSES (its sensor +# DEVICE_CLASS_UNITS; classes not listed take any unit). MQTT discovery refuses an entity whose +# unit is not in its class's list, exactly as written: "m3" or "kwh" make the entity disappear. +HA_DEVICE_CLASS_UNITS = { + "energy": ("Wh", "kWh", "MWh", "GWh", "TWh", "mWh", "J", "kJ", "MJ", "GJ", "cal", "kcal", "Mcal", "Gcal", "thm"), + "water": ("m³", "L", "gal", "ft³", "CCF", "MCF"), + "gas": ("m³", "L", "ft³", "CCF", "MCF"), + "volume": ("m³", "L", "mL", "gal", "fl. oz.", "ft³", "CCF", "MCF"), + "duration": ("d", "h", "min", "s", "ms", "μs"), + "power": ("mW", "W", "kW", "MW", "GW", "TW"), + "temperature": ("°C", "°F", "K"), +} READING = { "chars": "0123456789.,:-", # the reader may only output these characters # Quality tab: the periods it sums up (days, today included) and how many readings it lists. @@ -148,6 +201,8 @@ "export_max_mb": 24, "export_meter_max_chars": 300, # the optional "What meter is this?" text "export_wheel_top": 3, # wheel reader: the most likely positions kept per wheel + # The notification in Home Assistant when a camera's picture changed size (+ the sensor's slug) + "camera_notification_id": f"{APP_SLUG}_camera_changed_", } DETECTION = { @@ -339,7 +394,10 @@ # A light or switch that is turned on before a check takes its frame and off afterwards (a # lamp or flash next to the camera). Left alone when it is already on. "light_entity": "", - "light_delay_s": 1.0, # wait between switching on and taking the frame + # Wait between switching on and taking the frame. Frames are thrown away meanwhile (see + # Checks._grab_in_light); an ESP32 camera at 1600x1200 takes about a second per frame, so 1 s + # threw away only the stale one and the frame read was often dark. + "light_delay_s": 2.0, } TRIGGER_LIMITS = { "burst_interval_s": (0.5, 60.0), @@ -477,6 +535,14 @@ "history_max_gb": (0.0, 1000.0), } +# Training images moved from the media folder to the app's own storage (engine/history.py, +# move_samples). The copies left in the media folder are removed this many days after the move: until +# then, restoring the backup Home Assistant made before the update (it holds the app's storage, not +# the media folder) gives the older version back with every image it knew. +SAMPLES_MOVE = { + "keep_old_days": 30, +} + # --- Browsing the history (the History page and each sensor's History tab) --------------- HISTORY = { @@ -648,8 +714,20 @@ def db_path(self) -> Path: @property def samples_dir(self) -> Path: + """Training images (and the boxes taught to object sensors): in the app's own storage, like + the database that lists them, so a backup holds both.""" + return self.data_dir / "samples" + + @property + def old_samples_dir(self) -> Path: + """Where training images were kept before they moved to ``samples_dir`` (read until moved).""" return self.media_dir / "samples" + @property + def sample_thumbs_dir(self) -> Path: + """Thumbnails of training images: next to them, left out of backups (made again when asked for).""" + return self.data_dir / "thumbs" / "sample" + @property def history_dir(self) -> Path: return self.media_dir / "history" @@ -669,6 +747,7 @@ def ensure_dirs(self) -> None: self.heads_dir, self.faces_dir, self.samples_dir, + self.sample_thumbs_dir, self.history_dir, self.thumbs_dir, ): diff --git a/visionstate/backend/visionstate/storage.py b/visionstate/backend/visionstate/storage.py index d79ccad..d0da5bf 100644 --- a/visionstate/backend/visionstate/storage.py +++ b/visionstate/backend/visionstate/storage.py @@ -16,8 +16,21 @@ def __init__(self, settings: Settings): self.settings = settings def sample_path(self, sensor_id: int, filename: str) -> Path: + """A training image; one not moved yet from the media folder is read where it is.""" + path = self.new_sample_path(sensor_id, filename) + if not path.exists(): + old = self.old_sample_path(sensor_id, filename) + if old.exists(): + return old + return path + + def new_sample_path(self, sensor_id: int, filename: str) -> Path: return self.settings.samples_dir / str(sensor_id) / filename + def old_sample_path(self, sensor_id: int, filename: str) -> Path: + """Where older versions kept a training image (see Settings.old_samples_dir).""" + return self.settings.old_samples_dir / str(sensor_id) / filename + def face_path(self, filename: str) -> Path: """A taught face (db.Face): shared by every sensor, so not in a sensor's folder.""" return self.settings.faces_dir / filename @@ -26,7 +39,8 @@ def history_path(self, sensor_id: int, filename: str) -> Path: return self.settings.history_dir / str(sensor_id) / filename def thumb_path(self, kind: str, item_id: int) -> Path: - return self.settings.thumbs_dir / kind / f"{item_id}.jpg" + folder = self.settings.sample_thumbs_dir if kind == "sample" else self.settings.thumbs_dir / kind + return folder / f"{item_id}.jpg" @staticmethod def new_filename() -> str: @@ -38,7 +52,7 @@ def _write(self, path: Path, image: Image.Image) -> None: def save_sample(self, sensor_id: int, image: Image.Image) -> str: filename = self.new_filename() - self._write(self.sample_path(sensor_id, filename), image) + self._write(self.new_sample_path(sensor_id, filename), image) return filename def save_face(self, image: Image.Image) -> str: @@ -62,7 +76,8 @@ def thumbnail(self, kind: str, item_id: int, source: Path) -> Path: return target def delete_sample(self, sensor_id: int, sample_id: int, filename: str) -> None: - self.sample_path(sensor_id, filename).unlink(missing_ok=True) + self.new_sample_path(sensor_id, filename).unlink(missing_ok=True) + self.old_sample_path(sensor_id, filename).unlink(missing_ok=True) self.thumb_path("sample", sample_id).unlink(missing_ok=True) def delete_history(self, sensor_id: int, prediction_id: int, filename: str | None) -> None: @@ -83,7 +98,7 @@ def samples_bytes(self) -> int: return ( self._folder_bytes(self.settings.samples_dir) + self._folder_bytes(self.settings.faces_dir) - + self._folder_bytes(self.settings.thumbs_dir / "sample") + + self._folder_bytes(self.settings.sample_thumbs_dir) ) def delete_sensor(self, sensor_id: int) -> None: @@ -91,6 +106,7 @@ def delete_sensor(self, sensor_id: int) -> None: for directory in ( self.settings.samples_dir / str(sensor_id), + self.settings.old_samples_dir / str(sensor_id), self.settings.history_dir / str(sensor_id), ): shutil.rmtree(directory, ignore_errors=True) diff --git a/visionstate/config.yaml b/visionstate/config.yaml index 91c3af2..792b4a8 100644 --- a/visionstate/config.yaml +++ b/visionstate/config.yaml @@ -1,5 +1,5 @@ name: VisionState -version: "0.9.0" +version: "0.9.1" slug: visionstate description: Teach a local AI to recognise states (open, closed, …), find objects (people, cars, pets) or read numbers in your camera images url: https://github.com/oleost/VisionState @@ -23,6 +23,7 @@ map: read_only: false backup_exclude: - "models/*" + - "thumbs/*" watchdog: "http://[HOST]:[PORT:8099]/api/v1/status" options: log_level: info diff --git a/visionstate/frontend/e2e/ui.spec.ts b/visionstate/frontend/e2e/ui.spec.ts index 0a61357..63390a6 100644 --- a/visionstate/frontend/e2e/ui.spec.ts +++ b/visionstate/frontend/e2e/ui.spec.ts @@ -890,11 +890,46 @@ test('a counter has a rate window for its rate entity', async ({ page, request } errors.expectNone(); } finally { const sensor = await (await request.get(`api/v1/sensors/${id}`)).json(); - const { value: _v, last: _l, has_image: _h, ...reading } = sensor.reading; + const { value: _v, last: _l, has_image: _h, limit: _lim, camera_changed: _cc, ...reading } = sensor.reading; await request.patch(`api/v1/sensors/${id}`, { data: { reading: { ...reading, rate_window_min: 15 } } }); } }); +test('a counter learns how fast it may go up, or takes a fixed rate or change per reading', async ({ page, request }, info) => { + const errors = watchErrors(page); + const id = await seededReadingSensorId(request); + await page.goto(`#/sensors/${id}/settings`); + const choice = page.getByLabel('Limit how fast the counter may go up'); + await expect(choice).toHaveValue('auto'); + await expect(page.getByText(/Learning from its readings: \d+ of \d+ days/)).toBeVisible(); + // Until it learned, the standard rate applies, shown and open to change. + const config = await (await request.get('api/v1/config')).json(); + const before = (await (await request.get(`api/v1/sensors/${id}`)).json()).reading; + const standard = Number((config.counter_standard_steps_per_min * 10 ** -before.decimals * 60).toPrecision(6)); + await expect(page.getByLabel('Largest rise per hour')).toHaveValue(String(standard)); + await expect(page.getByLabel('Largest rise per hour')).toBeEnabled(); + await choice.selectOption('step'); + await expect(page.getByLabel('Largest rise per hour')).toHaveCount(0); + await expect(page.getByLabel('Largest rise per reading')).not.toHaveValue('0'); + await choice.selectOption('fixed'); + await page.getByLabel('Largest rise per hour').fill('2.4'); + await press(page.getByRole('button', { name: 'Save changes' }), info); + await expect(page.getByText('Settings saved')).toBeVisible(); + try { + const sensor = await (await request.get(`api/v1/sensors/${id}`)).json(); + expect([sensor.reading.change_limit, sensor.reading.max_rate]).toEqual(['fixed', 2.4]); + expect(sensor.reading.limit.allowed).not.toBeNull(); + await expectNoHorizontalOverflow(page); + await expectNoClipping(page, '.btn, .chip, .card'); + await page.screenshot({ path: path.join('test-results', 'pages', info.project.name, 'reading-limit.png'), fullPage: true }); + errors.expectNone(); + } finally { + const sensor = await (await request.get(`api/v1/sensors/${id}`)).json(); + const { value: _v, last: _l, has_image: _h, limit: _lim, camera_changed: _cc, ...reading } = sensor.reading; + await request.patch(`api/v1/sensors/${id}`, { data: { reading: { ...reading, change_limit: 'auto', max_rate: 0, max_step: 0 } } }); + } +}); + test('rejected readings: review queue and quality tab', async ({ page, request }, info) => { const errors = watchErrors(page); const garage = await seededSensorId(request); @@ -1306,6 +1341,28 @@ test('an answer in the review queue can be changed from the list', async ({ page errors.expectNone(); }); +test('faces: a new face model learning the taught faces shows on the cards, the Faces card and People', async ({ page, request }) => { + const errors = watchErrors(page); + const faceId = await seededFaceSensorId(request); + // Switching the face model embeds every taught face again; the status says how far it is. + await page.route('**/api/v1/status', async (route) => { + const status = await (await route.fetch()).json(); + status.faces = { ...status.faces, loaded: false, loading: true, embedding: { done: 3, total: 8 } }; + await route.fulfill({ json: status }); + }); + await page.goto('#/'); + const card = page.locator('.card', { has: page.getByRole('link', { name: 'Crew', exact: true }) }); + await expect(card.getByText('Learning faces… 3/8')).toBeVisible(); + await expectNoHorizontalOverflow(page); + await page.goto(`#/sensors/${faceId}/settings`); + await expect(page.locator('h1 ~ .chip', { hasText: 'Learning faces… 3/8' })).toBeVisible(); + await expect(page.getByTestId('faces-card').getByText(/Learning the taught faces .* 3 of 8/)).toBeVisible(); + await page.goto('#/people'); + await expect(page.getByTestId('faces-embedding')).toContainText('3 of 8'); + await expectNoHorizontalOverflow(page); + errors.expectNone(); +}); + test('faces: turned on with a confirmation, a person taught from the Live tab, shown under Quality', async ({ page, request }, info) => { const errors = watchErrors(page); const faceId = await seededFaceSensorId(request); @@ -1392,3 +1449,47 @@ test('faces: turned on with a confirmation, a person taught from the Live tab, s await expectNoClipping(page, '.btn, .butch'); errors.expectNone(); }); + +test('a camera that changed its picture size holds the readings until the region is checked', async ({ page, request }, info) => { + const errors = watchErrors(page); + const camera = `size-${info.project.name}`; // its own picture size: the projects run at the same time + const created = await request.post('api/v1/sensors', { + data: { + name: `Resized ${info.project.name}`, + kind: 'reading', + source_type: 'http', + source: `${COUNTER_URL(89931)}&camera=${camera}`, + roi: COUNTER_BOX, + reading: { mode: 'counter', display: 'counter', digits: 7, decimals: 3, change_limit: 'off' }, + interval_s: 3600, + debounce: 1, + }, + }); + expect(created.ok()).toBeTruthy(); + const id = (await created.json()).id; + const reading = async () => (await (await request.get(`api/v1/sensors/${id}`)).json()).reading; + try { + await expect.poll(async () => (await reading()).value, { timeout: 60_000 }).toBe('89.931'); + await request.get(`${CAMERA_URL}/set?wider=400&camera=${camera}`); + await request.post(`api/v1/sensors/${id}/classify`); + await expect.poll(async () => (await reading()).camera_changed?.to, { timeout: 30_000 }).toEqual([1200, 360]); + + await page.goto('#/'); + await expect(page.locator('.sensor', { hasText: `Resized ${info.project.name}` }).getByText('Camera changed')).toBeVisible(); + await page.goto(`#/sensors/${id}/live`); + const notice = page.getByTestId('camera-changed'); + await expect(notice).toContainText('from 800×360 to 1200×360 pixels'); + await expectNoHorizontalOverflow(page); + await expectNoClipping(page, '.btn, .chip, .notice'); + await page.screenshot({ path: path.join('test-results', 'pages', info.project.name, 'camera-changed.png'), fullPage: true }); + await expect(notice.getByRole('link', { name: 'Adjust the region' })).toHaveAttribute('href', `#/sensors/${id}/settings`); + + await press(notice.getByRole('button', { name: 'The region is right' }), info); + await expect(notice).toHaveCount(0, { timeout: 30_000 }); + expect((await reading()).camera_changed).toBeNull(); + errors.expectNone(); + } finally { + await request.get(`${CAMERA_URL}/set?wider=0&camera=${camera}`); + await request.delete(`api/v1/sensors/${id}`); + } +}); diff --git a/visionstate/frontend/src/lib/api.ts b/visionstate/frontend/src/lib/api.ts index 0199f82..912ae47 100644 --- a/visionstate/frontend/src/lib/api.ts +++ b/visionstate/frontend/src/lib/api.ts @@ -158,6 +158,8 @@ export const api = { request(`sensors/${id}`, send('PATCH', body)), deleteSensor: (id: number) => request(`sensors/${id}`, send('DELETE')), classify: (id: number) => request(`sensors/${id}/classify`, send('POST')), + /** A reading sensor's camera changed its picture size and the region is still right. */ + acceptPicture: (id: number) => request(`sensors/${id}/accept-picture`, send('POST')), retrain: (id: number) => request(`sensors/${id}/retrain`, send('POST')), async frame(id: number, cached = false): Promise { diff --git a/visionstate/frontend/src/lib/components/AnalysedFrame.svelte b/visionstate/frontend/src/lib/components/AnalysedFrame.svelte index 8164478..a89fd14 100644 --- a/visionstate/frontend/src/lib/components/AnalysedFrame.svelte +++ b/visionstate/frontend/src/lib/components/AnalysedFrame.svelte @@ -82,7 +82,9 @@ } onDestroy(() => { - if (objectUrl) URL.revokeObjectURL(objectUrl); + // Let go of the frame once the image is gone: revoked while it still loads, WebKit logs an error. + const last = objectUrl; + if (last) setTimeout(() => URL.revokeObjectURL(last), 1000); }); const teachOn = $derived(boxesFrame ?? shownFrame ?? fallbackFrame); diff --git a/visionstate/frontend/src/lib/components/CameraChanged.svelte b/visionstate/frontend/src/lib/components/CameraChanged.svelte new file mode 100644 index 0000000..dcea0b1 --- /dev/null +++ b/visionstate/frontend/src/lib/components/CameraChanged.svelte @@ -0,0 +1,67 @@ + + +{#if changed} +
+
+ + + The camera's picture changed from {changed.from[0]}×{changed.from[1]} to + {changed.to[0]}×{changed.to[1]} pixels {ago(changed.since)}, so the region may show something else. Readings + are held until the size is back, or until you check the region{onSettings ? ' below' : ''}. + +
+
+ + {#if !onSettings} + Adjust the region + {/if} +
+
+{/if} + + diff --git a/visionstate/frontend/src/lib/components/FacesCard.svelte b/visionstate/frontend/src/lib/components/FacesCard.svelte index 4837af9..202391b 100644 --- a/visionstate/frontend/src/lib/components/FacesCard.svelte +++ b/visionstate/frontend/src/lib/components/FacesCard.svelte @@ -125,6 +125,13 @@ + {:else if status?.embedding} +
+ + Learning the taught faces with the face model… {status.embedding.done} of {status.embedding.total} + + +
{:else if status && !status.loaded} Loading the face model… {:else if sensor.objects} diff --git a/visionstate/frontend/src/lib/components/LiveFrame.svelte b/visionstate/frontend/src/lib/components/LiveFrame.svelte index 1f26206..5336c51 100644 --- a/visionstate/frontend/src/lib/components/LiveFrame.svelte +++ b/visionstate/frontend/src/lib/components/LiveFrame.svelte @@ -84,7 +84,9 @@ onDestroy(() => { loop++; clearTimeout(timer); - if (url) URL.revokeObjectURL(url); + // Let go of the frame once the image is gone: revoked while it still loads, WebKit logs an error. + const last = url; + if (last) setTimeout(() => URL.revokeObjectURL(last), 1000); }); diff --git a/visionstate/frontend/src/lib/components/ReadingParams.svelte b/visionstate/frontend/src/lib/components/ReadingParams.svelte index 57984a3..11dad56 100644 --- a/visionstate/frontend/src/lib/components/ReadingParams.svelte +++ b/visionstate/frontend/src/lib/components/ReadingParams.svelte @@ -1,28 +1,62 @@ @@ -49,13 +83,68 @@ > - + {#if mode === 'counter'} + + {#if changeLimit === 'auto' || changeLimit === 'fixed'} + + {:else if changeLimit === 'step'} + + {/if} + {:else} + + {/if}