From 07a7f9f403856daba05ed1ee7d58dcfea03e8d7d Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Mon, 3 Aug 2026 13:31:30 +0200 Subject: [PATCH 1/8] docs(claude): record the two silent venv/packaging traps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both cost real time and neither had any build-time signal: 1. dh_virtualenv --python python3 resolved through PATH, so building on a box with pyenv active baked the builder's private interpreter into the package. On the target /usr/lib/cuems/bin/python is then a broken symlink and every CUEMS Python component fails to start — only visible after dpkg -i on a node, and even then not until the next restart. debian/rules now pins the absolute path; documented with the pre-ship verification command. 2. The .deb's site-packages copy precedes any .pth-appended path in sys.path, so an "editable install" is silently ignored while the package is present. That is why earlier editable deploys did nothing and source got rsync'd over site-packages instead (leaving rc13 files under an rc12 dist-info). Refs ClickUp 869ed00ya --- CLAUDE.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 7aff452..9d584fb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,3 +29,33 @@ All planning docs, implementation plans, agent prompts / reusable AI prompt temp - **The shared venv `/usr/lib/cuems` is one-way**: it has `include-system-site-packages = true`, so the venv sees system packages but the system `/usr/bin/python3` **cannot** import `cuemsutils`. Any tool whose entry points live in `/usr/bin` (dh-python3) must not `import cuemsutils` — reimplement with stdlib. (See the portable-.deb notes in the cuems-thermalmon CLAUDE.md.) - Many packages populate that shared venv (`cuemsutils`, pynng, lxml, **websockets** from cuems-utils; **pythonosc** from cuems-engine; plus editor/nodeconf/midi-connector). A `.deb` must not bundle anything another package already ships, or `dpkg -i` aborts on a file-overwrite conflict. lxml is bundled at 5.3.0 (CVE-2026-41066 fixed in 6.1.0; audited across all components — zero parser call sites, reachable risk nil). + +### Building the `.deb` — verify the venv interpreter before shipping + +Built from the `debian/bookworm` branch (merge `main` in, add a `debian/changelog` entry, then `dpkg-buildpackage -us -uc -b`; artifacts land in the parent dir). + +`debian/rules` pins `dh_virtualenv --python /usr/bin/python3` **by absolute path on purpose**. It used to say `--python python3`, which resolves through `PATH` — so building on a dev box with **pyenv** active baked the builder's private interpreter into the package: + +``` +pyvenv.cfg home = /home//.pyenv/versions/3.11.2/bin +bin/python -> /home//.pyenv/versions/3.11.2/bin/python3.11 +``` + +On every target host that symlink is broken, so `/usr/lib/cuems/bin/python` does not exist and **nothing in the shared venv can start** — every CUEMS Python component at once. There is no build-time error; it only surfaces after `dpkg -i` on a node, and even then the running process survives until the next restart. Always check before shipping: + +```bash +dpkg-deb --fsys-tarfile | tar -xO ./usr/lib/cuems/pyvenv.cfg | grep '^home' +# must be: home = /usr/bin (and bin/python -> ../../../bin/python3, relative) +``` + +### "Editable install" on a host is a no-op unless you remove the packaged copy + +The `.deb` installs `cuemsutils/` into `/usr/lib/cuems/lib/python3.11/site-packages/`, which comes **before** any `.pth`-appended path in `sys.path`. So a `_cuemsutils.pth` pointing at `/home/stagelab/src/cuems-utils/src` is silently ignored while the package is installed — which is why past "editable" deploys did nothing and people resorted to rsync-ing source over site-packages (leaving mismatched source/dist-info, e.g. rc13 files under an rc12 `dist-info`). To get a real editable install: + +```bash +/usr/lib/cuems/bin/pip uninstall -y cuemsutils +rm -rf /usr/lib/cuems/lib/python3.11/site-packages/cuemsutils # pip leaves non-RECORD strays +/usr/lib/cuems/bin/pip install -e /home/stagelab/src/cuems-utils --no-deps +``` + +Note `pip install -e` needs network for the build backend, so it is not an option on an offline node — install the `.deb` there instead. And an `apt install --reinstall cuems-utils` restores the packaged copy and silently re-shadows the editable tree. From 9c17418c4121f9fbd86eebda6682182634ead31d Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Mon, 31 Aug 2026 19:04:48 +0200 Subject: [PATCH 2/8] fix(fade): accept the engine-native curve names ease_in/ease_out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The frontend (8d67d08) now submits gradient-motiond's own CurveFactory names — its "exponential"/"logarithmic" labels send ease_in/ease_out, which are the implemented accelerating/decelerating shapes. But the FadeCurveType Python enum and the script.xsd enumeration only knew linear|exponential|logarithmic|sigmoid, so the first save of a non-linear fade from the new UI would fail validation. Add ease_in and ease_out to both; keep the legacy names accepted so projects authored by older editors still load (the frontend heals them to ease_in/ease_out on next save). Round-trip + legacy tests added. --- src/cuemsutils/cues/FadeCue.py | 19 ++++++++++++-- src/cuemsutils/xml/schemas/script.xsd | 11 ++++++++ tests/test_fade_cue.py | 38 +++++++++++++++++++++++++-- 3 files changed, 64 insertions(+), 4 deletions(-) diff --git a/src/cuemsutils/cues/FadeCue.py b/src/cuemsutils/cues/FadeCue.py index fa66948..92efedf 100644 --- a/src/cuemsutils/cues/FadeCue.py +++ b/src/cuemsutils/cues/FadeCue.py @@ -1,3 +1,7 @@ +# SPDX-FileCopyrightText: 2026 Stagelab Coop SCCL +# SPDX-License-Identifier: GPL-3.0-or-later +# SPDX-FileContributor: Ion Reguera + from enum import Enum from ..helpers import ensure_items, format_timecode @@ -8,12 +12,22 @@ class FadeCurveType(Enum): - """Enumeration of supported fade curve shapes.""" + """Enumeration of supported fade curve shapes. + + ``linear``, ``sigmoid``, ``ease_in`` and ``ease_out`` are the shapes + gradient-motiond's CurveFactory implements; the editor UI submits these. + ``exponential`` and ``logarithmic`` are legacy names kept so projects + authored by older editors still load — CurveFactory does NOT implement + them, and the frontend maps them onto ``ease_in``/``ease_out`` (the same + accelerating/decelerating shapes) on load, healing them on next save. + """ linear = 'linear' exponential = 'exponential' logarithmic = 'logarithmic' sigmoid = 'sigmoid' + ease_in = 'ease_in' + ease_out = 'ease_out' def __str__(self): return self.value @@ -125,7 +139,8 @@ def set_curve_type(self, curve_type) -> None: Args: curve_type (FadeCurveType | str): A FadeCurveType member or its - string value ('linear', 'exponential', 'logarithmic', 'sigmoid'). + string value ('linear', 'sigmoid', 'ease_in', 'ease_out', or + the legacy 'exponential'/'logarithmic'). Raises: ValueError: If curve_type is not a recognised value. diff --git a/src/cuemsutils/xml/schemas/script.xsd b/src/cuemsutils/xml/schemas/script.xsd index 7526a3c..675fdb4 100644 --- a/src/cuemsutils/xml/schemas/script.xsd +++ b/src/cuemsutils/xml/schemas/script.xsd @@ -1,4 +1,9 @@ + @@ -251,12 +256,18 @@ + + + diff --git a/tests/test_fade_cue.py b/tests/test_fade_cue.py index 452fda8..594dd5e 100644 --- a/tests/test_fade_cue.py +++ b/tests/test_fade_cue.py @@ -1,3 +1,7 @@ +# SPDX-FileCopyrightText: 2026 Stagelab Coop SCCL +# SPDX-License-Identifier: GPL-3.0-or-later +# SPDX-FileContributor: Ion Reguera + """Unit and integration tests for FadeCue.""" from os import path @@ -45,11 +49,11 @@ def test_fade_cue_explicit_construction(): assert fc.target_value == 75 -def _make_script_with_fade_cue(): +def _make_script_with_fade_cue(curve_type=FadeCurveType.linear): """Helper: build a minimal CuemsScript containing one FadeCue.""" fc = FadeCue({ 'action_target': TARGET_UUID, - 'curve_type': FadeCurveType.linear, + 'curve_type': curve_type, 'duration': '00:00:05.000', 'target_value': 50, }) @@ -128,6 +132,36 @@ def test_invalid_curve_type_raises(): fc.curve_type = 'triangle' +@pytest.mark.parametrize('curve', ['linear', 'sigmoid', 'ease_in', 'ease_out']) +def test_engine_native_curve_type_round_trip(curve): + """Every curve name the editor UI submits survives the XML round-trip. + + The frontend sends gradient-motiond's own CurveFactory names (its + "exponential"/"logarithmic" labels submit ease_in/ease_out), so both the + Python enum and the script.xsd enumeration must accept all four. + """ + script, _ = _make_script_with_fade_cue(FadeCurveType(curve)) + writer = XmlReaderWriter(schema_name='script', xmlfile=TMP_FILE) + writer.write_from_object(script) + assert writer.validate() is None + + reader = XmlReaderWriter(schema_name='script', xmlfile=TMP_FILE) + loaded_fc = reader.read_to_objects().cuelist.contents[0] + assert loaded_fc.curve_type == FadeCurveType(curve) + + +@pytest.mark.parametrize('curve', ['exponential', 'logarithmic']) +def test_legacy_curve_type_still_accepted(curve): + """Legacy curve names keep loading so old projects don't break. + + gradient-motiond never implemented these; the frontend heals them to + ease_in/ease_out on load, but utils must keep accepting them until then. + """ + fc = FadeCue() + fc.curve_type = curve + assert fc.curve_type == FadeCurveType(curve) + + def test_zero_duration_raises(): """T019: Setting duration to zero raises ValueError.""" fc = FadeCue() From bdf045f739acb56626c1009718d5d9d47e53ccf2 Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Fri, 2 Oct 2026 15:10:59 +0200 Subject: [PATCH 3/8] test(media): stored pixel dimensions and file size (869fat84r) (red) Media gains optional pixel_width / pixel_height / file_size, appended to MediaType after regions, so the engine stops running ffprobe on the arm path. Covers the setters, the writer (schema order whatever the dict order, no empty element for None) and the parse path, which never calls the setters. --- tests/unit/test_media_pixel_dimensions.py | 237 ++++++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 tests/unit/test_media_pixel_dimensions.py diff --git a/tests/unit/test_media_pixel_dimensions.py b/tests/unit/test_media_pixel_dimensions.py new file mode 100644 index 0000000..b7cff56 --- /dev/null +++ b/tests/unit/test_media_pixel_dimensions.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: 2026 Stagelab Coop SCCL +# SPDX-License-Identifier: GPL-3.0-or-later +# SPDX-FileContributor: Ion Reguera +"""Stored media dimensions: ``pixel_width``, ``pixel_height``, ``file_size``. + +The node engine used to run ``ffprobe`` for a video file's size on its first +arm since the engine started, under the command lock, so a GO sent right +after a cue selection started up to 400 ms late (ClickUp 869fat84r). The +editor now probes once at upload and stores the values in ``MediaType``, next +to the duration; ``file_size`` (bytes) lets the engine notice a file that was +replaced under the same name. + +The three elements are optional, appended after ``regions``. Two paths have to +hold, and they are tested separately because they share no code: + +* the object model (``Media``'s setters), used by code that builds a Media; +* the parse path (``CuemsParser`` / ``XmlReaderWriter``), which assigns keys + raw and never calls the setters, so validity and order there come from the + XSD and from ``MediaXmlBuilder``. + +Design: cuems-RELATIONS Plans/2026-10-01-engine-late-go-media-probe.md §3.1. +""" + +from __future__ import annotations + +import xml.etree.ElementTree as ET +from datetime import datetime, timezone +from pathlib import Path + +import pytest + +from cuemsutils.cues import CueList, CuemsScript, VideoCue +from cuemsutils.cues.MediaCue import Media, Region +from cuemsutils.xml import XmlReaderWriter +from cuemsutils.xml.Parsers import CuemsParser + +TMP_DIR = Path(__file__).parent.parent / "tmp" +TMP_DIR.mkdir(exist_ok=True) + +DIMS = ("pixel_width", "pixel_height", "file_size") +SCHEMA_ORDER = [ + "file_name", + "id", + "duration", + "regions", + "pixel_width", + "pixel_height", + "file_size", +] + + +def _media(**extra): + base = { + "file_name": "f.mov", + "id": "", + "duration": "00:00:10.000", + "regions": [Region({"id": 0, "loop": 1, "in_time": None, "out_time": None})], + } + return {**base, **extra} + + +def _script(media: Media) -> CuemsScript: + vc = VideoCue({"Media": media, "ui_properties": {"warning": None}}) + script = CuemsScript({"CueList": CueList({"contents": [vc]})}) + script.name = "pixel dimensions" + now = datetime.now(timezone.utc).isoformat() + script.created = now + script.modified = now + return script + + +def _write(script: CuemsScript, name: str) -> Path: + path = TMP_DIR / name + writer = XmlReaderWriter(schema_name="script", xmlfile=str(path)) + writer.write_from_object(script) + return path + + +def _media_children(path: Path) -> list[str]: + root = ET.parse(path).getroot() + media = next(el for el in root.iter() if el.tag.rsplit("}", 1)[-1] == "Media") + return [child.tag.rsplit("}", 1)[-1] for child in media] + + +def _read(path: Path): + return XmlReaderWriter(schema_name="script", xmlfile=str(path)).read_to_objects() + + +def _video_media(script): + return next(c for c in script.cuelist.contents if isinstance(c, VideoCue)).media + + +def _find_media_dicts(node): + if isinstance(node, dict): + for key, value in node.items(): + if key == "Media" and isinstance(value, dict): + yield value + else: + yield from _find_media_dicts(value) + elif isinstance(node, list): + for item in node: + yield from _find_media_dicts(item) + + +def _validate(path: Path): + return XmlReaderWriter(schema_name="script", xmlfile=str(path)).validate() + + +# --------------------------------------------------------------------------- +# Object model +# --------------------------------------------------------------------------- + + +class TestMediaSetters: + def test_absent_means_none(self): + media = Media(_media()) + assert media.pixel_width is None + assert media.pixel_height is None + assert media.file_size is None + assert not any(k in media for k in DIMS) + + def test_ints_and_digit_strings_are_stored_as_int(self): + media = Media(_media(pixel_width="1920", pixel_height=1080, file_size="4096")) + assert (media.pixel_width, media.pixel_height, media.file_size) == (1920, 1080, 4096) + assert all(type(media[k]) is int for k in DIMS) + + def test_none_removes_the_key(self): + media = Media(_media(pixel_width=1920, pixel_height=1080, file_size=10)) + media.pixel_width = None + media.file_size = None + assert "pixel_width" not in media and "file_size" not in media + assert media.pixel_height == 1080 + + @pytest.mark.parametrize("bad", [0, -1, "0", "-5", "abc", "", 1.5, "1.5", True, [1920]]) + def test_invalid_values_are_rejected(self, bad): + for key in DIMS: + with pytest.raises(ValueError): + Media(_media(**{key: bad})) + + +# --------------------------------------------------------------------------- +# Writing: schema order, no empty element, valid file +# --------------------------------------------------------------------------- + + +class TestWrite: + def test_written_in_schema_order_whatever_the_dict_order(self): + media = Media() + # Arrival order deliberately scrambled: the dimensions first. + media.file_size = 5000 + media.pixel_height = 1080 + media.pixel_width = 1920 + media.setter({k: v for k, v in _media().items()}) + path = _write(_script(media), "pixdims_order.xml") + assert _media_children(path) == SCHEMA_ORDER + assert _validate(path) is None + + def test_a_raw_dict_with_none_values_writes_no_empty_element(self): + """The parse path assigns keys raw: a Media can carry None for a + dimension (the editor strips them, but the writer must not depend on + it). would fail xs:positiveInteger.""" + media = Media(_media()) + dict.__setitem__(media, "pixel_width", None) + dict.__setitem__(media, "pixel_height", None) + dict.__setitem__(media, "file_size", None) + path = _write(_script(media), "pixdims_none.xml") + assert _media_children(path) == SCHEMA_ORDER[:4] + assert _validate(path) is None + + def test_a_project_without_dimensions_is_written_as_before(self): + path = _write(_script(Media(_media())), "pixdims_absent.xml") + assert _media_children(path) == SCHEMA_ORDER[:4] + assert _validate(path) is None + + +# --------------------------------------------------------------------------- +# Reading: the XSD and the parse path +# --------------------------------------------------------------------------- + + +class TestRead: + def test_round_trip_keeps_the_values_as_ints(self): + media = Media(_media(pixel_width=3840, pixel_height=2160, file_size=123456789)) + path = _write(_script(media), "pixdims_roundtrip.xml") + loaded = _video_media(_read(path)) + assert loaded.get("pixel_width") == 3840 + assert loaded.get("pixel_height") == 2160 + assert loaded.get("file_size") == 123456789 + assert all(type(loaded.get(k)) is int for k in DIMS) + + def test_written_again_after_reading_is_equivalent(self): + media = Media(_media(pixel_width=1280, pixel_height=720, file_size=42)) + first = _write(_script(media), "pixdims_rw1.xml") + second = _write(_read(first), "pixdims_rw2.xml") + assert _media_children(second) == SCHEMA_ORDER + assert _video_media(_read(second)) == _video_media(_read(first)) + + @pytest.mark.parametrize( + "element", ["0", "-3", + "wide", "", + "0"] + ) + def test_the_schema_rejects_invalid_values(self, element): + good = _write(_script(Media(_media())), "pixdims_bad_base.xml") + text = good.read_text() + bad = TMP_DIR / "pixdims_bad.xml" + bad.write_text(text.replace("", "" + element, 1)) + with pytest.raises(Exception): + _validate(bad) + + def test_the_schema_rejects_them_out_of_order(self): + good = _write(_script(Media(_media())), "pixdims_misorder_base.xml") + text = good.read_text() + bad = TMP_DIR / "pixdims_misorder.xml" + bad.write_text(text.replace("", "10", 1)) + with pytest.raises(Exception): + _validate(bad) + + def test_editor_json_with_dimensions_parses_and_writes(self): + """The editor's save path: frontend JSON -> CuemsParser -> XML. The + parser never calls the setters, so this is the path that matters for + order and types.""" + media = Media(_media(pixel_width=1920, pixel_height=1080, file_size=99)) + path = _write(_script(media), "pixdims_json_base.xml") + as_json = XmlReaderWriter(schema_name="script", xmlfile=str(path)).read() + # Put the dimensions FIRST in the Media dict: the writer, not the + # arrival order, must decide the order. + for media_dict in _find_media_dicts(as_json): + dims = {k: media_dict.pop(k) for k in DIMS} + rest = dict(media_dict) + media_dict.clear() + media_dict.update({**dims, **rest}) + parsed = CuemsParser(as_json).parse() + out = _write(parsed, "pixdims_json_out.xml") + assert _media_children(out) == SCHEMA_ORDER + assert _validate(out) is None + assert _video_media(_read(out)).get("pixel_width") == 1920 From ef590374b0132da3c7101500bab6835a2c8337b7 Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Fri, 2 Oct 2026 15:16:48 +0200 Subject: [PATCH 4/8] feat(media): store the media's pixel size and file size in the project (869fat84r) MediaType gains three optional elements after regions: pixel_width and pixel_height (the original size of the file's first video stream, as ffprobe reports it) and file_size (bytes, when they were measured). The editor measures them at upload, as it does the duration, so the node engine no longer runs ffprobe while arming a video cue under the command lock: a GO sent right after a cue selection started up to 400 ms late on that probe. - script.xsd: xs:positiveInteger, minOccurs=0; absent means unknown. - Media: pixel_width / pixel_height / file_size properties. A positive int or a string of digits is stored as int; None removes the key; anything else raises ValueError. - MediaXmlBuilder writes Media in schema order whatever the dict order (the editor appends the values after what the frontend sent, and the parsers keep arrival order) and writes no element for a None dimension (an empty fails the schema). Older utils lines reject a project carrying these elements; they are made to accept it by replacing only script.xsd (their parsers keep unknown Media keys as plain ints and their code never reads them). Design: cuems-RELATIONS Plans/2026-10-01-engine-late-go-media-probe.md --- src/cuemsutils/cues/MediaCue.py | 67 +++++++++++++++++++++++++++ src/cuemsutils/xml/XmlBuilder.py | 29 +++++++++++- src/cuemsutils/xml/schemas/script.xsd | 12 +++++ 3 files changed, 107 insertions(+), 1 deletion(-) diff --git a/src/cuemsutils/cues/MediaCue.py b/src/cuemsutils/cues/MediaCue.py index 602c5b9..1a9d3e4 100644 --- a/src/cuemsutils/cues/MediaCue.py +++ b/src/cuemsutils/cues/MediaCue.py @@ -1,3 +1,6 @@ +# SPDX-FileCopyrightText: 2026 Stagelab Coop SCCL +# SPDX-License-Identifier: GPL-3.0-or-later +# SPDX-FileContributor: Ion Reguera from typing import Tuple from .Cue import Cue @@ -237,6 +240,70 @@ def set_regions(self, regions): regions: list[Region] = property(get_regions, set_regions) + # --- Stored size of the media file (869fat84r) -------------------------- + # + # ``pixel_width`` / ``pixel_height`` are the original size of the file's + # first video stream, as ffprobe reports it, and ``file_size`` is the + # file's size in bytes when they were measured. The editor stores them at + # upload so the engine does not have to probe the file when it arms a + # cue; ``file_size`` lets the engine notice a file replaced under the + # same name. All three are optional: absent means unknown. + # + # Parsing a project never calls these setters (the parsers assign keys + # raw), so the schema and MediaXmlBuilder, not these, guard what is read + # from and written to a file. + + def _get_optional(self, key): + return self.get(key) + + def _set_positive_int(self, key, value): + """Store *value* as a positive ``int``; ``None`` removes the key. + + Raises: + ValueError: for anything that is not a positive integer or a + string of digits (``bool`` included: it is an ``int``). + """ + if value is None: + self.pop(key, None) + return + if isinstance(value, bool): + raise ValueError(f"Media {key} must be a positive integer, not {value!r}") + if isinstance(value, str) and value.isdigit(): + value = int(value) + if not isinstance(value, int) or value <= 0: + raise ValueError(f"Media {key} must be a positive integer, not {value!r}") + super().__setitem__(key, value) + + def get_pixel_width(self): + """Width in pixels of the media's first video stream, or ``None``.""" + return self._get_optional('pixel_width') + + def set_pixel_width(self, pixel_width): + """Set the width in pixels; ``None`` removes it.""" + self._set_positive_int('pixel_width', pixel_width) + + pixel_width = property(get_pixel_width, set_pixel_width) + + def get_pixel_height(self): + """Height in pixels of the media's first video stream, or ``None``.""" + return self._get_optional('pixel_height') + + def set_pixel_height(self, pixel_height): + """Set the height in pixels; ``None`` removes it.""" + self._set_positive_int('pixel_height', pixel_height) + + pixel_height = property(get_pixel_height, set_pixel_height) + + def get_file_size(self): + """Size in bytes of the file the dimensions were measured on, or ``None``.""" + return self._get_optional('file_size') + + def set_file_size(self, file_size): + """Set the file size in bytes; ``None`` removes it.""" + self._set_positive_int('file_size', file_size) + + file_size = property(get_file_size, set_file_size) + class MediaCue(Cue): """Base class for media-related cues (audio and video). diff --git a/src/cuemsutils/xml/XmlBuilder.py b/src/cuemsutils/xml/XmlBuilder.py index de3e137..c1872c1 100644 --- a/src/cuemsutils/xml/XmlBuilder.py +++ b/src/cuemsutils/xml/XmlBuilder.py @@ -1,3 +1,6 @@ +# SPDX-FileCopyrightText: 2026 Stagelab Coop SCCL +# SPDX-License-Identifier: GPL-3.0-or-later +# SPDX-FileContributor: Ion Reguera from enum import Enum from xml.etree.ElementTree import Element, ElementTree, SubElement, register_namespace @@ -170,10 +173,34 @@ def recurser(self, group, xml_tree): class CTimecodeXmlBuilder(GenericSimpleSubObjectXmlBuilder): pass +#: ``MediaType``'s element order in script.xsd. The writer follows it, not the +#: dict's order: the editor appends the stored dimensions after whatever the +#: frontend sent, and the parsers assign keys in arrival order (869fat84r). +MEDIA_ELEMENT_ORDER = ( + 'file_name', 'id', 'duration', 'regions', + 'pixel_width', 'pixel_height', 'file_size', +) + +#: Optional MediaType elements: a ``None`` value writes no element at all +#: (an empty ```` would fail ``xs:positiveInteger``). +MEDIA_OPTIONAL_ELEMENTS = frozenset({'pixel_width', 'pixel_height', 'file_size'}) + + +def _media_items_in_schema_order(media): + """``media``'s items, known keys in schema order, then any others in + arrival order (so the validator, not this writer, rejects them).""" + known = [(k, media[k]) for k in MEDIA_ELEMENT_ORDER if k in media] + other = [(k, v) for k, v in media.items() if k not in MEDIA_ELEMENT_ORDER] + return [ + (k, v) for k, v in known + other + if not (v is None and k in MEDIA_OPTIONAL_ELEMENTS) + ] + + class MediaXmlBuilder(GenericComplexSubObjectXmlBuilder): def build(self): if isinstance(self._object, dict): - for key, value in self._object.items(): + for key, value in _media_items_in_schema_order(self._object): if isinstance(value, VALUE_TYPES): cue_subelement = SubElement(self.xml_tree, key) cue_subelement.text = str(value) diff --git a/src/cuemsutils/xml/schemas/script.xsd b/src/cuemsutils/xml/schemas/script.xsd index 675fdb4..e2ef27a 100644 --- a/src/cuemsutils/xml/schemas/script.xsd +++ b/src/cuemsutils/xml/schemas/script.xsd @@ -186,6 +186,18 @@ SPDX-FileContributor: Ion Reguera + + + + From 6fcb7b5d4ea0137a461cc4ff383a6a0471283c38 Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Mon, 5 Oct 2026 11:09:38 +0200 Subject: [PATCH 5/8] test(media): file_md5 next to file_size, on every media type (869fat84r D18) (red) The file's MD5 identifies it unambiguously, for integrity checks: a corrupt or partial copy can have the right size. It is the MD5 the upload already carries. file_size and file_md5 apply to every media type; the pixel size stays VideoCue-only. An MD5 shaped like a number (all digits, or digits with one 'e') must survive the parser, which coerces such strings to int/float. --- tests/unit/test_media_pixel_dimensions.py | 96 ++++++++++++++++++++++- 1 file changed, 93 insertions(+), 3 deletions(-) diff --git a/tests/unit/test_media_pixel_dimensions.py b/tests/unit/test_media_pixel_dimensions.py index b7cff56..ab19eac 100644 --- a/tests/unit/test_media_pixel_dimensions.py +++ b/tests/unit/test_media_pixel_dimensions.py @@ -1,7 +1,8 @@ # SPDX-FileCopyrightText: 2026 Stagelab Coop SCCL # SPDX-License-Identifier: GPL-3.0-or-later # SPDX-FileContributor: Ion Reguera -"""Stored media dimensions: ``pixel_width``, ``pixel_height``, ``file_size``. +"""Stored media dimensions: ``pixel_width``, ``pixel_height``, ``file_size``, +and the file's ``file_md5``. The node engine used to run ``ffprobe`` for a video file's size on its first arm since the engine started, under the command lock, so a GO sent right @@ -18,7 +19,11 @@ raw and never calls the setters, so validity and order there come from the XSD and from ``MediaXmlBuilder``. -Design: cuems-RELATIONS Plans/2026-10-01-engine-late-go-media-probe.md §3.1. +``file_md5`` (D18) identifies the file unambiguously, for integrity checks; it +is the MD5 the upload already carries. ``file_size`` and ``file_md5`` apply to +every media type; the pixel size to VideoCues only. + +Design: cuems-RELATIONS Plans/2026-10-01-engine-late-go-media-probe.md §3.1, D18. """ from __future__ import annotations @@ -29,7 +34,7 @@ import pytest -from cuemsutils.cues import CueList, CuemsScript, VideoCue +from cuemsutils.cues import AudioCue, CueList, CuemsScript, VideoCue from cuemsutils.cues.MediaCue import Media, Region from cuemsutils.xml import XmlReaderWriter from cuemsutils.xml.Parsers import CuemsParser @@ -235,3 +240,88 @@ def test_editor_json_with_dimensions_parses_and_writes(self): assert _media_children(out) == SCHEMA_ORDER assert _validate(out) is None assert _video_media(_read(out)).get("pixel_width") == 1920 + + +# --------------------------------------------------------------------------- +# file_md5 (D18) +# --------------------------------------------------------------------------- + +MD5 = "d41d8cd98f00b204e9800998ecf8427e" +FULL_ORDER = SCHEMA_ORDER + ["file_md5"] + + +class TestFileMd5: + def test_absent_means_none(self): + assert Media(_media()).file_md5 is None + + def test_stored_lowercase(self): + assert Media(_media(file_md5=MD5.upper())).file_md5 == MD5 + + def test_none_removes_the_key(self): + media = Media(_media(file_md5=MD5)) + media.file_md5 = None + assert "file_md5" not in media + + @pytest.mark.parametrize("bad", ["xyz", MD5[:-1], MD5 + "0", "g" * 32, "", 123, True]) + def test_invalid_values_are_rejected(self, bad): + with pytest.raises(ValueError): + Media(_media(file_md5=bad)) + + def test_written_last_whatever_the_dict_order(self): + media = Media() + media.file_md5 = MD5 + media.file_size = 5000 + media.pixel_height = 1080 + media.pixel_width = 1920 + media.setter(_media()) + path = _write(_script(media), "pixdims_md5_order.xml") + assert _media_children(path) == FULL_ORDER + assert _validate(path) is None + + def test_round_trip_keeps_it_as_str(self): + media = Media(_media(pixel_width=1920, pixel_height=1080, file_size=7, file_md5=MD5)) + loaded = _video_media(_read(_write(_script(media), "pixdims_md5_rt.xml"))) + assert loaded.get("file_md5") == MD5 + + @pytest.mark.parametrize("md5", ["00000000000000000000000000000007", + "12e45678901234567890123456789012"]) + def test_a_number_shaped_md5_survives_the_parser(self, md5): + """All digits, or digits with one 'e', would be coerced to int/float + by str_to_value and no longer validate on the next save.""" + media = Media(_media(file_size=7, file_md5=md5)) + first = _write(_script(media), "pixdims_md5_num1.xml") + loaded = _read(first) + assert _video_media(loaded).get("file_md5") == md5 + second = _write(loaded, "pixdims_md5_num2.xml") + assert _validate(second) is None + as_json = XmlReaderWriter(schema_name="script", xmlfile=str(first)).read() + assert _video_media(CuemsParser(as_json).parse()).get("file_md5") == md5 + + def test_none_writes_no_element(self): + media = Media(_media(file_size=7)) + dict.__setitem__(media, "file_md5", None) + path = _write(_script(media), "pixdims_md5_none.xml") + assert _media_children(path) == SCHEMA_ORDER[:4] + ["file_size"] + assert _validate(path) is None + + @pytest.mark.parametrize("element", ["xyz", f"{MD5.upper()}", + "", f"{MD5}0"]) + def test_the_schema_rejects_invalid_values(self, element): + text = _write(_script(Media(_media())), "pixdims_md5_bad_base.xml").read_text() + bad = TMP_DIR / "pixdims_md5_bad.xml" + bad.write_text(text.replace("", "" + element, 1)) + with pytest.raises(Exception): + _validate(bad) + + def test_an_audio_cue_carries_size_and_md5(self): + """Size and MD5 apply to every media type (the pixel size does not).""" + cue = AudioCue({"Media": Media(_media(file_size=99, file_md5=MD5)), + "ui_properties": {"warning": None}}) + script = CuemsScript({"CueList": CueList({"contents": [cue]})}) + script.name = "audio md5" + now = datetime.now(timezone.utc).isoformat() + script.created = now + script.modified = now + path = _write(script, "pixdims_md5_audio.xml") + assert _media_children(path) == SCHEMA_ORDER[:4] + ["file_size", "file_md5"] + assert _validate(path) is None From 4c040eb819c15654138ea8d8d7e7fc7423b7065a Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Mon, 5 Oct 2026 11:09:39 +0200 Subject: [PATCH 6/8] feat(media): file_md5 in MediaType, kept a string by the parser (869fat84r D18) - script.xsd: optional file_md5 (Md5Type, 32 lowercase hex) after file_size; the comment states that file_size and file_md5 apply to every media type. - Media.file_md5: any case in, lowercase stored; None removes it; anything that is not 32 hex digits raises ValueError. - MediaXmlBuilder: file_md5 in the schema order, no element for None. - Parsers.STRING_TYPED_KEYS gains file_md5: str_to_value turned an all-digit MD5 into an int and digits-with-one-e into a float (about one MD5 in a million), which would then fail the schema on the next save. Design: cuems-RELATIONS Plans/2026-10-01-engine-late-go-media-probe.md D18 --- src/cuemsutils/cues/MediaCue.py | 25 ++++++++++++++++++++++++- src/cuemsutils/xml/Parsers.py | 6 ++++++ src/cuemsutils/xml/XmlBuilder.py | 4 ++-- src/cuemsutils/xml/schemas/script.xsd | 12 +++++++++++- 4 files changed, 43 insertions(+), 4 deletions(-) diff --git a/src/cuemsutils/cues/MediaCue.py b/src/cuemsutils/cues/MediaCue.py index 1a9d3e4..97069f2 100644 --- a/src/cuemsutils/cues/MediaCue.py +++ b/src/cuemsutils/cues/MediaCue.py @@ -244,7 +244,8 @@ def set_regions(self, regions): # # ``pixel_width`` / ``pixel_height`` are the original size of the file's # first video stream, as ffprobe reports it, and ``file_size`` is the - # file's size in bytes when they were measured. The editor stores them at + # file's size in bytes when they were measured. ``file_md5`` (below) + # identifies the file itself. The editor stores them at # upload so the engine does not have to probe the file when it arms a # cue; ``file_size`` lets the engine notice a file replaced under the # same name. All three are optional: absent means unknown. @@ -304,6 +305,28 @@ def set_file_size(self, file_size): file_size = property(get_file_size, set_file_size) + def get_file_md5(self): + """The file's MD5 (32 lowercase hex digits), or ``None``. Every media + type; it comes from the upload, which already carries it (D18).""" + return self._get_optional('file_md5') + + def set_file_md5(self, file_md5): + """Set the file's MD5 (any case, stored lowercase); ``None`` removes it. + + Raises: + ValueError: for anything that is not 32 hex digits. + """ + if file_md5 is None: + self.pop('file_md5', None) + return + if not isinstance(file_md5, str) or len(file_md5) != 32 or any( + c not in '0123456789abcdef' for c in file_md5.lower() + ): + raise ValueError(f"Media file_md5 must be 32 hex digits, not {file_md5!r}") + super().__setitem__('file_md5', file_md5.lower()) + + file_md5 = property(get_file_md5, set_file_md5) + class MediaCue(Cue): """Base class for media-related cues (audio and video). diff --git a/src/cuemsutils/xml/Parsers.py b/src/cuemsutils/xml/Parsers.py index f55a97a..29ce34b 100644 --- a/src/cuemsutils/xml/Parsers.py +++ b/src/cuemsutils/xml/Parsers.py @@ -1,3 +1,6 @@ +# SPDX-FileCopyrightText: 2026 Stagelab Coop SCCL +# SPDX-License-Identifier: GPL-3.0-or-later +# SPDX-FileContributor: Ion Reguera from ..cues import * from ..cues.FadeProfile import FadeFunctionParameter, FadeProfile from ..cues.MediaCue import Media, Region @@ -33,6 +36,9 @@ 'name', 'description', 'file_name', # defensive -- see above 'output_name', 'parameter_name', 'icon', 'color', 'unix_name', + # an MD5 is hex: about one in a million is all digits (-> int) or digits + # with one 'e' (-> float), and would no longer validate (869fat84r D18) + 'file_md5', }) class GenericDict(dict): diff --git a/src/cuemsutils/xml/XmlBuilder.py b/src/cuemsutils/xml/XmlBuilder.py index c1872c1..689fc8f 100644 --- a/src/cuemsutils/xml/XmlBuilder.py +++ b/src/cuemsutils/xml/XmlBuilder.py @@ -178,12 +178,12 @@ class CTimecodeXmlBuilder(GenericSimpleSubObjectXmlBuilder): #: frontend sent, and the parsers assign keys in arrival order (869fat84r). MEDIA_ELEMENT_ORDER = ( 'file_name', 'id', 'duration', 'regions', - 'pixel_width', 'pixel_height', 'file_size', + 'pixel_width', 'pixel_height', 'file_size', 'file_md5', ) #: Optional MediaType elements: a ``None`` value writes no element at all #: (an empty ```` would fail ``xs:positiveInteger``). -MEDIA_OPTIONAL_ELEMENTS = frozenset({'pixel_width', 'pixel_height', 'file_size'}) +MEDIA_OPTIONAL_ELEMENTS = frozenset({'pixel_width', 'pixel_height', 'file_size', 'file_md5'}) def _media_items_in_schema_order(media): diff --git a/src/cuemsutils/xml/schemas/script.xsd b/src/cuemsutils/xml/schemas/script.xsd index e2ef27a..4c8940e 100644 --- a/src/cuemsutils/xml/schemas/script.xsd +++ b/src/cuemsutils/xml/schemas/script.xsd @@ -194,13 +194,23 @@ SPDX-FileContributor: Ion Reguera unit-float width / height. file_size: the file's size in bytes when they were measured; the engine compares it with its own copy to notice a file replaced under the same name. Optional: absent means - unknown, and the engine then probes as before. VideoCues only. --> + unknown, and the engine then probes as before. The pixel size is + for VideoCues only; file_size and file_md5 for every media type. + file_md5: the file's MD5 (lowercase hex), from the upload, which + already carries it; identifies the file for integrity checks. --> + + + + + + + From f027750a3216e120b7862f954fd1e1867369e631 Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Fri, 2 Oct 2026 15:16:48 +0200 Subject: [PATCH 7/8] chore(release): 0.1.0rc15 Ships the stored media metadata (869fat84r): pixel_width / pixel_height (VideoCues) and file_size / file_md5 (every media type) in MediaType; and, from main, 9c17418: the engine-native fade curve names ease_in / ease_out. All are additive schema changes. Refs ClickUp 869fat84r --- src/cuemsutils/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/cuemsutils/__init__.py b/src/cuemsutils/__init__.py index de09bb4..7de948d 100644 --- a/src/cuemsutils/__init__.py +++ b/src/cuemsutils/__init__.py @@ -1,4 +1,5 @@ # SPDX-FileCopyrightText: 2025-present Adrià (StageLab) # # SPDX-License-Identifier: GPL-3.0 -__version__ = "0.1.0rc14" +# SPDX-FileContributor: Ion Reguera +__version__ = "0.1.0rc15" From 7778a5b8a713426d4e0f45a8028f1bd9351cbe20 Mon Sep 17 00:00:00 2001 From: Ion Reguera Date: Tue, 6 Oct 2026 20:19:04 +0200 Subject: [PATCH 8/8] release: cuems-utils 0.1.0rc15 --- debian/changelog | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/debian/changelog b/debian/changelog index 7c268b6..4db1d1a 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,11 +1,16 @@ -cuems-utils (0.1.0rc15) UNRELEASED; urgency=medium +cuems-utils (0.1.0rc15) unstable; urgency=medium + * Merge main into debian/bookworm (0.1.0rc15, 869fat84r): MediaType + gains optional pixel_width / pixel_height (VideoCue), file_size and + file_md5 (every media); the builder writes them in schema order and the + parser keeps file_md5 a string. Same code as cuemsutils 0.1.0rc15 on + PyPI (tag v0.1.0rc15). * Replaces: cuems-engine (<< 0.1.0rc6-2~). Engine debs up to 0.1.0rc6-1 also ship /usr/lib/cuems/bin/cffi-gen-src (cffi's console script), so upgrading cuems-utils on a box with any of them failed with "trying to overwrite" unless forced. cuems-engine 0.1.0rc6-2 stops shipping it. - -- Ion Reguera Tue, 06 Oct 2026 21:00:00 +0200 + -- Ion Reguera Tue, 06 Oct 2026 20:18:57 +0200 cuems-utils (0.1.0rc14) unstable; urgency=medium