Some Matroska/webm videos yield zero extracted frames (empty output / 500) #390

Activity

1. clams-bot

   added this to infraon Jun 17, 2026
2. github-project-automation

   moved this to Todo in infraon Jun 17, 2026
3. keighrim commented on Jun 18, 2026

   

   keighrim

   on Jun 18, 2026

   Last edited by keighrim

   MemberAuthor

   More actions

   The issue has three layers of problems, each caused by the one below it.

   The direct cause of the empty output is that the document's frameCount prop (written at first file read time by SDK) becomes 0 when being read (for image extraction), so the timeline computes as 0, the sampling window collapses to [0..0], no frames are extracted from zero-length sequence, and the empty result trips the TypeError. capture() (cv2) writes a correct frameCount, but the very next get_framerate() call can't see it ans returns 0.

   That was caused by get_framerate() needlessly re-opening the container. It decides whether fps is already known with if key in video_document; the test fails to match the value capture() just set, so it falls through to open_container(), whose PyAV path derives frameCount/duration from per-stream fields that some Matroska/webm files leave unset (stream.frames == 0, stream.duration is None) and writes 0, ignoring the container-level duration — overwriting the good value.

   And that membership test fails because of a bug in __contains__ membership check in the serialization layer. Document.add_property() cannot write into the document's own JSON (a Document is treated as read-only — issue #226), so app-added properties land in _props_pending. Document.get/get_property was overridden to read _props_pending, but __getitem__/__contains__ were not — they inherit Annotation.__getitem__, which consults only _props_original and _props_ephemeral. So a freshly added property is readable through get_property yet invisible to the in operator until the property is serialized and deserlized again. It becomes visible to in only after a serialize -> deserialize round trip, when it migrates _props_pending -> capital Annotation -> _props_ephemeral; the bug is therefore strictly intra-process.

   This answers the three questions in the issue: the reset comes from get_framerate() -> open_container(); it originates in the PyAV path overwriting cv2's value while the membership test silently fails; and the app should also guard the zero-frame case.

   Proposed fix

   The failure spans mmif.serialize and mmif.utils.video_document_helper. The serialize defect is the root cause and must be fixed regardless of anything else; the larger video-loader rework is orthogonal cleanup, not an alternative to it.

   Required: fix the getter/membership asymmetry (serialization layer)

   Override Document.__getitem__ (and therefore __contains__) to also consult _props_pending, so the getter and the in operator agree. This makes get_framerate()'s existing membership test correct without touching video_document_helper, and closes the whole class of "property added via add_property is invisible to in until serialized" bugs for every caller, not just SWT.

   Required: container-level duration fallback in open_container()

   Current implementation (mmif/utils/video_document_helper.py, in open_container()) — both frame_count and duration_ms fall through to 0 when the per-stream fields are unset, which is exactly what webm/mkv hit:

   mmif-python/mmif/utils/video_document_helper.py

   Lines 104 to 116 in 21ca1e0

   	if stream.frames > 0:
   	frame_count = stream.frames
   	elif stream.duration is not None and stream.average_rate is not None:
   	frame_count = int(round(float(stream.duration) * time_base
   	* float(stream.average_rate)))
   	else:
   	frame_count = 0
   	if stream.duration is not None:
   	duration_ms = int(round(float(stream.duration) * time_base * 1000))
   	elif frame_count > 0 and fps > 0:
   	duration_ms = int(round(frame_count / fps * 1000))
   	else:
   	duration_ms = 0

   (problematic matroska files land L110 and L116)

   We can add a new conditional branch to fall back to the container-level duration (container.duration / av.time_base, microseconds -> seconds) when the per-stream fields are absent, and derive frameCount from the resolved duration, so webm/mkv get non-zero duration/frameCount. MP4 behavior is unchanged (should still takes the stream.duration branch). See webm-duration-bug.md. This is needed independent of the getter fix: once get_framerate() stops needlessly re-opening the container, open_container() is still the metadata reader and must produce correct values on these files. Removing cv2 (below) makes it the only metadata reader, so its correctness matters more, not less.
4. clams-bot

   assigned

   keighrim

   on Jun 18, 2026
5. keighrim

   closed this as completedin #391on Jun 19, 2026
6. github-project-automation

   moved this from Todo to Done in infraon Jun 19, 2026
7. clams-bot

   unassigned

   keighrim

   on Jun 19, 2026