Repository navigation
questionable TP alignment of TD annotation #379
Description
Bug Description
I am worried about whether the TP to which the TD is aligned is the same TP that was the source of the image processed by the VLM.
I ran the captioner on input MMIF from SWT. It produced this in its output view:
{
"@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
"properties": {
"document": "d1",
"origin": "v_1:tf_18",
"provenance": "derived",
"mime": "application/json",
"text": {
"@value": "AL GALLETTA BLUEBERRY GROWER",
"@language": "en"
},
"id": "v_2:td_6"
}
},
{
"@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
"properties": {
"source": "v_0:tp_919",
"target": "v_2:td_6",
"id": "v_2:al_6"
}
},
Here are the TF and TP annotations that is referenced:
{
"@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v6",
"properties": {
"label": "chyron and person",
"classification": {
"chyron and person": 0.9387489855289459
},
"targets": [
"v_0:tp_919",
"v_0:tp_920",
"v_0:tp_921",
"v_0:tp_922",
"v_0:tp_923",
"v_0:tp_924",
"v_0:tp_925",
"v_0:tp_926"
],
"representatives": [
"v_0:tp_919"
],
"timeUnit": "milliseconds",
"id": "v_1:tf_18"
}
},
{
"@type": "http://mmif.clams.ai/vocabulary/TimePoint/v5",
"properties": {
"timePoint": 459026,
"label": "IN",
"classification": {
"GLOTW": 3.5924065741710365e-05,
"CR": 3.7618651731463615e-06,
"IN": 0.947616696357727,
"KU": 0.001030643587000668,
"B": 2.926498436818542e-25,
"S": 1.0204522123136162e-11,
"M": 5.168930283794282e-10,
"Y": 1.1250751413172111e-05,
"F": 1.7080129310897973e-08,
"E": 0.0006756898364983499,
"P": 0.050621531903743744,
"-": 4.5278543439053465e-06
},
"id": "v_0:tp_919"
}
},
However, this is the frame from the video sought for 459026 (and found at 00459025).
Note that there is no text in that image, not even faintly. However, here are some nearby frames
(from 00459892)
(from 00461661)
Question: Is it possible that the app is pulling an image from later than the time point it is seeking?
Here is the cataloging aid where we discovered this.
Reproduction steps
Full MMIF file: cpb-aacip-259-9c6s1g2d_NJN_News_pre-1984_2.mmif.json
Media file: cpb-aacip-259-9c6s1g2d.mp4
Expected behavior
The VLM should perform captioning on the exact frame referenced by the TP annotation.
Log output
Screenshots
No response
Additional context
No response
Activity
keighrim commentedon Mar 11, 2026
wait... If the TP alignment was wrong and VLM saw that non-text image, how did it get the OCR test out of it...?
keighrim commentedon Mar 11, 2026
Isn't this line pulling out the image too soon?
https://github.com/WGBH-MLA/visaid_builder/blob/36437a0d8753bf52eb2e2a7bd41b9a1924137fb3/visaid_builder/create_visaid.py#L168
keighrim commentedon Mar 11, 2026
keighrim commentedon Mar 11, 2026
closing as non-issue (see clamsproject/clams-vocabulary#15 (comment) possible root cause)
owencking commentedon Mar 19, 2026
It appears there is a real issue here, but probably not an issue with this app. @keighrim could you please take another look? If the issue is real, maybe it needs to be transferred to another repo?
From @keighrim's experiment above, it appears the MMIF utils functions (the time unit converter, followed by the frame extracter) are pulling the frame 13757.
Here is some code to see the msec time of that frame:
Here is the relevant range frame numbers and msec times:
Here are two candidate frames given that we were seeking the frame at 459026ms:
frame number 13757 at 459025ms (1ms before time sought)
frame number 13758 at 459058ms (32ms after time sought)
Notice that the latter is the image output in the message above. However, it is not at the 13757th frame in the video.
What is the discrepancy here? Maybe this is due to a difference between
avandcv2?Again, this is probably not an issue with the SmolVLM2 captioner. Maybe it relates to
video_document_helper.pyin the MMIF utils?keighrim commentedon Mar 20, 2026
Hi Owen, thanks for digging deeper into this. Looking back, I realize my earlier comments pointed at the cause but didn't really spell it out. Let me try to be more explicit this time.
I don't think this is a cv2-vs-PyAV difference; I believe it's an off-by-one in the counting script. That is, "my" frame 13757 is not "your" frame 13757.
In your PyAV script, you start counting at 1:
The MMIF
video_document_helper, on the other hand, counts from 0:mmif-python/mmif/utils/video_document_helper.py
Line 136 in 0dc017f
So every frame number you reported is shifted by 1 relative to what vdh uses. Your output:
What you call "frame 13758" (with the chyron) is exactly what vdh calls frame 13757: the same frame my code snippet extracted. There is no cv2-vs-PyAV discrepancy, both libraries decode the same frames via the same FFmpeg backend (I believe). The difference is purely in your numbering convention.
That being said, we need to understand that frame numbers are not absolute anchor points. More broadly, to my understanding, "frame number" is not even a thing in MPEG-4 (or most video containers). As far as I know, frames are identified by presentation timestamps (PTS), not by sequential indices (e.g. FFmpeg's
av_seek_frameoperates on timestamps, not indices). A "frame number" is always an abstraction imposed by whatever software is doing the decoding, and different tools can choose different starting points (0 vs 1) or even different counting semantics (decode order vs presentation order, B-frame handling, etc., as observed in clamsproject/clams-vocabulary#15 (comment)). I'm no MPEG expert, but this is my understanding of why comparing frame numbers across tools without first aligning on convention will always be fragile (I learned this from working on clamsproject/clams-vocabulary#15).To be clear, the original confusion you reported is real and worth investigating. When I closed this issue, I was trying to say that the problem isn't in the captioner or in
video_document_helper, but in the inconsistency between how the visaid and the CLAMS apps select frames for a given timepoint. I've opened a proper issue for that: WGBH-MLA/visaid-builder#4keighrim commentedon Mar 21, 2026
Also, I noticed that
while other videos, for example hichy-comp set,
So this particular video is literally missing "the first" frame, only audio between 0μs - 33367μs.
keighrim commentedon Mar 22, 2026
I spent way too much of this weekend in a rabbit hole of MPEG specs, OpenCV documentation and bug reports, video encoding history, NTSC timing, etc. And it turns out the problem runs deeper than I originally thought.
vdh has a real bug
In my last comment I said the discrepancy was "purely in your numbering convention" (the off-by-one). That was only half the story. After digging into the container metadata:
That
start 0.033367means the first frame's PTS (timestamp recorded in the video file at the encoding time) is 1001 ticks (at timescale 30000), not 0. This is1001/30000 = 0.033367seconds -- exactly one frame duration.The way vdh is implemented based on cv2, it reads frames sequentially (
cur_f=0, 1, 2, ...), where cv2's 0-indexed frame N corresponds to the "physical" frame with PTS = (N+1) x 1001 ticks. But vdh'sframe/fpsconversion computes time assuming frame 0 = time 0, which is not the case for this video.13757 / 29.97 --> 459026msround(round(f/fps, 3) * 1000)13758 x 1001 / 30000459058.6ms13757 x 1001 / 30000459025.233...msThe MMIF says 459026ms. The frame vdh actually extracted and classified is at PTS 459058.6ms. Any tool that seeks by closest timestamp (as visaid does) will likely land on 459025.233...ms instead. (Note:
frame.timein Owen's code snippet showed integer millisecond values, which themselves still suffer from rounding errors in this conversion.)This can't be fixed within cv2
The obvious fix is
time = frame / fps + start, accounting for the PTS start offset. But cv2 does not expose the container's PTS start offset directly. One could readCAP_PROP_POS_MSECafter decoding a frame, but that property has documented reliability problems (opencv#12112, opencv#15749, opencv#27819). The root issue is that cv2 hides the container's timestamp model behind a frame-number abstraction that doesn't map cleanly to how MPEG-4 containers actually work.Transferring to mmif-python
This is a bug in
video_document_helper, not in this app. Transferring to themmif-pythonrepo to track the fix at the SDK level. The fix will involve replacing cv2 with direct FFmpeg access (PyAV, ffmpeg subprocess, or another binding -- still exploring options) so that vdh can read actual PTS from decoded frames instead of computing timestamps from frame numbers.In addition to this bug fix, I'm proposing to eliminate the
timeUnitproperty from the MMIF vocabulary once and for all -- see clamsproject/clams-vocabulary#15 for more evidence of the evilness of frame numbers as timestamps.keighrim commentedon Mar 22, 2026
Fix plan
Replace cv2 with PyAV for video I/O
vdh uses cv2 only for reading container metadata and decoding frames, nothing from actual CV toolkit. PyAV provides the same capabilities (
frame.to_ndarray(),container.seek()) while also exposingframe.time(presentation timestamp mapped to seconds, derived directly fromframe.pts * stream.time_base). This lets vdh read actual timestamps from decoded frames instead of computing them from frame indices.PyAV uses the same ffmpeg backend that our opencv (in pre-built container images) uses. So it doesn't need a new backend configuration, it just removes the layer that hides the timestamps.
This change also means we can drop cv2 as a dependency at the SDK level. vdh is the only reason
mmif-python[cv]requires opencv. Looking at downstream apps, onlyapp-east-wrapper,app-barsdetection, andapp-parseq-wrapperimport cv2 directly in their own code (for actual CV operations like image preprocessing -- but these are old and largely abandoned anyway). All other apps that use theclams-python-opencv4base image pull in opencv solely because vdh depends on it. Once vdh switches to PyAV, those apps no longer need opencv at all, and the three apps that do use it can declare it as their own dependency rather than inheriting it from the SDK (the image already has ffmpeg installed, so adding PyAV is just one pip-install away)Note: this might also let us stop shipping the
clams-python-*-opencvcontainer images, since opencv is not that relevant for LLM/VLM-based operations (PyAV + PIL covers "decode frame, pass to model" completely). But that's a separate discussion and should be tracked in aclams-pythonissue.Eliminate frame numbers as intermediate conversion
Any vdh code that converts
ms --> fnum --> do_stuff --> fn --> msshould be rewritten to stay in the time domain. No public API should expose frame-number-based timestamp calculation. Frame numbers may still exist internally (the decoder necessarily produces frames in sequence), but they should never leak into the public interface or be used as a proxy for timestamps.Here's the current public API categorized by what needs to happen:
Remove or deprecate (fnum-centric APIs)
framenum_to_millisecond(),framenum_to_second(),millisecond_to_framenum(),second_to_framenum(): these are theframe / fpsconversion functions at the heart of the bug. They should not exist as public API.sample_frames(start_frame, end_frame, sample_rate): operates entirely in frame-number space. Should be replaced with a time-domain equivalent that accepts start/end in ms.get_mid_framenum(),get_representative_framenum(),get_representative_framenums(): already deprecated in favor ofextract_frames_by_mode(), but their internal equivalents (_get_mid_framenum,_tp_ids_to_framenums,_timeframe_to_frame_range,_sample_all,_sample_single,_sample_representatives) still convert to frame numbers as an intermediate step. These internal functions need rewriting too.Rewrite the implementation
capture(): currently readsCAP_PROP_FPSandCAP_PROP_FRAME_COUNTfrom cv2, computes duration asround(fc / fps, 3) * 1000. Should instead read stream metadata directly (streamtime_base,start_time,duration) and store PTS-derived values. ThefpsandframeCountdocument properties it currently sets are part of the fnum worldview and need rethinking.extract_frames_as_images(video_document, framenums): the main frame extraction function. Currently accepts frame numbers, reads sequentially via cv2 with a skip-threshold hack (theskip_threadhold = 1000path that usesset(CAP_PROP_POS_FRAMES)). Should accept timestamps (ms) instead, seek by timestamp, and decode. Internally it can still track position sequentially for efficiency, but the interface should be time-based.extract_frames_by_mode(),extract_target_frames(): these are the high-level APIs apps actually call. Their signatures don't expose frame numbers directly (they accept TimeFrame annotations), but internally they go through_tp_ids_to_framenums()and thenextract_frames_as_images(framenums). The internal plumbing needs to stay in the time domain end-to-end.convert_timepoint(),convert_timeframe(): currently delegate totimeunit_helper.convert()which usesround(round(t / fps, 3) * 1000)for frame-to-ms. If we eliminateframesas a valid time unit (per the clams-vocabulary#15 proposal), these simplify to ms-to-ms or ms-to-seconds conversion with no fps involved. Theget_framerate()dependency goes away for time conversion.Keep as-is
_resolve_video_document(): resolves a video document from a TimeFrame annotation. No fnum dependency.SamplingModeenum and related constants: these describe how to select frames, not what coordinate system to use. Still valid.Migration path
Two options:
Either way, this results in a breaking version. Downstream apps that use vdh for video access (which is most of them -- see the
clams-python-opencv4base image usage) will need updates.keighrim commentedon Apr 22, 2026
this side of patch is done via #382, remaining TODOs are blocked by clamsproject/clams-vocabulary#15 , closing as done for now.