vdh.extract_frames_by_mode should also return "source" information #384
Description
New Feature Summary
vdh.extract_frames_by_mode() (added via #373 released in 1.3.0) returns a list of frames (np.ndarray or PIL.Image) but discards the source TimePoint IDs in the process. Apps that consume the results have no way to know which TP inside the input TimeFrame produced each frame.
This is causing provenance regressions in downstream apps. For example, in clamsproject/app-smolvlm2-captioner#7, Owen reported that the captioner used to emit Alignment annotations whose source pointed to the per-frame TP, but after switching to extract_frames_by_mode(), there's no way to record the time points in the Alignment annotations. This is paritcularly bad for mode=reps since when multiple representatives exist, the captioner would emit multiple captions per TF without finer grounding. The same problem will hit any LLM-based app processing TF inputs (OCR, captioning, classification-via-LLM, etc.).
Proposed solution
Add a new function alongside the existing extract_frames_by_mode():
def extract_frames_by_mode_with_sources(...) -> List[Tuple[Frame, Union[str, int]]]:
"""
Same behavior as extract_frames_by_mode(), but returns each frame paired
with a grounding point: the long_id (str) of the source TP annotation if
one is available, otherwise the sampled timestamp (int, in milliseconds)
that the frame was extracted from. Either way the caller has a way to
anchor the frame to a specific point in the source media.
"""Per-mode behavior:
representativesmode: every frame paired with its source TP long_id (alwaysstr, since the mode skips TF without representatives)singlemode: paired with the middle representative's long_id (str) when representatives exist; paired with the midpoint timestamp in ms (int) when falling back to the start/end intervalallmode: each frame paired with its source TP long_id (str) whentargetsare present; paired with the sampled timestamp in ms (int) for each frame when no targets exist and stream-rate sampling is used
The existing extract_frames_by_mode() stays unchanged. Apps that need provenance migrate to the new function; apps that don't care keep using the existing one. No need to bump the major version.
Related
Implicit assumption to note
This proposal assumes that values in a TF's representatives (and targets) properties are always TP annotation IDs. That assumption is currently only ad-hoc; representatives is not a formally defined property on TF; it's an "additional property" used by convention. The formalization is being discussed at clamsproject/clams-vocabulary#12. The new helper depends on this convention being preserved in whatever the formal definition ends up being.
Alternatives
No response
Additional context
- Output MMIF structure and semantics for referring to captioned
TimePointandTimeFrameapp-smolvlm2-captioner#7 — concrete regression motivating this request Alignmenttype needs some improvement along with definition of "anchors" clams-vocabulary#12 — formalization ofrepresentatives/targets
Activity
keighrim commentedon Apr 27, 2026
#385 is now open against this issue. The implementation diverges from the original proposal in a few places. Summarizing here for future reference and to clarify the rationale for the changes.
Return shape: tuple-of-lists, not list-of-pairs
The issue proposed
List[Tuple[Frame, Union[str, int]]](list-of-pairs). The implementation usesTuple[List[Image], List[Union[str, int]]](tuple-of-two-parallel-lists), matching the existingextract_target_framesshape — which is nowextract_images_by_count_with_sourcesand would otherwise need a return-shape change on top of the rename.Rename of all extract-functions
Reviewing the existing extract-functions (
extract_timepoints_as_images,extract_target_frames,extract_frames_by_mode), I noticed they don't share a naming pattern, andframe(as always) is overloaded across different semantics. The PR fixes the naming alongside #384 since they touch the same functions:extract_timepoints_as_imagesextract_images_from_timepointsextract_target_frames->Tuple[List[Image], List[str]]extract_images_by_count_with_sources->Tuple[List[Image], List[str]](this always works ontargetslist and the second return value is target ID list)extract_images_by_count->List[Image]extract_frames_by_mode->List[Image]extract_images_by_mode->List[Image]extract_images_by_mode_with_sources->Tuple[List[Image], List[Union[str, int]]](this can work either with or withoutreps; see below for the second return value semantics)Pattern:
extract_images_<from|by>_<noun>.from <input>for direct point-driven extraction,by <criterion>for dispatch-driven extraction. Source-emitting variants append_with_sources(plural).The three pre-rename names live as soft-deprecated aliases that delegate to the new names and emit
DeprecationWarning, scheduled for removal in 2.0.0. No downstream app should break on upgrade.Implementation note
The internal samplers (
_sample_all_timepoint_pairs_ms,_sample_representatives_timepoint_pairs_ms,_sample_single_timepoint_pair_ms) were rewritten to returnList[Tuple[int, Optional[str]]]—Nonefor the source slot signals a fallback path (no rep / no target). Both bareextract_images_by_*and_with_sourcesvariants consume the same sampler output; the bare form strips[0], the_with_sourcesform materializesint(ms)forNonesources._with_sourcesper-mode source resolution (matches the original proposal):REPRESENTATIVES.id(alwaysstr)SINGLEwith reps.id(str)SINGLEno repsint)ALLwithtargets.id(str)ALLno targetsint)