`vdh.extract_frames_by_mode` should also return "source" information

[keighrim]

New Feature Summary

vdh.extract_frames_by_mode() (added via #373 released in 1.3.0) returns a list of frames (np.ndarray or PIL.Image) but discards the source TimePoint IDs in the process. Apps that consume the results have no way to know which TP inside the input TimeFrame produced each frame.

This is causing provenance regressions in downstream apps. For example, in clamsproject/app-smolvlm2-captioner#7, Owen reported that the captioner used to emit Alignment annotations whose source pointed to the per-frame TP, but after switching to extract_frames_by_mode(), there's no way to record the time points in the Alignment annotations. This is paritcularly bad for mode=reps since when multiple representatives exist, the captioner would emit multiple captions per TF without finer grounding. The same problem will hit any LLM-based app processing TF inputs (OCR, captioning, classification-via-LLM, etc.).

Proposed solution

Add a new function alongside the existing extract_frames_by_mode():

def extract_frames_by_mode_with_sources(...) -> List[Tuple[Frame, Union[str, int]]]:
    """
    Same behavior as extract_frames_by_mode(), but returns each frame paired
    with a grounding point: the long_id (str) of the source TP annotation if
    one is available, otherwise the sampled timestamp (int, in milliseconds)
    that the frame was extracted from. Either way the caller has a way to
    anchor the frame to a specific point in the source media.
    """

Per-mode behavior:

• representatives mode: every frame paired with its source TP long_id (always str, since the mode skips TF without representatives)
• single mode: paired with the middle representative's long_id (str) when representatives exist; paired with the midpoint timestamp in ms (int) when falling back to the start/end interval
• all mode: each frame paired with its source TP long_id (str) when targets are present; paired with the sampled timestamp in ms (int) for each frame when no targets exist and stream-rate sampling is used

The existing extract_frames_by_mode() stays unchanged. Apps that need provenance migrate to the new function; apps that don't care keep using the existing one. No need to bump the major version.

Related

Implicit assumption to note

This proposal assumes that values in a TF's representatives (and targets) properties are always TP annotation IDs. That assumption is currently only ad-hoc; representatives is not a formally defined property on TF; it's an "additional property" used by convention. The formalization is being discussed at clamsproject/clams-vocabulary#12. The new helper depends on this convention being preserved in whatever the formal definition ends up being.

Alternatives

No response

Additional context

• Output MMIF structure and semantics for referring to captioned TimePoint and TimeFrame app-smolvlm2-captioner#7 — concrete regression motivating this request
• Alignment type needs some improvement along with definition of "anchors" clams-vocabulary#12 — formalization of representatives/targets

[keighrim]

#385 is now open against this issue. The implementation diverges from the original proposal in a few places. Summarizing here for future reference and to clarify the rationale for the changes.

Return shape: tuple-of-lists, not list-of-pairs

The issue proposed List[Tuple[Frame, Union[str, int]]] (list-of-pairs). The implementation uses Tuple[List[Image], List[Union[str, int]]] (tuple-of-two-parallel-lists), matching the existing extract_target_frames shape — which is now extract_images_by_count_with_sources and would otherwise need a return-shape change on top of the rename.

Rename of all extract-functions

Reviewing the existing extract-functions (extract_timepoints_as_images, extract_target_frames, extract_frames_by_mode), I noticed they don't share a naming pattern, and frame (as always) is overloaded across different semantics. The PR fixes the naming alongside #384 since they touch the same functions:

Before	After
extract_timepoints_as_images	extract_images_from_timepoints
extract_target_frames -> Tuple[List[Image], List[str]]	extract_images_by_count_with_sources -> Tuple[List[Image], List[str]] (this always works on targets list and the second return value is target ID list)
(new)	extract_images_by_count -> List[Image]
extract_frames_by_mode -> List[Image]	extract_images_by_mode -> List[Image]
(new)	extract_images_by_mode_with_sources -> Tuple[List[Image], List[Union[str, int]]] (this can work either with or without reps; see below for the second return value semantics)

Pattern: extract_images_<from|by>_<noun>. from <input> for direct point-driven extraction, by <criterion> for dispatch-driven extraction. Source-emitting variants append _with_sources (plural).

The three pre-rename names live as soft-deprecated aliases that delegate to the new names and emit DeprecationWarning, scheduled for removal in 2.0.0. No downstream app should break on upgrade.

Implementation note

The internal samplers (_sample_all_timepoint_pairs_ms, _sample_representatives_timepoint_pairs_ms, _sample_single_timepoint_pair_ms) were rewritten to return List[Tuple[int, Optional[str]]] — None for the source slot signals a fallback path (no rep / no target). Both bare extract_images_by_* and _with_sources variants consume the same sampler output; the bare form strips [0], the _with_sources form materializes int(ms) for None sources.

_with_sources per-mode source resolution (matches the original proposal):

Mode	Source per emitted image
REPRESENTATIVES	rep TP .id (always str)
SINGLE with reps	middle rep TP .id (str)
SINGLE no reps	midpoint ms (int)
ALL with targets	each target TP .id (str)
ALL no targets	each sampled ms (int)