Add TwelveLabs Pegasus clip finder as opt-in alternative#18
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Add TwelveLabs Pegasus clip finder as opt-in alternative#18mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
This PR adds
PegasusClipFinder, an opt-in alternative to the existing transcript-onlyClipFinder. The defaultClipFinderfinds clips from the words alone using TextTiling;PegasusClipFinderinstead uses TwelveLabs Pegasus, a video-language model that analyzes the actual video (visuals, audio, pacing). It proposes highlight time ranges which are aligned to the transcript's character offsets, so the returnedClipobjects are fully interchangeable with those produced byClipFinder.Why this helps clipsai
The README notes the library is tuned for "audio-centric, narrative-based videos." Pegasus extends usefulness to videos where highlights are visual rather than purely verbal, without changing the existing pipeline or output type.
Non-breaking / opt-in
ClipFinderand its defaults are untouched.twelvelabsSDK is an optional extra (pip install "clipsai[twelvelabs]"), imported lazily only insidefind_clips, so importing clipsai stays dependency-free for everyone not using it.ConfigManagersubclass for validation,ClipFinderErrorfor errors, NumPy-style docstrings, returnslist[Clip].Usage
How it was tested
tests/test_pegasus_clipfinder.py(no network): config validation, API-key handling, response parsing (incl. JSON wrapped in prose/markdown), time-range ->Clipmapping (filtering, clamping to media end, sorting, malformed-range skipping), andfind_clipswiring with a mocked SDK client.blackandflake8clean against the repo'ssetup.cfgconfig.analyze(model_name=, video={"type":"url","url":...}, prompt=, max_tokens=)request shape and.dataresponse field. The Pegasus request envelope was verified end-to-end (auth + model + video URL contract all accepted).You can grab a free API key at https://twelvelabs.io — there's a generous free tier.