Add TwelveLabs Pegasus as a hosted captioning backend - #4
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Add TwelveLabs Pegasus as a hosted captioning backend#4mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Adds an opt-in pegasus-1.5 model preset that uses TwelveLabs Pegasus to caption videos server-side instead of a local VLM. The model loader gains a third 'pegasus' strategy (backend/twelvelabs_provider.py) that uploads the source video as a TwelveLabs asset, waits for it to be ready, and calls analyze() with the user's prompt. No GPU or local weights are required. The key is read from TWELVELABS_API_KEY in the environment and is never persisted. Defaults are unchanged (Qwen3-VL stays the default preset) and the local strategies are untouched. Docs and the frontend preset type are updated to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
What this adds
An opt-in
pegasus-1.5model preset that uses TwelveLabs Pegasus to caption videos server-side, as an alternative to the local VLM presets (Qwen3-VL, Gemma 4).The model loader already dispatches on a preset's
loaderfield, so this slots in cleanly as a third strategy:backend/twelvelabs_provider.py—load_pegasus(validates the API key, builds the SDK client) andgenerate_pegasus(uploads the source video as a TwelveLabs asset, waits for it to be ready, then callsclient.analyze(...)with the user's prompt).backend/model_presets.py— newpegasus-1.5preset (loader: "pegasus",is_hosted,requires_api_key).backend/model_loader.py—load_model/generate_captiondispatch to the hosted strategy;generate_captiongains an optionalvideo_path(local strategies ignore it, Pegasus needs the file).backend/processing.py— threads the sourcevideo_paththrough togenerate_captionin both the sequential and parallel paths.backend/config.py/backend/schemas.py/frontend/src/types/settings.ts— env-only API key + preset metadata fields.Why it helps
Pegasus needs no GPU and no local weights — it analyses the whole video server-side. That lets the suite run on machines without a capable GPU and offloads captioning for large batches, while the local presets remain the default and are completely untouched.
Opt-in / non-breaking
qwen3-vl-8b; nothing changes unless you explicitly select Pegasus.TWELVELABS_API_KEYin the environment and is never written tosettings.json/user_config.json.twelvelabs>=1.2.8is added to requirements as an optional dep (only imported when the Pegasus preset is loaded).How it was tested
backend/tests/test_twelvelabs_provider.py: no-network unit tests (preset registration, meta-shape parity, guard rails) plus a live test gated onTWELVELABS_API_KEY(skipped without it). All pass; the 13 pre-existingtest_api.pyerrors are unrelated (a MagicMock f-string issue present onmaster).generate_pegasusagainst a public sample video — upload → poll-until-ready →analyzereturned a real caption ("The video shows a street with cars and a bus moving from left to right, bordered by a park...").python -m py_compileclean on all changed backend modules.Docs updated:
documentation/CONFIGURATION.md,documentation/ARCHITECTURE.md, and the README.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.