Question about codec preprocessing internals and source availability

[pfeibai]

Hi, thanks for releasing this work.

I have a few questions about the codec preprocessing pipeline used in LLaVA-OneVision-2 / codec-video-prep.

From the Python API layer, I can trace the flow roughly as:

processor(..., video_backend="codec")
_run_cv_preinfer(...)
cv-preinfer
Python pipeline code such as api.py, pipeline.py, readiness.py, topk_2x2_bitcost.py
From these files, it looks like the current online codec path mainly uses bitcost / readiness-based grouping and patch selection.

However, I could not find the actual extraction logic for:

• motion vectors (MV)
• residual_y

lower-level codec-domain parsing / decoder instrumentation
So I want to ask:

1. Is the current released codec pipeline for LLaVA-OneVision-2 using only bitcost, or does it also use MV / residual information somewhere in the lower-level backend?

2. If MV / residual_y are used, where is that extraction logic implemented?

3. Is the low-level backend source code public?
   For example, the source for cv_reader_fast
   Any FFmpeg patches / decoder modifications used to export bitcost / MV / residual-related signals
   If that code is not public yet, is there any plan to release it?

I can see the Python orchestration code in the wheel, but I could not find the underlying C/C++ source or FFmpeg patch code, so I am not sure whether I am missing another public repository.

Thanks.

[YunyaoYan]

Hi, thanks for your interest and for checking the pipeline in detail.

For the current released codec path, the online preprocessing mainly uses bitcost/readiness-based grouping and patch selection. It does not explicitly extract MV or residual_y in the current fast path.

The reason is that bitcost is computed after entropy coding codec syntax elements, including motion-related and residual-related syntax. So although we do not directly expose MV/residual_y in the current pipeline, bitcost can still reflect part of the information carried by motion vectors and residuals.

The open-source codec-video-prep code is available here: https://github.com/YunyaoYan/codec-video-prep

In earlier internal versions, we did experiment with explicit MV/residual extraction. However, for codec-video-prep, we removed that part from the current released fast path mainly for runtime efficiency and deployment simplicity. After the runtime performance becomes more stable, we plan to gradually add MV/residual extraction back into the pipeline 🚀.

Thanks again for the question! 😊

[Ironieser]

Sharing what I found, since both #182 and #184 ask about the codec preprocessing source.

The source is public at https://github.com/YunyaoYan/codec-video-prep:

• native/cv_reader_fast.cpp + ffmpeg_patch/bitcost_only/* - the FFmpeg decoder patches that export per-MB / per-CTU bit-cost.
• codec_selector/ - the readiness grouping + top-K 2x2 patch-selection pipeline.

Heads-up on the published wheel: codec-video-prep-legacy-exact==0.2.5.post2 is stale and its bit-cost path raises Argument Error. The repo main is already fixed (it uses cv_reader_fast). Tracked at YunyaoYan/codec-video-prep#4.

---

— Sixun Dong (investigation done with Claude in Cursor)