Skip to content

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code - #147

Merged
ZTWHHH merged 6 commits into
mm-evaluation:mainfrom
ZTWHHH:add-five-new-model-series
Jun 17, 2026
Merged

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code#147
ZTWHHH merged 6 commits into
mm-evaluation:mainfrom
ZTWHHH:add-five-new-model-series

Conversation

@ZTWHHH

@ZTWHHH ZTWHHH commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Add 6 new multimodal model series to the framework. All inference files default to max_new_tokens=2048. Companion test-result PR: #148.

Series HF model Tested variants Modalities
kimi_vl moonshotai/Kimi-VL-A3B-{Instruct,Thinking} Both text, multi-image-interleave
minicpm_v_4d5 openbmb/MiniCPM-V-4_5 All text, multi-image-interleave
glm_4d1v THUDM/GLM-4.1V-9B-{Thinking,Base} Both text, multi-image-interleave
ovis2 AIDC-AI/Ovis2-{1B,2B,4B,8B} All registered text, multi-image-interleave
aria rhymes-ai/Aria All text, multi-image-interleave
qwen3d5 Qwen/Qwen3.5-{0.8B,2B,4B,9B} All registered text, multi-image-video-interleave

Qwen3.5 is the official unified vision-language family (Qwen3_5ForConditionalGeneration + Qwen3VLProcessor); see https://qwen.ai/blog?id=qwen3.5 and https://github.com/QwenLM/Qwen3.5.

This PR contains infer/model-code changes only.

Files:

  • mmeval/infer/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria,qwen3d5}.py
  • env_files/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria,qwen3d5}_requirements.txt
  • mmeval/registry.py

Type of Change

  • Bug fix
  • New model support
  • New dataset support
  • Documentation update
  • Other (specify):

Testing

  • Tested locally
  • Added example script (if new model/dataset)

Test command used:

PYTHONPATH=. ENV_DIR=/raid/ztw/envs python mmeval/run.py \
  --model_name_or_path <HF model id> \
  --dataset local@json \
  --infile tests/samples/<modality>.json \
  --img_dir tests/media/448 \
  --out_dir <work_dir> \
  --gpu_per_parallel 1 --parallel_per_task 1 \
  --max_new_tokens 2048

Checklist

  • Updated mmeval/registery.py (if new model)
  • Updated README (if new model/dataset)
  • Code follows existing style

ZTWHHH and others added 2 commits June 10, 2026 15:23
- mmeval/infer/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria}.py
- env_files/{kimi_vl,minicpm_v_4d5,glm_4d1v,ovis2,aria}_requirements.txt
- mmeval/registry.py: series_mapping + series_infer_env_mapping entries

Models:
- kimi_vl: moonshotai/Kimi-VL-A3B-Instruct, Kimi-VL-A3B-Thinking
- minicpm_v_4d5: openbmb/MiniCPM-V-4_5
- glm_4d1v: THUDM/GLM-4.1V-9B-Thinking, GLM-4.1V-9B-Base
- ovis2: AIDC-AI/Ovis2-{1B,2B,4B,8B,16B,34B}
- aria: rhymes-ai/Aria

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the official Qwen3.5 multimodal series to the same branch alongside
the previous 5 series. Qwen3.5 is a unified vision-language model (no
separate -VL suffix); architecture: Qwen3_5ForConditionalGeneration with
Qwen3VLProcessor.

Models: Qwen/Qwen3.5-{0.8B,2B,4B,9B,27B,35B-A3B,122B-A10B,397B-A17B}
and Base / FP8 / GPTQ-Int4 variants.

HF:  https://huggingface.co/Qwen/Qwen3.5-9B
GH:  https://github.com/QwenLM/Qwen3.5

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@ZTWHHH ZTWHHH changed the title Add 5 new model series: kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code Jun 10, 2026
- ovis2: registered variants narrowed to Ovis2-{1B,2B,4B,8B} (all tested).
  Larger Ovis2-{16B,34B} variants are de-registered to keep the registry
  to tested-only variants.
- qwen3d5: registered variants narrowed to Qwen3.5-{0.8B,2B,4B,9B}. Larger
  variants (27B / 35B-A3B / 122B-A10B / 397B-A17B and their Base/FP8/GPTQ
  flavors) are de-registered for the same reason; they can be reintroduced
  in a follow-up once tested.
- env_files/glm_4d1v_requirements.txt: add av (PyAV) — the Glm4v processor's
  video path requires it via torchvision.io.
ZTWHHH added 2 commits June 11, 2026 08:10
The previous parse_input concatenated all text fragments and appended
them at the end of the content list, breaking interleave order for
prompts where images appear between text chunks (e.g. 'Image A <image>
and Image B <image>'). Append text chunks in position so the multi-image
interleave path matches the official model.chat() expectations.
Previous defaults (512/1024/8192) replaced with a uniform 2048 for the
six newly added series. 2048 is the project-wide normal upper bound;
no model needs a lower limit.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds six new multimodal model series integrations to simple-mmeval, wiring each series into the registry and providing per-series inference runners plus environment requirement files (defaulting to max_new_tokens=2048).

Changes:

  • Register 6 new model series (kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5) in mmeval/registry.py and map them to env + infer entrypoints.
  • Add inference implementations for the 6 series under mmeval/infer/.
  • Add per-series environment requirement files under env_files/.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
mmeval/registry.py Adds series name→model-name registry entries and env/infer mappings for the 6 new series.
mmeval/infer/kimi_vl.py Adds inference runner for Kimi-VL-A3B variants (text + multi-image interleave).
mmeval/infer/minicpm_v_4d5.py Adds inference runner for MiniCPM-V-4_5 (text + multi-image interleave).
mmeval/infer/glm_4d1v.py Adds inference runner for GLM-4.1V (text + multi-image/video interleave).
mmeval/infer/ovis2.py Adds inference runner for Ovis2 (text + multi-image interleave).
mmeval/infer/aria.py Adds inference runner for Aria (text + multi-image interleave).
mmeval/infer/qwen3d5.py Adds inference runner for Qwen3.5 unified VL family (text + multi-image/video interleave).
env_files/kimi_vl_requirements.txt Defines per-series Python deps for kimi_vl.
env_files/minicpm_v_4d5_requirements.txt Defines per-series Python deps for minicpm_v_4d5.
env_files/glm_4d1v_requirements.txt Defines per-series Python deps for glm_4d1v.
env_files/ovis2_requirements.txt Defines per-series Python deps for ovis2.
env_files/aria_requirements.txt Defines per-series Python deps for aria.
env_files/qwen3d5_requirements.txt Defines per-series Python deps for qwen3d5.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +1 to +4
torch==2.6.0
torchvision==0.21.0
transformers @ git+https://github.com/huggingface/transformers.git@main
accelerate

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. I considered the reproducibility benefit, but reverted the fixed transformers commit pin to keep the integration closer to the official Qwen3.5 dependency setting (the model card explicitly asks for the latest transformers from main). Since there is no confirmed functional issue requiring a pinned internal commit, I prefer to follow the official setup here.

Comment thread mmeval/infer/kimi_vl.py
Comment on lines +94 to +100
text = self.processor.apply_chat_template(
user_message, add_generation_prompt=True, return_tensors="pt",
)
proc_kwargs = {"text": text, "return_tensors": "pt", "padding": True, "truncation": True}
if images:
proc_kwargs["images"] = images
inputs = self.processor(**proc_kwargs).to(self.model.device)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review. I checked this against the official Kimi-VL README, which uses the same processor.apply_chat_template(..., return_tensors="pt")processor(images=image, text=text, ...) pattern; the Kimi-VL processor accepts pre-tokenized ids here. The tests pass end-to-end, so I kept the official pattern to preserve upstream fidelity.

@ZTWHHH
ZTWHHH force-pushed the add-five-new-model-series branch from 215af03 to 2911c6b Compare June 17, 2026 02:55
- mmeval/infer/aria.py: strip a single trailing <|im_end|>.
- mmeval/infer/glm_4d1v.py: strip a single trailing <|user|>.

Useful structure such as </think>, <answer>...</answer>, and
<|begin_of_box|>...<|end_of_box|> is preserved. mmeval/infer/qwen3d5.py
does not postprocess model output.
@ZTWHHH
ZTWHHH force-pushed the add-five-new-model-series branch from 2911c6b to 28f7071 Compare June 17, 2026 03:19
@ZTWHHH
ZTWHHH merged commit f2d672a into mm-evaluation:main Jun 17, 2026
1 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants