MUSE: A Unified Agentic Harness for MLLMs

A multimodal unified structured execution harness that improves MLLMs by optimizing the execution scaffold rather than the model itself.

---

Why MUSE?

Even frontier MLLMs catastrophically fail on tasks humans solve in a fraction of a second. The dominant response has been to push harder on the model itself (instruction-tune, train visual CoT, RL self-verify). Rather than retraining the model, we ask a complementary question: how much capability can be elicited from a frozen MLLM purely by improving the execution scaffold around it? In this work, we:

• propose MUSE, a multimodal unified structured execution harness that improves frozen MLLMs by optimizing the execution scaffold rather than the model itself. To the best of our knowledge, MUSE is the first unified agentic harness tailored for frozen MLLMs in vision-centric tasks.

• introduce a modular inference pipeline that unifies task representation, visual preprocessing, structured parsing, deterministic verification, and iterative repair under a single abstraction.

• demonstrate consistent improvements across diverse benchmarks and models. These findings highlight the agentic multimodal harness as a critical yet underexplored design dimension, offering an orthogonal avenue for improving MLLMs beyond model-centric optimization.

Quick Start

bash vir_env_setup.sh
conda activate MUSE

# 2. Set provider keys (only the providers you actually call)
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

# 3. Run BASE vs MUSE on a small slice
bash run_examples.sh

Extending — Add Your Own Task

The harness layer never branches on dataset in the runner — you wire in via three registries:

# 1) Adapter: yield RolloutTask per row
class MyAdapter:
    name = "my_dataset"
    def iter_tasks(self, data_root, *, llm_input, split=None, options=None):
        for row in load(data_root):
            yield RolloutTask(dataset=self.name, task_type="my_task",
                              prompt_text=row["q"], image_paths=(row["img"],),
                              ground_truth={"answer": row["a"]}, ...)

# 2) Task hooks: prompt + token budget + repair hints + tool budget
register_task_runtime_hooks("my_task", TaskRuntimeHooks(
    extract_answer_for_verify=_default_extract,
    max_llm_tokens=384,
    build_initial_prompt=my_prompt_builder,
    harness_repair_action_hints=_json_qa_repair_action_hints,
    required_tools=(),
))

# 3) Outcome policy: verify model output
class MyPolicy:
    def verify(self, task, raw, moves) -> VerifyResult:
        ok = my_check(raw, task.ground_truth["answer"])
        return VerifyResult(ok=ok, detail="..." if ok else
                            "Verifier: answer did not pass. Re-examine and revise.",
                            error_type=None if ok else VerifyErrorType.TASK_LOGIC_ERROR)
register_outcome_policy_for_task_type("my_task", MyPolicy())

Citation

If you use this codebase or build on the design, please cite:

@article{lu2026muse,
  title   = {MUSE: A Unified Agentic Harness for MLLMs},
  author  = {Lu, Jianglin and Wang, Hailing and Ma, Xu and Dong, Qihua and Zhang, Mingyuan and Wang, Yizhou and Fu, Yun},
  journal = {arXiv preprint arXiv:2606.03005},
  year    = {2026}
}

Acknowledgments

Built on LiteLLM and the public datasets VSP, BLINK, CoMT, and TIR-Bench.