Project Page | Project Brief | Roadmap

TL;DR: CutVerse is a compositional GUI-agent benchmark for media post-production editing, targeting long-horizon, cross-application, multimodal workflows in real professional software.

Updates

• 2026-06-08: Merged the runnable OSWorld-Win benchmark code into the public repository.
• 2026-05-09: Upgraded repository homepage to feature the paper figures and a promotion-ready layout.
• 2026-05-08: Repository homepage aligned with paper scope and benchmark statistics.
• 2026-04-24: Public intro repository initialized.

Why CutVerse

CutVerse benchmarks what current GUI-agent evaluations often miss: dense creative interfaces, strict temporal synchronization, and cross-modal alignment under long trajectories.

• Real software environments instead of synthetic task abstractions.
• End-to-end workflows from asset preparation to export.
• Milestone-driven trajectory verification beyond binary pass/fail.
• Scalable Windows-VM infrastructure for reproducible online evaluation.

Task and Software Coverage

Infrastructure

CutVerse is built as a robust and scalable benchmark stack:

1. Windows VM execution engine with resettable checkpoints.
2. Parser that synchronizes screen recordings and low-level interaction logs.
3. Milestone QA evaluator for fine-grained online trajectory assessment.

CutVerse overview

Data and evaluation pipeline

Runnable Benchmark Code

This repository now also contains the Windows OSWorld-style benchmark harness used to drive real desktop environments through the Local Engine computer-use API.

Component	Default URL / port	Runs on	Purpose
Agent FastAPI	http://127.0.0.1:8326	host PC	/models, /run_task, planning loop, model calls
Local Engine	http://<vm-ip>:8324 or http://127.0.0.1:8324	VM or host PC	screen, screenshot, mouse, keyboard, clipboard actions
Electron UI	Vite 3000	host PC	BenchmarkRunner UI and VNC viewer
VNC proxy	ws://127.0.0.1:16080	host PC	noVNC WebSocket bridge to the VM

Model API Base URL is not the Agent URL. Configure model endpoints in the UI Base URL field or in .env; the UI itself talks to the Agent FastAPI.

Install

git clone https://github.com/CUC-MIPG/CutVerse.git
cd CutVerse

uv sync --extra all
cd engineering_agent\frontend
pnpm install
cd ..\..

Create local configuration:

copy .env.example .env
notepad .env

Do not commit .env. Put real API keys in .env or enter them in the UI.

Start the UI and Agent

engineering_agent\scripts\start_all.bat

This starts:

• uvicorn agent.api:app on AGENT_HOST:AGENT_PORT
• Electron BenchmarkRunner frontend

Default AGENT_PORT is 8326. If another project already uses that port:

set AGENT_PORT=18326
set VITE_AGENT_URL=http://127.0.0.1:18326
engineering_agent\scripts\start_all.bat

VM Mode

Use this when actions should run inside a Hyper-V Windows VM.

1. Start the VM, for example UseIt-Dev-VM-14.
2. Ensure the in-VM scheduled task UseItLocalEngine is running.
3. Confirm Local Engine health from the host:

uv run osworld-bench-test-vm --vm-name UseIt-Dev-VM-14

The smoke test checks Hyper-V access, VM IP discovery, GET /screen, POST /screenshot, and POST /step.

Host-PC Local Engine Mode

Use this when actions should execute on the host PC instead of in a VM.

Terminal A:

engineering_agent\scripts\start_local_engine.bat

Terminal B:

set LOCAL_ENGINE_URL=http://127.0.0.1:8324
engineering_agent\scripts\start_all.bat

In the UI, leave VM restore disabled and set Computer / Local Engine URL to http://127.0.0.1:8324.

Model Configuration

The UI API key (optional) field has priority over environment variables. If it is empty, the agent falls back to vendor-specific env vars such as:

• KIMI_API_KEY
• ANTHROPIC_API_KEY
• OPENAI_API_KEY
• GOOGLE_API_KEY / GEMINI_API_KEY
• DASHSCOPE_API_KEY
• DOUBAO_API_KEY

For Kimi, set:

set KIMI_API_KEY=...
set KIMI_BASE_URL=https://api.moonshot.cn/v1

or enter both in the UI model panel. kimi-k2* profiles use temperature=1 as required by the current Kimi API behavior.

For OpenAI-compatible vLLM servers, use the UI Base URL, for example:

http://172.17.34.44:9889/v1

and set the model name to the deployed model alias.

Common Commands

# list model profiles served by the Agent
irm http://127.0.0.1:8326/models

# run a CLI task against configured VM or LOCAL_ENGINE_URL
uv run osworld-bench --task "Open Notepad and type Hello" --max-steps 10

# check env, deps, Hyper-V, VM and snapshot
uv run osworld-bench-check

# VM smoke test
uv run osworld-bench-test-vm --vm-name UseIt-Dev-VM-14

Important Paths

Path	Purpose
agent/	FastAPI, benchmark loop, config, Hyper-V VM manager
AI_Run/.../gui_v3/benchmark/	model planners and model profiles
engineering_agent/local_engine/	Local Engine server and computer-use controller
engineering_agent/frontend/	Electron + React BenchmarkRunner UI
engineering_agent/scripts/	Windows launch/build scripts
docs/P6_QUICKSTART.md	detailed Windows + Hyper-V quick start
REVIEWLOG.md	chronological fix and verification notes

Repository Status

• Included now: project overview, benchmark scope, release roadmap, and runnable Windows benchmark harness.
• Do not commit .env, real API keys, virtual environments, caches, logs, runs, screenshots, built executables, frontend dist, or VM service bundles.

Citation

@article{hu2026cutverse,
  title={CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing},
  author={Hu, Haobo and Guo, Xiangwu and Chen, Zhiheng and Gao, Difei and Liu, Haotian and Jin, Libiao and Mao, Qi},
  journal={arXiv preprint},
  year={2026}
}

Contact

For collaboration or early benchmark access, please open an issue in this repository.