LychSim is a highly controllable, interactive simulation framework built on Unreal Engine 5, designed to lower the technical barrier of using a modern game engine for computer vision research. It is organized around three pillars: a streamlined Python API that abstracts away engine complexities, a procedural data pipeline that generates diverse high-fidelity environments — with controllable out-of-distribution visual challenges paired with rich 2D / 3D ground truth — and a native MCP integration that turns the simulator into a closed-loop playground for reasoning agentic LLMs. It powers downstream applications spanning synthetic data generation, RL-based adversarial examiners, and interactive, language-driven scene layout generation.
from lychsim.api import LychSim
sim = LychSim(server_name="localhost", port=9000, width=1920, height=1080)
rgb = sim.get_cam_lit(cam_id=0)
depth = sim.get_cam_depth(cam_id=0)
annots = sim.get_obj_annots() # per-object category, location, rotation, AABB/OBB/bounds
sim.close()LychSim: A Controllable and Interactive Simulation Framework for Vision Research
Wufei Ma, Chloe Wang, Siyi Chen, Jiawei Peng, Patrick Li, and Alan Yuille
Johns Hopkins University
Important
We are looking for regular maintainers and project collaborators to join our team. Please send me an email if interested!
LychSim ships with a Model Context Protocol (MCP) server, so an LLM agent (Claude Code, Gemini CLI, …) can read and act on a live Unreal scene through the same Python API.
Example. A user prompts Claude:
Move the table next to the window.
Claude resolves the request through a sequence of MCP tool calls against the running scene:
┌─ user ─────────────────────────────────────────────────┐
│ "move the table next to the window" │
└───────────────────────┬────────────────────────────────┘
↓
list_objects() → ["Table_0", "Window_3", "Chair_1", ...]
get_object_location(["Window_3"]) → {"location": [320, -110, 90]}
get_object_location(["Table_0"]) → {"location": [ 50, 40, 90]}
set_object_location("Table_0", [310, -90, 90]) → ok
↓
┌─ scene ────────────────────────────────────────────────┐
│ Table_0 is now beside Window_3 │
└────────────────────────────────────────────────────────┘
Each tool returns plain JSON the model can reason over:
{
"status": "ok",
"outputs": [
{
"object_id": "Window_3",
"status": "ok",
"location": [320.0, -110.0, 90.0]
}
]
}LychSim ships a lychsim command-line tool for managing Unreal projects, launching them in the background, and inspecting running instances — so you don't have to drive subprocess.Popen by hand.
- Detached background launches with auto-bumped ports — multiple UE instances run side-by-side without fighting over
unrealcv.ini. - Per-handle runs registry with pid + exe-match liveness — powers
ps/logs/stopaccurately, with no ghosts after pid recycling. - Two configuration registries:
envfor projects and binaries,enginefor UE installs, both with auto-discovery on each platform and an arrow-key picker (lychsim runwith no args). lychsim doctorflags configuration issues — missing paths, disabled LychSim plugins, no GPU, port conflicts.
$ lychsim env add /path/to/MyProject.uproject # register a project (or .exe, or a directory of either)
$ lychsim run MyProject # launch in background, prints ip:port when ready
$ lychsim ps # show running instances
$ lychsim logs <handle> -f # tail the redirected UE log
$ lychsim stop all # tear down, with confirmation
See the CLI reference for the full list of subcommands and flags.
We release two complementary annotation datasets on the scenes and objects used in LychSim. Both datasets can be loaded with a single datasets.load_dataset call and are integrated with the built-in procedural generation pipeline.
Object annotations
For each 3D asset that appears in our scenes we annotate its semantic category, canonical scale, and pose alignment (the calibration yaw that puts the asset's "front" along +X), together with a precomputed mesh_offset so that spawning at loc + mesh_offset lands the visual bbox bottom-center exactly on loc. These annotations are critical for producing semantically aligned ground-truth 3D object poses and for programmatic object placement and scene manipulation.
Scene-level procedural rules
For each scene we capture structural priors — navigable floor spaces, road areas, pedestrian walks, and dynamic vehicle / pedestrian trajectories — as structured records keyed to the underlying placed actors. These spatial priors guide the procedural generation process, ensuring that newly synthesized layouts remain faithful to the original scene semantics.
from datasets import load_dataset
objects = load_dataset("wufeim/lychsim_objects")
scenes = load_dataset("wufeim/lychsim_scenes")Note
The currently published versions are preview data. The full release will follow.
For a quick try-it-out — no Unreal Engine 5 install required — install LychSim straight from GitHub, then grab one of our pre-built scene binaries and launch it with lychsim run:
pip install git+https://github.com/wufeim/LychSim.gitSee the Quick Start guide for the full walkthrough.
For a full installation (UE5 setup, building scenes from source, integrating your own projects), refer to the installation guide.
LychSim's full documentation lives at wufeim.github.io/LychSim — start there for an overview of the project. From the landing page you can dive into the tutorials for step-by-step walkthroughs (installation, first scene, capturing data, MCP setup), or jump straight to the API reference for the full surface of the C++ commands and Python API/MCP packages.
See roadmap and releases here.
The following research works have been built with LychSim:
- Unreal3DSpace
- Perceptual Taxonomy
If you find our work useful for your research, please consider citing our work:
@article{ma2026lychsim,
title={LychSim: A Controllable and Interactive Simulation Framework for Vision Research},
author={Ma, Wufei and Wang, Chloe and Chen, Siyi and Peng, Jiawei and Li, Patrick and Yuille, Alan},
journal={arXiv preprint arXiv:2605.12449},
year={2026}
}
LychSim is built on the architecture of UnrealCV (Qiu et al., 2017), which exposes Unreal Engine to external Python clients. LychSim extends the plugin into a full interactive simulation framework with new functionalities, procedural generation, and native Python/MCP integration for agentic research.