MSD Benchmark: Multi-Factor Sequential Disentanglement

MSD is a benchmark for evaluating disentangled representation learning on sequential data (e.g., videos, audio, time-series). It supports datasets with both static and dynamic factors, and includes tools for automatic annotation, model evaluation, and visualization.

---

🛠️ Installation

We recommend using a conda environment:

conda create -n msd python=3.9
conda activate msd
pip install -r requirements.txt

After installation, edit the meta.yaml file and set the msd_root field to the absolute path where all outputs (e.g., logs, models, results) should be saved.

---

📂 Components

MSD is modular and consists of the following major components:

• Datasets: Loaders for synthetic and real-world data. See docs/datasets.md
• Methods: Plug-and-play training pipelines for disentanglement models. See docs/methods.md
• Latent Exploration: Analyzes latent codes to discover factor-aligned dimensions. See docs/latent_exploration.md
• Evaluation: Metrics and visualizations for judging model performance. See docs/evaluation_metrics.md
• VLM Tools: Use vision-language models (e.g., GPT-4) for automatic annotation and factor classification. See docs/vlm_module.md
• Configuration: YAML-driven setup for all runs. See docs/configuration.md

---

🚀 Running Experiments

All training and evaluation is handled by the run.py script.

Train a model:

python run.py --run_config configurations/methods/ssm_skd/ssm_skd_sprites.yaml --train

Evaluate a model:

python run.py --run_config configurations/methods/ssm_skd/ssm_skd_sprites.yaml --eval

The model will be automatically loaded from the checkpoint_dir path specified in the configuration file.

You must also specify a meta.yaml file with global variables like msd_root. This is optional if your meta.yaml is located at configurations/meta.yaml.

---

🧠 Automatic Annotation

Use auto_annotate.py to automatically discover and label factor spaces in new datasets using a vision-language model.

python auto_annotate.py \
  --ds_path /path/to/dataset.h5 \
  --subset train \
  --n_exploration 500 \
  --n_annotation 500 \
  --out_dir /path/to/output \
  --ds_name my_dataset

For details, see docs/vlm_module.md

---

📎 Citation

TBD.

---

📬 Contact

For questions or contributions, feel free to open an issue or contact the authors.