PhotoFramer: Multi-modal Image Composition Instruction

Zhiyuan You, Ke Wang, He Zhang, Xin Cai, Jinjin Gu, Tianfan Xue^#, Chao Dong^#, Zhoutong Zhang

^#Corresponding author

Homepage | Model Weights (preview) | Paper

Composition Instruction Task Paradigm

Composition Instruction Results

🛠️ Setup

• Setup environment.

git clone git@github.com:zhiyuanyou/PhotoFramer-Code.git
cd PhotoFramer-Code
conda create -n photoframer python=3.10 -y
conda activate photoframer
pip install -r requirements.txt
pip install flash_attn==2.5.8 --no-build-isolation

• Download pretrained Bagel weights from BAGEL-7B-MoT, then arrange the folders as follows.

|-- PhotoFramer-Code
  |-- ModelZoo
    |-- BAGEL-7B-MoT

• Download pretrained PhotoFramer-preview weights, then arrange the folders as follows.

|-- PhotoFramer-Code
  |-- ModelZoo
    |-- PhotoFramer-preview

💪🏻 Training & Inference

Datasets

We first release the dataset construction scripts in build_dataset.

Please note that these scripts are still rough and are mainly intended for users who would like to construct their own training datasets. We will further clean up and reorganize them in future updates.

Training

We will update this part after we release the training datasets.

Inference

# Args: <config.yaml> <shard_id> <total_shards> <gpu_id>

sh scripts/infer.sh test_images/configs/test_local.yaml 0 1 0

• The results of the three tasks are saved in test_images/results/shift, test_images/results/zoomin, and test_images/results/viewchange/, respectively.

• Under each directory, there are three types of files, i.e., *_orig.jpg (original image), *_edit.jpg (editted image, i.e., good-composition example), and *_reason.txt (text instruction).

• Try tools/vis_results_load.html to better view the results.

🤝 Acknowledgement

This work is based on Bagel. Sincerely thanks for this awesome work.

⭐️ Citation

If you find our work useful for your research and applications, please cite using the BibTeX:

@inproceedings{photoframer,
    title={PhotoFramer: Multi-modal Image Composition Instruction},
    author={You, Zhiyuan and Wang, Ke and Zhang, He and Cai, Xin and Gu, Jinjin and Xue, Tianfan and Dong, Chao and Zhang, Zhoutong},
    booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2026}
}