ICU-Bench: Benchmarking Continual Unlearning in Multimodal Large Language Models

This repository provides the dataset and experimental code for the paper "ICU-Bench: Benchmarking Continual Unlearning in Multimodal Large Language Models".

This repository uses a single entry script: run.sh. All training and evaluation commands are already prepared in that file as templates.

1. Environment Setup

Option A: Conda (recommended)

cd /data/2026NIPS_ICUBench_code
conda env create -f environment.yml
conda activate icubench

If environment.yml is not complete for your machine, install from requirements.txt:

pip install -r requirements.txt

Option B: Existing virtual environment

cd /data/2026NIPS_ICUBench_code
pip install -r requirements.txt

2. Project Structure (runtime related)

• run.sh: unified launcher and command templates.
• base_method/: training scripts for methods (GA, GA-Diff, KL-Min, NPO, NPO-Diff, MANU, MMU).
• base_method/eval/: evaluation scripts.
• dataset/: ICUBench data root.
• vanilla_model/: base model directory.
• adapter_output/: output adapters results.
• eval_output/: output of evaluation scripts (e.g. generated responses).

3. How to Run

All runnable commands are in run.sh, but they are commented out by default. You need to do 3 things:

1. Edit path and runtime variables in run.sh.
2. Uncomment the command block you want to run.
3. Execute bash run.sh.

3.1 Edit Variables in run.sh

At the top of run.sh, check these variables first:

• BASE_MODEL / VANILLA_DIR: base model path.
• ICU_BENCH_DIR: dataset path (dataset).
• OUTPUT_ROOT: output root (adapter_output).
• GPU_IDS: visible GPU IDs, e.g. "0" or "0,1".
• BATCH_IDX, TASK_IDX: used by evaluation scripts.

4. Training Commands (in run.sh)

Available method blocks:

• GA
• GA-Diff
• KL-Min
• NPO
• NPO-Diff
• MANU
• MMU

Uncomment one block (or multiple blocks, sequentially) and run:

bash run.sh

5. Evaluation Commands (in run.sh)

Available evaluation blocks:

• Forget evaluation: base_method/eval/task_forget_eval.py
• Retain evaluation: base_method/eval/batch_retain_eval.py
• Generation evaluation: base_method/eval/generation_eval.py
• Real-world evaluation: base_method/eval/realworld_eval.py

For each evaluation block:

1. Set --adapter_root to your trained adapter directory.
2. Confirm BATCH_IDX and TASK_IDX.
3. Uncomment the block and run bash run.sh.

6. Minimal Example (NPO + Forget Eval)

In run.sh:

1. Set GPU_IDS, BASE_MODEL, ICU_BENCH_DIR, OUTPUT_ROOT.
2. Uncomment the NPO training block.
3. Run:

bash run.sh

After training:

1. Comment back the NPO block (optional).
2. Uncomment Forget evaluation block.
3. Ensure --adapter_root "${NPO_ADAPTER}" is correct.
4. Run again:

bash run.sh

7. Notes

• run.sh starts with set -e; execution stops immediately if any command fails.
• realworld_eval.py requires lmms-eval installed and configured.
• You can keep this repo as a command hub by only editing variables and uncommenting blocks.