LMR-CL : Learning Modality-Fused Representations with Contrastive Loss for Multimodal Emotion Recognition

Project Summary

---

1. Duration: 2023.02 ~ 2023.07
2. Role: Project Leader, Text Feature Extraction, Contrastive Loss Design & Modeling
3. Tech Stack: Pytorch, HuggingFace
4. Results & Awards:
   • Patent Filed: Device and Method for Recognizing Emotions Based on Multimodal Data Fusion (Application No.: 10-2023-0160075)
   • ETRI Human Understanding AI Paper Competition -- Minister of Science and ICT Award [link]
   • Published Paper [link]
5. Overview: A deep-learning-based emotion recognition model that classifies 7 emotions (including neutral) from multimodal data -- text, audio, and biosignals -- collected during free conversation between two speakers using the KEMDy20 dataset. We propose Intra-modal and Inter-modal Contrastive Losses to align multimodal representations and achieved the highest macro F1-score in the competition.
6. Press: **[news1] [news2] [news3]**

---

The "Han Hye-sung" team, composed of students from the Graduate School of Software Convergence at Kyung Hee University, won the Minister of Science and ICT Award at the '2nd ETRI Human Understanding AI Paper Competition' held by the Electronics and Telecommunications Research Institute (ETRI).

(link : https://www.gttkorea.com/news/articleView.html?idxno=5685)

paper link : https://www.dbpia.co.kr/pdf/pdfView.do?nodeId=NODE11488658&googleIPSandBox=false&mark=0&ipRange=false&b2cLoginYN=false&isPDFSizeAllowed=true&accessgl=Y&language=ko_KR&hasTopBanner=true

---

Repository Structure

LMR-CL/
├── config.py              # Configuration and hyperparameters (CLI arguments)
├── utils.py               # Utility functions and custom loss modules (DiffLoss, CMD, IAMC, IEMC, F1_Loss)
├── dataloader.py           # PyTorch Dataset & DataLoader for BERT and WAV modalities
├── train.py               # Unified training entry point (supports --finetuning flag)
├── models/                # Model architectures
│   ├── bert.py            # LMR model (Text + Biosignal + Acoustic with contrastive loss)
│   └── wav.py             # Wav2Vec2-based audio classification model
├── solvers/               # Training and evaluation logic
│   ├── solver_bert.py     # Solver for text (BERT) modality training
│   └── solver_wav.py      # Solver for audio (WAV) modality training
├── Data/                  # Data preprocessing and dataset creation
│   ├── create_dataset.py  # KemDY20 dataset builder (bert/wav/wav_finetuning)
│   └── preprocess/
│       └── bio_feature_extractor.py  # Biosignal feature extraction pipeline
├── checkpoints/           # Saved model checkpoints
├── environment.yml        # Conda environment specification
├── LICENSE                # MIT License
└── README.md

Key changes from previous structure:

• Single-file folders (config/, utils/, dataloaders/) were flattened to root-level scripts (config.py, utils.py, dataloader.py).
• The train/ folder (with separate scripts for wav, wav_finetuning, and bert) was merged into a single train.py with a --finetuning flag.
• solver_wav_finetuning.py was merged into solver_wav.py.
• previous_code_files/ (legacy code) was removed.
• Import paths were simplified (e.g., from config import get_config instead of from config.config import get_config).

Prerequisites

• Conda (Miniconda or Anaconda)
• Python 3.7+
• CUDA 10.1 (for GPU-enabled training)

Setup

1. Clone the repository:

   git clone https://github.com/hannn0403/LMR-CL.git
   cd LMR-CL

2. Create the Conda environment:

   conda env create -f environment.yml
   conda activate lmr_cl

Data Preparation

1. Download the KEMDy20_v1_1 dataset and place it in the Data/ directory:

   Data/
   └── KEMDy20_v1_1/
       ├── audio/
       ├── text/
       └── biosignal/
   

2. config.py will look for this path by default; adjust --data_dir if your directory differs.

Configuration

All hyperparameters and experiment settings are defined via CLI arguments in config.py. Key options include:

--mode [train|eval]         # Run mode
--data kemdy20             # Dataset name
--use_bert True            # Enable text modality
--use_cmd_sim True         # Use contrastive similarity loss
--batch_size 256
--n_epoch 3
--learning_rate 1e-4
--optimizer Adam
--activation relu
--diff_weight 0.1          # Contrastive loss weight
--recon_weight 0.3         # Reconstruction loss weight

To view all options:

python -c "from config import get_config; get_config(parse=True)"

Training

Audio (WAV) Training

python train.py \
  --mode train \
  --data kemdy20 \
  --name wav_experiment

Audio (WAV) Fine-Tuning

python train.py --finetuning \
  --mode train \
  --data kemdy20 \
  --name wav_ft_experiment

Text (BERT) Training

BERT training is handled directly through the solver:

python solvers/solver_bert.py \
  --mode train \
  --data kemdy20 \
  --use_bert True \
  --name bert_experiment

Checkpoints will be saved under checkpoints/<experiment_name>/.

Evaluation

Use the corresponding solver to evaluate on the validation/test split:

# BERT evaluation
python solvers/solver_bert.py \
  --mode eval \
  --load_checkpoint checkpoints/bert_experiment/best_model.pth

# WAV evaluation
python solvers/solver_wav.py \
  --mode eval \
  --load_checkpoint checkpoints/wav_experiment/best_model.pth

License

This project is licensed under the MIT License. See the LICENSE file for details.

References

• Paper: Han et al., Learning Modality-Fused Representations with Contrastive Loss for Multimodal Emotion Recognition.
• Competition: Minister of Science and ICT Award, 2nd ETRI Human Understanding AI Paper Competition.
• Dataset: KEMDy20

---