Speech Emotion Recognition with Transformers

A deep learning project for Speech Emotion Recognition (SER) using Transformer-based architectures on the CREMA-D dataset.

Current Progress

Completed

• Data Pipeline

  • Download and preprocessing script for CREMA-D dataset
  • Audio loading with librosa
  • Mel spectrogram feature extraction (64 mel bins, 4-second clips)
  • Train/validation/test split (70/10/20)

• Model Architectures

  • CNN + Transformer hybrid model (CNNTransformerModel)
  • Pure Spectrogram Transformer model (SpectrogramTransformer)
  • Positional encoding, attention mechanisms, classification heads

• Training Pipeline

  • PyTorch training loop with GPU support
  • AdamW optimizer with learning rate scheduling
  • Early stopping with patience
  • Model checkpointing (best model saved)

• Evaluation Pipeline

  • Accuracy, F1-score (macro/weighted) metrics
  • Per-class accuracy analysis
  • Confusion matrix visualization
  • Classification report generation

• Exploratory Notebook

  • Data exploration and visualization
  • Audio playback functionality
  • Label distribution analysis

TODO / Future Work

• Data Augmentation

  • Time stretching, pitch shifting
  • SpecAugment (frequency/time masking)
  • Noise injection

• Model Improvements

  • Implement attention visualization
  • Add model ensembling
  • Experiment with pre-trained audio models (wav2vec2, HuBERT)

• Advanced Training

  • Cross-validation
  • Hyperparameter tuning (learning rate, model size, etc.)
  • Mixed precision training (FP16)
  • Gradient accumulation for larger effective batch sizes

• Deployment

  • ONNX export for inference
  • Real-time inference demo
  • Web API endpoint

• Documentation

  • Add detailed model architecture diagrams
  • Benchmark comparisons with other SER methods
  • Hyperparameter sensitivity analysis

---

Project Structure

audio-transformer-project/
├── data/
│   └── download_data.py        # Script to download CREMA-D dataset
├── notebooks/
│   └── module_notebook.ipynb   # Data exploration and preprocessing
├── src/
│   ├── models/
│   │   ├── __init__.py
│   │   ├── cnn_with_trans.py   # CNN + Transformer model
│   │   └── spect_transformer.py # Pure Transformer model
│   ├── preprocessing/
│   │   ├── __init__.py
│   │   ├── data_load.py        # Dataset class for loading audio
│   │   └── preprocess.py       # Audio preprocessing utilities
│   ├── train.py                # Training script
│   └── evaluate.py             # Evaluation script
├── checkpoints/                # Saved model checkpoints (gitignored)
├── results/                    # Evaluation results (gitignored)
├── requirements.txt
├── LICENSE
└── README.md

---

Dataset

CREMA-D (Crowd-sourced Emotional Multimodal Actors Dataset)

• 7,442 audio clips from 91 actors
• 6 emotion classes: angry, disgust, fear, happy, neutral, sad
• Sample rate: 16kHz

---

Installation

1. Clone the repository

git clone https://github.com/yourusername/audio-transformer-project.git
cd audio-transformer-project

2. Create virtual environment

python -m venv venv

# Windows
.\venv\Scripts\Activate

# Linux/Mac
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

4. Install FFmpeg (required for audio processing)

• Windows: Download from ffmpeg.org and add to PATH
• Linux: sudo apt install ffmpeg
• Mac: brew install ffmpeg

---

Usage

1. Download Dataset

python data/download_data.py

This downloads the CREMA-D dataset and saves audio files to data/crema_d/audio/ with labels in data/crema_d/labels.csv.

2. Preprocess Features

Run the Jupyter notebook to generate mel spectrogram features:

jupyter notebook notebooks/module_notebook.ipynb

This saves preprocessed features to src/features/X.npy and src/features/y.npy.

3. Train Model

python src/train.py --epochs 50 --batch_size 32 --lr 0.001

Training Arguments:

Argument	Default	Description
--epochs	50	Number of training epochs
--batch_size	32	Batch size
--lr	0.001	Learning rate
--features_dir	src/features	Path to features directory
--checkpoint_dir	checkpoints	Path to save checkpoints

4. Evaluate Model

python src/evaluate.py --checkpoint checkpoints/best_model.pt

Evaluation Arguments:

Argument	Default	Description
--checkpoint	checkpoints/best_model.pt	Path to model checkpoint
--batch_size	32	Batch size for inference
--output_dir	results	Directory to save results
--plot	True	Generate confusion matrix plot

---

Model Architectures

1. CNN + Transformer (CNNTransformerModel)

Input: Mel Spectrogram (64 x T)
    ↓
CNN Feature Extractor (3 conv layers)
    ↓
Linear Projection → d_model
    ↓
Positional Encoding
    ↓
Transformer Encoder (4 layers)
    ↓
Global Average Pooling
    ↓
Classification Head → 6 classes

2. Spectrogram Transformer (SpectrogramTransformer)

Input: Mel Spectrogram (64 x T)
    ↓
Linear Projection (per frame) → d_model
    ↓
[CLS] Token + Positional Encoding
    ↓
Transformer Encoder (6 layers, pre-norm)
    ↓
CLS Token Output
    ↓
Classification Head → 6 classes

Model Hyperparameters:

Parameter	CNN+Transformer	Spectrogram Transformer
d_model	256	256
nhead	8	8
num_layers	4	6
dim_feedforward	512	512
dropout	0.1	0.1

---

Requirements

• Python 3.10+
• PyTorch 2.0+
• CUDA (optional, for GPU acceleration)

See requirements.txt for full dependencies.

---

License

This project is licensed under the terms specified in the LICENSE file.

---

Acknowledgments

• CREMA-D Dataset
• TensorFlow Datasets
• Librosa