ASR Adapter-Based Fine-Tuning for Low-Resource Languages

A professional implementation of parameter-efficient fine-tuning using bottleneck adapters for Automatic Speech Recognition (ASR) on the Kinyarwanda language.

📋 Table of Contents

• Overview
• Features
• Architecture
• Installation
• Quick Start
• Usage
• Project Structure
• Configuration
• Results
• Submission Checklist
• References

🎯 Overview

This project implements adapter-based fine-tuning for ASR models, specifically targeting low-resource languages like Kinyarwanda. The approach allows efficient domain adaptation while keeping the pre-trained model frozen, preventing catastrophic forgetting.

Challenge Objective

Reduce Word Error Rate (WER) on the ASR Fellowship Challenge Dataset by training small adapter modules while keeping the base ASR model frozen.

Key Benefits

✅ Parameter Efficient: Only 1-5% of parameters are trainable ✅ Prevents Catastrophic Forgetting: Base model knowledge is preserved ✅ Fast Training: Significantly faster than full fine-tuning ✅ Low Memory: Reduced GPU memory requirements ✅ Modular: Easy to swap adapters for different domains

✨ Features

• Bottleneck Adapter Implementation: Based on state-of-the-art research
• Configurable Architecture: Easy to adjust adapter parameters via YAML
• Comprehensive Evaluation: WER and CER metrics with detailed logging
• Production-Ready Code: Clean, documented, and fully typed
• Reproducible: Fixed seeds and deterministic training
• Modular Design: Separation of concerns for easy maintenance

🏗️ Architecture

Bottleneck Adapter

The adapter module consists of:

Input (hidden_size)
    ↓
Down-projection (hidden_size → bottleneck_size)
    ↓
Non-linearity (GELU)
    ↓
Dropout
    ↓
Up-projection (bottleneck_size → hidden_size)
    ↓
Residual Connection (+)
    ↓
Output (hidden_size)

Integration Strategy

Adapters are inserted after the feedforward layers in each transformer encoder layer:

Transformer Layer
├── Multi-Head Attention
├── Layer Norm
├── Feedforward Network
├── ⚡ ADAPTER ← Inserted here
└── Layer Norm

🚀 Installation

Prerequisites

• Python 3.8 or higher
• CUDA-capable GPU with 8GB+ VRAM (recommended)
• 20GB+ disk space

Setup

# Clone the repository
git clone [REPOSITORY_URL]
cd ASR

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

🏃 Quick Start

1. Download Dataset

python scripts/download_data.py --config config/config.yaml

2. Train Adapters

python scripts/train_adapters.py --config config/config.yaml

3. Evaluate Models

# Evaluate base model (no adapters)
python scripts/evaluate.py --config config/config.yaml --model_type base

# Evaluate fine-tuned model (with adapters)
python scripts/evaluate.py --config config/config.yaml --model_type finetuned

4. Generate Report

python scripts/generate_report.py \
    --config config/config.yaml \
    --name "Your Name" \
    --email "your.email@example.com" \
    --phone "+250 XXX XXX XXX" \
    --base_wer 0.45 \
    --finetuned_wer 0.32 \
    --trainable_params 5234560 \
    --total_params 317000000

📖 Usage

Training Configuration

Edit config/config.yaml to customize:

# Adapter configuration
adapter:
  type: "bottleneck"
  bottleneck:
    reduction_factor: 16  # Higher = smaller adapter
    non_linearity: "gelu"
    adapter_dropout: 0.1

# Training configuration
training:
  num_train_epochs: 10
  learning_rate: 3e-4
  per_device_train_batch_size: 4

Custom Model

To use a different base model:

model:
  name: "facebook/wav2vec2-large-xlsr-53"  # or "openai/whisper-small"

Advanced Usage

Resume Training

python scripts/train_adapters.py \
    --config config/config.yaml \
    --resume_from_checkpoint checkpoints/checkpoint-1000

Custom Evaluation

from src.models import ASRModelWithAdapters
from src.evaluation import evaluate_model

# Load model
model_wrapper = ASRModelWithAdapters(config)

# Evaluate on custom dataset
metrics = evaluate_model(
    model=model_wrapper.get_model(),
    processor=model_wrapper.get_processor(),
    dataset=your_dataset,
    device="cuda"
)

📁 Project Structure

ASR/
├── config/
│   └── config.yaml              # Main configuration file
├── src/
│   ├── data/
│   │   ├── dataset.py           # Dataset loading and preprocessing
│   │   └── data_collator.py     # Data collation for batching
│   ├── models/
│   │   ├── adapters.py          # Adapter module implementation
│   │   └── asr_with_adapters.py # Model wrapper with adapters
│   ├── training/
│   │   └── trainer.py           # Training logic
│   └── evaluation/
│       └── metrics.py           # Evaluation metrics (WER, CER)
├── scripts/
│   ├── download_data.py         # Download dataset
│   ├── train_adapters.py        # Train adapter modules
│   ├── evaluate.py              # Evaluate models
│   └── generate_report.py       # Generate PDF report
├── models/
│   ├── base/                    # Base model weights
│   └── adapters/                # Trained adapter weights
├── outputs/
│   ├── base_transcriptions.txt
│   └── finetuned_transcriptions.txt
├── requirements.txt
├── README.md
└── rapport.pdf                  # Generated report

⚙️ Configuration

Key Configuration Options

Parameter	Description	Default
model.name	Pre-trained model to use	facebook/wav2vec2-xls-r-300m
adapter.bottleneck.reduction_factor	Bottleneck size divisor	16
training.learning_rate	Learning rate for adapters	3e-4
training.num_train_epochs	Number of training epochs	10
training.fp16	Use mixed precision training	true

Adapter Size Calculation

bottleneck_size = hidden_size / reduction_factor
adapter_params = 2 * hidden_size * bottleneck_size

Example (hidden_size=1024, reduction_factor=16):
bottleneck_size = 1024 / 16 = 64
adapter_params = 2 * 1024 * 64 = 131,072 parameters per adapter

📊 Results

Expected Performance

Model	WER	CER	Trainable Params
Base Model	~45%	~20%	0 (frozen)
With Adapters	~32%	~14%	~5M (1.6%)
Improvement	-13%	-6%	Efficient!

Note: Actual results depend on dataset and training configuration.

Performance Metrics

The model is evaluated using:

• WER (Word Error Rate): Primary metric for the challenge
• CER (Character Error Rate): Additional character-level metric

📝 Submission Checklist

Required Files

• base_transcriptions.txt - Transcriptions from base model
• finetuned_transcriptions.txt - Transcriptions from fine-tuned model
• src/ - Complete source code (documented and reproducible)
• models/base/ - Base model weights
• models/adapters/ - Trained adapter weights
• rapport.pdf - Comprehensive report including:
  • Full name and contact information
  • WER results (base vs fine-tuned)
  • Number of trainable parameters
  • Architecture description
  • Training strategy
  • Step-by-step reproduction instructions

Verification

# Verify base model is frozen
python -c "from src.models import ASRModelWithAdapters; import yaml; \
config = yaml.safe_load(open('config/config.yaml')); \
model = ASRModelWithAdapters(config); \
model.print_trainable_parameters()"

# Should show: Trainable parameters: ~5M (1-2% of total)

🔬 Technical Details

Adapter Initialization

Adapters are initialized near zero to start close to the identity function:

# Near-zero initialization
nn.init.normal_(down_project.weight, std=1e-3)
nn.init.zeros_(down_project.bias)
nn.init.normal_(up_project.weight, std=1e-3)
nn.init.zeros_(up_project.bias)

Training Strategy

1. Warmup Phase: Linear warmup for 10% of training
2. Main Training: Cosine learning rate decay
3. Early Stopping: Patience of 3 epochs on validation WER
4. Mixed Precision: FP16 for faster training (if GPU supports)

Data Processing

• Audio resampled to 16kHz
• Text normalized (lowercase, unicode normalization)
• Dynamic padding for efficient batching
• CTC loss with blank token

📚 References

1. Thomas, B., Kessler, S., and Karout, S. (2022). "Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognition." arXiv:2202.03218.

2. Hou, W., et al. (2021). "Exploiting Adapters for Cross-lingual Low-resource Speech Recognition." arXiv:2105.11905.

3. Houlsby, N., et al. (2019). "Parameter-Efficient Transfer Learning for NLP." arXiv:1902.00751.

4. Baevski, A., et al. (2020). "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations." NeurIPS 2020.

🤝 Contributing

This is a submission for the ASR Fellowship Challenge. For questions or issues, please contact the candidate.

📄 License

This project is submitted as part of the ASR Fellowship Challenge evaluation process.

👤 Author

[Your Name]

• Email: [your.email@example.com]
• Phone: [+250 XXX XXX XXX]
• Date: 2025-11-26

🙏 Acknowledgments

• Digital Umuganda for providing the Kinyarwanda dataset
• Meta AI for the Wav2Vec2 XLS-R model
• HuggingFace for the Transformers library
• The adapter research community

---

Good luck with the ASR Fellowship Challenge! 🚀