🎙️ Acoustic Language Identification via Deep Recurrent Networks

A deep learning system for identifying Urdu, English, Urdu-English code-switched speech, and Arabic directly from raw audio.

---

📌 Overview

This project presents an end-to-end Acoustic Language Identification (ALI) framework that classifies short speech recordings into four language categories:

Class ID	Language
0	Urdu
1	English
2	Urdu-English Code-Switched Mix
3	Arabic

The system processes .wav audio files, converts them into Mel-Frequency Cepstral Coefficient (MFCC) representations, and predicts the spoken language using a recurrent neural network.

Three deep learning architectures were trained and compared:

• Deep Multi-Layer Perceptron (MLP)
• Long Short-Term Memory Network (LSTM)
• Gated Recurrent Unit Network (GRU)

The final Streamlit application uses the selected LSTM model to provide a predicted language and a confidence distribution across all four classes.

---

✨ Key Features

• Custom and crowdsourced multilingual speech dataset
• Four-class acoustic language classification
• Support for Urdu-English code-switched speech
• Automatic mono conversion and 16 kHz resampling
• MFCC-based acoustic feature extraction
• Variable-length speech sequence processing
• MLP, LSTM and GRU model comparison
• Masking and global temporal pooling
• Saved .keras models for reuse
• Interactive Streamlit dashboard
• Audio playback inside the application
• Per-class confidence visualization
• Clean dark-mode interface

---

🖼️ Application Screenshots

1. System Ready — Dashboard Idle State

The application successfully loads the trained LSTM model and waits for a .wav file.

---

2. Audio Uploaded — Inference Processing

The user uploads a multilingual speech sample and the application begins real-time MFCC extraction and recurrent model evaluation.

---

3. Prediction and Confidence Distribution

Example result for an English audio recording:

• Predicted class: English
• Confidence: 81.27%
• The dashboard also displays probabilities for Urdu, Urdu-English Mix and Arabic.

---

🧠 Problem Statement

Modern voice systems often assume that each recording contains one clean, monolingual language. This assumption breaks in multilingual environments where speakers naturally switch between Urdu and English or record speech using different devices, accents and background conditions.

The project addresses this problem by classifying speech directly from its acoustic patterns without requiring transcription or text-based language detection.

Potential applications include:

• multilingual voice assistants;
• automated call routing;
• speech translation pipelines;
• localized accessibility tools;
• conversational AI systems;
• language-aware speech preprocessing.

---

📊 Dataset Profile

The project was trained on 873 audio recordings collected from custom voice samples and crowdsourced sources.

Language	Samples	Dataset Share
Urdu	285	32.65%
English	262	30.01%
Urdu-English Mix	233	26.69%
Arabic	93	10.65%
Total	873	100%

Dataset Availability

The raw dataset is not included publicly because it contains human voice recordings and may involve privacy, consent, licensing and repository-size considerations.

A description of the dataset is available in:

dataset/README.md

The dataset may be shared for academic review upon reasonable request, subject to contributor consent and source-license restrictions.

---

🔬 Feature Engineering

The audio preprocessing pipeline standardizes each recording before classification.

Common Audio Processing

Raw WAV Audio
      ↓
Mono Channel Conversion
      ↓
16,000 Hz Resampling
      ↓
13 MFCC Coefficients

MLP Feature Track

For the MLP baseline, the system calculates:

• 13 base MFCC coefficients
• first-order delta coefficients
• second-order delta-delta coefficients
• mean and standard deviation summaries

This produces a fixed vector of:

78 features per recording

Sequential Feature Track

For LSTM and GRU models, temporal order is preserved:

(Time Steps, 13 MFCC Features)

During training, sequences were post-padded to a maximum length of:

(720, 13)

A masking layer prevents padded zero values from affecting recurrent learning.

---

🏗️ Model Architectures

Model 1 — Deep MLP Baseline

Input: 78 Features
      ↓
Dense 256 + Batch Normalization + Dropout
      ↓
Dense 128 + Batch Normalization + Dropout
      ↓
Dense 64 + Dropout
      ↓
Softmax Output: 4 Classes

The MLP achieved the highest raw test accuracy but showed clear overfitting because flattened statistics discard temporal speech order.

Model 2 — LSTM Sequence Model

Input Sequence
      ↓
Masking Layer
      ↓
LSTM (64 Units, return_sequences=True)
      ↓
GlobalAveragePooling1D
      ↓
Dropout
      ↓
Dense 32
      ↓
Softmax Output: 4 Classes

The LSTM was selected for deployment because it achieved the highest macro precision while preserving long-range acoustic patterns.

Model 3 — GRU Sequence Model

Input Sequence
      ↓
Masking Layer
      ↓
GRU (64 Units, return_sequences=True)
      ↓
GlobalAveragePooling1D
      ↓
Dropout
      ↓
Dense 32
      ↓
Softmax Output: 4 Classes

The GRU provided a more compact recurrent alternative but achieved lower evaluation scores on this dataset.

---

📈 Model Performance

The models were evaluated on 175 unseen test recordings.

Model	Test Accuracy	Macro Precision	Macro Recall	Macro F1
Deep MLP Baseline	74.29%	76.88%	77.12%	76.81%
Recurrent LSTM	73.14%	77.39%	76.66%	76.66%
Gated GRU	65.71%	70.54%	70.45%	68.48%

Why LSTM Was Deployed

Although the MLP produced slightly higher accuracy, its training behavior indicated stronger overfitting. The LSTM was selected because:

• it achieved the highest macro precision;
• it preserves chronological acoustic information;
• masking protects the model from padded sequence regions;
• global pooling captures information across the full recording;
• it is more suitable for variable-duration speech signals.

---

🔄 End-to-End System Pipeline

User Uploads WAV File
        ↓
Librosa Loads Audio
        ↓
Audio Resampled to 16 kHz and Converted to Mono
        ↓
13 MFCC Features Extracted
        ↓
Temporal Sequence Sent to LSTM Model
        ↓
Softmax Probability Distribution Generated
        ↓
Predicted Language and Confidence Displayed

---

🛠️ Technology Stack

Category	Technology
Language	Python
Deep Learning	TensorFlow, Keras
Audio Processing	Librosa
Numerical Computing	NumPy
Machine Learning Utilities	Scikit-learn
Data Analysis	Pandas
Visualization	Matplotlib
User Interface	Streamlit
Training Environment	Google Colab / GPU
Model Format	Keras .keras
Serialized Preprocessing	Pickle

---

📁 Recommended Repository Structure

Acoustic-Language-Identification/
│
├── app.py
├── README.md
├── requirements.txt
├── .gitignore
│
├── best_lstm_model.keras
├── baseline_mlp_model.keras
├── best_gru_model.keras
├── scaler.pkl
│
├── notebooks/
│   └── Deep_Learning_Project.ipynb
│
├── dataset/
│   └── README.md
│
├── screenshots/
│   ├── 01-dashboard-idle.png
│   ├── 02-audio-processing.png
│   └── 03-english-prediction.png
│
└── docs/
    ├── project-report.pdf
    └── project-presentation.pdf

The deployed Streamlit application expects best_lstm_model.keras and scaler.pkl to remain in the same directory as app.py.

---

🚀 Running the Application Locally

1. Clone the Repository

git clone https://github.com/fayzliaqat/Acoustic-Language-Identification.git
cd Acoustic-Language-Identification

2. Create a Virtual Environment

Windows

python -m venv .venv
.venv\Scripts\activate

macOS / Linux

python3 -m venv .venv
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

4. Run the Streamlit App

streamlit run app.py

The application will normally open at:

http://localhost:8501

5. Upload a Test File

Upload a mono or stereo .wav recording. The application automatically resamples it to 16 kHz and displays:

• predicted language;
• highest confidence score;
• confidence percentage for every class;
• audio playback controls.

---

📦 Suggested requirements.txt

streamlit
tensorflow
numpy
librosa
scikit-learn
pandas
matplotlib
soundfile

---

📓 Training Notebook

The complete experimentation and model-training pipeline is available in:

notebooks/Deep_Learning_Project.ipynb

The notebook includes:

• dataset loading;
• filename-based label extraction;
• feature engineering;
• class distribution analysis;
• train/validation/test splitting;
• MLP training;
• LSTM training;
• GRU training;
• learning curves;
• confusion matrices;
• macro evaluation metrics;
• model serialization.

---

📄 Documentation

Supporting documentation is available in:

docs/project-report.pdf
docs/project-presentation.pdf

The report explains the research background, methodology, architecture, evaluation and limitations. The presentation provides a concise visual summary of the project.

Remove student registration IDs or any sensitive information before publishing academic documents publicly.

---

⚠️ Limitations

• The dataset is relatively small for speech deep learning.
• Arabic has fewer samples than the other classes.
• Background noise and microphone variation can affect predictions.
• Urdu and Urdu-English code-switched speech may overlap acoustically.
• The system supports only four language classes.
• The model accepts .wav input only.
• Speaker-independent generalization requires broader testing.
• Confidence scores are model probabilities, not guaranteed certainty.
• The current application performs classification only and does not transcribe speech.

---

🔮 Future Improvements

• Add pitch shifting and background-noise augmentation
• Expand the Arabic class and balance all categories
• Add spectrogram-based CNN or CRNN models
• Fine-tune Whisper, Wav2Vec 2.0 or HuBERT
• Add microphone recording directly inside the app
• Add multilingual speech transcription
• Add real-time streaming classification
• Add downloadable inference reports
• Deploy the application online
• Add automated tests and continuous integration
• Publish an anonymized and consent-cleared dataset subset

---

🔐 Privacy and Responsible Use

Human voice recordings may contain personal or biometric information.

Before sharing the dataset:

• obtain consent from contributors;
• remove names and identifying metadata;
• verify licensing for external recordings;
• avoid publishing sensitive voice notes;
• document the collection source and intended use;
• provide a clear process for data removal requests.

The system should be treated as an academic prototype and should not be used for high-stakes decisions.

---

🏷️ Suggested GitHub Topics

deep-learning
speech-processing
audio-classification
language-identification
lstm
gru
tensorflow
keras
librosa
streamlit
mfcc
code-switching
urdu
arabic
python

---

👥 Project Team

• Fayz Liaqat
• Syed Muhammad Imad
• Abdulrehman Mirza

Academic Supervision

• Faizan Younas

---

📜 Academic Context

This project was developed as part of a university Deep Learning course and explores practical multilingual speech classification through custom dataset engineering, comparative neural architectures and interactive deployment.

---

Built with speech signals, recurrent networks, and an unreasonable number of MFCC matrices. 🎙️