Temporal Action Localization for Distracted Driver Behavior Detection

A complete implementation of a temporal action localization framework for detecting hazardous driving behaviors from in-cabin video streams. This system combines VideoMAE-based feature extraction with an Augmented Self-Mask Attention (AMA) detector, enhanced by a Spatial Pyramid Pooling–Fast (SPPF) module to capture multi-scale temporal features.

Abstract

The identification of hazardous driving behaviors from in-cabin video streams is essential for enhancing road safety and supporting the detection of traffic violations and unsafe driver actions. However, current temporal action localization techniques often struggle to balance accuracy with computational efficiency.

This work develops and evaluates a temporal action localization framework tailored for driver monitoring scenarios, particularly suitable for periodic inspection settings such as transportation safety checkpoints or fleet management assessment systems. Our approach follows a two-stage pipeline that combines VideoMAE-based feature extraction with an Augmented Self-Mask Attention (AMA) detector, enhanced by a Spatial Pyramid Pooling–Fast (SPPF) module to capture multi-scale temporal features.

Project Objective

The objective of this project is to successfully implement the distracted driver action localization pipeline proposed by Zhang et al., which combines YOLO-based region extraction, VideoMAE feature encoding, and the Augmented Self-Mask Attention (AMA) mechanism for temporal modeling. In the context of enhancing road safety, this framework serves as the foundation for detecting traffic violations and dangerous driving behaviors from in-cabin video streams, particularly suited for deployment in periodic vehicle inspection stations or transportation safety assessment systems.

Beyond reproducing the original system, the project aims to develop a lighter and more efficient variant by experimenting with different VideoMAE backbone sizes and replacing the neck component - originally implemented as an identity or feature pyramid structure - with a Spatial Pyramid Pooling Fast (SPPF) module. This modification is intended to enrich multi-scale spatial representations prior to the self-attention stage, thereby improving the model's ability to capture subtle driver actions while maintaining favorable computational efficiency.

Overview

This repository provides a full end-to-end pipeline for detecting and classifying driver actions in in-cabin video footage. The system consists of four main components:

1. Preprocessing: Driver detection and video cropping using YOLOv5
2. Feature Extraction: Video feature extraction using VideoMAE models (ViT-Base and ViT-Giant)
3. Action Detection: Temporal action detection using AMA (Augmented Self-Mask Attention) Transformer with SPPF enhancement
4. Post-processing: Action proposal filtering, ensemble methods, and submission file generation

Additionally, a web-based demonstration system is included for experimental deployment in periodic inspection or transportation monitoring environments.

Project Structure

Final_Source/
├── AMA/                           # AMA model implementation
│   ├── configs/                  # Model configuration files (vitb, vitg, sppf variants)
│   ├── configs_test/            # Test/deployment configurations
│   ├── libs/                     # Core libraries
│   │   ├── core/                 # Configuration management
│   │   ├── datasets/             # Dataset loaders (AICity, ActivityNet, Charades, THUMOS14)
│   │   ├── modeling/             # Model architectures (backbones, necks, meta-archs)
│   │   └── utils/                # Utilities (NMS, metrics, training utils)
│   ├── train.py                  # Training script
│   ├── eval.py                   # Evaluation/inference script
│   └── requirements.txt          # Python dependencies
│
├── feature_extraction/           # VideoMAE feature extraction module
│   ├── models/                  # Model definitions (ViT-Base, ViT-Giant)
│   ├── dataset/                 # Dataset loaders and video transforms
│   ├── inference_video_feature_vitb.py  # ViT-Base feature extraction
│   ├── inference_video_feature_vitg.py  # ViT-Giant feature extraction
│   ├── extract_tad_feature.py    # Temporal action detection feature extraction
│   └── requirements.txt
│
├── preprocess/                   # Data preprocessing pipeline
│   ├── yolov5/                  # YOLOv5 for driver detection and tracking
│   ├── get_json_for_data_A1.py  # Generate JSON annotations for training/validation
│   ├── get_json_for_data_A2.py  # Generate JSON annotations for test set
│   ├── get_json_for_sl_data.py  # Generate JSON for self-learning data
│   └── get_splited_videos.py    # Split videos by action labels
│
├── post_process/                 # Post-processing and ensemble
│   ├── ensemble.py              # Ensemble multiple model predictions
│   ├── generate_txt.py          # Convert CSV predictions to submission format
│   └── submit/                  # Generated submission files
│
├── web_app/                      # Web application for driver behavior analysis
│   ├── app.py                   # FastAPI backend server
│   ├── inference_utils.py       # Feature extraction + AMA inference pipeline
│   ├── frontend/                # React frontend application
│   │   ├── src/                 # React components
│   │   └── public/              # Static assets
│   ├── driver_analysis.db       # SQLite database
│   └── requirements.txt
│
├── data/                         # Data files and labels
│   ├── label_A1-train_A1-val.json
│   └── label_submit.json
│
└── docs/                         # Documentation
    ├── PREPROCESS.md
    ├── FEATURE_EXTRACTION.md
    ├── AMA.md
    ├── POST_PROCESS.md
    └── TEST.md

System Requirements

• Python: 3.8 or higher
• CUDA: CUDA-capable GPU recommended (for training and inference)
• PyTorch: 1.12+ (with CUDA support recommended)
• Node.js: 16+ (for web application frontend)
• Memory: At least 8GB RAM (16GB+ recommended for training)
• Storage: Sufficient space for videos, features, and model checkpoints

Installation

1. Clone the repository

git clone <repository-url>
cd Final_Source

2. Install Python dependencies

AMA module:

cd AMA
pip install -r requirements.txt
cd libs/utils
python setup.py install --user
cd ../..

Feature extraction module:

cd feature_extraction
pip install -r requirements.txt

Web application:

cd web_app
pip install -r requirements.txt

3. Install Node.js dependencies (for web app)

cd web_app/frontend
npm install

4. Download pretrained weights

Feature Extraction weights and extracted features:

• Download finetuned VideoMAE weights and pre-extracted features:
  • Finetuned weights and extracted features
• Extract and place the weights in feature_extraction/weights/
• Place extracted features in data/extracted_features/ if using pre-extracted features

AMA model training weights:

• Download trained AMA model checkpoints:
  • AMA training weights
• Extract and place checkpoint files (.pth.tar) in AMA/ckpt/ directory

Usage

Complete Pipeline

The typical workflow consists of four main steps:

Step 1: Data Preprocessing

Prepare your video data by detecting and cropping driver regions.

export BASE_DIR=/path/to/Final_Source
cd preprocess

# Install YOLOv5 dependencies
cd yolov5
pip install -r requirements.txt
cd ..

# Detect and crop driver regions from videos
python yolov5/driver_tracking.py \
    --vid_path $BASE_DIR/data/raw_videos/A1 \
    --out_file $BASE_DIR/data/crop_videos/A1

# Generate JSON annotation files
python get_json_for_data_A1.py \
    --video_path $BASE_DIR/data/crop_videos/A1 \
    --label_path $BASE_DIR/data/raw_videos/labels&instructions/A1 \
    --output_path $BASE_DIR/data/label_A1-train_A1-val.json

# Split videos by action labels (optional, for training)
python get_splited_videos.py \
    --video_path $BASE_DIR/data/crop_videos/A1 \
    --save_path $BASE_DIR/data/splited_videos/A1 \
    --csv_output $BASE_DIR/data/splited_videos \
    --label_path $BASE_DIR/data/raw_videos/labels&instructions/A1

For detailed preprocessing instructions, see docs/PREPROCESS.md.

Step 2: Feature Extraction

Extract video features using VideoMAE models.

cd feature_extraction

# Extract features using ViT-Giant (recommended)
python inference_video_feature_vitg.py \
    --video_dir $BASE_DIR/data/crop_videos/A1 \
    --ckpt_pth weights/vit_g_hybrid_pt_1200e_k710_ft.pth \
    --output_dir $BASE_DIR/data/extracted_features/A1

# Or extract features using ViT-Base
python inference_video_feature_vitb.py \
    --video_dir $BASE_DIR/data/crop_videos/A1 \
    --ckpt_pth weights/vit_b_k710_dl_from_giant.pth \
    --output_dir $BASE_DIR/data/extracted_features/A1

For detailed feature extraction instructions, see docs/FEATURE_EXTRACTION.md.

Step 3: AMA Training and Inference

Train or run inference with the AMA model.

Training:

cd AMA
python train.py ./configs/vitg.yaml --output ckpt/output_dir/

Inference/Evaluation:

cd AMA
python eval.py \
    ./configs_test/aicity_ego_vitg_deploy.yaml \
    ckpt/aicity_ego_vitg_ckpt/model_dir/ \
    --output_csv $BASE_DIR/post_process/predictions.csv

For detailed AMA instructions, see docs/AMA.md.

Step 4: Post-processing

Generate submission files and optionally ensemble multiple predictions.

cd post_process

# Generate submission TXT file from CSV predictions
python generate_txt.py \
    --csv_path predictions.csv \
    --out_file submit/submission.txt

# Ensemble multiple model predictions
python ensemble.py --out_file submit/ensemble_submission.txt

For detailed post-processing instructions, see docs/POST_PROCESS.md.

Web Application

The repository includes a web application for interactive driver behavior analysis.

Start the backend:

cd web_app
python app.py
# Backend runs on http://localhost:5000

Start the frontend (in a new terminal):

cd web_app/frontend
npm start
# Frontend runs on http://localhost:3000

The web application allows you to:

• Upload videos and driver profiles
• Run feature extraction and action detection
• View detected actions and key frames
• Query driver history

For detailed web app instructions, see web_app/README.md.

Experimental Results

Experimental results reveal a distinct trade-off between model capacity and efficiency:

Feature Extraction Stage

• ViT-Giant backbone: Delivers superior representations with 88.09% Top-1 test accuracy
• ViT-Base variant: Achieves 82.55% accuracy with significantly lower computational fine-tuning costs (101.85 GFLOPs/segment compared to 1584.06 GFLOPs/segment for Giant)

Temporal Action Localization Stage

• SPPF integration: Consistently improves performance across all configurations
• ViT-Giant + SPPF: Achieves peak mAP of 92.67%
• ViT-Base configuration: Maintains robust results with favorable computational efficiency

Evaluation on the AICity2024 Track 3 dataset confirms the viability of the proposed method, showing that the framework can effectively adapt to both high-performance and resource-constrained driver monitoring scenarios.

Key Components

AMA (Augmented Self-Mask Attention Transformer)

• Temporal action detection model for video sequences
• Supports multiple backbone architectures:
  • ViT-Base (vitb) - Lightweight, efficient variant
  • ViT-Giant (vitg) - High-performance variant
  • SPPF variants - Enhanced with Spatial Pyramid Pooling-Fast for multi-scale feature capture
• Configurable through YAML configuration files

Feature Extraction (VideoMAE)

• Based on VideoMAEv2 architecture
• Extracts spatiotemporal features from video clips
• Supports ViT-Base and ViT-Giant backbones
• Processes videos frame-by-frame with temporal modeling
• Enables trade-off between accuracy and computational cost

SPPF Module Enhancement

• Spatial Pyramid Pooling-Fast (SPPF) module integrated into the neck component
• Captures multi-scale temporal features before self-attention stage
• Replaces original identity or feature pyramid structures
• Improves model's ability to capture subtle driver actions

Preprocessing

• YOLOv5-based driver detection and tracking
• Automatic video cropping to driver region
• Video splitting by action labels
• JSON annotation file generation for training and evaluation

Post-processing

• Non-Maximum Suppression (NMS) for action proposals
• Score thresholding and filtering
• Ensemble methods for combining multiple model predictions
• Submission file generation in required format

Configuration

AMA Configuration

Model configurations are stored in YAML files under AMA/configs/:

• vitb.yaml: ViT-Base configuration
• vitg.yaml: ViT-Giant configuration
• vitb_sppf.yaml: ViT-Base with SPPF neck
• vitg_sppf.yaml: ViT-Giant with SPPF neck

Test/deployment configurations are in AMA/configs_test/.

Feature Extraction Configuration

Feature extraction parameters can be adjusted in the inference scripts:

• Frame sampling rate
• Input resolution
• Batch size
• Device (CPU/GPU)

Data Format

Input Videos

• Supported formats: MP4, AVI, MOV, MKV
• Recommended: MP4 with H.264 codec
• Videos should contain in-cabin driver footage

Annotation Format

JSON annotation files follow ActivityNet format:

{
  "version": "AI CITY 2024 track-3",
  "database": {
    "video_name": {
      "annotations": [
        {
          "label": "Drinking",
          "segment": [start_time, end_time]
        }
      ]
    }
  }
}

Action Classes

The system recognizes 16 action classes:

• Normal Forward Driving
• Drinking
• Phone Call (left/right)
• Eating
• Texting (left/right)
• Reaching behind
• Adjust control panel
• Pick up from floor (driver/passenger)
• Talk to passenger (right/backseat)
• Yawning
• Hand on head
• Singing or Dancing

Testing on New Data

For testing on dataset B or new data, follow the instructions in docs/TEST.md.

Troubleshooting

Common Issues

1. CUDA out of memory: Reduce batch size in configuration files
2. Missing checkpoints: Ensure model weights are downloaded and placed in correct directories
3. Import errors: Install all dependencies from requirements.txt files
4. Video processing errors: Check video codec compatibility, convert to MP4 if needed

Performance Tips

• Use GPU for faster inference
• Batch process multiple videos when possible
• Use ViT-Giant for better accuracy (slower) or ViT-Base for faster inference
• Pre-extract features to avoid redundant computation

Main Contributions

The primary contributions of this work include:

1. Complete end-to-end pipeline for distracted driver behavior understanding
2. Extensive experiments analyzing architectural trade-offs between accuracy and efficiency
3. Web-based demonstration system designed for experimental deployment in periodic inspection or transportation monitoring environments
4. SPPF enhancement to improve multi-scale feature representation while maintaining computational efficiency

Dataset

This framework is evaluated on the AICity2024 Track 3: Naturalistic Driving Action Recognition dataset, which contains in-cabin video footage of drivers performing various actions.

Applications

This system is particularly suited for:

• Periodic vehicle inspection stations: Automated detection of dangerous driving behaviors
• Fleet management assessment systems: Monitoring driver safety across vehicle fleets
• Transportation safety checkpoints: Real-time or batch processing of driver video footage
• Driver behavior research: Analysis of naturalistic driving scenarios

License

Please refer to the LICENSE files in respective subdirectories. This codebase includes components from:

• VideoMAEv2 (OpenGVLab)
• YOLOv5 (Ultralytics)

References

This implementation is based on the work by Zhang et al. on Augmented Self-Mask Attention Transformer for Naturalistic Driving Action Recognition.

Acknowledgments

This implementation references and builds upon:

• VideoMAEv2: OpenGVLab/VideoMAEv2
• YOLOv5: ultralytics/yolov5

Keywords

Temporal action localization, Driver behavior detection, VideoMAE, Augmented Self-Mask Attention (AMA), Spatial Pyramid Pooling–Fast (SPPF), Vision Transformers, Naturalistic driving