GIF Analysis Tool

A tool for analyzing and optimizing GIF files, with a focus on efficient batch processing optimized for Mac M2 chips.

Features

• Analyze GIF files for quality metrics (SSIM, PSNR, VMAF)
• Compress GIFs using multiple compression engines
• Generate detailed reports on compression efficiency
• Smart batch processing with adaptive sizing
• Optimized for Mac M2 chips

New Feature: Adaptive Batch Sizing System

The latest version includes an intelligent adaptive batch sizing system that dynamically adjusts the number of GIFs processed in parallel based on:

1. GIF Complexity Analysis: Automatically detects GIF complexity by examining:

   • Frame differences
   • File size
   • Resolution and frame count
   • Color complexity

2. Memory Management: Monitors system resources to prevent out-of-memory errors:

   • Dynamically adjusts batch sizes based on available RAM
   • Prevents VMAF calculation failures due to memory limitations
   • Ensures optimal resource utilization

3. Optimization by Complexity Grouping:

   • Processes simple GIFs in larger batches (maximum efficiency)
   • Processes complex GIFs in smaller batches (ensures reliability)
   • Balances throughput with stability

How Adaptive Batch Sizing Works

When processing a large number of GIFs (e.g., 9,000 files), the system:

1. First samples each GIF to estimate its complexity (quick examination)

2. Categorizes GIFs into complexity groups (low, medium, high)

3. Calculates optimal batch sizes for each group based on:

   • Available system memory
   • Base batch size (configurable)
   • Estimated memory requirements per GIF

4. Creates optimized processing batches to maximize throughput while preventing errors

5. Monitors performance during processing to make adjustments

Benefits for VMAF Processing

The adaptive system is particularly valuable for VMAF calculations, which are memory-intensive:

• Prevents out-of-memory errors during VMAF processing
• Dynamically reduces batch size for complex GIFs
• Adjusts to system load for consistent performance
• Maintains high throughput by processing simple GIFs in larger batches

Usage

Run the optimized analysis script:

./run_analysis_mac_optimized.py --batch-size 5

The --batch-size parameter sets the base batch size, which will be automatically adjusted based on GIF complexity.

Advanced Options

./run_analysis_mac_optimized.py --help

For VMAF-specific processing:

./run_analysis_mac_optimized.py --selective --fields vmaf_all_frames,vmaf_first_frame,vmaf_min,vmaf_max,vmaf_harmonic_mean

Recommendations for Batch Size

• For general processing: Use batch size 5-6 (default is 6)
• For memory-constrained systems: Use batch size 3-4
• For VMAF-heavy processing: Use batch size 4-5
• For maximum throughput on high-end systems: Use batch size 8

The adaptive system will automatically scale down batch sizes for complex GIFs, so these are just starting points.

Compression Engines

The tool supports multiple compression engines:

• Gifsicle: Industry standard GIF optimization tool
  • Uses optimization levels 1-3 and lossy values 0-300
  • Encoded as: (optimization_level * 1000) + lossy_value
  • See GIFSICLE_ENCODING.md for detailed information
• Animately: Advanced GIF compression engine with superior compression ratios
  • Uses 11 compression levels (0-100)
  • Usually provides 10-16% better compression than gifsicle at equivalent levels

Compression Engine Performance

Based on our testing, the Animately engine typically outperforms Gifsicle:

Engine	Level	Compression Ratio	Notes
Gifsicle	0	1.01	Minimal compression, lossless
Gifsicle	50	0.94	Good compression, some quality loss
Animately	0	0.93	Better than gifsicle level 50
Animately	50	0.84	Best compression, acceptable quality

System Requirements

• macOS on Apple Silicon (optimized for M2)
• Python 3.8+
• Required Python packages:
  • numpy
  • pandas
  • Pillow
  • psutil (for memory monitoring)
  • tqdm (for progress tracking)

Overview

This tool analyzes GIF files to calculate quality metrics and optimize file size while maintaining visual quality. It implements Netflix's Video Multimethod Assessment Fusion (VMF) algorithm to assess the perceptual quality of compressed GIFs.

Features

• Quality Assessment: Calculate SSIM, PSNR, and VMF scores for GIFs at different compression levels
• Compression Optimization: Test multiple compression levels to find optimal quality/size ratio
• Parallel Processing: Process multiple files simultaneously for faster analysis
• Detailed Reports: Generate comprehensive reports with quality metrics and file size data
• Adaptive Batch Sizing: Smart batch processing with adaptive sizing

Installation

Prerequisites

• Python 3.8 or higher
• Pip package manager

Setup

1. Clone the repository:

   git clone https://github.com/yourusername/gif-analysis.git
   cd gif-analysis
   

2. Install the package and dependencies:

   python install.py
   

   or

   pip install -e .
   

Usage

Command Line Interface

The tool can be run from the command line:

python main.py [OPTIONS]

Options:

• --input-dir, -i: Directory containing GIF files to analyze
• --output-dir, -o: Directory where compressed GIFs will be saved
• --levels, -l: Comma-separated list of compression levels to test (0-300)
• --engine, -e: Engine to use: 'animately', 'gifsicle', or 'all' (default)
• --parallel, -p: Enable parallel processing
• --no-parallel: Disable parallel processing
• --verbose, -v: Enable verbose logging
• --max-files, -m: Maximum number of GIF files to process (for testing)
• --safe-levels: Use only safe compatible levels (0-100) for all engines

Shorthand Scripts

For convenience, three shorthand scripts are provided to run the tool with predefined settings:

Mac M2 Optimized Script (Recommended)

./run_analysis_mac_optimized.py [ADDITIONAL_OPTIONS]

a way to run the script with selective optimizations:

./run_analysis_mac_optimized.py --selective

a way to run the script with overwrite:

./run_analysis_mac_optimized.py --overwrite

a way to run the script with batch size:

./run_analysis_mac_optimized.py --batch-size 5

a way to run the script with batch size and selective:

./run_analysis_mac_optimized.py --batch-size 3 --selective 

This script provides the most optimized experience, especially for Mac M2 systems, with these specific optimizations:

• Automatically configures CPU thread usage based on available cores (leaving 2 cores free for system processes)
• Sets optimal batch size for M2 Mac memory configurations
• Enables hardware acceleration for video processing using Apple's VideoToolbox
• Configures NumPy and BLAS libraries for optimal performance
• Provides enhanced logging and graceful interruption handling (Ctrl+C to pause/resume)
• Supports all the same options as other scripts

Shell Script (macOS/Linux)

./analyze.sh [ADDITIONAL_OPTIONS]

Python Wrapper Script (Cross-Platform)

./run_analysis.py [ADDITIONAL_OPTIONS]

All scripts:

• Use the default input directory (gif_analysis/data)
• Use the default output directory (gif_analysis/output/compressed_gifs)
• Enable verbose logging
• By default, use all available compression engines with their optimal levels:
  • Gifsicle: 33 encoded levels (1000-3300)
  • Animately: 11 levels (0-100)
• Pass any additional options to the main script

Special Options for Shorthand Scripts

• --gifsicle-encoded: Use only gifsicle with its 33 encoded compression levels
• --animately-only: Use only the animately engine with its 11 compression levels
• --safe-levels: Use all engines but with safe compatible levels (0-100) for both

Example:

# Run analysis with maximum of 5 files
./analyze.sh --max-files 5

# Use only the animately engine
./run_analysis.py --animately-only

# Use only gifsicle with encoded compression levels
./analyze.sh --gifsicle-encoded

# Use both engines with safe compatible levels
./run_analysis.py --safe-levels

Examples

Basic usage:

python main.py -i ./gif_analysis/data -o ./gif_analysis/output/compressed_gifs

Specific compression levels:

python main.py -i ./gif_analysis/data -o ./gif_analysis/output/compressed_gifs -l 30,60,90,120

Disable parallel processing:

python main.py -i ./gif_analysis/data -o ./gif_analysis/output/compressed_gifs --no-parallel

Limit processing to a few files for testing:

python main.py -i ./gif_analysis/data -o ./gif_analysis/output/compressed_gifs -l 0,50 --max-files 5

Configuration

The tool uses a YAML configuration file located at gif_analysis/config.yaml. You can modify this file to customize:

• Input and output directories
• Default compression levels
• Report paths
• Other parameters

Project Structure

gif-analysis/                # Repository root
│
├── gif_analysis/            # Python package directory
│   ├── __init__.py          # Package initialization
│   ├── config.yaml          # Configuration file
│   ├── core/                # Core functionality
│   │   ├── __init__.py      # Core subpackage initialization
│   │   ├── analyzer.py      # Main analysis module
│   │   ├── compression_manager.py  # Handles compression engines
│   │   └── utils.py         # Utility functions
│   ├── data/                # Input data directory
│   │   ├── unused/          # Storage for unused GIF files
│   │   └── tagged_examples/ # Categorized example GIFs
│   └── output/              # Output directory
│       ├── compressed_gifs/ # Compressed GIFs
│       │   └── gifsicle/    # Gifsicle engine outputs
│       ├── logs/            # Log files
│       ├── data/            # Analysis data (CSV files)
│       └── reports/         # Analysis reports
│
├── main.py                  # CLI entry point
├── analyze.sh               # Shell script runner
├── run_analysis.py          # Python script runner
├── install.py               # Installation script
├── migrate_data.py          # Data migration script
├── setup.py                 # Package setup file
├── tests/                   # Test files
├── cleanup_plan.md          # Documentation of directory cleanup
├── LICENSE                  # License file
└── README.md                # This file

Directory Structure Notes

The tool follows a standard Python package structure:

1. All core functionality is inside the gif_analysis package
2. Data files (GIFs to analyze) should be placed in gif_analysis/data/
3. Output files are stored in gif_analysis/output/ with dedicated subdirectories
4. Configuration is managed through gif_analysis/config.yaml

Data Migration

If you're upgrading from a previous version with a different directory structure, you can use the migrate_data.py script to help migrate your files:

python migrate_data.py

This script will:

1. Copy files from old directories to the new package structure
2. Help you clean up redundant directories
3. Guide you through the migration process

Technical Details

Quality Metrics

• SSIM (Structural Similarity Index): Measures the similarity of two images
• PSNR (Peak Signal-to-Noise Ratio): Measures the ratio between maximum possible power of a signal and the power of noise
• VMF (Video Multimethod Assessment Fusion): Netflix's algorithm for perceptual quality assessment that combines multiple metrics with content-adaptive weighting

Compression Engines

• Gifsicle: High-quality GIF optimization with lossy compression options
  • Uses 33 encoded compression levels (1000-3300) across 3 optimization levels
  • Each optimization level (1-3) has 11 lossy values (0-300)
  • Encoded as: (optimization_level * 1000) + lossy_value
  • See GIFSICLE_ENCODING.md for detailed information
• Animately: Advanced GIF compression engine with superior compression ratios
  • Uses 11 compression levels (0-100)
  • Usually provides 10-16% better compression than gifsicle at equivalent levels
  • Maintains high perceptual quality while achieving smaller file sizes

Compression Engine Performance

Based on our testing, the Animately engine typically outperforms Gifsicle:

Engine	Level	Compression Ratio	Notes
Gifsicle	0	1.01	Minimal compression, lossless
Gifsicle	50	0.94	Good compression, some quality loss
Animately	0	0.93	Better than gifsicle level 50
Animately	50	0.84	Best compression, acceptable quality

Testing Compression Engines

You can test both compression engines on a single GIF file using:

python test_engines.py

This will compress the test.gif file with both Gifsicle and Animately engines at different compression levels and show detailed results.

License

MIT License

Acknowledgments

• Netflix Technology Blog for VMF research: "Toward A Practical Perceptual Video Quality Metric"
• The Gifsicle project for their excellent GIF optimization tool
• Animately team for their advanced GIF compression technology