Sentiment Analysis Pipeline

Production-ready sentiment analysis supporting VADER and DistilBERT models with REST API, monitoring, and automated deployment.

Features

• 🚀 REST API with health checks and metrics
• 📊 Monitoring with Prometheus and Grafana
• 🐳 Docker containerization with multi-stage builds
• 🔄 CI/CD pipeline with automated testing and deployment
• ⚡ High Performance batch processing (12K+ texts/sec)
• 🧪 Comprehensive Testing with 55% test coverage

Quick Start

Development Setup

# Extract and go to directory
cd inference_pipeline

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install development dependencies (includes production deps)
pip install -r requirements-dev.txt

# Install package in development mode
pip install -e .

# Run tests
pytest tests/ -v

# Run with coverage
pytest --cov=inference_pipeline --cov-report=html

Production Setup

# Production environment - install only production dependencies
pip install -r requirements.txt
pip install -e .

# Or use Docker for production deployment
docker-compose up -d

Local Development Usage

# Setup virtual environment first (see Development Setup above)
python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt
pip install -e .

# Start API server
python -m inference_pipeline.main serve --port 8080

# Test endpoints
curl http://localhost:8080/health
curl -X POST http://localhost:8080/predict \
  -H "Content-Type: application/json" \
  -d '{"texts": ["This is amazing!", "I hate this."]}'

# Try batch processing with included sample data  
python -m inference_pipeline.main run \
  --input data/sample_reviews.csv \
  --output data/my_results.csv \
  --model-type vader

Docker Deployment

# Build the Docker image
docker build -t sentiment-inference-pipeline:latest .

# Quick demo
docker run --rm sentiment-inference-pipeline:latest python -m inference_pipeline.main demo

# Demo with DistilBERT model
docker run --rm sentiment-inference-pipeline:latest python -m inference_pipeline.main demo --model-type distilbert

# Start API server
docker run --rm -p 8080:8080 sentiment-inference-pipeline:latest python -m inference_pipeline.main serve --port 8080

# Production deployment with monitoring
./scripts/deploy.sh

# Or manually with docker-compose
docker-compose up -d                          # API only
docker-compose --profile monitoring up -d     # With Prometheus/Grafana

API Endpoints

Endpoint	Method	Description
/health	GET	Service health status
/predict	POST	Sentiment predictions
/metrics	GET	Performance metrics

Request Example

curl -X POST http://localhost:8080/predict \
  -H "Content-Type: application/json" \
  -d '{
    "texts": ["Great product!", "Terrible service"],
    "model_type": "vader"
  }'

Response Format

{
  "predictions": [
    {"text": "Great product!", "sentiment": "POSITIVE", "confidence": 0.89},
    {"text": "Terrible service", "sentiment": "NEGATIVE", "confidence": 0.92}
  ],
  "model_type": "vader",
  "processing_time": 0.003
}

Models

Model	Speed	Accuracy	Use Case
VADER	~12K texts/sec	Good	Real-time, high-volume
DistilBERT	~70 texts/sec	Excellent	High-accuracy analysis

Batch Processing

# Process large CSV files
python -m inference_pipeline.main run \
  --input data/sample_reviews.csv \
  --output data/results.csv \
  --model-type vader \
  --batch-size 100

# Process with DistilBERT for higher accuracy
python -m inference_pipeline.main run \
  --input data/sample_reviews.csv \
  --output data/results.csv \
  --model-type distilbert \
  --batch-size 50

Dataset Downloads

Download real-world datasets from Kaggle for testing and analysis:

# Download both datasets with default limits
python -m inference_pipeline.main download

# Download with custom limits
python -m inference_pipeline.main download --amazon-limit 1000 --sentiment140-limit 50000

# Download smaller samples for testing
python -m inference_pipeline.main download --amazon-limit 500 --sentiment140-limit 5000

# See all options
python -m inference_pipeline.main download --help

Available Datasets

Dataset	Source	Default Size	Description
Amazon Reviews	Kaggle	50K reviews	Product reviews with ratings (1-5)
Sentiment140	Kaggle	100K tweets	Twitter sentiment data with labels

Note: Requires Kaggle API credentials (~/.kaggle/kaggle.json)

Included Sample Data

The repository includes sample data for immediate testing:

• data/sample_reviews.csv - 1,000 Amazon product reviews for testing

Development

Setup

# Development environment
python -m venv .venv
source .venv/bin/activate  # or .venv\Scripts\activate on Windows
pip install -r requirements-dev.txt
pip install -e .

Testing

# Run all tests
python -m pytest tests/ -v --cov=inference_pipeline

# Health check
./scripts/health_check.py --verbose

# Performance benchmarks
python -m pytest tests/test_pipeline.py::TestPerformance -v

Code Quality

# Format and lint
black inference_pipeline/ tests/
isort inference_pipeline/ tests/
flake8 inference_pipeline/ tests/

Monitoring

Access monitoring dashboards when running with --profile monitoring:

• API: http://localhost:8080
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)

Production Deployment

Automated CI/CD Pipeline

The project includes a complete GitHub Actions workflow (.github/workflows/ci-cd.yml) that automatically:

1. Continuous Integration:

   • Runs tests on Python 3.10 and 3.11
   • Performs code quality checks (black, flake8, isort)
   • Runs security scanning with bandit
   • Builds and tests Docker containers
   • Generates test coverage reports

2. Continuous Deployment:

   • Builds production Docker images
   • Pushes to container registry
   • Deploys to staging/production environments

Workflow Triggers:

• Push to main or develop branches
• Pull requests to main branch

Manual Deployment Options

Option 1: Docker Compose (Recommended)

# Production deployment with monitoring
docker-compose up -d                          # API only
docker-compose --profile monitoring up -d     # With Prometheus/Grafana

# Scale horizontally
docker-compose up --scale inference-pipeline=3 -d

Option 2: Single Container

# Build and run single container
docker build -t sentiment-inference-pipeline:latest .
docker run -d -p 8080:8080 --name sentiment-api sentiment-inference-pipeline:latest python -m inference_pipeline.main serve --port 8080

# With volume mounting for data processing
docker run -d -p 8080:8080 \
  -v $(pwd)/data:/app/data \
  --name sentiment-api \
  sentiment-inference-pipeline:latest python -m inference_pipeline.main serve --port 8080

Option 3: Direct Python Deployment

# Production server setup
pip install -r requirements.txt
pip install -e .

# Start with production WSGI server (e.g., Gunicorn)
pip install gunicorn
gunicorn --bind 0.0.0.0:8080 --workers 4 inference_pipeline.api:app

Cloud Deployment

AWS ECS/Fargate

# Build and push to ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <account>.dkr.ecr.us-east-1.amazonaws.com
docker build -t inference-pipeline .
docker tag inference-pipeline:latest <account>.dkr.ecr.us-east-1.amazonaws.com/inference-pipeline:latest
docker push <account>.dkr.ecr.us-east-1.amazonaws.com/inference-pipeline:latest

Google Cloud Run

# Deploy to Cloud Run
gcloud run deploy inference-pipeline \
  --image gcr.io/PROJECT-ID/inference-pipeline \
  --platform managed \
  --region us-central1 \
  --allow-unauthenticated

Azure Container Instances

# Deploy to Azure
az container create \
  --resource-group myResourceGroup \
  --name inference-pipeline \
  --image myregistry.azurecr.io/inference-pipeline:latest \
  --cpu 2 --memory 4 \
  --ports 8080

Environment Variables

MODEL_TYPE=distilbert           # Model to use
PYTHONUNBUFFERED=1             # Logging
OMP_NUM_THREADS=4              # CPU optimization

Health Checks & Validation

# Verify deployment
curl http://localhost:8080/health

# Test API endpoint
curl -X POST http://localhost:8080/predict \
  -H "Content-Type: application/json" \
  -d '{"texts": ["Test message"], "model_type": "vader"}'

# Check metrics
curl http://localhost:8080/metrics

Resource Requirements

• Memory: 1-2GB (VADER), 2-4GB (DistilBERT)
• CPU: 1-2 cores recommended
• Storage: 500MB for models and dependencies

Horizontal Scaling

The service is stateless and can be horizontally scaled:

# Docker Compose scaling
docker-compose up --scale inference-pipeline=3

# Kubernetes scaling
kubectl scale deployment inference-pipeline --replicas=5

Project Structure

inference_pipeline/
├── inference_pipeline/     # Main package
│   ├── models/            # Model implementations
│   ├── api.py            # Flask API server
│   ├── pipeline.py       # Batch processing
│   ├── metrics.py        # Monitoring
│   ├── main.py           # CLI interface
│   └── dataset_downloader.py  # Kaggle data fetching
├── data/                  # Sample datasets
│   └── sample_reviews.csv # 1K Amazon reviews (input)
├── tests/                # Test suite
├── scripts/              # Deployment scripts  
├── monitoring/           # Prometheus config
├── .github/workflows/    # CI/CD pipeline
├── requirements.txt      # Production dependencies
├── requirements-dev.txt  # Development dependencies (includes prod)
├── setup.py             # Package configuration
├── pyproject.toml       # Build system config
└── docker-compose.yml   # Multi-service deployment

Requirements

Environment Configuration

The project uses environment variables for configuration:

1. Copy the example environment file:

   cp .env.example .env

2. Edit .env with your settings:

   • MODEL_TYPE: Choose between vader or distilbert
   • BATCH_SIZE: Adjust based on your system memory
   • USE_GPU: Set to true if you have CUDA-capable GPU
   • LOG_LEVEL: Set logging verbosity (DEBUG, INFO, WARNING, ERROR)

3. For Kaggle dataset downloads (optional):

   # Add your Kaggle credentials to .env
   KAGGLE_USERNAME=your_username
   KAGGLE_KEY=your_api_key

Dependency Management

• requirements.txt: Production dependencies only
• requirements-dev.txt: Development tools + production deps
  • Includes testing framework (pytest, pytest-cov)
  • Code quality tools (black, flake8, isort)
  • Security scanning (bandit)

System Requirements

• Python: 3.8+ (tested on 3.8, 3.9, 3.10)
• Core Dependencies:
  • PyTorch 2.0+ (ML models)
  • Transformers 4.30+ (DistilBERT)
  • Flask 2.3+ (REST API)
  • Pandas 2.0+ (data processing)
• System: 1GB RAM minimum, 2GB recommended