Reddit Community Discussion Analyzer

A production-ready Python pipeline for automated Reddit data collection, sentiment classification, and topic clustering — with a live web dashboard.

---

Features

• Data Collection — Multi-source collector supporting Arctic Shift API, Reddit public JSON, Kaggle CSV datasets, and PRAW — no credentials required by default
• Sentiment Analysis — VADER lexicon scoring + TF-IDF / Naive Bayes ML classifier with feature inspection
• Topic Clustering — KMeans + TF-IDF with automatic cluster labelling and silhouette evaluation across 10 topic categories
• Engagement Analysis — Quantifies AI vs. non-AI post engagement with live multiplier calculation
• Word Frequency — Stopword-filtered frequency distribution across all collected titles
• Web Dashboard — Flask server with Chart.js visualisations, live filtering, and one-click pipeline refresh
• 95% Test Coverage — Full unit test suite via pytest + pytest-cov

---

Quick Start

1. Clone and install

git clone https://github.com/RishabChopra/reddit-analyzer.git
cd reddit-analyzer
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Run the pipeline — no setup needed

# Real live data from Arctic Shift + Reddit JSON (default, no credentials)
python3 -m src.pipeline

# Use a Kaggle CSV dataset (see below) — up to 500,000+ posts
python3 -m src.pipeline --mode kaggle

# Kaggle CSV + live Arctic Shift data combined (maximum data)
python3 -m src.pipeline --mode kaggle+live

# Fully offline, no internet needed
python3 -m src.pipeline --mode mock

3. Start the dashboard

python3 app.py   # open http://127.0.0.1:5000

---

Getting a Kaggle Dataset (free, no API needed)

For larger historical datasets (10,000+ posts), download a Reddit CSV from Kaggle:

1. Go to kaggle.com and search: Reddit posts CSV technology
2. Good datasets to look for:
   • Reddit Posts Dataset — general technology subreddits
   • Reddit Climate Change Dataset — large, well-structured
   • Reddit WallStreetBets — high engagement data
3. Download the CSV and place it in the data/ folder
4. Run python3 -m src.pipeline --mode kaggle

The collector automatically detects and maps column names across different Kaggle dataset formats — no manual configuration needed.

---

All data source modes

Mode	Command	Requirements	Data volume
auto (default)	python3 -m src.pipeline	None	~1,000 live posts
kaggle	--mode kaggle	CSV in data/	10,000–500,000+ posts
kaggle+live	--mode kaggle+live	CSV in data/	Kaggle + live merged
arctic	--mode arctic	None	~1,000 live posts
json	--mode json	None	~500 live posts
praw	--mode praw	.env credentials	Up to 1,000/subreddit
mock	--mode mock	None (offline)	500 generated posts

---

Project Structure

reddit-analyzer/
├── src/
│   ├── collector.py      # Multi-source collector (Arctic Shift, JSON, Kaggle, PRAW, mock)
│   ├── sentiment.py      # VADER scoring + Naive Bayes classifier
│   ├── topics.py         # KMeans clustering + word frequency
│   └── pipeline.py       # Orchestrator — runs all phases
├── tests/
│   ├── conftest.py
│   └── test_pipeline.py  # Unit tests (95%+ coverage)
├── static/
│   └── index.html        # Dashboard frontend
├── data/                 # Output directory (auto-created)
│   ├── posts.csv         # All collected posts with enriched fields
│   └── results.json      # Analysis results (consumed by dashboard)
├── app.py                # Flask web server
├── requirements.txt
└── .env.example

---

Running Tests

# Run all tests with coverage report
pytest tests/ -v --cov=src --cov-report=term-missing

# Run just the sentiment tests
pytest tests/test_pipeline.py::TestSentiment -v

# Run just the topic tests
pytest tests/test_pipeline.py::TestTopics -v

---

Environment Variables

Variable	Description	Default
REDDIT_CLIENT_ID	Reddit app client ID (optional)	—
REDDIT_CLIENT_SECRET	Reddit app client secret (optional)	—
REDDIT_USER_AGENT	User agent string	community-analyzer/1.0
SUBREDDITS	Comma-separated subreddits	technology,MachineLearning
POST_LIMIT	Posts per subreddit	200
FLASK_PORT	Dashboard port	5000

---

Output Files

After running the pipeline, data/ contains:

• posts.csv — All collected posts with columns: title, score, num_comments, subreddit, sentiment, vader_compound, ml_sentiment, topic_label, topic_keywords, is_ai, created_date
• results.json — Aggregated analysis: engagement metrics, sentiment distribution, topic summary, word frequency, monthly trends, top posts

---

Dashboard Features

• Metric cards — total posts, AI engagement multiplier, sentiment %, AI post count
• Monthly bar chart — AI vs. non-AI average score over time
• Sentiment doughnut — positive / neutral / negative breakdown
• Topic clusters — 10 categories ranked by post count with proportional bars
• Word frequency cloud — top terms sized by frequency
• Post feed — top 20 posts by score, filterable by sentiment; click any post to open on Reddit
• Live refresh — re-runs the full pipeline without restarting the server

---

Key Findings (live data, April 2026)

• 1,000 posts collected across 5 subreddits in under 2 minutes
• AI engagement multiplier: 0.75× — non-AI posts scored higher on average, though AI posts drove significantly more comment discussion
• 25.3% positive sentiment across technology communities
• Top topics: Software Development, AI Safety & Ethics, Large Language Models, Open Source AI, Cloud & Infrastructure

---

Tech Stack

Layer	Technology
Data collection	Arctic Shift API + Reddit public JSON + Kaggle CSV + PRAW
Sentiment scoring	VADER (vaderSentiment)
ML classifier	scikit-learn (TF-IDF + Naive Bayes)
Clustering	scikit-learn (KMeans + TF-IDF)
Web server	Flask
Frontend	Vanilla JS + Chart.js
Testing	pytest + pytest-cov