Fake vs. Real News Classification

Compare whether fake news is easier to detect using full article text or only article titles using Random Forest and TF-IDF features.

Course: DS 4002
Group Name: DLC
Group Leader: Lauren Medica
Group Members: Lauren Medica, Dev Patel, Caroline Lingle

---

Overview

This project compares two machine learning approaches:

• Full article text classification
• Title-only classification

We evaluate performance primarily using weighted F1 score.

Key Result:
Full-text model (F1 = 0.9874) outperforms title-only model (F1 = 0.9475).

---

Installation

Clone the repository:

git clone <your-repo-link>
cd <repo-name>

Install required packages:

pip install -r requirements.txt

If needed, download NLTK data:

import nltk
nltk.download("stopwords")
nltk.download("wordnet")
nltk.download("omw-1.4")

---

Software & Platform

Language: Python 3

Libraries Used:

• pandas
• numpy
• scikit-learn
• nltk
• matplotlib

---

Repository Structure

PROJECT_ROOT/
│
├── README.md
├── requirements.txt
│
├── DATA/
│   └── raw/
│       ├── fake.csv
│       └── real.csv
│
├── SCRIPTS/
│   ├── 01_load_data.py
│   ├── 02_preprocessing.py
│   ├── 03_full_text_model.py
│   ├── 04_title_model.py
│   └── 05_model_comparison.py
│
├── OUTPUT/
│   ├── metrics_full_text.csv
│   ├── metrics_title.csv
│   └── confusion_matrices/
│
└── Project_1.ipynb

---

Reproducing Results

Step 1 — Download Data

Download dataset from Kaggle:

https://www.kaggle.com/datasets/clmentbisaillon/fake-and-real-news-dataset

Place the following files into:

DATA/raw/

• fake.csv
• real.csv

---

Step 2 — Install Dependencies

From the project root directory, run:

pip install -r requirements.txt

---

Step 3 — Run Scripts (in order)

From the project root directory, execute:

python SCRIPTS/01_load_data.py
python SCRIPTS/02_preprocessing.py
python SCRIPTS/03_full_text_model.py
python SCRIPTS/04_title_model.py
python SCRIPTS/05_model_comparison.py

---

Step 4 — View Output

Results will appear in:

OUTPUT/

Including:

• F1 scores
• Accuracy
• Precision / Recall
• Confusion matrices

---

Modeling Approach

• TF-IDF vectorization
• 80/20 stratified train-test split
• Random Forest classifier
• Primary evaluation metric: Weighted F1 score

---

Results

Model	F1 Score
Full Text → Full Text	0.9874
Title → Title	0.9475
Full Text → Title	0.4137

---

Notes

• The same train/test split is used for both models.
• A key design decision was whether to train a separate model for titles.
• Titles contain strong predictive signals but lack contextual depth.

---

Future Work

• Logistic Regression comparison
• Deep learning models (BERT, LSTM)
• Source-level analysis
• Satire detection

---

Acknowledgements

Dataset: C. Bisaillon (Kaggle)
Instructor: Karsten Siller
TA: Cole Whittington