A smart machine learning tool that can tell whether a tweet is reporting a real disaster or just using metaphors and exaggeration.
The dataset contains approximately 11,000 tweets, including:
- Volcanic eruptions
- Australian bushfires
- Early COVID-19 news
- Accidents, storms, floods, and other disasters
Each tweet includes text, a location, a keyword, and a label: disaster or not.
The raw dataset had a lot of redundancy in keywords. For example, people would use "wildfire", "bush fire", "forest fire", and "wild fire" - all referring to the same disaster type.
I handled this in two ways:
- Manual mapping: I created a dictionary that groups similar keywords together. So "wildfire", "bush fire", "forest fire" all map to "fire". This reduced noise and helped the model learn from more examples of each category.
- Lemmatization: I used spaCy's natural language processing to reduce words to their root form. This way, "burning", "burn", "burned" all become "burn".
Real disaster tweets don't look like metaphorical ones. By analyzing the dataset, I discovered several reliable indicators:
-
URLs are important: About 65% of real disasters include links to news articles, while only 50% of non-disasters do. When someone tweets about an actual emergency, they usually link to a news source or official statement.
-
News terminology matters: Words like "via", "breaking", "reported", "update", and "live" appear much more in real disaster tweets. These are the kinds of words journalists and news accounts use.
-
Punctuation tells a story: Real emergencies tend to be reported matter-of-factly. Metaphorical tweets about disasters ("My life is a disaster!!!") use way more exclamation marks.
-
Personal references are surely metaphoric: Tweets saying "my heart is on fire" or "my life is a disaster" are almost always metaphors, not actual emergencies.
Before feeding tweets to the model, I needed to standardize them:
- Removed noise: Deleted URLs, @mentions, and emojis
- Normalized text: Converted all text to lowercase, removed accents, handled special characters.
- Tokenized: Used NLTK's TweetTokenizer.
- Removed stop words
- Kept meaningful tokens: Only kept words with 3+ letters that are purely alphabetic.
I tested several different machine learning algorithms to see which performed best and Random Forest with SMOTE emerged as the top performer.
- Accuracy: 90%
- F1-score for disasters: 0.64–0.70
Confusion Matrix:
| Pred 0 | Pred 1 | |
|---|---|---|
| True 0 | 1733 | 118 |
| True 1 | 100 | 219 |
- The dataset had mislabeled examples - tweets marked as disasters that clearly weren't. For instance, "My heart is on fire" was labeled as a real disaster.
- Only 18% of the dataset were actual disasters - the rest were normal tweets using disaster-related keywords metaphorically. Random Forest with SMOTE helped here.
- Some tweets genuinely live in the gray zone and are hard to classify, such as: celebrity accidents, political sarcasm and historical references.