Create mini project idea - phuong anh phi.pdf

Final Project Presentation

@@ -0,0 +1 @@
+https://www.canva.com/design/DAG5Qai-T24/VjfLbkUXnMKQGKwN_-4Sww/edit?utm_content=DAG5Qai-T24&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton

ReadMe - Charlene & Phuong Anh.ipynb

@@ -0,0 +1,171 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "881211e0",
+   "metadata": {},
+   "source": [
+    "# Data 400 Project - Charlene Vu & Phuong Anh Phi\n",
+    "# Decoding Tech Hiring Trends: A Keyword Analysis of LinkedIn Job Postings 🤓💼\n",
+    "\n",
+    "## Overview\n",
+    "\n",
+    "This project politely stalks (aka scrapes) LinkedIn job postings, cleans up messy job descriptions, and then uses topic modeling (LDA) to figure out what kinds of work, skills, and values are actually being described.\n",
+    "\n",
+    "Instead of asking how many jobs exist or which skill appears the most, this project focuses on the language of job ads — what themes keep showing up, and how those themes change depending on job context like work setting, seniority, and location.\n",
+    "\n",
+    "In short: I make the computer read a bunch of job postings so I don’t have to.\n",
+    "\n",
+    "\n",
+    "## Why This Topic? 🤔📚\n",
+    "\n",
+    "As a student applying to jobs, it’s hard to tell what a role really means beyond the title. Two postings can look similar on the surface but describe totally different expectations, skills, and work styles.\n",
+    "\n",
+    "Job descriptions aren’t just skill checklists — they also reflect how companies define work, responsibility, culture, and impact. This project uses topic modeling to explore those hidden patterns and see how the language of work changes across contexts, like junior vs senior roles or remote vs on-site jobs.\n",
+    "\n",
+    "\n",
+    "## Research Question 🔍\n",
+    "\n",
+    "Rather than asking “how many jobs are out there?”, this project asks What themes show up in LinkedIn job postings, and how do those themes differ by job context?\n",
+    "\n",
+    "More specifically:\n",
+    "\n",
+    "* What themes commonly appear in job descriptions?\n",
+    "* How do these themes differ by work setting (remote, hybrid, on-site)?\n",
+    "* How does job language change between junior and senior roles?\n",
+    "* How do ideas like culture, impact, and responsibility show up in job ads?\n",
+    "\n",
+    "Using Latent Dirichlet Allocation (LDA), the goal is to uncover the hidden structure of how jobs are described.\n",
+    "\n",
+    "\n",
+    "## Project Structure 🗂️\n",
+    "\n",
+    "Because chaos is not a strategy:\n",
+    "\n",
+    "* `scraping_notebook.ipynb` – Scrapes LinkedIn job postings\n",
+    "* `Vu C-Phi A-DATA400 Final Project.ipynb` – Data cleaning, EDA, and topic modeling\n",
+    "* `dataset/jobs.csv` – Cleaned dataset used for analysis\n",
+    "* `slides/presentation.pdf` – Summary of methods and findings\n",
+    "* `README.md` – You’re reading it 👀\n",
+    "\n",
+    "\n",
+    "## Data Collection 🕵️‍♀️\n",
+    "\n",
+    "* Source: LinkedIn job postings\n",
+    "* How: Python scraping (LinkedIn was… not a fan)\n",
+    "* Data collected includes:\n",
+    "\n",
+    "  * Job title\n",
+    "  * Job description\n",
+    "  * Company\n",
+    "  * Location\n",
+    "  * Job URL\n",
+    "\n",
+    "### Limitations 😅\n",
+    "\n",
+    "* Salary data was mostly missing and not used\n",
+    "* Scraping limits reduced the size of the dataset\n",
+    "* Job descriptions vary a lot in length and quality\n",
+    "\n",
+    "\n",
+    "## Data Processing 🧹\n",
+    "\n",
+    "Turning chaos into slightly less chaos:\n",
+    "\n",
+    "* Removed duplicate job postings\n",
+    "* Dropped postings with missing job descriptions\n",
+    "* Cleaned job text by:\n",
+    "\n",
+    "  * Lowercasing\n",
+    "  * Removing punctuation, numbers, and special characters\n",
+    "  * Removing stopwords\n",
+    "* Filtered very short job descriptions (LDA does not like those)\n",
+    "\n",
+    "\n",
+    "## Feature Engineering 🧠✨\n",
+    "\n",
+    "* Created a cleaned text column specifically for NLP\n",
+    "* Inferred job context features from text:\n",
+    "\n",
+    "  * Work setting (remote / hybrid / on-site)\n",
+    "  * Seniority level (junior vs senior, based on keywords)\n",
+    "* Converted job descriptions into a document–term matrix (DTM) using `CountVectorizer`\n",
+    "\n",
+    "Basically: turning words into numbers so LDA can do its thing.\n",
+    "\n",
+    "\n",
+    "## Modeling & Analysis 📊\n",
+    "\n",
+    "The main analysis uses **Latent Dirichlet Allocation (LDA)** to identify common themes in job descriptions.\n",
+    "\n",
+    "Steps:\n",
+    "\n",
+    "* Built DTMs from cleaned job descriptions\n",
+    "* Trained LDA models with different numbers of topics\n",
+    "* Selected a topic count that balanced interpretability and coherence\n",
+    "* Examined top keywords per topic to interpret themes\n",
+    "* Compared how topic prevalence differs across:\n",
+    "\n",
+    "  * Work settings\n",
+    "  * Seniority levels\n",
+    "\n",
+    "\n",
+    "## Results & Insights 🔍\n",
+    "\n",
+    "What the model revealed:\n",
+    "\n",
+    "* Job postings consistently cluster around themes like:\n",
+    "\n",
+    "  * Technical skills\n",
+    "  * Communication and collaboration\n",
+    "  * Responsibility and ownership\n",
+    "  * Company culture and impact\n",
+    "* Senior roles emphasize leadership, ownership, and strategy\n",
+    "* Junior roles focus more on learning, support, and execution\n",
+    "* Remote roles tend to highlight autonomy and communication more than on-site roles\n",
+    "\n",
+    "Job titles don’t tell the full story — the language does.\n",
+    "\n",
+    "\n",
+    "## Tools & Tech 🛠️\n",
+    "\n",
+    "* Python\n",
+    "* pandas, numpy\n",
+    "* matplotlib\n",
+    "* scikit-learn\n",
+    "* Jupyter Notebook\n",
+    "\n",
+    "\n",
+    "## Limitations 🙃\n",
+    "\n",
+    "* Dataset size is limited by scraping restrictions\n",
+    "* Topic interpretation is subjective (LDA gives themes, not labels)\n",
+    "* Results reflect a snapshot in time, not long-term trends\n",
+    "\n",
+    "\n",
+    "## Future Improvements 🚀\n",
+    "\n",
+    "If I had more time (and fewer LinkedIn blocks):\n",
+    "\n",
+    "* Collect more data across industries and regions\n",
+    "* Use dynamic topic modeling to track changes over time\n",
+    "* Compare LDA with embedding-based topic models\n",
+    "* Connect themes more directly to job outcomes (like seniority or pay)\n",
+    "\n",
+    "\n",
+    "## Ethical Stuff 🧑‍⚖️\n",
+    "\n",
+    "* Only publicly available job postings were used\n",
+    "* No personal data was collected\n",
+    "* This project is for academic and educational purposes only (promise)\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}