diff --git a/DATA400 - FINAL PROJECT PROGRESS PRESENTATION.pdf b/DATA400 - FINAL PROJECT PROGRESS PRESENTATION.pdf new file mode 100644 index 0000000..1bf0957 Binary files /dev/null and b/DATA400 - FINAL PROJECT PROGRESS PRESENTATION.pdf differ diff --git a/Idea2.md b/Idea2.md new file mode 100644 index 0000000..ba39f88 --- /dev/null +++ b/Idea2.md @@ -0,0 +1,53 @@ +# Idea 2: Detecting Fake Disaster Tweets + +## 1) Research Question + +How can we develop an accurate machine learning model to classify disaster-related tweets as real or fake? + +## 2) Data Source + +The primary data source for this project would be the ["Natural Language Processing with Disaster Tweets"](https://www.kaggle.com/competitions/nlp-getting-started/data) dataset from Kaggle. This dataset contains: + +- Labeled tweets classified as real disaster tweets or fake/non-disaster tweets +- Tweet text +- Keywords +- Location information (where available) + +Additional data could potentially be gathered from Twitter's API to expand the dataset if needed. + +## 3) Discussion + +This project aims to develop a machine learning model that can accurately distinguish between real disaster-related tweets and fake or non-disaster tweets. This has important applications for disaster response organizations, news agencies, and emergency services who rely on social media for real-time information during crises. + +Key steps in the project would include: + +1. **Exploratory Data Analysis** + - Analyze the distribution of real vs fake tweets + - Examine tweet length, word frequencies, and other text characteristics + - Visualize common words and n-grams in real and fake tweets + +2. **Data Preprocessing** + - Clean text data (remove URLs, special characters, etc.) + - Tokenization and lemmatization + - Remove stop words + - Convert text to numerical features (e.g., using TF-IDF or word embeddings) + +3. **Model Development** + - Split data into training and test sets + - Experiment with different algorithms: + - Traditional ML: Naive Bayes, Random Forest, SVM + - Deep Learning: LSTM, Bidirectional LSTM, BERT + - Use techniques like cross-validation and hyperparameter tuning + +4. **Model Evaluation** + - Compare model performance using metrics like accuracy, precision, recall, F1-score + - Analyze confusion matrix to understand model errors + - Perform error analysis on misclassified tweets + +5. **Deployment** + - Develop a simple web application to classify new tweets + - Deploy the model into pickle file + +Potential challenges include dealing with class imbalance, handling out-of-vocabulary words, and generalizing to new disaster types. The project could be extended by incorporating additional features like user metadata or tweet engagement metrics. + +This project would provide valuable insights into the characteristics of real and fake disaster tweets and contribute to more effective social media monitoring during emergencies. diff --git a/Idea3.md b/Idea3.md new file mode 100644 index 0000000..75b7112 --- /dev/null +++ b/Idea3.md @@ -0,0 +1,60 @@ +# Detecting Problematic Internet Usage in Adolescents + +## 1) Research Question + +How can we develop a machine learning model to accurately detect problematic internet usage patterns in adolescents using smartphone sensor data? + +## 2) Data Source + +The primary data source for this project is the [dataset](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/data) provided by the Child Mind Institute in their Kaggle competition "Detect Problematic Internet Usage". This dataset includes: + +- Accelerometer data +- Gyroscope data +- Screen state (on/off) information +- Timestamps for each data point +- Labels indicating periods of problematic internet usage + +## 3) Discussion + +This project aims to create a machine learning model that can identify patterns of problematic internet usage in adolescents using smartphone sensor data. This has important applications in mental health, particularly in early intervention and prevention of internet addiction among young people. + +Key steps in the project would include: + +1. **Exploratory Data Analysis** + - Analyze the distribution of problematic usage periods + - Examine patterns in accelerometer and gyroscope data + - Visualize screen state patterns over time + - Investigate correlations between sensor data and problematic usage + +2. **Data Preprocessing** + - Handle missing data points + - Normalize accelerometer and gyroscope readings + - Create time-based features (e.g., time of day, day of week) + - Generate rolling statistics (mean, variance) over different time windows + +3. **Feature Engineering** + - Extract relevant features from raw sensor data + - Create features based on screen state transitions + - Develop features that capture prolonged periods of inactivity or constant motion + - Incorporate frequency domain features using Fourier transforms + +4. **Model Development** + - Split data into training and validation sets + - Experiment with different algorithms: + - Traditional ML: Random Forest, Gradient Boosting, SVM + - Deep Learning: LSTM, 1D CNN, Transformer models + - Implement time series cross-validation + - Use techniques like hyperparameter tuning and ensemble methods + +5. **Model Evaluation** + - Evaluate models using competition metric (Average Precision) + - Analyze false positives and false negatives + - Assess model performance across different times of day and days of week + +6. **Interpretability and Insights** + - Use techniques like SHAP values to understand feature importance + - Identify key patterns that indicate problematic usage + - Provide actionable insights for intervention strategies + +Potential challenges include dealing with imbalanced data, handling varying smartphone usage patterns across individuals, and ensuring privacy and ethical considerations in model deployment. +. diff --git a/Long Bui - Idea.md b/Long Bui - Idea.md new file mode 100644 index 0000000..c248f7e --- /dev/null +++ b/Long Bui - Idea.md @@ -0,0 +1,45 @@ +**Research Question:** How does the presence and funding of technology institutions correlate with incomes, living standards, development rates, and educational outcomes in different communities? + +**Data Sources:** +1. National Center for Education Statistics (NCES) - Data on technology-focused institutions, funding, enrollment +2. U.S. Census Bureau - American Community Survey data on income levels, educational attainment, poverty rates +3. Bureau of Economic Analysis - GDP and economic growth data by county/metro area +4. National Science Foundation - Research funding data for institutions +5. Department of Education - College Scorecard data on student outcomes + +**Data Collection and Variables:** +- Identify technology-focused institutions (e.g. technical colleges, research universities with strong STEM focus) using NCES data +- Collect data on institution funding, enrollment, research output over time +- Gather community-level data on: + - Median household income + - Educational attainment rates + - Poverty rates + - GDP growth + - Employment in tech sectors + - Patent applications +- Use geographic identifiers to match institution and community data + +**Analysis Plan:** +1. Exploratory Data Analysis: + - Visualize trends in tech institution presence/funding and community outcomes over time + - Examine correlations between variables + +2. Regression Analysis: + - Use panel data models to estimate the impact of tech institutions on community outcomes, controlling for other factors + - Model specification example: + log(Income_it) = β0 + β1(TechInstitutionFunding_it) + β2(Controls_it) + αi + γt + εit + Where i represents communities, t represents time periods + +3. Geospatial Analysis: + - Create maps showing the distribution of tech institutions and economic outcomes + - Analyze spillover effects to nearby communities + +4. Causal Inference: + - Exploit policy changes or openings/closures of institutions as natural experiments + - Use difference-in-differences or instrumental variable approaches to estimate causal effects + +5. Heterogeneity Analysis: + - Examine differential impacts across urban/rural areas, regions, types of institutions since urban may witness positive correlations but rural may not. + +**Conclusion:** +This project would provide insights into how technology-focused educational institutions contribute to local economic development and human capital formation. It combines elements of education policy, economic development, and labor economics while utilizing diverse data sources and analytical techniques. \ No newline at end of file