Data Cleaning Automation CLI This is a command-line application built with Python and Pandas to automate the essential steps of data cleaning for .csv and .xlsx files.
The script inspects a given dataset for duplicate records and missing values, processes them based on defined rules, and saves the cleaned data and a separate file with the removed duplicates.
Features File Type Agnostic: Works seamlessly with both CSV (.csv) and Excel (.xlsx) files.
Duplicate Detection: Identifies and counts all duplicate rows in the dataset.
Duplicate Segregation: Saves a copy of all identified duplicate rows to a separate CSV file for review.
Duplicate Removal: Creates a new, clean dataset with all duplicates removed.
Missing Value Analysis: Detects and reports the total number of missing values and provides a column-by-column breakdown.
Smart Null Value Handling:
For numeric columns (integers, floats), it replaces null values with the column's mean.
For non-numeric columns (objects, strings, etc.), it drops any row containing a null value.
Organized Output: All generated files (cleaned data, duplicates) are saved in a Results folder, which is created automatically if it doesn't exist.
How to Use Prerequisites:
Make sure you have Python installed.
Install the required libraries:
pip install pandas numpy openpyxl xlrd
Prepare Your Data:
Place your dataset (.csv or .xlsx) in a known location (e.g., a folder named Datasets).
Run the Application:
Execute the script from your terminal:
python data_cleaner.py
Follow the Prompts:
The application will ask for two inputs:
Dataset Path: Enter the full path to your file (e.g., Datasets/sales.xlsx).
Dataset Name: Provide a short, descriptive name without spaces or extensions (e.g., sales_data). This name will be used as a prefix for the output files.
Check the Results:
After the script finishes, a Results folder will be created in your project directory.
Inside, you will find:
your_data_name_cleaned.csv: The main cleaned dataset.
your_data_name_duplicates.csv: The file containing only the removed duplicate rows (if any were found).
Example Input:
Please enter the full path to your dataset (e.g., Datasets/sales.xlsx): Datasets/sales.xlsx Please enter a short name for your dataset (e.g., sales_data): sales_report_q1
Output files in Results folder:
sales_report_q1_cleaned.csv
sales_report_q1_duplicates.csv