A production-grade, end-to-end MLOps pipeline and interactive auditing cockpit designed for corporate insider threat detection and financial internal control. Built natively to run entirely on cost-free, open-source components under strict CPU-only enterprise constraints, this platform processes millions of raw event logs, engineers advanced behavioral vectors, and executes a synchronized dual-layer machine learning engine to catch and classify rogue user actions.
The platform relies on a decoupled, pipeline-centric architecture to ingest, process, train, and visualize enterprise audit logs safely without soluton overheads.
- Challenge: Processing the massive
http.csvfile (over 22 million rows of dense web logs) causes the standardpd.read_csv()to trigger silent Linux Kernel Out-Of-Memory (OOM) crashes. - Solution: Re-engineered the ETL layer into an iterative Group-by-Chunk engine (
chunksize=500000). It extracts, tokenizes, and aggregates textual payload metadata locally within a streaming buffer, locking memory consumption permanently below 2 GB.
- Challenge: Ground-truth insider threats occur on only 1,364 out of 330,452 user-days (~0.41%). Standard tree models drop anomaly categorization down to 0.00% to maximize global accuracy shortcuts.
- Solution: Implemented a stratified K-Nearest Neighbor SMOTE oversampling step exclusively on the isolated training split, combined with an automated heterogeneous bagging array (LightGBM, CatBoost etc.) using dynamic depth and feature fraction cycling. This raised the Anomaly Precision to 0.64 and Recall to 0.61.
- Challenge: User-day metrics like
_min,_max,_median, and_meandurations are perfectly collinear because users typically operate a single designated workstation. This splits and distorts feature importance assignments. - Solution: Built an automated pre-modeling Pearson correlation matrix selector (
threshold > 0.95) to automatically prune redundant twins, condensing the inputs into a crisp, high-variance feature selection matrix.
Ensure your local Linux workspace directory has python configured, then install the optimized open-source stack:
# Clone the repository workspace
git clone https://github.com
cd InternalMonitoringSystem
# Initialize and activate the python virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install the exact production dependencies
pip install pandas numpy lightgbm catboost scikit-learn imbalanced-learn shap plotly streamlit joblib pyarrow# Step 1: Run low-memory streaming chunk aggregation over your data downloads
python feature_*.py
# Step 2: Clear multi-collinearity noise and rank feature informativeness
python select_features.py
# Step 3: Prepare dataset for ML Engines
python preparing_dataset.py
# Step 4: Run the SMOTE resampling and train the dual-layer model grid
python train_*.py# Fire up the Streamlit frontend web cockpit server
streamlit run app_v3.pyOpen http://localhost:8501 in your browser window. Toggle the sidebar filter to "High-Risk Alerts Only", select an employee timeline, and execute live auditing inferences.
Auditors can log comments directly onto the compliance panel. The interface uses a custom multi-model TreeSHAP average calculator to combine the feature importance matrices of your bagging boosters into a unified horizontal bar plot. Clicking the download link compiles a formal, secure text report file complete with timestamped telemetry evidence strings and final disciplinary action routing stamps, providing ironclad audit logs for corporate compliance reviews.
Distributed under the MIT License. See LICENSE for more information.