English | 中文

RiskScore

Credit Risk Scoring

WOE/IV · XGBoost/LightGBM · Leakage-free · SHAP Interpretable

🏠 Primary: Gitee · GitHub: MeaFew/riskscore

English | 中文

---

Headline

On real Kaggle Home Credit Default Risk data (307,511 samples / 141 features / 8.07% default rate), XGBoost achieves the following under 5-fold stratified cross-validation with leakage-free out-of-fold (OOF) evaluation:

OOF AUC 0.766 · KS 0.399 · Gini 0.533 — clearly above the single-table logistic-regression baseline (AUC 0.626) and approaching the competition Top 10% band (~0.795, which requires multi-table features + ensembling).

Model	AUC	KS	Gini
Logistic Regression	0.626	0.192	0.251
Random Forest	0.746	0.365	0.491
XGBoost	0.766	0.399	0.533
LightGBM	0.766	0.398	0.532

Source: reports/model_results.json — mean of 5-fold stratified CV on single-table features (application_train); XGBoost OOF AUC = 0.766 (each row scored by a model that never saw it, a faithful generalization estimate). Target encoding is fitted per CV fold inside a sklearn Pipeline on the fold's training split only, so validation rows never encode their own target.

Leakage-free Design (core selling point)

A credit-risk model is only as trustworthy as its evaluation is honest. This pipeline corrects three common leakage patterns:

• Target-encoding leakage: an earlier version computed target encoding on the full training set, then fed it into 5-fold CV — so a validation row's own target leaked into its encoded feature and inflated AUC. Target encoding now lives in a sklearn Pipeline and is fit on each fold's training split only. IV-based feature selection was removed from the modeling path (kept only as the analytic data/processed/iv_report.csv) to avoid the same class of leak via "select features with the full-set target, then use them inside CV folds".
• Fabricated test AUC: Home Credit's application_test.csv has no TARGET, so there is no labeled holdout. An earlier version split the training set 80/20, retrained on the full data, and evaluated — producing a 0.80 "test AUC" that was a resubstitution number and, worse, higher than the honest CV AUC. This metric was removed; we now report OOF AUC.
• Preprocessing leakage: outlier caps and median imputation are now fit on train only, then transformed onto test.

Overview

End-to-end credit risk scoring pipeline built on the Kaggle Home Credit Default Risk dataset: professional-grade feature engineering (WOE/IV) → model comparison (LR → RF → XGBoost → LightGBM) → SHAP-based interpretability → a Streamlit risk-calculator dashboard.

Quick Start

# Clone from GitHub
git clone https://github.com/MeaFew/riskscore.git
# or from Gitee (faster in China): git clone https://gitee.com/zeroonei1/riskscore.git
cd riskscore

# Create and activate a Python 3.11 virtual environment
python -m venv .venv
# Linux / macOS: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1

# Install locked dependencies, the package, and development tools
make setup
# Windows without GNU Make: python -m pip install -r requirements.lock
#                           python -m pip install -e ".[dev]"

# Download real dataset (GitHub Releases, ~40MB)
bash download_data.sh

# Run full pipeline
make all
# Windows without GNU Make: python run_all.py

# Or step by step
make preprocess
make features
make train
make evaluate
make shap

# Launch dashboard
make dashboard

# Quality gates
make verify

Key Highlights

• Feature Engineering: WOE binning (analytic reference), per-fold target encoding (leakage-free), cross-features
• Model Stack: Logistic Regression (baseline) → Random Forest → XGBoost → LightGBM
• Evaluation: AUC, KS, Gini, calibration curves, confusion matrix at optimal threshold
• Interpretability: SHAP summary, dependence plots, force plot for individual cases
• Delivery: Streamlit dashboard with risk calculator

Tech Stack

Layer	Tools	Notes
ETL	pandas, scikit-learn	Missing value imputation, outlier capping
Feature Eng	Custom WOE/IV	Quantile-based binning with smoothing
Modeling	XGBoost, LightGBM, sklearn	5-fold stratified CV
Interpretability	SHAP	TreeExplainer for gradient boosting models
Evaluation	scipy, sklearn	AUC, KS, Gini, PR curve, calibration
Delivery	Streamlit	Interactive risk calculator + model comparison
Quality	pytest, ruff, GitHub Actions	CI runs lint + tests on every push

Project Structure

.
├── src/riskscore/
│   ├── generate_mock_data.py      # Synthetic data generator (for CI)
│   ├── preprocess.py              # Data cleaning & missing value handling
│   ├── feature_engineering.py     # Cross-features + WOE/IV analytic report (target encoding moved into CV Pipeline)
│   ├── train_models.py            # LR / RF / XGB / LGBM with CV
│   ├── evaluate.py                # ROC, PR, calibration, confusion matrix
│   ├── shap_analysis.py           # SHAP summary, dependence, force plots
│   ├── aggregate_auxiliary_features.py
│   └── merge_auxiliary_features.py
├── dashboard/
│   └── app.py                     # Streamlit interactive dashboard
├── tests/
│   └── test_pipeline.py           # Unit + integration tests
├── Makefile                       # Workflow orchestration
└── requirements.lock              # Reproducible dependency pins

Benchmark

Based on Kaggle Home Credit Default Risk (7,190+ teams, metric: AUC-ROC).

Reference	AUC	Notes
Kaggle Starter Baseline	0.688	Official starter notebook, no feature engineering
Single-table Logistic Regression	0.748	application_train only + GridSearchCV
Single-table LightGBM	0.749	Same as above, gradient boosting
Competition Median	~0.72-0.75	Leaderboard median
Competition Top 10%	~0.795	Multi-table features + ensemble
This Project (single-table)	0.766	Per-fold target encoding (leakage-free) + XGBoost/LightGBM (5-fold CV / OOF)

Note: the competition Private Leaderboard is closed. Scores above are from local 5-fold stratified cross-validation plus leakage-free out-of-fold (OOF) evaluation on real Kaggle data (307,511 train samples, application_train). Auxiliary-table aggregation modules are included, but no multi-table score is shown until it has been validated under the same protocol.

Additional evaluation plots

Related Projects

Project	Repo	Description
E-commerce User Analytics	MeaFew/shoplytics	29M real user behavior records, 10 analytical modules
Marketing Attribution & MMM	MeaFew/attributor	MMM + multi-touch attribution + budget optimization
Multivariate Time Series	MeaFew/foresight	LSTM / Transformer / XGBoost time series forecasting
Graph Fraud Detection	MeaFew/graphguard	GNN illicit transaction detection (Elliptic)

License

MIT