Personal Information

Low Voon Bin Robin
e1406565@u.nus.edu

Folder Structure

aiip8-low-voon-bin-robin-885D/
│
├── .github/ 
│   └── workflows
│       └── github-actions.yml
│
├── src/     
│   ├── __init__.py
│   ├── data_loader.py
│   ├── evaluate.py       
│   ├── feature_importance.py     
│   ├── main.py
│   ├── models.py
│   └── preprocess.py
│
├── README.md     
├── eda.ipynb         
├── .gitignore
├── requirements.txt
└── run.sh

Executing Pipeline and Modifying Parameters

To execute the full pipeline, run it from the repository root using bash run.sh (Linux/WSL). The script loads the dataset from data/phishing.db, performs a stratified train/validation/test split (70/10/20), applies preprocessing (median imputation + standardisation for numeric features; most-frequent imputation + one-hot encoding for categorical features), then trains and evaluates three models (Logistic Regression, Random Forest, HistGradientBoosting). For evaluation, it reports the confusion matrix, classification report, and ROC-AUC on both validation and test sets.

The main configurable parameter is MIN_RECALL_PHISH in src/main.py (currently 0.85), which controls the threshold selection policy: the threshold is chosen on the validation set to achieve at least the specified minimum phishing recall (label 0), trading off higher phishing detection against increased false positives. Adjusting MIN_RECALL_PHISH (e.g., 0.90 for stricter phishing detection or 0.80 for fewer false alarms) changes the operating point without retraining the models.

Pipeline flow (end-to-end)

Below is the logical flow of the pipeline from data loading to final evaluation.

1. Load dataset

• Read the phishing_data table from the SQLite database at data/phishing.db into a pandas DataFrame.
• The dataset contains features describing websites and a binary label:
  • label = 0 → phishing
  • label = 1 → legitimate

2. Separate features and target

• Set y = df["label"] as the prediction target.
• Set X = df.drop(["Unnamed: 0", "label"], axis=1) as model inputs. We drop Unnamed: 0 as it appears to be an ID counter.

3. Preprocessing (ColumnTransformer)

We process numeric and categorical columns differently:

• Numeric columns (12 cols)

  1. Impute missing values using median
  2. Standardise with StandardScaler i.e. feature scaling and mean normalisation (mean ≈ 0, std ≈ 1)

• Categorical columns (2 cols)

  1. Impute missing values using most frequent category
  2. Apply One-Hot Encoding (handle_unknown="ignore")

These transformed outputs are horizontally concatenated (stitched) into a single feature matrix for model training.

4. Train/Validation/Test split

We use stratified splits to preserve label balance:

• Split the data into:
  • Train+Val (80%) and Test (20%)
• Split Train+Val into:
  • Train (70% of total) and Validation (10% of total)
    This ensures the test set is only used once for final evaluation.

5. Model training

Each model is trained using the same training split:

1. Logistic Regression (baseline linear model)
2. Random Forest (non-linear tree ensemble)
3. HistGradientBoostingClassifier (boosted trees)

All models are wrapped in a single sklearn Pipeline: preprocess -> classifier

6. Threshold selection

Default classification uses a threshold of 0.5, but phishing detection often prioritises catching phishing over avoiding false alarms.

To reflect this, we select a threshold on the validation set to enforce a minimum phishing recall:

• MIN_RECALL_PHISH = 0.85
• We scan thresholds on P(label=1) (legitimate probability) and pick the threshold that:
  • achieves Recall(phishing=0) ≥ MIN_RECALL_PHISH, and
  • among those thresholds, maximises Precision(phishing=0) (fewer false alarms for the same recall)

The chosen threshold is frozen and then applied to the test set (no re-tuning on test).

7. Evaluation

For each model, we report metrics on:

• Validation (used for threshold selection / model comparisons)
• Test (final, unbiased results)

Metrics:

• Confusion Matrix
• Classification Report (precision/recall/F1 for both classes)
• ROC-AUC (threshold-independent ranking performance)

---

Flow chart

+---------------------+
|  Load SQLite table  |
|  data/phishing.db   |
+----------+----------+
           |
           v
+---------------------+
|  Split X and y      |
|  drop id + label    |
+----------+----------+
           |
           v
+-------------------------------+
|  Preprocess (ColumnTransformer)|
|  - Numeric: median + scale     |
|  - Categorical: mode + one-hot |
+----------+--------------------+
           |
           v
+-------------------------------+
| Stratified split              |
| Train (70%) / Val (10%) /     |
| Test (20%)                    |
+----------+--------------------+
           |
           v
+-------------------------------+
| Train 3 models on Train       |
| - Logistic Regression         |
| - Random Forest               |
| - HistGradientBoosting        |
+----------+--------------------+
           |
           v
+-------------------------------+
| Threshold tuning on Validation|
| Rule A: Recall(phish) >= 0.85 |
| Pick best precision(phish)    |
+----------+--------------------+
           |
           v
+-------------------------------+
| Final evaluation on Test      |
| (apply same threshold)        |
| Confusion matrix / report /   |
| ROC-AUC                       |
+-------------------------------+

Task 1 EDA summary and resulting pipeline choices

EDA was performed in the accompanying eda.ipynb notebook; this section provides a brief summary of the key findings and how they informed the modelling pipeline.

Key EDA findings

• Dataset size and target balance: The dataset contains 10,500 rows. The target labels are reasonably balanced (both classes have substantial support, ~55-45 split), so modelling can rely on standard classification metrics (precision/recall/F1, ROC-AUC) without requiring heavy rebalancing.
• Missing/invalid values: A small number of missing values were observed (e.g., missing values in the LineOfCode column) and some invalid entries (e.g., negative values in NoOfImage). These issues motivated robust handling of missing data in the preprocessing stage.
• Categorical features are informative: The Industry and HostingProvider columns showed noticeable differences in phishing rates across categories e.g. high phishing rates in banking despite low absolute figures, indicating that categorical context contains predictive signal.
• Numeric feature distributions are skewed: Several numeric features exhibited long-tailed / skewed distributions. Visual comparisons (including log-scaled plots) suggested that non-linear models may capture patterns better than purely linear decision boundaries.
• Class-conditional separation: Overlay plots comparing label=0 vs label=1 suggested partial separability in several numeric features, but not perfectly linear separation, motivating the use of tree-based models in addition to a linear baseline.

Pipeline choices based on EDA

• Train/Val/Test split with stratification: We use a stratified 70/10/20 split to preserve label proportions across splits and to keep the test set untouched for final reporting.
• Missing value handling:
  • Numeric columns: median imputation (robust to outliers)
  • Categorical columns: most-frequent (mode) imputation
• Encoding categorical columns: Industry and HostingProvider are one-hot encoded (handle_unknown="ignore") to preserve category-level signal observed in EDA.
• Scaling numeric features: Numeric columns are standardised (feature scaling and mean normalisation) to support stable optimisation for Logistic Regression and to keep a consistent preprocessing pipeline across models.
• Model selection reflecting non-linearity: In addition to Logistic Regression (baseline), we included Random Forest and HistGradientBoosting to capture non-linear relationships suggested by EDA.
• Feature engineering: No additional manual feature engineering was applied beyond:
  • cleaning steps performed in Task 1 (e.g., trimming categorical whitespace and addressing obvious invalid values), and
  • standard preprocessing transformations (imputation, scaling, one-hot encoding).
    The goal was to keep the pipeline reproducible and to let the models learn interactions directly from the provided features.

Full plots, distribution checks, and category-level analyses are documented in eda.ipynb.

Feature processing summary

The pipeline applies different preprocessing steps depending on whether a feature is numeric or categorical. The transformations are implemented using an sklearn ColumnTransformer so the same logic is consistently applied during training and inference.

Feature type	How features are identified	Missing value handling	Transformation	Output format	Why this is done
Numeric (e.g., counts/lengths)	X.select_dtypes(include=["number"])	Median imputation (SimpleImputer(strategy="median"))	Standardisation (StandardScaler)	Dense numeric columns (same count as input numeric cols)	Median is robust to outliers; scaling helps stable optimisation for LR and keeps numeric magnitudes comparable
Categorical (e.g., Industry, HostingProvider)	All non-numeric columns	Most-frequent imputation (SimpleImputer(strategy="most_frequent"))	One-hot encoding (OneHotEncoder(handle_unknown="ignore"))	Expanded binary indicator columns (one per category)	Preserves category-level signal; handle_unknown="ignore" prevents errors when unseen categories appear at inference time

Notes:

• The transformed numeric and one-hot encoded categorical matrices are horizontally concatenated into a single feature matrix used by all models.
• Missing value handling for categorical features was actually not applicable as none of these columns had missing values. However, this is present for data integrity.

Choice of Models

We treat this as a binary classification problem where label=0 indicates a phishing website and label=1 indicates a legitimate website. The dataset contains a mix of numeric features (e.g., counts/lengths) and categorical features (e.g., industry, hosting provider), so we evaluated models that handle tabular data well and provide complementary strengths:

1. Logistic Regression (baseline)

We use Logistic Regression as a simple, interpretable baseline. It provides a strong starting point for binary classification, is fast to train, and its linear decision boundary makes it easier to reason about which features contribute to the prediction. This baseline helps quantify how much performance gain is achieved by more expressive non-linear models.

2. Random Forest (non-linear ensemble)

Random Forest is an ensemble of decision trees that can capture non-linear patterns and feature interactions that Logistic Regression cannot. It typically performs well on tabular datasets without heavy feature engineering, is robust to noise, and provides feature importance measures. We included Random Forest to test whether non-linear relationships in the features improve phishing detection performance.

3. HistGradientBoostingClassifier (boosted trees)

HistGradientBoostingClassifier is a gradient-boosted decision tree model that sequentially improves performance by correcting previous errors. Boosting models often achieve strong results on structured/tabular data and can produce better probability ranking (useful for threshold tuning) compared to a single model. We included it as a stronger non-linear model to compare against Random Forest in terms of both thresholded metrics (accuracy/F1) and ranking quality (ROC-AUC).

Evaluation of Models developed

At first glance, the standard approach would be to use a threshold = 0.5 for preliminary analysis of accuracy, precision and recall. However, accuracy does not provide value contextually as there is disproportionate weight distribution between false positives and false negatives that is unaccounted for. As such, a threshold selection policy is determined below.

Threshold selection policy: In phishing detection, false negatives (missed phishing) are typically more costly than false positives (legitimate sites flagged). We therefore tuned the decision threshold on the validation set to enforce a minimum phishing recall of 0.85. We selected 0.85 as a practical operating point to prioritise detection while avoiding an excessive drop in overall accuracy and an overly high false-alarm rate, which would reduce usability. The chosen threshold is determined on the validation set and then fixed for final evaluation on the test set.

We then enforced Recall(phish=0) ≥ 0.85 on validation, then chose the threshold that maximised phishing precision subject to it.

Model	Threshold	Accuracy	Phish Precision	ROC-AUC
HistGB	0.55	0.8338	0.8058	0.8894
RandomForest	0.66	0.8305	0.7854	0.8833
Logistic	0.70	0.7000	0.6200	0.8018

We observe that for all three measurements (accuracy, phish precision and ROC_AUC), Hist Gradient Boosting performs better than RandomForest. Logistic Regression performed the worst on all statistics.

We then calculated permutation importance for HistGB to determine the importance of each feature. Below are the top 5 features based on Importance Mean.

Feature	Importance Mean	Importance Std
LineOfCode	0.213613	0.009170
NoOfImage	0.024650	0.004959
NoOfSelfRef	0.018128	0.002672
NoOfExternalRef	0.002350	0.003080
HostingProvider	0.001875	0.001324

Permutation importance indicates how much ROC-AUC decreases when a feature is shuffled. LineOfCode was the most influential feature (largest AUC drop), followed by NoOfImage and NoOfSelfRef. Features with near-zero importance contributed little to ranking performance in this model.

Other considerations for deployment

1. Threshold and risk tolerance (operating point)

The same trained model can behave very differently depending on the probability threshold used to convert scores into class labels. In phishing detection, missed phishing (false negatives) often has higher cost than false alarms (false positives). In deployment, the threshold should be selected based on the organisation’s risk tolerance and monitored over time (e.g., enforcing a minimum phishing recall while keeping false-alarm rates acceptable). Threshold tuning should be performed on a validation set and updated only after careful evaluation.

2. Monitoring for data drift and performance decay

Website patterns and attacker behaviour evolve, so feature distributions can drift over time (concept drift). Production monitoring should track:

• input drift (feature distribution changes)
• output drift (changes in predicted probability distributions)
• performance drift (precision/recall using delayed labels when available) When drift is detected, the model may need recalibration, threshold updates, or retraining.

3. Handling unseen categories and schema changes

Categorical columns such as Industry and HostingProvider may introduce new categories over time. The pipeline uses one-hot encoding with handle_unknown="ignore" to avoid runtime failures, but performance should be monitored when unseen categories become frequent. Schema validation (checking column presence/types) should be added to prevent silent failures if upstream data formats change.

4. Latency, throughput, and resource constraints

Tree ensembles (Random Forest / Gradient Boosting) generally require more compute than Logistic Regression. Deployment should consider:

• inference latency requirements
• batch vs real-time scoring
• model size and memory footprint

If latency is strict, Logistic Regression may be preferred despite lower accuracy.

5. Security and adversarial considerations

Phishing detection is adversarial: attackers may intentionally manipulate inputs to evade detection. Defensive considerations include:

• rate limiting and abuse detection around the scoring endpoint
• sanity checks on feature ranges to detect malformed inputs
• regular retraining and monitoring of new evasion patterns
• logging and review workflows for suspicious borderline cases

6. Privacy and compliance

If any features are derived from user or sensitive data, ensure:

• data minimisation (collect only what is needed)
• retention policies and compliance with applicable regulations (e.g. PDPA)