Dual-Encoder Weakly Supervised Biomarker Prediction from Histopathology Whole Slide Images

Overview

This project implements an end-to-end weakly supervised computational pathology pipeline for predicting molecular biomarkers directly from H&E Whole Slide Images (WSIs).

The pipeline uses:

• GigaTIME and GigaPath for pathology feature extraction (two different pipelines)
• CLAM-based Multiple Instance Learning (MIL) for slide-level prediction
• Repeated Stratified K-Fold Cross Validation for robust evaluation
• Attention heatmaps for interpretability

Current experiments focus primarily on P53 biomarker prediction.

---

Problem Statement

Molecular biomarkers such as P53 play an important role in diagnosis and treatment planning of glioma. Conventional biomarker identification relies on molecular assays, which can be expensive and time-consuming.

Histopathology images contain latent morphological patterns that may correlate with molecular information.

Challenges:

• Whole Slide Images are gigapixel scale
• Patch-level labels are unavailable
• Biomarkers are not visually obvious
• Limited dataset size

This work addresses these challenges using weakly supervised Multiple Instance Learning.

---

Dataset

Dataset used:

IPD-Brain (IIIT Hyderabad)

Dataset details:

Attribute	Value
Dataset	IPD-Brain
Total Slides Used	~63
Biomarker Focus	P53
Image Format	.vsi
Labels	Slide-level labels

The baseline paper used approximately 547 slides, making the current experiments more challenging due to limited data availability.

---

Pipeline Architecture

Overall workflow:

Whole Slide Image
        ↓
Tissue Detection
        ↓
Patch Extraction
        ↓
   [ Feature Encoding Branch ]
   ↙                         ↘
GigaTIME                 Prov-GigaPath
(UNet++ Base)              (ViT Base)
   ↘                         ↙
        Slide Feature Matrix
        ↓
Slide Feature Matrix
        ↓
CLAM Multiple Instance Learning
        ↓
Biomarker Prediction
        ↓
Attention Heatmaps

---

Step 1: Tissue Detection and Patch Extraction

Whole Slide Images are extremely large and cannot be directly processed on GPUs.

Processing steps:

1. Generate low-resolution slide thumbnail
2. Convert image to HSV
3. Apply Otsu thresholding
4. Create tissue mask
5. Remove background regions
6. Extract patches

Patch configuration:

Parameter	Value
Patch Size	256×256(GigaTime), 224x224(GigaPath)
Patch Naming	x_y.jpg
Background Removal	HSV + Otsu

Output:

patches/
    slide001/
        0_0.jpg
        256_0.jpg
        512_0.jpg

---

Step 2: Foundation Model Feature Extraction

The pipeline supports dual projection methods to represent the tissue mathematically.

Route A: GigaTIME Feature Extraction

GigaTIME is a pathology-oriented deep learning model based on UNet++ architecture.

Input:

RGB patch
[3×256×256]

Output:

23 biological feature channels

Feature generation process:

out=model(image)

pooled=out.mean(dim=[2,3])

Final patch representation:

23-dimensional vector

For a slide with N patches:

Slide representation:

[N × 23]

Route B : GigaPath Feature Extraction

Prov-GigaPath is a massive Vision Transformer (ViT) pre-trained on over 1.3 billion pathology image patches, excelling at capturing global, complex contextual relationships across tissue types.

Input:

patch : [224x224]

Output & Final patch representation:

1536-dimensional high-fidelity embedding vector

For a slide with N patches:

Slide representation:

[N x 1536]

---

Step 3: Feature Expansion (only for GigaTime)

To increase representational capacity, feature vectors were expanded.

Original:

[N × 23]

Expanded:

features
features²
log(abs(features))

Final:

[N × 69]

---

Step 4: CLAM Multiple Instance Learning

Since labels exist only at slide level:

• Patch = instance
• Slide = bag

CLAM architecture:

Input Features (69 for GigaTime, 1536 for GigaPath) 

        ↓

Fully Connected Layer

        ↓

ReLU

        ↓

Dropout

        ↓

Multi-head Attention

        ↓

Top-K Patch Selection

        ↓

Weighted Aggregation

        ↓

Classifier

        ↓

P53 Prediction

Model configuration:

Parameter	Value
Hidden Dimension	128
Attention Heads	4
Top-K Patches	50
Dropout	0.3

---

Training Configuration

Parameter	Value
Optimizer	Adam
Learning Rate	1e-4
Epochs	20
K-folds	5
Repeats	2
Primary Metric	AUC

---

Cross Validation Strategy

Repeated Stratified K-fold validation:

Repeat 1:
    Fold 0
    Fold 1
    Fold 2
    Fold 3
    Fold 4

Repeat 2:
    Fold 0
    Fold 1
    Fold 2
    Fold 3
    Fold 4

Each split:

• Training = 80%
• Validation = 20%

Stratification preserves class distribution.

---

Experimental Results (GigaTime)

Repeat 1

Fold	AUC
Fold 0	0.9286
Fold 1	0.8345
Fold 2	0.7500
Fold 3	0.7222
Fold 4	0.4338

---

Repeat 2

Fold	AUC
Fold 0	0.9762
Fold 1	0.7143
Fold 2	0.3889
Fold 3	0.8889
Fold 4	0.9167

---

Final Performance

Mean AUC:

0.7554

Peak AUC:

0.9762

---

Experimental Results (GigaPath)

Repeat 1

Fold	AUC
Fold 0	0.8333
Fold 1	0.8810
Fold 2	0.9761
Fold 3	0.8333
Fold 4	0.8333

---

Repeat 2

Fold	AUC
Fold 0	0.8095
Fold 1	0.9048
Fold 2	0.8095
Fold 3	0.9722
Fold 4	0.7778

---

Final Performance

Mean AUC:

0.863

Peak AUC:

0.9761

---

Observations

Although the average AUC was approximately 0.755 (GigaTime) and 0.863 (GigaPath) , individual folds achieved performance near 0.97.

Performance variation likely occurs because:

• Dataset size is small
• P53 signal is weak
• Certain folds contain difficult cases
• Some validation sets become imbalanced

Despite using only ~63 slides:

• Mean performance exceeded the baseline (~0.73)
• Peak performance was significantly higher

---

Ensemble Prediction

To improve robustness:

Top-performing models can be combined:

Prediction =
Average(
Fold0_Rep1,
Fold0_Rep2,
Fold4_Rep2
)

Advantages:

• Reduces fold variance
• Improves stability
• Better generalization

---

Heatmap Generation

Attention scores from CLAM are used to generate heatmaps.

Pipeline:

Features
    ↓

Attention scores
    ↓

Gamma correction
    ↓

Gaussian smoothing
    ↓

Turbo color mapping
    ↓

Heatmap

Heatmaps allow visualization of tissue regions contributing to predictions.

---

Directory Structure

gigatime_pipeline/

├── config.py
├── gigatime_model.py

├── scripts/
│   ├── extract_gigatime_features.py
│   ├── train_clam_spatial_mb.py
│   ├── evaluate_saved_models.py
│   ├── predict_ensemble.py
│   ├── heatmap.py
│   └── check_features.py

├── models/
├── features/
├── patches/
├── heatmaps/
├── raw_data/


gigapath_pipeline/

├── config.py
├── gigapath_model.py

├── scripts/
│   ├── extract_patches.py
│   ├── normalize_patches.py
│   ├── extract_features.py
│   ├── model.py
│   ├── train.py
│   ├── evaluate.py 
│   ├── predict.py
│   └── heatmap.py

├── models/
├── features/
├── patches/
├── heatmaps/
├── raw_data/

---

Run Pipeline

Feature extraction:

python gigatime_pipeline_scripts/extract_gigatime_features.py (for GigaTime features)
python gigapath_pipeline_scripts/extract_features.py (for GigaPath features)

Training:

python gigatime_pipeline/scripts/train_clam_spatial_mb.py
python gigapath_pipeline/scripts/train.py

Evaluate saved models:

python gigatime_pipeline_scripts/evaluate_saved_models.py
python gigapath_pipeline_scripts/evaluate.py

Generate heatmaps:

python scripts/heatmap.py

---

Future Improvements

Potential future work:

• Larger dataset collection
• External validation
• Multi-biomarker prediction
• Spatially-aware MIL
• Graph-based aggregation
• GigaPath slide encoder experiments

---