EvalForge

An Open-Source Model Health & Evaluation Intelligence Engine

EvalForge is a pre-deployment reliability engine for machine learning models. It evaluates robustness, calibration, drift, stability, and hidden failure modes, summarized into a unified Model Health Score (0-100).

---

Current Capabilities

• Model Health Score (0-100): Weighted analysis across accuracy, calibration, fragility, drift, stability, and blind spots.
• Python API (ModelAuditor): Clean, object-oriented interface for orchestrating evaluations programmatically.
• CLI Analysis: Robust command-line tool with flags for stability and cluster-wise blind spot mapping.
• Diagnostics Layer:
  • Confidence-Accuracy Mismatch Detection.
  • Adversarial Fragility Scoring.
  • Distribution Drift Detection (KS test).
  • Seed Stability Analysis (Mean/Std across runs).
  • Cluster-wise Blind Spot Mapping (K-Means).
• Reporting: Structured markdown report cards and shareable HTML exports with visual evidence.

Health Score Weights

• Accuracy / core model performance: 35%
• Calibration / mismatch analysis: 15%
• Fragility / robustness under perturbation: 15%
• Drift detection: 15%
• Stability across runs or seeds: 10%
• Blind spot clustering penalty: 10%

When some diagnostics are not run, EvalForge redistributes only the active weights rather than penalizing the model for missing optional checks.

---

Installation

pip install .

---

Quick Start

Python API Usage

from evalforge.api import ModelAuditor
import pandas as pd
import pickle

# Load your model and data
with open("model.pkl", "rb") as f:
    model = pickle.load(f)
df_test = pd.read_csv("test_data.csv")

# Initialize and evaluate
auditor = ModelAuditor(model, target_col="target")
health_score = auditor.evaluate(df_test, run_blind_spots=True)

# Generate and export report
report = auditor.get_report()
auditor.export_report("reports/my_report.html")

CLI Usage

evalforge analyze \
  --model model.pkl \
  --data test.csv \
  --target "target" \
  --stability "0.91,0.89,0.92" \
  --blind-spots \
  --export-html

Regression CLI Usage

evalforge analyze \
  --model regressor.pkl \
  --data test.csv \
  --target "target" \
  --task-type regression \
  --train-data train.csv \
  --stability "0.82,0.84,0.83"

Documentation

• Usage Guide
• Architecture Notes

---

Next Steps

1. Classification and regression parity: extend regression-safe support for blind spots, reporting language, and fairness diagnostics.
2. Configurable policy layer: allow teams to customize score weights, fail thresholds, and enabled diagnostics per project.
3. Release polish: move packaging metadata to pyproject.toml, publish cleaner developer setup, and separate runtime from dev dependencies.
4. CI hardening: expand beyond the demo smoke check into stronger packaging, compatibility, and release validation.

---

Developer Guide

Environment Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -e .[dev]

Running Tests

python3 -m pytest tests/

• Please ensure your docstrings have a tiny, simple 1-line joke :)
• Stick to the SOLID principles.
• Do not use emojis in the documentation.