A production-minded ML system that finds and masks cancelled / scribbled-out regions on handwritten answer sheets before an OCR/grading model reads them — with a statistically certified guarantee on the one error that actually matters: hiding a real answer.
- The hard part isn't detection — it's safety. Missing a scribble is cheap; hiding a student's real answer is not. The system is built precision-first and abstains when unsure.
- Certified ≤ 2% chance of hiding a valid answer at 95% confidence (conformal risk control); empirically ~0.1% on a frozen, human-labeled test set.
- RF-DETR transformer detector, trained cost-efficiently on serverless GPUs, with post-hoc confidence calibration so a masking threshold is actually trustworthy.
- A two-tier architecture (cheap detector + VLM-as-teacher) validated with controlled experiments — including the negative results that ruled out the wrong paths.
| 🎯 Safety as a guarantee, not a hope | Conformal prediction (Hoeffding–Bentkus RCPS) turns a noisy confidence score into a provable error bound. |
| 🔬 Calibration done right | Per-class isotonic / Platt scaling; reliability diagrams; the masking threshold is derived from a risk target, not hand-picked. |
| 💸 Cost-aware engineering | L4 over A100 (the model isn't compute-bound); bulk inference ≈ pennies per 1k pages; the expensive VLM is used only offline as a teacher. |
| 🧪 Honest evaluation | Frozen, leakage-safe split; confusion matrices (detection and decision); compression-robustness; documented negative results. |
Students cross out, scribble over, and cancel work on exam pages. An OCR/grading model that reads that noise produces wrong transcriptions and wrong grades. The task: detect cancelled regions and mask them before OCR — cheaply enough to run on every page, and safely enough to (almost) never hide a real answer.
A cost asymmetry drives every design choice:
- False negative (miss a scribble) → grader sees struck-out work. Mild, recoverable.
- False positive (mask a real answer) → possible wrong grade. Dangerous.
→ optimize precision + a calibrated abstain policy, not raw accuracy.
- Data pipeline — normalized multi-reviewer annotations into a clean 6-class dataset; frozen train/val/test split, leakage-safe (1,195 / 149 / 150 pages).
- Detector — fine-tuned RF-DETR (DINOv2-backbone transformer detector) on serverless GPUs; benchmarked against YOLO across model sizes and resolutions.
- Confidence calibration — per-class isotonic / Platt scaling so a score of "0.8" means ~80% precision (reliability diagram below).
- Certified threshold — Conformal Risk Control (Hoeffding–Bentkus) selects the most aggressive masking threshold that still provably keeps the valid-answer-hidden rate ≤ α.
- Two-tier architecture — a cheap detector for small marks (runtime) + a VLM-as-teacher for large cancelled blocks (offline labeling → distillation). Both directions validated experimentally.
- Evaluation harness — detection + decision confusion matrices, per-class precision/recall, compression robustness; re-runnable on the frozen set for regression tracking.
Safety vs. coverage is an explicit, certified dial:
| Guaranteed valid-answer-hidden | Coverage (cancellations caught) | Empirical harm |
|---|---|---|
| ≤ 2% (95% conf) | ~24% | ~0.1% |
| ≤ 3% | ~28% | ~0.1% |
| ≤ 5% | ~46% | ~0.9% |
Left — mask/keep decision matrix: 0 valid answers hidden at the safe threshold. Right — the safety↔coverage curve: coverage is a tunable dial, bounded by the safety guarantee.
Calibration works — raw confidence is miscalibrated; post-hoc calibration pulls it onto the diagonal:
- Cost-aware to the cent. Chose L4 over A100 (small model → not compute-bound → ~3–5× cost for ~1.3× speed). Bulk inference over 10k pages ≈ $0.25–1.20.
- Smoke-test before spend. A 1-epoch probe + reading the resolved training config caught that a proposed focal-loss / repeat-factor-sampling change was a no-op (the model already uses IA-BCE; the kwargs were silently ignored) — saving a wasted multi-hour GPU run.
- Knowing what doesn't work is the result. Off-the-shelf VLMs are useless as box detectors (0% box recall, and they masked whole valid answers) but excellent as offline teachers for large cancelled blocks (54% recall / 78% precision). That asymmetry is the architecture.
- Calibration ≠ accuracy. Calibration made the threshold trustworthy; raising coverage needs more data — diagnosed, not conflated.
Python · RF-DETR / PyTorch · serverless GPU (Modal) · scikit-learn (isotonic / Platt) · conformal prediction (RCPS) · OpenCV / Pillow · vision-language models (offline teacher)
scripts/ training, calibration, confusion-matrix, inference, VLM probe
knowledge_base/ distilled, fully-grounded engineering docs (00–13)
research/ raw research outputs (calibration methods, citations)
docs/assets/ charts used in this README
- Proof-of-concept, not deployed. The detector is safe-but-conservative (~24% coverage at the safe threshold); raising coverage is a data problem, not a threshold one.
- A ≤1% guarantee isn't certifiable yet (needs ~300+ labeled calibration pages; currently 149).
- Every number is reported at a stated operating point on frozen, human-labeled data — no cherry-picking.
Trained and evaluated on anonymized handwritten answer-sheet data. No PII or student work is included in this repository. All metrics are measured on a frozen, leakage-safe split.