Spike: Enhance OCR payload validation using Pydantic validators

[kvithayathil]

Summary

Spike to explore enhancing validation on OCR payload models using Pydantic's built-in validation capabilities. The current OCREntry / OCRData models accept raw strings and ints with only Field(description=...) metadata — no constraints, no custom validators, no reask/retry logic for bad LLM output.

Motivation

OCR extraction relies on LLM providers (OpenAI, Gemini, Mistral) returning structured JSON that deserializes into OCREntry/OCRData. Today, invalid data (empty names, malformed addresses, negative wards, nonsensical dates) passes through silently and only surfaces as mismatches downstream. Stronger validation at the Pydantic layer would:

• Catch bad OCR output immediately after extraction
• Provide actionable error messages that could be fed back to the LLM for re-asking (Instructor-style)
• Reduce downstream matching failures caused by garbage data

Current State

Key models with no validation beyond type coercion:

• OCREntry (app/ocr/data/data_models.py:6-12) — Name: str, Address: str, Date: str, Ward: int — bare types, no constraints
• OCRData (app/ocr/data/data_models.py:15-16) — wraps list[OCREntry], no min items
• Duplicate OCREntry in app/ocr/ocr_client_factory.py:83-89
• OcrMatchResults (app/matching/response_adapter.py:31-43) — has one field_validator for column sorting, only example of custom validation in the OCR pipeline
• extract_from_encoding_async() (app/ocr/ocr_client_factory.py:211-247) — catches exceptions but does not retry on validation errors

Pydantic Capabilities to Evaluate

Reference: Pydantic Validators docs | Instructor Reask Validation

1. Field-level validators (AfterValidator / field_validator)

from typing import Annotated
from pydantic import AfterValidator, Field

def normalize_name(v: str) -> str:
    if not v.strip():
        raise ValueError("Name must not be empty")
    return v.strip()

class OCREntry(BaseModel):
    Name: Annotated[str, AfterValidator(normalize_name)] = Field(...)
    Ward: int = Field(ge=0, description="Ward number must be non-negative")

2. Model validators (model_validator)

Cross-field checks (e.g., Date format consistency with campaign date range):

@model_validator(mode='after')
def validate_entry_consistency(self) -> Self:
    # Cross-field validation logic
    return self

3. Field constraints (Field(ge=, le=, min_length=, pattern=))

Built-in constraints for common cases without writing custom validators:

Name: str = Field(min_length=2, description="...")
Ward: int = Field(ge=0, le=8, description="DC wards are 0-8")

4. Reask / retry pattern (Instructor-inspired)

When a provider returns data that fails validation, re-prompt with the validation error so the LLM can self-correct:

from pydantic import ValidationError

async def extract_with_reask(config, base64_image, max_retries=2):
    for attempt in range(max_retries + 1):
        raw = await extractor(config, base64_image)
        try:
            data = OCRData.model_validate({"Data": raw})
            return data
        except ValidationError as e:
            if attempt == max_retries:
                raise
            # Re-ask with error feedback (Instructor pattern)
            messages.append({"role": "assistant", "content": json.dumps(raw)})
            messages.append({"role": "user", "content": f"Correct these validation errors:\n{e}"})

5. Context-aware validation (ValidationInfo.context)

Pass runtime context (e.g., expected ward range for the campaign) into validators:

@field_validator('Ward')
@classmethod
def validate_ward_range(cls, v, info):
    wards = info.context.get('valid_wards') if info.context else None
    if wards and v not in wards:
        raise ValueError(f"Ward {v} not in valid range {wards}")
    return v

Spike Deliverables

• Audit all OCR-related Pydantic models for validation gaps
• Propose specific validators / constraints for OCREntry fields (Name, Address, Date, Ward)
• Evaluate deduplication of OCREntry (exists in both data_models.py and ocr_client_factory.py)
• Prototype a reask/retry wrapper around extract_from_encoding_async using ValidationError feedback
• Assess whether instructor library should be added as a dependency, or if the reask pattern can be implemented with raw Pydantic + provider clients
• Document trade-offs: latency impact of retries vs. data quality improvement
• Write findings as a comment on this issue for team review

Scope

• Research and prototype only — no production changes
• Focus on OCREntry / OCRData validation; snip do not redesign the full OCR pipeline
• Time-boxed: ~1 day of investigation

References

• Pydantic Validators
• Instructor Reask Validation
• Pydantic Field Constraints
• Pydantic Model Validators