Skip to content

Repair the fixture evaluator and make semantic regression a CI gate #101

Description

@jmcte

Problem

The documented evaluation command currently exits 1 while the unit and scheduled CI lanes remain green:

  • classification: 0/40
  • semantic contract: 0/1
  • noise suppression: 0/0

The evaluator and fixtures use different field/wrapper shapes, suppression reads the wrong corpus, and tests assert report shape rather than a passing, non-empty evaluation.

Relevant code

  • scripts/evaluate.py:21-40,43-77,80-137
  • fixtures/classification/cases.json
  • fixtures/semantic/golden_outputs.json
  • fixtures/noise_suppression/messages.json
  • tests/test_evaluate.py
  • scripts/ci/run-extended-validation.sh
  • .github/workflows/extended-validation.yml:53-102

Scope

  • Normalize evaluator loading to the actual fixture schemas.
  • Load semantic artifacts from the top-level artifacts collection.
  • Evaluate the dedicated noise-suppression corpus.
  • Fail closed when an expected corpus is absent or empty.
  • Assert expected case counts and overall_passed == true.
  • Add per-lane counts and false-promotion/false-suppression visibility.
  • Run the evaluator in extended validation.
  • Include pyproject.toml, src/**, tests/**, and fixtures/** in main-push validation scope.
  • Avoid rerunning a single unit test as the only “extended” behavior.

Non-goals

  • Do not tune classifier rules merely to force a green result unless the fixture expectation is demonstrably wrong.
  • Do not add live or private mailbox content.
  • Do not make an empty evaluation pass.

Acceptance criteria

  • Classification evaluates all 40 current cases with stable IDs.
  • Semantic evaluation processes each artifact, not its wrapper object.
  • Noise suppression evaluates a nonzero dedicated corpus.
  • Missing/empty expected fixture corpora fail clearly.
  • Unit tests assert exact corpus counts and overall pass state.
  • Extended validation runs the evaluator and uploads or prints an inspectable summary.
  • Source, test, fixture, and packaging changes cannot skip relevant post-merge validation.
  • A deliberately wrong fixture expectation makes CI fail.

Validation

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.12 scripts/evaluate.py --fixtures-dir fixtures --report-json /tmp/mpi-eval.json
PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.12 -m unittest tests.test_evaluate -v
bash scripts/ci/run-extended-validation.sh

Parent

Part of #98.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:dataData, schema, storage, privacy, or migration surface.area:infraInfrastructure, CI, release, governance, scripts, or repo setup.bugSomething isn't workingpriority:P1Codex Connector P1; blocks execution until Athena and Ares validate.ready-for-agentIssue is approved for worker-lane execution.risk:highHigh-risk change; validation required.status:ready-for-agentIssue is approved for worker-lane execution.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions