Problem
The documented evaluation command currently exits 1 while the unit and scheduled CI lanes remain green:
- classification: 0/40
- semantic contract: 0/1
- noise suppression: 0/0
The evaluator and fixtures use different field/wrapper shapes, suppression reads the wrong corpus, and tests assert report shape rather than a passing, non-empty evaluation.
Relevant code
scripts/evaluate.py:21-40,43-77,80-137
fixtures/classification/cases.json
fixtures/semantic/golden_outputs.json
fixtures/noise_suppression/messages.json
tests/test_evaluate.py
scripts/ci/run-extended-validation.sh
.github/workflows/extended-validation.yml:53-102
Scope
- Normalize evaluator loading to the actual fixture schemas.
- Load semantic artifacts from the top-level
artifacts collection.
- Evaluate the dedicated noise-suppression corpus.
- Fail closed when an expected corpus is absent or empty.
- Assert expected case counts and
overall_passed == true.
- Add per-lane counts and false-promotion/false-suppression visibility.
- Run the evaluator in extended validation.
- Include
pyproject.toml, src/**, tests/**, and fixtures/** in main-push validation scope.
- Avoid rerunning a single unit test as the only “extended” behavior.
Non-goals
- Do not tune classifier rules merely to force a green result unless the fixture expectation is demonstrably wrong.
- Do not add live or private mailbox content.
- Do not make an empty evaluation pass.
Acceptance criteria
Validation
PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.12 scripts/evaluate.py --fixtures-dir fixtures --report-json /tmp/mpi-eval.json
PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.12 -m unittest tests.test_evaluate -v
bash scripts/ci/run-extended-validation.sh
Parent
Part of #98.
Problem
The documented evaluation command currently exits 1 while the unit and scheduled CI lanes remain green:
The evaluator and fixtures use different field/wrapper shapes, suppression reads the wrong corpus, and tests assert report shape rather than a passing, non-empty evaluation.
Relevant code
scripts/evaluate.py:21-40,43-77,80-137fixtures/classification/cases.jsonfixtures/semantic/golden_outputs.jsonfixtures/noise_suppression/messages.jsontests/test_evaluate.pyscripts/ci/run-extended-validation.sh.github/workflows/extended-validation.yml:53-102Scope
artifactscollection.overall_passed == true.pyproject.toml,src/**,tests/**, andfixtures/**in main-push validation scope.Non-goals
Acceptance criteria
Validation
Parent
Part of #98.