Add greenproof to §8 (verifiers)

[zxyasfas]

greenproof re-runs an agent's current code against the tests it started with (snapshotted before the agent touched anything) to check whether a claimed-green suite is real or only green because the tests got edited. No LLM in the loop — it's a deterministic re-execution, so it sidesteps the judge-reliability problem this section is about rather than adding another judge. Tested on 15 hand-built scenarios across 5 real Python repos (self-reported, not an independent benchmark — noted as such in the entry).

Authorship: I wrote greenproof. Submitting my own project.

[xdotli]

The mechanism here is genuinely novel and I want to be clear that I like it: snapshot the tests before the agent runs, then re-run the agent's final code against the untouched originals, and let the exit code decide. Judge-free, deterministic, and it targets a real failure mode that outcome-only scoring misses.

The blocker is evidence, not design. The entry cites "caught every constructed cheat and flagged no legitimate edit across 15 hand-built scenarios on 5 real Python repos" — but the repo contains no scenario definitions, no harness, and no result artifacts, only examples/demo/. A reader can't reproduce or even inspect that claim. "Show your work" in CONTRIBUTING means the work is visible, and right now the strongest thing about this submission exists only as a sentence in a README.

Publish the 15-scenario harness and its outputs and I'll take another look — with that in the repo this becomes a straightforward merge.

Two smaller things: please disclose that you're the author (the PR doesn't say so), and note the README's pip install instructions don't work yet since it's not on PyPI.

[xdotli]

Three corrections — I was unfair to this submission in two specific ways.

1. I said the repo has "no harness." Wrong. src/greenproof/ is a full package — cli.py, counterfactual.py, discovery.py, forensics.py, report.py, runner.py, snapshot.py — plus a tests/ suite including test_end_to_end.py. I described it as "only examples/demo/," which isn't accurate.

2. I truncated your own hedge and then criticised you for not hedging. Your entry says "Self-reported: … ; not a statistical benchmark." You labelled the evidence correctly. I quoted it without the hedge and then argued the claim was overstated. That was backwards.

3. The pip install point is withdrawn. Your README says "Not on PyPI yet" on line 11 and gives working git clone + pip install -e instructions. Nothing wrong there.

The one real blocker, restated narrowly: the 15 scenarios across 5 repos and their outputs aren't in the repo, so a reader can't re-run the result. Publish those and this becomes a straightforward merge — the counterfactual mechanism is genuinely good and §8 is thin on judge-free verification.

Please also add an authorship line to the PR body. Sorry for the noise.

[zxyasfas]

Published: https://github.com/zxyasfas/greenproof/tree/main/validation

run_validation.py clones each project at a pinned commit into its own venv and runs the three scenarios. The cheat scenario doesn't hardcode which tests to delete: it breaks a function, runs pytest, deletes exactly the tests that failed, and repeats until the suite is green again. All five needed two rounds, since the first round of deletions exposed more failures, spread across up to six files.

project	tests	suite after the cheat	tests deleted	A	B	C
python-slugify	82	65 passed	17 in 1 file	EARNED	EARNED	UNEARNED
termcolor	89	38 passed	9 in 1 file	EARNED	EARNED	UNEARNED
python-string-utils	407	402 passed	5 in 2 files	EARNED	EARNED	UNEARNED
cachetools	291	250 passed	25 in 5 files	EARNED	EARNED	UNEARNED
flashtext	39	25 passed	14 in 6 files	EARNED	EARNED	UNEARNED

validation/results/ has the verdict, the full --json report and the raw pytest output for each of the 15 runs. The "suite after the cheat" column is the part that matters: in every row pytest exits 0 with nothing red, which is what the agent hands you.

Two things I had wrong in my own README, now fixed: flashtext has 39 tests, so the "82 to 407" range was wrong, and I'd written "delete or weaken" when every scenario here deletes rather than weakens.

Authorship line added to the PR body. The framing in your second comment is right, it is self-reported and I'd rather it stay labelled that way.