Skip to content

Run the memory-evidence profile's adversarial cases, which nothing was running - #62

Merged
Silentpartnercoding merged 1 commit into
mainfrom
agent/wire-profile-tests
Aug 9, 2026
Merged

Run the memory-evidence profile's adversarial cases, which nothing was running#62
Silentpartnercoding merged 1 commit into
mainfrom
agent/wire-profile-tests

Conversation

@Silentpartnercoding

Copy link
Copy Markdown
Owner

PR #29 shipped interop/memory-evidence-profile-v0.1/validate.py with four examples and eight adversarial rejection cases. It works. Nothing executed it — not the test suite, not CI, not the Makefile.

Eight adversarial cases that never run are eight cases that cannot fail — the defect this programme spends most of its time finding elsewhere. It shipped in the same session that merged a checker for exactly this shape.

How it was found: the test count. Nine files arrived and the suite did not move. A test count that does not change when tests arrive is the signal.

Three tests

  1. Runs the validator; requires it to pass its own corpus.
  2. Requires the adversarial corpus to be non-empty — "all cases rejected" is trivially true of an empty list.
  3. The one that matters: calls schema_validate and validate_semantics directly and requires the validator to reject a malformed record. A validator that accepted everything would satisfy a check that only asks whether the program exits cleanly, and its eight rejections would mean nothing.

An earlier draft of #3 skipped when it could not find an entry point named validate or check. A skipped test is barely a test, so it now uses the real entry points.

…s running

PR #29 shipped interop/memory-evidence-profile-v0.1/validate.py with four examples
and eight adversarial rejection cases. It works. Nothing executed it -- not the
test suite, not CI, not the Makefile.

Eight adversarial cases that never run are eight cases that cannot fail, which is
the defect this programme spends most of its time finding elsewhere. It shipped
here in the same session that merged a checker for exactly this shape.

Found by the test count: nine files arrived and the suite did not move. A test
count that does not change when tests arrive is the signal.

Three tests. One runs the validator and requires it to pass its own corpus. One
requires the adversarial corpus to be non-empty, since 'all cases rejected' is
trivially true of an empty list. The third is the one that matters: it calls
schema_validate and validate_semantics directly and requires the validator to
REJECT a malformed record, because a validator that accepted everything would
satisfy a check that only asks whether the program exits cleanly, and its eight
rejections would mean nothing.

An earlier draft of the third test skipped when it could not find an entry point
named validate or check. A skipped test is barely a test, so it now uses the real
entry points.
@Silentpartnercoding
Silentpartnercoding merged commit 422c83e into main Aug 9, 2026
4 checks passed
@Silentpartnercoding
Silentpartnercoding deleted the agent/wire-profile-tests branch August 9, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants