Negative schema tests

[awwright]

The JSON Schema test suite needs support for describing "negative tests": values that are not valid schemas, and are supposed to cause an error (according to the spec). For example, { "minimum": null } is invalid, and an example of a negative schema test.

Negative schema tests are necessary to test that implementations implement the interoperability rules.

In an earlier comment, I suggest that "tests" should be null for this case—but since the value for "schema" is deliberately invalid, and therefore not technically a schema, it should be specified in a separate property, e.g.

[
    {
        "description": "number in enum is invalid",
        "negativeSchema": { "enum": 4 }
    }
]

Related issues #386

[gregsdennis]

I think we have some of these already. They define a schema as data and validate against the metaschema. I agree it'd be good to fill those out to ensure keyword coverage.

[Julian]

There are two separate kinds of tests. The tests which use a metaschema to validate indeed we have some already for. #354 is a PR in need of splitting up which contains more.

The other kind is tests which indeed cause an implementation to error (an even more egregious example is a schema which is not even valid JSON or something). Those kinds of tests we don't have any of currently. #386 is indeed such an example, but there are a bunch of previous issues which have asked for this.

We wouldn't re-use the schema key (or put these in the same place), as doing so is a big backwards incompatible change for everyone using the suite, but it's very easy to create a second (or now third) directory like output-tests (for output tests) containing these kinds of tests and then to use whatever structure we'd like for the file.

We also already have #589 tracking this idea, so I'm going to close this as a duplicate since unless I misunderstand something this is the same sort of idea, but obviously fully agree we should do this.

[santhosh-tekuri]

{ "$ref": "#/$defs/a"}

the above schema validates with metaschema, but we expect it as invalid schema because $ref resolves no nothing.

current test suite has no way testing this

[Julian]

Yep, same sort of example (as #589).