Should the test suite test validation of invalid schemas which are not invalid under the metaschema they come from

[ssilverman]

There are a number of constraints that the specification places on schemas which make them (the schemas) valid or invalid. Many or most of these constraints are specified in the draft's metaschema. As a concrete example, schemas must be objects or booleans, and the metaschema includes a type validator to ensure as such. There are however a number of constraints the draft specifications place on schemas which are not represented in a metaschema. They are directed (presumably at schema authors) as things they MUST NOT do, but no validatable entity exists which implements these constraints.

Because no such entity exists, it's not currently possible to test these constraints under the current structure of the suite.

Known examples

• $schema
  • URI must be normalized
  • must only appear at the root level of a schema
• Test needed to differentiate between JSON Pointer or named schema in URI fragment #449 (comment)
  more to be added

If the decision is made that the suite should test this behavior (e.g., because implementers should implement validation of this behavior), then a structure change may be necessary to the suite, in which case one needs to be designed in a way to reduce impact to users of the test suite.

There are a few ways to do this. Here are two:

1. Add a new property, "isMeta" or something similar, to a test set, as a sibling to the "schema" element.
2. Put schemas-being-tested-beyond-the-meta-schema tests into a specific directory in a draft's test suite, for example in "meta/". Then the test runner knows to handle them specially.

Both cases basically tell the test runner, "apply additional schema rules to the data instances too".

[Julian]

Going to modify the description, but if I change your meaning, obviously lemme know / revert.