feat(pipeline): SparqlItemSelector.maxResults + drop requireNonEmptyData

[ddeboer]

Summary

Two related changes to how @lde/pipeline-based pipelines express sampling caps and conformance:

1. feat(pipeline): add a maxResults option to SparqlItemSelector that genuinely caps the total bindings yielded across all paginated pages.
2. fix(pipeline-shacl-sampler): use that new option so samplesPerClass actually limits the sampled subjects per sh:targetClass. Was a real bug — the cap wasn't applied.
3. refactor(pipeline)!: remove the requireNonEmptyData decorator. With the proper sampler cap in place, the right semantic for ‘no target class matched’ is SHACL’s vacuous-truth (conforms: true), not a synthesised non-conformance. Consumers that need to distinguish ‘untested’ from ‘tested and passed’ can read quadsValidated > 0.

Why

The sampling cap bug

Until now, shaclSampleStages built its subject selector with LIMIT N in the SELECT, intending it as a total cap on samples. SparqlItemSelector interprets a query-level LIMIT as page size, not as a total — and proceeds to paginate with OFFSET until the source is exhausted. The effect: a target class with thousands of instances was fully walked rather than capped at N.

Concrete impact: a recent end-to-end run against lod.uba.uva.nl/Cinema-Context/Cinema-Context validated 890K quads instead of an expected handful, and the on-disk SHACL report file reached 258 MB.

maxResults

Adds an explicit option to SparqlItemSelector for the "I want at most N items total" case:

new SparqlItemSelector({
  query: 'SELECT DISTINCT ?s WHERE { ?s a <Class> }',
  maxResults: 50,
});

When set:

• The first page's LIMIT is clamped to maxResults so the endpoint never returns more than the cap.
• Pagination stops as soon as maxResults bindings have been yielded.
• Independent of any in-query LIMIT, which still controls page size in the multi-page case.
• Setting maxResults: 0 is a valid no-op — the selector yields nothing without issuing any fetch.

The sampler now uses this in place of the buggy in-query LIMIT.

Removing requireNonEmptyData

The decorator flipped a validator's report to non-conforming when quadsValidated === 0. After more thought (with the sampling cap fixed), that semantic conflates two genuinely different categories of dataset:

Dataset	quadsValidated	Honest verdict
Uses SCHEMA-AP-NDE classes, has violations	> 0	non-conforming
Uses SCHEMA-AP-NDE classes, valid	> 0	conforming
Doesn’t use SCHEMA-AP-NDE at all (e.g. Linked.Art)	0	not applicable

requireNonEmptyData reported the third case as ‘non-conforming’, which is dishonest — a Linked.Art dataset isn’t non-conformant to SCHEMA-AP-NDE any more than it’s non-conformant to FOAF or DCAT-AP-SL. It’s a different model.

The proper consumer-side filter is quadsValidated > 0 AND conforms = true for ‘tested and passed’. SHACL’s vacuous-truth default for empty targets is the more honest signal; the decorator was an opt-in deviation with limited downstream value.

Pre-release, so the removal lands without a back-compat shim. Documented in packages/pipeline/README.md so consumers know which signal to read for which question.

Breaking change

requireNonEmptyData is removed from @lde/pipeline’s exports. Consumers that imported it (notably the DKG pipeline) need to drop the import and rely on quadsValidated for their ‘untested’ filter. Coordinated update in dataset-knowledge-graph PR #280.

Notes

• 4 new tests in selector.test.ts cover the maxResults paths (cap across pages, first-page clamp, no extra page request, zero-maxResults no-op).
• sampleStages.test.ts updated for the now-LIMIT-less subject selector query.
• Coverage thresholds on @lde/pipeline adjusted downward (3 fewer functions; less code → slightly lower coverage ratio).