feat: Recurse into nested BIDS datasets beyond derivatives/, and normalize invalid DatasetType

[yarikoptic]

Co-Authored-By: Claude Code 2.1.116 / Claude Opus 4.7 (1M context) noreply@anthropic.com

Summary

Two orthogonal-but-related fixes around nested BIDS datasets and the
DatasetType field:

1. Recurse into nested BIDS datasets under rawbids/ and sourcedata/,
   not just derivatives/. The previous recursion logic scanned only
   derivatives/<name>/dataset_description.json. It silently ignored
   two other shapes that appear in real datasets:

   • Single nested: the container directory itself is a BIDS dataset
     (e.g. rawbids/dataset_description.json, which was added as a
     study-level container in
     bids-standard/bids-specification#2191,
     and the historical sourcedata/raw/dataset_description.json form).
   • Multiple nested: sub-directories of the container are each BIDS
     datasets (e.g. derivatives/fmriprep/,
     sourcedata/ds00003/dataset_description.json as used by OpenNeuro
     study-style layouts).

   With this change, -r now discovers and validates all of these.
   Bugs planted inside a rawbids/ or sourcedata/ds00003/ subdataset
   are caught; previously they went silently undetected.

2. Normalize an invalid DatasetType for internal rule lookups.
   The setter previously defaulted only when DatasetType was falsy; a
   truthy-but-out-of-enum value (e.g. "Raw" — capitalization typo in
   bids-examples/proj-nipoppy/rawbids/dataset_description.json) made
   rules.directories[DatasetType] resolve to undefined, cascading
   into spurious NOT_INCLUDED errors on legitimate sub-*/ses-*/
   subjects on top of the legitimate JSON_SCHEMA_VALIDATION_ERROR.

   Fall back to the spec default for any out-of-enum value. Valid enum
   is pulled from schema.objects.metadata.DatasetType.enum (no
   hardcoded list). Normalization is done on a shallow copy so the
   memoized JSON is untouched — JSON schema validation still flags the
   enum violation exactly once.

User-visible behavior changes

• -r now recurses into nested BIDS datasets found under any of
  derivatives/, rawbids/, or sourcedata/, at either the container's
  own level or one level down.
• Text output label "Derivative:" is now "Nested dataset:".
• The derivativesSummary field on ValidationResult is preserved for
  API stability but now covers all nested BIDS datasets (not only
  derivatives). Documented in the validate JSDoc.
• Datasets with an out-of-enum DatasetType still get the
  JSON_SCHEMA_VALIDATION_ERROR they already got, but no longer
  cascade into spurious NOT_INCLUDED on legitimate subjects.

Not in this PR (flagged for follow-up)

• Compliance checks currently unenforced: a raw (or derivative) root
  containing a nested BIDS dataset under rawbids/ or sourcedata/,
  and a non-study root containing rawbids/ at all. Likely best paired
  with an upstream schema proposal to declare nested-BIDS capability on
  directory rules (e.g. nested_bids: single | multiple | none), so
  the validator stops hardcoding the container set.
  (@yarikoptic has even missed that and needs to get back to that PR to see if we discussed this or not to be also part of 'derivatives' dataset type!)
• Pre-existing: filememoize (src/utils/memoize.ts) keys by
  (file.parent.path, file.path), which collides across rerooted trees.
  When multiple nested BIDS datasets are validated in parallel under
  -r, whichever loadJSON runs first wins and others reuse its
  content. Affects multi-derivative recursion too. Tracked separately.

Test plan

• deno test --allow-env --allow-read --allow-write --allow-run --allow-net src/ — 336/336 pass.
• Synthetic dataset reproducing the nested-rawbids + invalid-DatasetType case:
  • -r surfaces the planted bug inside rawbids/sub-002/bogus_file.txt.
  • -r surfaces the planted bug inside sourcedata/ds00003/sub-A/stray.dat.
  • JSON_SCHEMA_VALIDATION_ERROR still fires for DatasetType: "Raw".
  • No spurious NOT_INCLUDED on legitimate sub-001.
• Existing derivatives recursion behavior unchanged (covered by
  src/tests/local/derivatives.test.ts).

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.33%. Comparing base (5c0b1cd) to head (62fed66).

Files with missing lines	Patch %	Lines
src/validators/bids.ts	86.11%	3 Missing and 2 partials ⚠️
src/schema/context.ts	70.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #390      +/-   ##
==========================================
- Coverage   90.45%   90.33%   -0.13%     
==========================================
  Files          51       51              
  Lines        4316     4345      +29     
  Branches      721      726       +5     
==========================================
+ Hits         3904     3925      +21     
- Misses        323      328       +5     
- Partials       89       92       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[yarikoptic]

let me know if I should reduce commenting here

[effigies]

It's definitely a lot. I don't think we need the branching logic, though, validTypes.includes(undefined) will also return False.

// shallow copy to avoid polluting memoize cache
value = {...value}
const validTypes = this.schema?.objects?.metadata?.DatasetType?.enum as Array
// Unknown types (including capitalizations) fall back to defaults
// Per spec: 'the default value is `"raw"`'.
// Before DatasetType was defined, `GeneratedBy` was used as a derivatives indicator
if (validTypes && !validTypes.includes(value.DatasetType as string)) {
  value.DatasetType = value.GeneratedBy ? 'derivative' : 'raw'
}
this.#dataset_description = value

I do wonder if we should drop the GeneratedBy behavior, since the spec does not say to do this, and GeneratedBy is now valid in raw datasets.

[yarikoptic]

yeah, I thought about that too! Moreover GeneratedBy is imho very applicable to "raw" datasets as they are typically generated by some tool + human(s) and thus we did provide it for "raw" (made it explicit!) BIDS dataset and I think frankly it should even be RECOMMENDED for "raw" (not study) : https://github.com/OpenNeuroDatasets/ds005256/blob/main/dataset_description.json#L37

[yarikoptic]

so let's kick it out in a separate PR?

[effigies]

Sure, can be a separate PR.

[yarikoptic]

since we want this one to land and not wait -- just did right here.

[yarikoptic]

this made me pause -- should we separate out into sourcedatasetsSummary (would include rawbids/) from derivativesSummary? it would add a field but overall be "compatible" if anyone uses that, right @effigies ?

[yarikoptic]

decided to give claude a try for this, pushing... moving to draft - will need to test etc