migrate: V_epsilon target + NDI-layer stimulus_bath → bath assembly by stevevanhooser · Pull Request #829 · VH-Lab/NDI-matlab

stevevanhooser · 2026-06-21T22:24:07Z

Summary

Adds the NDI-layer half of the Brainstorm-E (V_epsilon) migration: the
per-document DID converter (DID-matlab #149) defers the classes it cannot
complete from a single document, and this PR resolves them where the whole
session/element graph is in hand.

The companion DID-matlab work splits treatment → manipulation tiers and
ontology_table_row → observation tiers (1 → N) per document, but a
stimulus_bath → bath cannot be completed per-document: a manipulation
must be emitted complete, and the resulting bath needs two things only
reachable by following the stimulator element — its subject and an
epoch_bounded_reference on the stimulator's epoch. The DID converter
therefore defers stimulus_bath with did2:convert:needsSessionContext;
this PR does the session-aware assembly.

What's here

ndi.migrate.internal.bodyResolver(bodies) — indexes the open body
set and answers the questions a single document can't:
- subjectOfElement(elementId) — the element's subject_id, following
  underlying_element_id up the derivation chain (cycle-guarded).
- epochClockOfElement(elementId, epochId) — the clocktype for the
  element's epoch: the matching element_epoch, then any
  epochclocktimes for the epoch (favouring dev_local_time), then the
  conventional dev_local_time default.
ndi.migrate.internal.stimulusBathToBath (scaffolded earlier) is now
actually invoked: it assembles the epoch_bounded_reference + the bath
(pharmacological_manipulation.mixture + bath.kind/location) from the
legacy stimulus_bath using the resolver.
ndi.migrate.local — new TargetVersion option:
- Default 'V_delta' preserves the existing, tested behaviour and the
  V_delta.sqlite destination — no change for current callers.
- 'V_epsilon' writes V_epsilon.sqlite, threads TargetVersion into
  v1_to_v2, and runs a guarded second pass (resolveDeferred) that
  re-assembles every needsSessionContext deferral via the resolver, folds
  the assembled V_epsilon bodies back through v1_to_v2 (which
  short-circuits already-target bodies to pad + validate, not re-migrate),
  and moves the successes from quarantine into migrated. The pass is wrapped
  in try/catch so a resolver failure degrades to leaving the deferrals
  quarantined rather than failing the run.

Testing

No local MATLAB was available; correctness is vetted by CI. The DID side is
exercised by did2.unittest.testMigratorsE (including
testStimulusBathDefersToNdiLayer and testAlreadyEpsilonBodyShortCircuits,
which guards the fold-back short-circuit this PR relies on).

Pairs with DID-matlab #149 (the per-document V_epsilon migrators and the
isAlreadyTarget / TargetVersion plumbing).

🤖 Generated with Claude Code

Generated by Claude Code

Adds ndi.migrate.internal.stimulusBathToBath: the session-aware build that the per-document converter defers (did2:convert:needsSessionContext). It assembles a complete `bath` (pharmacological_manipulation) from a legacy stimulus_bath: - mixture (concentrations) + location from the document, - subject_id resolved from the stimulator element's subject, - time_reference = epoch_bounded_reference on the stimulator's epoch (the stimulator is the time referent; no other connection kept). Session/element lookups come via a `resolver` struct (.subjectOfElement / .epochClockOfElement) that ndi.migrate.local will supply from the open session's element graph. STATUS: scaffold (authored without local MATLAB; assembly shape complete, the resolver wiring + the ndi.migrate.local second pass over deferred docs are TODO). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o

…#782) Finish the NDI side of the stimulus_bath -> bath migration so the context-dependent deferrals the per-document DID converter emits are actually resolved during ndi.migrate.local: - ndi.migrate.internal.bodyResolver(bodies): index the open body set and answer the session-aware questions the per-document converter cannot -- subjectOfElement (following underlying_element_id up the derivation chain) and epochClockOfElement (element_epoch, then epochclocktimes, then the conventional dev_local_time default). - ndi.migrate.local: add a TargetVersion option (default 'V_delta', preserving the tested behaviour and V_delta.sqlite destination). Under 'V_epsilon' it writes V_epsilon.sqlite, threads TargetVersion into v1_to_v2, and runs a guarded second pass (resolveDeferred) that re-assembles every needsSessionContext deferral via bodyResolver + stimulusBathToBath, folds the assembled V_epsilon bodies back through v1_to_v2 (which short-circuits already-target bodies to pad+validate), and moves the successes from quarantine into migrated. The second pass is wrapped in try/catch so a resolver failure degrades to leaving the deferrals quarantined rather than failing the whole run. Verified in NDI CI (no local MATLAB). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o

The full NDI suite (run-tests.yml) installs the stable released did-matlab, whose did2.convert.v1_to_v2 predates the TargetVersion name-value option (that option lives on the unmerged DID-matlab #149 branch). Passing 'TargetVersion' unconditionally therefore broke every TestMigrateLocal case -- including the default V_delta path -- with an arguments-block rejection. Forward TargetVersion only when it differs from the default 'V_delta', so the default migration call stays byte-identical to before and remains compatible with the stable did-matlab. A 'V_epsilon' run adds the name-value and requires the newer did-matlab, which it needs regardless. The V_epsilon second pass and the per-target destination path were already guarded behind strcmp(TargetVersion,'V_epsilon'), so the default path is fully restored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o

…#782) The context-dependent stimulus_bath -> bath migration had only synthetic unit tests; nothing proved the assembled bath VALIDATES against V_epsilon or that the resolver reads the right subject/epoch from a live body set. - TestMigrateLocalEpsilon: builds a complete v1 session on disk (stimulator element + element_epoch + stimulus_bath) and runs ndi.migrate.local with TargetVersion='V_epsilon', Validate=true, asserting the stimulus_bath is NOT left deferred, a `bath` + `epoch_bounded_reference` are produced, and the bath carries the stimulator's subject_id + a time_reference and the reference carries the epoch + clock. Gated on mksqlite + NDI_TEST_EPSILON + DID_SCHEMA_PATH so it skips everywhere the V_epsilon environment is absent. - test-epsilon-migrate.yml: assembles that environment (matching DID-matlab E branch via head_ref + assembled V_epsilon schema set + NDI_TEST_EPSILON) and runs the gated test -- neither run-tests.yml (stable DID, V_delta) nor test-vnext.yml (V_delta) can. - stimulusBathToBath.parseMixture: guarantee >= 1 mixture record (blank fallback). pharmacological_manipulation.mixture is mustBeNonEmpty, so an empty mixture_table would otherwise make the assembled bath fail validation -- exactly the kind of gap this end-to-end test exists to find. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o

github-actions · 2026-06-22T22:27:11Z

Test Results

672 tests +1 671 ✅ ±0 5m 46s ⏱️ +19s
112 suites +1 1 💤 +1
1 files ±0 0 ❌ ±0

Results for commit aa162f6. ± Comparison against base commit eefcaa0.

♻️ This comment has been updated with latest results.

codecov · 2026-06-22T22:27:32Z

Codecov Report

❌ Patch coverage is 3.03030% with 224 lines in your changes missing coverage. Please review.
✅ Project coverage is 23.43%. Comparing base (eefcaa0) to head (aa162f6).

Files with missing lines	Patch %	Lines
src/ndi/+ndi/+migrate/+internal/bodyResolver.m	0.00%	93 Missing ⚠️
...c/ndi/+ndi/+migrate/+internal/stimulusBathToBath.m	0.00%	75 Missing ⚠️
src/ndi/+ndi/+migrate/local.m	11.11%	56 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##            Vnext     #829      +/-   ##
==========================================
- Coverage   23.59%   23.43%   -0.17%     
==========================================
  Files         715      717       +2     
  Lines       31177    31404     +227     
==========================================
+ Hits         7357     7360       +3     
- Misses      23820    24044     +224

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-06-22T22:28:31Z

Vnext NDI unit test results

672 tests +672 655 ✅ +655 5m 52s ⏱️ + 5m 52s
112 suites +112 1 💤 + 1
1 files + 1 0 ❌ ± 0 16 🔥 +16

For more details on these errors, see this check.

Results for commit aa162f6. ± Comparison against base commit eefcaa0.

♻️ This comment has been updated with latest results.

The first real run of TestMigrateLocalEpsilon (issue #782) surfaced the bug it was written to catch: ndi.migrate.local feeds bodyResolver the v1 bodies straight from did2.convert.readers.sqliteV1, which returns RAW JSON char strings, not structs. bodyResolver's isstruct-based index therefore indexed nothing, subjectOfElement could not resolve, and resolveDeferred's per-entry catch silently left the stimulus_bath deferred (no bath, no epoch_bounded_reference produced). Normalise the body set at the top of bodyResolver: decode any char/string entry via jsondecode, keep scalar structs. Handles both the fresh read path (cellstr from the readers) and the idempotent re-run path (structs from toStruct). This is the gap end-to-end CI verification exists to find. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o

claude and others added 6 commits June 21, 2026 22:10

Update GitHub badges [skip ci]

462e88d

Update GitHub badges [skip ci]

2817ea4

stevevanhooser merged commit a02e214 into Vnext Jul 4, 2026
8 of 11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

migrate: V_epsilon target + NDI-layer stimulus_bath → bath assembly#829

migrate: V_epsilon target + NDI-layer stimulus_bath → bath assembly#829
stevevanhooser merged 7 commits into
Vnextfrom
claude/optimistic-ritchie-f06jvs

stevevanhooser commented Jun 21, 2026

Uh oh!

github-actions Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

stevevanhooser commented Jun 21, 2026

Summary

What's here

Testing

Uh oh!

github-actions Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

codecov Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vnext NDI unit test results

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 22, 2026 •

edited

Loading

codecov Bot commented Jun 22, 2026 •

edited

Loading

github-actions Bot commented Jun 22, 2026 •

edited

Loading