Remove NDI doc_data workarounds, use DID-python's native SQL search#38
Merged
Conversation
DID-python now populates fields/doc_data tables in _do_add_doc via doc2sql (matching MATLAB) and provides SQL-based search with brute-force fallback. This removes all NDI-side workarounds: - Remove _flatten_document, _get_or_create_field_idx, _populate_doc_data - Simplify add() — DID handles doc_data population - Simplify bulk_add() — route through DID's add_docs - Simplify update() — DID handles cleanup and repopulation - Switch find() from brute-force field_search to DID's SQL search - Remove outdated DID_PYTHON_UPDATE_INSTRUCTIONS.md https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
The Python make_artifacts test was only creating subject and subjectmeasurement documents, but MATLAB's equivalent also creates a full DAQ system (filenavigator, daqreader, daqsystem). This meant the Python-Python roundtrip never tested daqsystem_load. Add filenavigator, daqreader, and daqsystem documents to the session so the read_artifacts test verifies DAQ system search and loading works end-to-end through DID-python's SQL search. https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
- Fix doc_superclass() to handle MATLAB's single-dict format (vs list) - Use NDI class identifiers (e.g. "ndi.file.navigator") in documents instead of Python class names (e.g. "FileNavigator") for cross-language database compatibility - Fix FileNavigator._load_from_document() to handle dict document properties (not just DotDict attribute access) - Support MATLAB '#' wildcard in file patterns (convert to glob '*') - Handle set/frozenset from eval of MATLAB cell array syntax in _normalize_fileparameters() - Add excludeFields parameter to compareSessionSummary for skipping runtime-dependent fields like epoch nodes - Filter .epochid.ndi files from summary comparison (runtime side-effect) - Use NDI class names in sessionSummary via NDI_*_CLASS constants https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
Epoch ID filename fixes:
- Parse MATLAB cell array syntax preserving element order instead of
eval() to set (which loses ordering)
- Hash the raw fileparameters document string to match MATLAB's hash
- Strip file extension from base filename in epochidfilename() to
match MATLAB's format (.{stem}.{hash}.epochid.ndi)
- Order epoch files by pattern match index (MATLAB pattern order)
rather than alphabetically
Unified class registry:
- Add ndi.class_registry module mapping NDI class identifiers (e.g.
"ndi.probe.timeseries.mfdaq", "ndi.daq.reader.mfdaq.intan") to
their Python implementation classes
- Replace scattered registries in session_base._document_to_object()
and daq/system._load_from_document() with unified get_class() calls
https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
- Normalize document JSON loaded from database to ensure superclasses and depends_on are always lists (handles MATLAB single-element encoding) - Fix unused import in class_registry.py (ruff F401) - Apply black formatting to reader_base.py, navigator/__init__.py, session_summary.py https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
Replace raw SQL queries and manual _normalize_loaded_props calls in SQLiteDriver.find() with DID-python's get_docs() and get_docs_by_branch() methods. This moves all serialization/deserialization responsibility into DID-python where it belongs. Depends on DID-python PR: bulk get_docs + MATLAB normalization https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
Replace CamelCase class names with mechanical underscore-based names derived from the fully-qualified MATLAB class name: - MATLAB '.' -> Python '_' - MATLAB '_' -> Python '__' Examples: DirSession -> ndi_session_dir ProbeTimeseriesMFDAQ -> ndi_probe_timeseries_mfdaq EpochProbeMapDAQSystem -> ndi_epoch_epochprobemap__daqsystem This eliminates the need for lookup tables, YAML bridge mappings, and per-class matlab_class attributes. The class name IS the mapping. 73 classes renamed across 216 files. All 1526 tests pass. Documentation added in docs/developer_notes/class_naming_convention.md. https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
Three sequential stages on a single runner (shared temp artifacts): 1. MATLAB makeArtifacts - install MATLAB/matbox/NDI-matlab, generate artifacts 2. Python make + read - generate Python artifacts, verify both MATLAB and Python 3. MATLAB readArtifacts - verify Python-generated artifacts from MATLAB Triggers on push/PR to main and manual workflow_dispatch. https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
…cell-array format
MATLAB's ndi.file.navigator accesses fileparameters.filematch via dot
indexing, which requires the stored string to be parseable into a struct.
Python was storing fileparameters as repr() (e.g. "['#.rhd']") or empty
string, neither of which MATLAB can parse into a struct.
Changes:
- Add _to_matlab_cell_str() helper that converts ['#.rhd'] to "{ '#.rhd' }"
- Update navigator.newdocument() to use MATLAB cell-array format
- Set fileparameters and epochprobemap_fileparameters in symmetry test's
filenavigator document to match MATLAB's Intan DAQ system setup
https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47
stevevanhooser
pushed a commit
that referenced
this pull request
Mar 16, 2026
Resolve conflicts between class rename PR (#38) and our functional fixes (epoch discovery, DID API, serialization helpers). Keep both the new MATLAB-mirror class names and all epoch/DAQ reconstruction improvements. https://claude.ai/code/session_01HbgrZ69hyXBhiCx9tcAuA6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DID-python now populates fields/doc_data tables in _do_add_doc via doc2sql (matching MATLAB) and provides SQL-based search with brute-force fallback. This removes all NDI-side workarounds:
https://claude.ai/code/session_01RBSaNF8SSrGqnpnP5vUT47