Refactor Dataset to mirror MATLAB architecture with base/subclass pattern#21
Merged
Merged
Conversation
…ions Three issues made downloaded datasets unusable in Python vs MATLAB: 1. ndi.query.all() raised AttributeError because all() was a static method on the Query class but ndi.query is a module in Python (vs a class in MATLAB). Added module-level all(), none(), from_search() wrappers. 2. D.reference returned the cloud dataset ID (directory name) instead of the human-readable name. downloadDataset now passes ds_info["name"] to the Dataset constructor. 3. sessions=0 after download because no session_in_a_dataset tracking docs were created. Added _discover_sessions() to Dataset.__init__ (mirroring MATLAB ndi.dataset.dir constructor) which scans the database for session-type documents and registers untracked sessions. https://claude.ai/code/session_01ByofAviQrhGPLQ9BLquGjV
Major restructuring to match the MATLAB codebase: **ndi.dataset (base class):** - reference/id() delegate to session (matching MATLAB) - build_session_info() discovers sessions from session_in_a_dataset docs filtered by base.session_id == dataset.id() (matching MATLAB) - Static methods: repairDatasetSessionInfo, addSessionInfoToDataset, removeSessionInfoFromDataset (matching MATLAB) - database_search() searches session._database directly + linked sessions - database_add() routes docs by session_id (matching MATLAB) **ndi.dataset.dir (DatasetDir subclass):** - Constructor mirrors MATLAB ndi.dataset.dir with 1/2/3-arg forms - Hidden documents parameter for downloadDataset (MATLAB 3rd arg) - Session discovery: dataset_session_info → session_in_a_dataset → session docs, then re-creates session with correct ref/id - _ensure_session_tracking() creates session_in_a_dataset docs for untracked sessions found in the database - Uses dataset path directly for session (.ndi/ at root, not .ndi_dataset/) **ndi.query module-level functions:** - Added all(), none(), from_search() at module level so ndi.query.all() works like MATLAB (where ndi.query is a class, not a module) **downloadDataset:** - Now passes documents to DatasetDir constructor (matching MATLAB pattern) - Reference discovered from session documents, not cloud API name https://claude.ai/code/session_01ByofAviQrhGPLQ9BLquGjV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR refactors the
Datasetclass to mirror the MATLABndi.datasetarchitecture, introducing a baseDatasetclass and aDatasetDirsubclass for directory-backed datasets. The refactoring improves code organization, adds MATLAB compatibility documentation, and simplifies session management through a new session info tracking system.Key Changes
Base/Subclass Architecture: Split
Datasetinto an abstract base class andDatasetDirsubclass (mirrors MATLAB'sndi.datasetandndi.dataset.dir)Datasetclass now contains core session management logicDatasetDirhandles directory-specific initialization and session discoveryndi.dataset.Datasetis aliased toDatasetDirSession Info Management: Replaced document-based session tracking with in-memory
_session_infoand_session_arraystructuresbuild_session_info()method to populate session tracking from database documentsaddSessionInfoToDataset(),removeSessionInfoFromDataset(),repairDatasetSessionInfo()_session_arraySimplified Constructor:
DatasetDirnow supports multiple calling conventionsDatasetDir(path)— open existing datasetDatasetDir(reference, path)— MATLAB-style creationDatasetDir(path, reference='name')— keyword-styledocumentsparameter for bulk-loading (used bydownloadDataset)Session Discovery: Added
_discover_correct_session()and_ensure_session_tracking()methodssession_in_a_datasettracking records for existing sessionsdataset_session_infodocumentsEnhanced Documentation: Added MATLAB equivalents throughout
MATLAB equivalent:commentsQuery Module Improvements: Added module-level convenience functions
ndi.query.all(),ndi.query.none(),ndi.query.from_search()now work like MATLABUpdated
downloadDataset(): Uses newDatasetDirconstructor with bulk document loadingndi.dataset.dir([], folder, docs)patternImplementation Details
Idoclass from baseDataset_session_cachedict to_session_arraylist for consistency with MATLAB_sessionattribute is now set by subclass (was previously set in base constructor)session_id(dataset's own vs. other sessions)https://claude.ai/code/session_01ByofAviQrhGPLQ9BLquGjV