diff --git a/.cursorrules b/.cursorrules new file mode 100644 index 0000000..1736f73 --- /dev/null +++ b/.cursorrules @@ -0,0 +1,18 @@ +# NDI Cursor Rules + +## 1. Project Context +This is the NDI (Neuroscience Data Interface) Python port. It is a "Lead-Follow" project where the MATLAB codebase is the Source of Truth. + +## 2. Core Instructions +Before proposing or writing any code, you MUST read and adhere to the following project-specific files: + +- **Entry Point:** Read `AGENTS.md` at the root for a high-level overview of your role. +- **Porting Logic:** Read `docs/developer_notes/PYTHON_PORTING_GUIDE.md` for technical implementation rules (Pydantic, Naming, etc.). +- **Parity Principles:** Read `docs/developer_notes/ndi_xlang_principles.md` for indexing vs. counting rules and data handling. +- **The Contract:** Locate and read the `ndi_matlab_python_bridge.yaml` in the directory you are currently working in. + +## 3. Strict Naming Policy +Do not attempt to convert MATLAB camelCase to snake_case. Maintain the casing defined in the bridge YAML files. + +## 4. Active Maintenance +If a function is missing from the bridge YAML, add it based on the MATLAB source and notify the user: "INTERFACE UPDATE: I have modified the bridge contract for [Function Name]." diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..2dc139e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,38 @@ +# AGENTS.md + +## 1. Role & Mission + +You are an AI Developer for the NDI (Neuroscience Data Interface) project. Your mission is to maintain 1:1 functional and semantic parity between the mature MATLAB (Source of Truth) codebase and the Python port. + +## 2. The Mandatory Knowledge Base + +Before proposing, writing, or refactoring any code, you MUST read the following files in order: + +1. **Porting Protocol:** `docs/developer_notes/PYTHON_PORTING_GUIDE.md` + - Focus: Technical workflow, naming rules, Pydantic validation, and linting requirements. + +2. **Universal Principles:** `docs/developer_notes/ndi_xlang_principles.md` + - Focus: High-level logic rules (e.g., 0-vs-1 indexing, Semantic Parity for scientific counting, and NumPy usage). + +## 3. The Local Contract: The Bridge File + +Every sub-package contains a file named `ndi_matlab_python_bridge.yaml`. + +- **Rule 1: Consult the Bridge First.** This file defines the exact names, input arguments, and output tuples for that namespace. +- **Rule 2: Active Maintenance.** If a function or class exists in MATLAB but is missing from the bridge file, you must: + 1. Analyze the MATLAB `.m` file. + 2. Add the new entry to the `ndi_matlab_python_bridge.yaml`. + 3. **Notify the User:** You must state: "INTERFACE UPDATE: I have modified the bridge contract for [Function Name] to reflect the MATLAB source." +- **Rule 3: Strict Naming.** You are forbidden from "Pythonizing" names (e.g., changing `ListAllDocuments` to `list_all_documents`) unless the bridge file explicitly instructs you to do so in the `decision_log`. + +## 4. Technical Constraints + +- **Validation:** All public API functions must use the `@pydantic.validate_call` decorator. +- **Counting:** Any user-facing concept (Epochs, Channels, Trials) uses 1-based counting in Python to match MATLAB. +- **Internal Access:** Use 0-based indexing for internal Python data structures (lists, NumPy arrays). +- **Formatting:** Code must pass `black` and `ruff check --fix` before completion. + +## 5. Directory Mapping Reference + +- **MATLAB Source:** `VH-Lab/NDI-matlab` (GitHub) +- **Python Target:** `src/ndi/[namespace]/` (Mirrors MATLAB `+namespace/`) diff --git a/PYTHON_PORTING_GUIDE.md b/PYTHON_PORTING_GUIDE.md deleted file mode 100644 index 5dee328..0000000 --- a/PYTHON_PORTING_GUIDE.md +++ /dev/null @@ -1,146 +0,0 @@ -# MATLAB to Python Porting Rules - -## 1. The Core Philosophy: Lead-Follow Architecture - -The MATLAB codebase is the **Source of Truth**. The Python version is a "faithful mirror." When a conflict arises between "Pythonic" style and MATLAB symmetry, **symmetry wins**. - -## 2. Function & Variable Naming (The "Strict Mirror" Rule) - -**MATLAB function names are the source of truth.** Do **not** attempt to "translate" MATLAB names into Python PEP 8 (`snake_case`). - -- **Exact Match:** Every function name in Python must be an identical string match to the MATLAB function name. -- **Case Sensitivity:** If the MATLAB function is `ListAllDocuments`, the Python function must be `ListAllDocuments`. If the MATLAB function is `get_dataset_id`, the Python function must be `get_dataset_id`. If the MATLAB function is `savetofile`, the Python method must be `savetofile` (not `save_to_file`). -- **No Aliasing:** Do not create `snake_case` aliases unless explicitly requested. The user should be able to copy-paste function names between environments. -- **Verification:** When porting or reviewing code, always check the MATLAB source at [VH-Lab/NDI-matlab](https://github.com/VH-Lab/NDI-matlab) to confirm the exact function name. The MATLAB `.m` file's `function` line is the canonical reference. - -## 3. Namespace and Directory Structure - -MATLAB namespaces (`+` packages) must be mapped 1:1 to Python packages and modules to ensure discoverability. - -- **Hierarchy:** Every MATLAB `+namespace` folder must become a Python directory containing an `__init__.py`. -- **File Mapping:** If a MATLAB function exists as `+ndi/+fun/+X/Y.m`, the Python equivalent must be located at `ndi/fun/X/Y.py`. -- **Sub-modules:** For functions inside a namespace that aren't in their own file, group them into a `.py` file named after the MATLAB namespace level. - -## 4. Input Validation: Pydantic is Mandatory - -To replicate the robustness of the MATLAB `arguments` block, use Pydantic for all public-facing API functions. - -- **Decorator:** Use the `@pydantic.validate_call` decorator on all functions. -- **Type Mirroring:** - - MATLAB `double` or `numeric` → Python `float` or `int` - - MATLAB `char` or `string` → Python `str` - - MATLAB `{member1, member2}` → Python `Literal["member1", "member2"]` -- **Coercion:** Allow Pydantic's default behavior of casting (e.g., allowing a string `"1"` or integer `1` to satisfy a `bool` type). -- **Arbitrary Types:** When a function accepts types that Pydantic cannot serialise natively (e.g. `numpy.ndarray`, file-like `IO[bytes]`), pass `config=ConfigDict(arbitrary_types_allowed=True)` to the decorator. -- **Constraints:** Use `Annotated[type, pydantic.Field(...)]` to express MATLAB `arguments`-block constraints such as `mustBePositive` (`gt=0`), `mustBeNonnegative` (`ge=0`), and `mustBeInteger`. - -### 4a. Reusable Validators in `ndi.validators` - -MATLAB centralises custom validation functions in the `+ndi/+validators/` namespace. Python must do the same in the `ndi/validators/` package. - -- **When to create a validator:** Any type check, format check, or constraint that appears (or is likely to appear) in more than one function should be extracted into its own module under `ndi/validators/` instead of being written inline. -- **Naming convention:** MATLAB-originated validators keep their exact MATLAB name (e.g. `mustBeID`). Python-specific validators that have no MATLAB counterpart use `snake_case` prefixed with a descriptive verb (e.g. `is_ndarray`, `is_iso8601`). -- **Signature pattern:** Each validator takes a single value, raises `ValueError` on failure, and returns the validated value unchanged: - ```python - def is_ndarray(val: object) -> np.ndarray: - if not isinstance(val, np.ndarray): - raise ValueError("Input must be a numpy.ndarray") - return val - ``` -- **Registration:** Every new validator must be imported and listed in `ndi/validators/__init__.py` so it is accessible as `ndi.validators.`. - -## 5. Code Style & Linting - -All Python code must pass **`black`** formatting and **`ruff`** linting before being committed. - -- **Formatter:** [Black](https://black.readthedocs.io/) is the project's sole code formatter. -- **Linter:** [Ruff](https://docs.astral.sh/ruff/) is the project's linter. -- **Before committing:** Always run both checks and fix any issues: - 1. `ruff check src/ tests/` — fix any lint errors (unused imports, undefined names, etc.). Use `ruff check --fix src/ tests/` for auto-fixable issues. - 2. `black --check src/ tests/` — verify formatting. Run `black src/ tests/` to fix. -- **Line length:** Use Black's default (88 characters). -- **No manual formatting overrides:** Do not use `# fmt: off` / `# fmt: on` or `# noqa` unless absolutely necessary. - -## 6. Error Handling - -- If a MATLAB function throws an error for a specific condition, the Python version must raise a corresponding Exception (`ValueError`, `TypeError`, or a custom `NDIError`). -- The goal is to ensure that a user providing bad input gets a **"Hard Fail"** at the function entry point in both languages. - -## 7. Documentation (Docstring Symmetry) - -- Include the original MATLAB documentation in the Python docstring. -- Note any Python-specific requirements (like specific library dependencies) at the bottom of the docstring. - ---- - -## Namespace Coverage Status - -Verified coverage of each MATLAB namespace against the Python port. See -[MATLAB_MAPPING.md](MATLAB_MAPPING.md) for the full function-by-function mapping. - -### `ndi.cloud.api` — Fully Ported - -**Verified:** 2026-03-11 against `VH-Lab/NDI-matlab` branch `main`. - -| Submodule | MATLAB funcs | Python funcs | Coverage | Notes | -|-----------|:------------:|:------------:|:--------:|-------| -| `+datasets` (14) | 14 | 14 + 2 | **100 %** | Python adds `listAllDatasets` (auto-paginator), `listDeletedDatasets` | -| `+documents` (15) | 15 | 15 + 1 | **100 %** | `countDocuments` subsumes MATLAB's `documentCount`; Python adds `bulkUpload` | -| `+files` (6) | 6 | 6 + 2 | **100 %** | Python adds `putFileBytes`, `getBulkUploadURL` | -| `+users` (3) | 3 | 3 | **100 %** | | -| `+compute` (6) | 6 | 6 | **100 %** | | -| `+auth` (8) | 8 | 8 | **100 %** | `loginOriginal`/`logoutOriginal` (legacy) intentionally skipped; auth funcs live in `ndi.cloud.auth` | -| `call.m` / `url.m` | 2 | — | **N/A** | Replaced by `CloudClient` + `CloudConfig` (architectural change) | -| `+implementation/*` (50 classes) | 50 | — | **N/A** | Eliminated; single `CloudClient` replaces all impl classes | - -**Architectural differences from MATLAB:** - -- MATLAB uses an abstract `call` base class with 50 concrete implementation classes (one per endpoint). Python replaces this with `CloudClient`, a thin `requests.Session` wrapper with `get`/`post`/`put`/`delete` methods. -- MATLAB `url.m` builds endpoint URLs from a name→template dictionary. Python uses `CloudConfig.api_url` + path templates in each function. -- All Python API functions use `@pydantic.validate_call` for input validation (matching MATLAB `arguments` blocks) and `@_auto_client` to make the `client` parameter optional. - -**Not ported (intentional):** - -| MATLAB | Reason | -|--------|--------| -| `ndi.cloud.uilogin` | MATLAB GUI | -| `ndi.cloud.ui.dialog.selectCloudDataset` | MATLAB GUI dialog | -| `ndi.cloud.utility.createCloudMetadataStruct` | MATLAB struct validator; `CloudConfig` replaces | -| `ndi.cloud.utility.mustBeValidMetadata` | MATLAB struct validator; type hints replace | - -### `ndi.validators` — Fully Ported - -**Verified:** 2026-03-11 against `VH-Lab/NDI-matlab` branch `main`. - -| MATLAB | Python | Coverage | -|--------|--------|:--------:| -| 11 functions | 11 + 2 | **100 %** | - -All 11 MATLAB `arguments`-block validators ported 1:1 with matching -function names. Python equivalents accept Python types (``list`` for -cell array, ``dict`` for struct, ``pd.DataFrame`` for table). - -Python adds two reusable validators with no direct MATLAB counterpart: - -| Python | Purpose | -|--------|---------| -| `is_ndarray` | Validates value is a `numpy.ndarray` | -| `is_iso8601` | Validates string is parseable ISO 8601 | - -### `ndi.util` — Fully Ported - -**Verified:** 2026-03-11 against `VH-Lab/NDI-matlab` branch `main`. - -| Category | MATLAB funcs | Python funcs | Coverage | Notes | -|----------|:------------:|:------------:|:--------:|-------| -| Data/time utilities (8) | 8 | 8 | **100 %** | | -| `+openminds` (2) | 2 | 2 | **100 %** | Ported in `ndi.openminds_convert` | -| GUI / MATLAB-specific (3) | 3 | — | **N/A** | `choosefile`, `choosefileordir`, `toolboxdir` | - -**Not ported (intentional):** - -| MATLAB | Reason | -|--------|--------| -| `ndi.util.choosefile` | MATLAB GUI dialog (`inputdlg`) | -| `ndi.util.choosefileordir` | MATLAB GUI dialog (`inputdlg`) | -| `ndi.util.toolboxdir` | MATLAB-specific path resolution | diff --git a/docs/developer_notes/PYTHON_PORTING_GUIDE.md b/docs/developer_notes/PYTHON_PORTING_GUIDE.md new file mode 100644 index 0000000..13ef13d --- /dev/null +++ b/docs/developer_notes/PYTHON_PORTING_GUIDE.md @@ -0,0 +1,55 @@ +# NDI MATLAB to Python Porting Guide + +## 1. The Core Philosophy: Lead-Follow Architecture + +The MATLAB codebase is the **Source of Truth**. The Python version is a "faithful mirror." When a conflict arises between "Pythonic" style and MATLAB symmetry, **symmetry wins**. + +- **Lead-Follow:** MATLAB defines the logic, hierarchy, and naming. +- **The Contract:** Every package contains an `ndi_matlab_python_bridge.yaml`. This file is the binding contract for function names, arguments, and return types for that specific namespace. + +## 2. Naming & Discovery (The Mirror Rule) + +Function and class names must match MATLAB exactly. + +- **Naming Source:** Refer to the local `ndi_matlab_python_bridge.yaml`. +- **Missing Entries:** If a function is not in the bridge file, refer to the MATLAB source to determine the name, add the entry to the bridge file, and notify the user of the addition for their review. +- **Case Preservation:** Use `ListAllDocuments`, not `list_all_documents`. Use `savetofile`, not `save_to_file`. +- **Directory Parity:** Python file paths must mirror MATLAB `+namespace` paths (e.g., `+ndi/+cloud` → `src/ndi/cloud/`). + +## 3. The Porting Workflow (The Bridge Protocol) + +To port or update a function, agents must follow these steps: + +1. **Check the Bridge:** Open the `ndi_matlab_python_bridge.yaml` in the target package. +2. **Sync the Interface:** If the function is missing or outdated, update the YAML entry first based on the MATLAB `.m` file. +3. **Implement:** Write the Python code to satisfy the `input_arguments` and `output_arguments` defined in the YAML. +4. **Log & Notify:** Document intentional divergences in the YAML's `decision_log`. Explicitly tell the user what changes were made to the bridge file so they can review the contract. + +## 4. Input Validation: Pydantic is Mandatory + +To replicate the robustness of the MATLAB `arguments` block, use Pydantic for all public-facing API functions. + +- **Decorator:** Use the `@pydantic.validate_call` decorator on all functions. +- **Type Mirroring:** + - MATLAB `double`/`numeric` → Python `float | int` + - MATLAB `char`/`string` → Python `str` + - MATLAB `{member1, member2}` → Python `Literal["member1", "member2"]` +- **Union Types:** Implement multiple allowed types as a Type Union (e.g., `str | int`). +- **Coercion:** Allow Pydantic's default casting (e.g., allowing a string `"1"` to satisfy a `bool` type). +- **Arbitrary Types:** For types like `numpy.ndarray`, use `config=ConfigDict(arbitrary_types_allowed=True)`. + +## 5. Multiple Returns (Outputs) + +MATLAB allows multiple return values natively. In Python, these must be returned as a **tuple** in the exact order defined in the `output_arguments` section of the bridge YAML. + +## 6. Code Style & Linting + +All Python code must pass formatting and linting before being committed. + +- **Black:** The sole code formatter. Use default line length (88). +- **Ruff:** The primary linter. Run `ruff check --fix` before committing. + +## 7. Error Handling & Documentation + +- **Hard Fails:** If a MATLAB function throws an error, the Python version must raise a corresponding Exception (`ValueError`, `TypeError`, or `NDIError`). +- **Docstring Symmetry:** Include the original MATLAB documentation in the Python docstring. Add a "Python-specific Notes" section at the bottom for library-specific details. diff --git a/docs/developer_notes/ndi_matlab_python_bridge.yaml b/docs/developer_notes/ndi_matlab_python_bridge.yaml new file mode 100644 index 0000000..afdfaf1 --- /dev/null +++ b/docs/developer_notes/ndi_matlab_python_bridge.yaml @@ -0,0 +1,83 @@ +# The NDI Bridge Protocol: YAML Specification + +# ============================================================================= +# 1. File Purpose & Placement +# ============================================================================= +# Name: ndi_matlab_python_bridge.yaml +# Location: One file per sub-package directory +# (e.g., src/ndi/session/ndi_matlab_python_bridge.yaml). +# Role: This is the Primary Contract. It defines how MATLAB names and types +# map to Python. If a function is not here, it does not officially +# exist in the Python port. + +# ============================================================================= +# 2. Standard Header +# ============================================================================= +# Every bridge file must begin with this metadata to orient the agent: + +project_metadata: + bridge_version: "1.1" + naming_policy: "Strict MATLAB Mirror" + indexing_policy: "Semantic Parity (1-based for user concepts, 0-based for internal data)" + +# ============================================================================= +# 3. The "Active Maintenance" Instruction +# ============================================================================= +# When an agent or developer works on a function: +# +# 1. Check: Does the function/class exist in the YAML? +# 2. Add/Update: If it is missing or the MATLAB signature has changed, +# update the YAML first. +# 3. Notify: The agent MUST explicitly tell the user: +# "I have updated the ndi_matlab_python_bridge.yaml to include +# [Function Name]. Please review the interface contract." + +# ============================================================================= +# 4. Structure for Classes and Functions +# ============================================================================= + +# --- Example: Class --- +# - name: NDI_Session # Exact MATLAB Name +# type: class +# matlab_path: "+ndi/Session.m" +# python_path: "ndi/session.py" +# +# properties: +# - name: reference +# type_matlab: "char" +# type_python: "str" +# decision_log: "Mirroring property name exactly." +# +# methods: +# - name: get_epoch_data +# input_arguments: +# - name: epoch_id +# type_python: "int" +# decision_log: "Semantic Parity: User provides 1-based ID." + +# --- Example: Standalone Function --- +# - name: ListAllDocuments +# type: function +# matlab_path: "+ndi/+cloud/ListAllDocuments.m" +# python_path: "ndi/cloud/ListAllDocuments.py" +# input_arguments: +# - name: DatasetID +# type_matlab: "string | numeric" +# type_python: "str | int" +# output_arguments: +# - name: docs +# type_matlab: "struct array" +# type_python: "list[dict]" +# decision_log: "Python returns a list of dictionaries to mimic MATLAB struct arrays." + +# ============================================================================= +# 5. Summary of Field Rules +# ============================================================================= +# Field | Rule +# ------------------|---------------------------------------------------------- +# name | Strict Match. Case-sensitive match to the MATLAB .m file. +# input_arguments | Used to generate Pydantic @validate_call. +# output_arguments | Defines the order of the Return Tuple. +# type_python | Use Python 3.10+ syntax (e.g., str | int). +# decision_log | Mandatory for any divergence (e.g., "Shifted to +# | 0-indexing for internal array access"). diff --git a/docs/developer_notes/ndi_xlang_principles.md b/docs/developer_notes/ndi_xlang_principles.md new file mode 100644 index 0000000..e4750de --- /dev/null +++ b/docs/developer_notes/ndi_xlang_principles.md @@ -0,0 +1,54 @@ +# NDI Cross-Language (MATLAB/Python) Principles + +- **Status:** Active +- **Scope:** Universal (Applies to all NDI implementations) +- **Goal:** Zero-friction cognitive switching for researchers. + +## 1. Indexing & Counting (The Semantic Parity Rule) + +We distinguish between **Computer Science Indexing** and **Scientific Counting**. + +- **Implementation (0-vs-1):** + - Python implementations MUST use 0-indexing for internal data structures (lists, arrays, dataframes). + - MATLAB implementations MUST use 1-indexing for internal structures. +- **User-Facing Concepts (Counting from 1):** + - **The Rule:** Any NDI concept that is "counted" (Epochs, Channels, Trials, Probes) MUST use 1-based numbering in both languages. + - **Reasoning:** If a scientist records "Channel 1," it must be called "Channel 1" in both MATLAB and Python. If Python used "Channel 0," it would create a dangerous "off-by-one" error when comparing results across platforms. + - **Implementation:** Python code must accept `channel_number=1` from the user, but internally map it to `data[0]`. + +## 2. Data Containers + +- **Decision:** Prefer NumPy over Lists. +- **Rule:** Any MATLAB `double` array or matrix should be represented as a `numpy.ndarray` in Python, not a native Python list. +- **Rationale:** Neuroscience data is high-dimensional. NumPy provides the performance and slicing capabilities that match MATLAB's matrix engine. + +## 3. Multiple Returns (Outputs) + +- **Decision:** Explicit Tuple Returns. +- **Rule:** Python functions must return multiple values as a tuple in the exact order specified in the MATLAB function signature. +- **Parity:** If MATLAB returns `[data, information]`, Python must return `(data, information)`. + +## 4. Logical Values (Booleans) + +- **Decision:** Strict Boolean Mirroring. +- **Rule:** MATLAB `1`/`0` (logical) must be Python `True`/`False`. +- **Pydantic Role:** Use Pydantic to allow the string `"true"` or `"false"` to be coerced into Python booleans for API robustness. + +## 5. Character Arrays vs. Strings + +- **Decision:** Universal String Handling. +- **Rule:** MATLAB `char` and `string` are both treated as Python `str`. +- **Handling Lists:** A MATLAB cell array of strings must be a Python `list[str]`. + +## 6. Error Philosophy (Hard Fail) + +- **Decision:** No Silent Failures. +- **Rule:** If MATLAB issues an `error`, Python MUST raise an exception (e.g., `ValueError`, `TypeError`). +- **Parity:** Ensure the exception is raised at the same logical checkpoint as the MATLAB error. + +## Instructions for AI Agents + +1. **Data Indexing:** Use 0-indexing for Python list/array access. +2. **Scientific Counting:** Maintain 1-based counting for Epochs, Channels, and Trials. If a user passes `epoch_id=1`, do not subtract 1 unless you are using it to index an internal array. +3. **NumPy:** Always prioritize `numpy.ndarray` for numerical data. +4. **Bridge File:** Consult the local `ndi_matlab_python_bridge.yaml` to see if specific counting overrides exist for a function.