feat: Add Disaggregated COT ingestion pipeline

.github/workflows/python-test.yml

@@ -0,0 +1,32 @@
+name: Python package
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install -e ".[dev]"
+    - name: Run tests
+      run: |
+        # Create a dummy store for the test environment to prevent COTDATA_STORE missing errors
+        mkdir -p /tmp/cotdata_store
+        export COTDATA_STORE=/tmp/cotdata_store
+        pytest tests/

CONTRIBUTING.md

@@ -0,0 +1,56 @@
+# Contributing to cotdata
+
+Thank you for your interest in contributing to `cotdata`! This project provides the canonical data layer for quantitative analysis by fetching CFTC COT and Norgate price data into a shared, offline, cross-platform parquet store.
+
+## How to Contribute
+
+1. **Reporting Bugs**: Open an issue and provide as much detail as possible, including your OS (Windows vs Mac is critical here due to Norgate Data), Python version, and clear steps to reproduce the issue.
+2. **Suggesting Enhancements**: Feel free to open an issue or start a Discussion on GitHub. If you're adding support for a new data provider, please propose the architecture first.
+3. **Submitting Pull Requests**:
+   - Fork the repository and create your feature branch (`git checkout -b feature/amazing-feature`).
+   - Write clear, documented code.
+   - Run the local test suite (see below) before submitting your PR.
+   - Open the PR against the `main` branch.
+
+## Development Setup
+
+This project uses `uv` (or standard `pip` environments) for dependency management.
+
+1. **Clone the repository:**
+   ```bash
+   git clone https://github.com/your-username/cotdata.git
+   cd cotdata
+   ```
+
+2. **Set up a virtual environment and install dev dependencies:**
+   ```bash
+   uv venv
+   uv pip install -e ".[dev]"
+   ```
+   *Note: If you are on Windows and intend to work on the Norgate integration, you should install the Norgate extras: `uv pip install -e ".[dev,norgate]"`.*
+
+3. **Set the temporary datastore:**
+   For local development, you should point the store to a local temporary folder to avoid overwriting your live data:
+   ```bash
+   export COTDATA_STORE=/tmp/cotdata_test_store  # Mac/Linux
+   # OR
+   $env:COTDATA_STORE = "C:\temp\cotdata_test_store"  # Windows
+   ```
+
+## Running Tests
+
+We use `pytest` for all unit and integration tests.
+
+```bash
+pytest tests/
+```
+
+### Important Note on Norgate Testing
+The Norgate integration (`src/cotdata/providers/norgate.py` and `tests/test_adjustment.py`) **only runs on Windows** because it requires the proprietary Norgate Data Updater application to be installed, running, and authenticated locally.
+- If you submit a PR on a Mac or Linux machine, only the generic CFTC parsing tests will run locally.
+- The CI pipeline will attempt to run cross-platform tests where possible, but cannot test live Norgate API calls. Please be extremely careful when modifying Norgate adapter logic.
+
+## Code Style
+
+- We prefer standard PEP8 formatting. 
+- Try to keep the file sizes of the generated parquet data as small as possible. The `cotdata` pipeline aggressively drops irrelevant columns (like CFTC concentration ratios) to ensure the data lake remains highly performant for downstream pandas/XGBoost models. If you are adding a new data feature, only retain the columns necessary for quantitative analysis.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Matt S.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

OPEN_SOURCE_TODO.md

@@ -0,0 +1,21 @@
+# Remaining Open Source Tasks
+
+This checklist tracks the remaining tasks to get `cotdata` fully ready for public release and PyPI publication.
+
+## Code & Repository Updates
+- [ ] **Update `pyproject.toml` Metadata**: Add the required fields to publish to PyPI.
+  - `authors` list (e.g., `[{name = "Matt S.", email = "..."}]`)
+  - `urls` block (e.g., `Homepage`, `Repository`, `Issues`)
+  - `classifiers` (e.g., `License :: OSI Approved :: MIT License`, `Programming Language :: Python :: 3`)
+- [ ] **Update `README.md` Installation Instructions**: Add a generic `pip install cotdata` and PyPI link, moving the local `uv` workspace instructions to an "Advanced/Development" section.
+- [ ] **Add Badges to `README.md`**: Add markdown shields at the top of the README for CI Status, PyPI Version, and MIT License.
+- [ ] **Create Issue Templates**: Add `.github/ISSUE_TEMPLATE/bug_report.md` and `feature_request.md` (make sure they ask the user to specify their OS since Norgate requires Windows).
+- [ ] **Create Pull Request Template**: Add `.github/PULL_REQUEST_TEMPLATE.md` to ensure PRs reference related issues and confirm tests have passed.
+
+## GitHub Settings Configuration
+*(To be done in the repository's Settings tab on github.com)*
+- [ ] **Branch Protection Rules**: Protect `main`. Require PRs before merging and require the "test" GitHub Action status check to pass.
+- [ ] **Security & Analysis**: Enable Dependabot alerts and security updates for dependencies.
+- [ ] **Enable Discussions**: Turn on GitHub Discussions in settings to field general usage/data questions away from the Issue tracker.
+- [ ] **Repository Details**: Add a project description, website link, and tags/topics (e.g., `algotrading`, `cot-data`, `cftc`, `norgatedata`) to the repository's about section.
+- [ ] **Create First Release**: Tag `v0.1.0` and use GitHub Releases to generate the release notes once ready to publish.

README.md

@@ -8,10 +8,10 @@ Canonical data layer for the COT / futures-strategy stack. It exists so that qua
                        │  writes   │
                        ▼           ▼
         ┌───────────────────────────────────────┐
-        │  CANONICAL STORE   ($COTDATA_STORE)    │
-        │  prices/*.parquet  cot/*.parquet       │   ← synced (rsync / Dropbox / S3)
-        │  manifest.json     (the contract)      │
-        └───────────────────────────────────────┘
+        │  CANONICAL STORE   ($COTDATA_STORE)            │
+        │  prices/*.parquet  cot_legacy/*.parquet        │   ← synced (rsync / Dropbox / S3)
+        │  cot_disagg/*.parquet  manifest.json           │
+        └────────────────────────────────────────────────┘
                        ▲           ▲   reads (offline, cross-platform)
         ┌──────────────┴───┐  ┌────┴──────────────┐
         │   COT analyzer   │  │  analysis toolset │      both:  import cotdata
@@ -52,7 +52,7 @@ Swapping a vendor is a producer-only change.
 
 - `prices/{symbol}_{adjustment}.parquet` — Open/High/Low/Close/Volume/Open Interest,
   tz-naive `Date` index. `adjustment` ∈ {`backadj`, `unadj`}. Close = exchange settlement.
-- `cot/{code}.parquet` — weekly CFTC positioning.
+- `cot_legacy/{code}.parquet` — weekly CFTC Legacy positioning.
 - `manifest.json` — per-table `last_date`, `n_rows`, `source`, `updated_at`, `schema_version`.
 
 ## Consumer
@@ -61,7 +61,8 @@ Swapping a vendor is a producer-only change.
 import cotdata
 df = cotdata.get_prices("ES", adjustment="backadj")   # USE THIS FOR SIGNALS + STOPS
 sz = cotdata.get_prices("ES", adjustment="unadj")     # USE FOR POSITION SIZING / POINT VALUE
-cot = cotdata.get_cot("ES")
+cot_legacy = cotdata.get_cot("ES", report="legacy")   # USE FOR COMM/NON-COMM
+cot_disagg = cotdata.get_cot("ES", report="disagg")   # USE FOR TRADER COUNTS (MM/SD/OR)
 ```
 
 Set `COTDATA_STORE` to the synced store directory. 
@@ -72,7 +73,8 @@ Set `COTDATA_STORE` to the synced store directory.
 
 ```
 COTDATA_STORE=/store  cotdata-update --prices --symbols ES NQ    # Norgate (Windows)
-COTDATA_STORE=/store  cotdata-update --cot                       # CFTC (cross-platform)
+COTDATA_STORE=/store  cotdata-update --cot-legacy                # CFTC Legacy (cross-platform)
+COTDATA_STORE=/store  cotdata-update --cot-disagg                # CFTC Disaggregated (cross-platform)
 ```
 
 Schedule nightly (prices, after the Norgate Data Updater) and weekly (COT Friday releases).
@@ -137,8 +139,8 @@ The primary source for price history (Norgate Data). Indexed by tz-naive `Date`.
 | `Open Interest` | float | Total open interest. |
 | `Delivery Month` | float | Expiration month of the active contract (e.g. `202609`). Used to detect contract rolls. |
 
-### COT Data (`cot/{code}.parquet`)
-The primary source for positioning data (CFTC Legacy Futures Report). Indexed by tz-naive `Report_Date_as_MM_DD_YYYY`.
+### COT Legacy Data (`cot_legacy/{code}.parquet`)
+The primary source for Legacy positioning data (CFTC Legacy Futures Report). Indexed by tz-naive `Report_Date_as_MM_DD_YYYY`.
 
 > [!NOTE]
 > **Legacy Reports**: The Legacy reports are broken down by exchange. These reports have a futures only report and a combined futures and options report. Legacy reports break down the reportable open interest positions into two classifications: non-commercial and commercial traders. The `cotdata` pipeline strictly downloads the **Futures-only** reports (located at `https://www.cftc.gov/files/dea/history/dea_fut_xls_{YEAR}.zip`).
@@ -158,3 +160,9 @@ The primary source for positioning data (CFTC Legacy Futures Report). Indexed by
 | `NonComm_Positions_Short_All` | float | Non-Commercial (Large Speculator) Short positions. |
 | `NonRept_Positions_Long_All` | float | Non-Reportable (Small Speculator) Long positions. |
 | `NonRept_Positions_Short_All` | float | Non-Reportable (Small Speculator) Short positions. |
+
+### COT Disaggregated Data (`cot_disagg/{code}.parquet`)
+The primary source for entity-specific positioning and trader counts (CFTC Disaggregated Futures-Only Report, dating back to 2006). Indexed by tz-naive `Report_Date_as_MM_DD_YYYY`.
+
+> [!NOTE]
+> **Lossless Image**: Unlike the Legacy schema which filters down to 10 specific columns, the Disaggregated parquets are a **lossless image** of the source CFTC `txt` files. They contain all granular entity groups (Money Manager, Swap Dealer, Producer/Merchant, Other Reportable) and their respective `Traders_*` counts (e.g., `Traders_Tot_All`, `Traders_M_Money_Long_All`). This is the required store for computing Position Size and Clustering metrics.

docs/spec_disaggregated_ingest.md

@@ -0,0 +1,28 @@
+# Spec: Disaggregated Futures-Only COT Ingest
+
+This document dictates the canonical data layer extraction for the Disaggregated Futures-Only CFTC report.
+
+## 1. Why Disaggregated?
+The existing Legacy report (`cot/`) provides Commercial and NonCommercial positional breakdowns. However, it does not include trader counts, nor does it split out Swap Dealers (SD) or Money Managers (MM). 
+To compute Keenan's Positioning Trio (Concentration, Clustering, and Position Size), we must ingest the Disaggregated report, which includes `Traders_*` columns natively.
+
+## 2. Ingest Architecture
+This pipeline operates strictly independently from the Legacy pipeline.
+*   **Producer Module**: `src/cotdata/providers/cftc_disagg.py`
+*   **Source Files**: `fut_disagg_txt_{year}.zip` (TXT/CSV is significantly faster to parse and natively supported by `pandas.read_csv` without the heavy `xlrd` dependency required by Legacy `.xls`).
+*   **First Available Year**: 2006
+*   **Fidelity**: Unlike Legacy, the Disaggregated pipeline is **near-lossless**. It preserves every `Traders_*` column directly from the zip without sub-setting, only normalizing `Report_Date_as_MM_DD_YYYY` to datetimes and padding `CFTC_Contract_Market_Code` to 6-digits.
+*   **Store Namespace**: To prevent collision, it writes to a new `$COTDATA_STORE/cot_disagg/` subdirectory and is tracked under a `"cot_disagg"` key in `manifest.json`.
+
+## 3. Public API
+The consumer API remains `cotdata.get_cot(name)`, but now accepts a `report` argument:
+```python
+df_legacy = cotdata.get_cot("ES", report="legacy")  # existing behavior
+df_disagg = cotdata.get_cot("ES", report="disagg")  # new lossless disaggregated DataFrame
+```
+Predecessor contract stitching (`hist_codes`) applies identically to both reports.
+
+## 4. Reconciliation
+Because Disaggregated is parallel to Legacy, we validate it by checking that the summed entities match the Legacy groups:
+*   `Money_Manager` + `Other_Reportable` ≈ `NonCommercial` (Large Speculators)
+*   `Producer_Merchant` + `Swap_Dealer` ≈ `Commercial` (Hedgers)

src/cotdata/config.py

@@ -19,8 +19,12 @@ def prices_dir() -> Path:
     return store_root() / "prices"
 
 
-def cot_dir() -> Path:
-    return store_root() / "cot"
+def cot_legacy_dir() -> Path:
+    return store_root() / "cot_legacy"
+
+
+def cot_disagg_dir() -> Path:
+    return store_root() / "cot_disagg"
 
 
 def manifest_path() -> Path:

src/cotdata/cot.py

@@ -12,7 +12,7 @@
 _CODE_COL = "CFTC_Contract_Market_Code"   # Legacy schema contract-code column
 
 
-def get_cot(name: str) -> pd.DataFrame:
+def get_cot(name: str, report: str = "legacy") -> pd.DataFrame:
     """Weekly COT for an internal symbol OR a raw CFTC code. Empty if absent.
 
     If the resolved symbol declares hist_codes (predecessor exchange listings of
@@ -24,14 +24,17 @@ def get_cot(name: str) -> pd.DataFrame:
     sym = REGISTRY.get(name)
     if sym is None:                       # allow lookup by primary CFTC code, not just symbol
         sym = next((s for s in REGISTRY.values() if s.cftc_code == name), None)
+
+    read_fn = store.read_cot_disagg if report == "disagg" else store.read_cot_legacy
+
     if sym is None or not sym.cftc_code:
-        return store.read_cot(name)
-    primary = store.read_cot(sym.cftc_code)
+        return read_fn(name)
+    primary = read_fn(sym.cftc_code)
     if not sym.hist_codes:
         return primary
     frames = [primary]                    # primary first → wins de-duplication on overlaps
     for hc, scale in hist_code_scales(sym.hist_codes):
-        h = store.read_cot(hc)
+        h = read_fn(hc)
         if h.empty:
             continue
         h = h.copy()

src/cotdata/providers/cftc.py

@@ -40,7 +40,7 @@
 
 
 def _cache_dir() -> Path:
-    d = config.store_root() / "_cache" / "cot"
+    d = config.store_root() / "_cache" / "cot_legacy"
     d.mkdir(parents=True, exist_ok=True)
     return d
 
@@ -116,8 +116,6 @@ def update(codes=None, first_year: int = FIRST_YEAR, last_year=None) -> None:
         if sub.empty:
             print(f"{code}: no rows")
             continue
-        # Index by report date (DatetimeIndex → manifest last_date), keeping the
-        # Legacy column name so a consumer's reset_index() gets the exact schema.
         sub = sub.sort_values(REPORT_DATE).set_index(REPORT_DATE)
-        store.write_cot(code, sub, source="cftc")
+        store.write_cot_legacy(code, sub, source="cftc")
         print(f"{code}: {len(sub):5d} weeks -> store")