feat(prices): add propadj (ratio) adjustment; salvage DC Class III Milk

CHANGELOG.md

@@ -0,0 +1,19 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres
+to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- **`propadj` price adjustment** — a proportional (ratio) back-adjusted view
+  derived on read from the stored `unadj` + `backadj` series via
+  `get_prices(symbol, adjustment="propadj")`. It preserves daily percentage
+  returns and stays strictly positive, unlike Norgate's additive `backadj`.
+  Motivated by **DC (Class III Milk)**: additive back-adjustment drove 46.7% of
+  `DC_backadj` closes ≤ 0 (range −9.83 to 23.09), making price-based stops and
+  R-multiples unusable; `propadj` yields a strictly-positive DC series
+  (4.68–25.01) over the full 1997–2026 history. Recommended for low-priced,
+  long-history contracts. Derived from already-stored series — no producer
+  re-run or schema change required. ([#23](https://github.com/mspinola/cotdata/pull/23))

README.md

@@ -76,7 +76,7 @@ The **store is the API boundary** — not Python imports. Producers write Parque
 
 The store layout:
 
-- `prices/{symbol}_{adjustment}.parquet` — Open/High/Low/Close/Volume/Open Interest, tz-naive `Date` index. `adjustment` ∈ {`backadj`, `unadj`}. Close = exchange settlement.
+- `prices/{symbol}_{adjustment}.parquet` — Open/High/Low/Close/Volume/Open Interest, tz-naive `Date` index. `adjustment` ∈ {`backadj`, `unadj`} on disk; `propadj` is a third view **derived on read** (not stored). Close = exchange settlement.
 - `cot_legacy/{symbol}_{code}.parquet` — weekly CFTC Legacy positioning.
 - `cot_disagg/{symbol}_{code}.parquet` — weekly CFTC Disaggregated positioning.
 - `cot_tff/{symbol}_{code}.parquet` — weekly CFTC Traders in Financial Futures positioning.
@@ -94,6 +94,7 @@ import cotdata
 # Prices — pick the adjustment that matches your use:
 signals = cotdata.get_prices("ES", adjustment="backadj")  # signals + stops (gap-free rolls)
 sizing  = cotdata.get_prices("ES", adjustment="unadj")    # position sizing (true dollar prices)
+milk    = cotdata.get_prices("DC", adjustment="propadj")  # ratio-adjusted: strictly positive, %-return preserving
 
 # COT — three CFTC report families:
 legacy  = cotdata.get_cot("ES", report="legacy")   # Commercial / Non-Commercial
@@ -257,10 +258,17 @@ COT tables are stored per code as **`{symbol}_{code}`** (e.g. `RTY_23977A`), so
 
 ### Back-adjusted vs unadjusted prices
 
-Futures contracts expire, forcing traders to "roll" into the next contract, which usually trades at a slightly different price. Simply stitching contracts together creates artificial price gaps, so cotdata stores two series:
+Futures contracts expire, forcing traders to "roll" into the next contract, which usually trades at a slightly different price. Simply stitching contracts together creates artificial price gaps, so cotdata stores two series and derives a third:
 
-- **`backadj` (signals & stops).** Gap-free arithmetic rolls shift historical prices to align with the new contract, preserving the *true shape* and percentage moves. Use this for indicators, signals, and stop-losses to avoid false triggers on rollover gaps.
+- **`backadj` (signals & stops).** Gap-free *arithmetic* (additive) rolls shift historical prices to align with the new contract, preserving *absolute* daily point moves. Use this for indicators, signals, and stop-losses to avoid false triggers on rollover gaps.
 - **`unadj` (position sizing).** Back-adjustment shifts historical prices (sometimes negative), so you can't use it for dollar values. Use `unadj` (raw, real-life prices) for that day to compute true dollar risk and contract counts.
+- **`propadj` (proportional / ratio adjustment — strictly positive).** Derived on read from `unadj` + `backadj`; preserves daily *percentage* returns and never goes non-positive. Use it for **low-priced, long-history contracts where additive back-adjustment accumulates roll gaps below zero** and breaks price-based stops and R-multiples. See *Class III Milk (DC)* below.
+
+#### Why `propadj` exists — Class III Milk (DC)
+
+Norgate publishes continuous futures in only two forms: unadjusted and **additive** back-adjusted (`_CCB`) — there is no native ratio-adjusted series. Additive adjustment subtracts each roll's calendar spread from all prior history, and for a low-priced, seasonal, ~29-year contract like **DC (Class III Milk, ~$15–20/cwt)** those gaps accumulate past zero: **46.7% of `DC_backadj` closes are ≤ 0** (range −9.83 to 23.09). A price-based stop, an R-multiple, or a percentage return is meaningless on a non-positive series, so CMR cannot use DC's `backadj` at all — even though DC is the flagship *new-asset-class* (Dairy) held-out generalization market.
+
+`propadj` salvages it. Because the additive series `B` and unadjusted series `U` differ by an offset `O = B − U` that steps only at rolls, each roll's calendar spread is recoverable (`s = O[r−1] − O[r]`) and convertible to a multiplicative roll ratio `k = (U[r−1] + s)/U[r−1]`. Scaling each historical segment by the cumulative product of `k` (most-recent segment anchored to actual prices) yields a series that is **strictly positive over the full 1997–2026 history** (DC range 4.68–25.01), preserves within-segment percentage returns exactly, and is sign-identical to `backadj` on every day including rolls. It is a pure function of two already-stored series, so it needs no producer re-run — `get_prices("DC", adjustment="propadj")` works today. Recommendation: **CMR reads DC (and any similarly low-priced contract) with `adjustment="propadj"`.** Restricting DC to its positive-price era (2011→present, ~15y) or dropping it were the fallbacks; neither is needed.
 
 ### Providers & authentication
 

src/cotdata/prices.py

@@ -2,20 +2,110 @@
 expects (the old fetch_daily_ohlc contract). No network, cross-platform."""
 from typing import Optional
 
+import numpy as np
 import pandas as pd
 
 from . import store
 
 _COLS = ["Open", "High", "Low", "Close", "Volume", "Open Interest"]
+_OHLC = ["Open", "High", "Low", "Close"]
+
+
+def _ratio_adjust(symbol: str) -> pd.DataFrame:
+    """Derive a proportional (ratio) back-adjusted OHLC series from the stored
+    unadjusted + additive-back-adjusted series. Empty if either is missing.
+
+    WHY. Norgate only publishes ADDITIVE (arithmetic) back-adjustment — the
+    ``_CCB`` continuous (see providers/norgate.py). For a low-priced, long-history
+    contract like DC (Class III Milk, ~$15–20/cwt over ~29y) the additive
+    accumulation of roll gaps drives ~47% of back-adjusted closes ≤ 0 (down to
+    −9.83). A close-based stop, an R-multiple, or a % return is meaningless on a
+    non-positive series, so CMR cannot use DC's ``backadj`` at all. A ratio-adjusted
+    series preserves percentage returns and stays strictly positive.
+
+    This is a consumer-side derivation (no network, no Windows/norgatedata), in the
+    same spirit as the reconstructed-volume view: the transform lives in the data
+    layer so consumers ask for what they want. It is a pure function of two series
+    already in the store, so it needs no producer re-run or schema bump.
+
+    METHOD. The additive series ``B`` and the unadjusted series ``U`` differ by an
+    offset ``O = B − U`` that is piecewise-constant and steps only at rolls (each
+    roll's step is Norgate's stitched calendar spread; verified on DC — every step
+    >$0.0001 lands on a Delivery-Month change). At roll ``r`` the recovered spread,
+    measured on the last day the OLD contract is front (day ``r−1``), is
+    ``s = O[r−1] − O[r] = F_new(r−1) − F_old(r−1)``; the roll ratio is
+    ``k = (U[r−1] + s) / U[r−1] = F_new/F_old``. Each historical segment is scaled
+    by the cumulative product of ``k`` for all rolls at/after it, anchoring the
+    most-recent segment to 1 (actual prices). Result: identical daily % returns to
+    ``U`` within a segment, gap-free true returns across each roll (sign-identical
+    to ``B``), strictly positive, anchored to the real current price. O/H/L/C are
+    scaled by the per-row segment factor; Volume/Open Interest/Delivery Month and
+    the reconstruction columns pass through from the unadjusted frame unchanged.
+    """
+    U = store.read_prices(symbol, "unadj")
+    B = store.read_prices(symbol, "backadj")
+    if U.empty or B.empty or "Close" not in U or "Close" not in B:
+        return pd.DataFrame()
+
+    U = U.copy()
+    U.index = pd.to_datetime(U.index).tz_localize(None).normalize()
+    B.index = pd.to_datetime(B.index).tz_localize(None).normalize()
+    U = U.sort_index()
+    idx = U.index.intersection(B.index.sort_values())
+    if len(idx) == 0:
+        return pd.DataFrame()
+    U = U.loc[idx]
+    b_close = B["Close"].reindex(idx)
+
+    # Roll boundaries: prefer the semantic Delivery-Month change (matches
+    # roll_dates); fall back to material offset steps when it is absent.
+    offset = b_close - U["Close"]
+    if "Delivery Month" in U.columns:
+        dm = U["Delivery Month"]
+        roll = dm.ne(dm.shift()) & dm.shift().notna()
+    else:
+        roll = offset.diff().abs() > 1e-4
+        roll.iloc[0] = False
+
+    seg = roll.cumsum()                    # segment id, increments at each roll
+    spread = (-offset.diff()).where(roll, 0.0)   # F_new(r−1) − F_old(r−1)
+    u_prev = U["Close"].shift(1)
+    ratio = pd.Series(1.0, index=idx)
+    ok = roll & (u_prev > 0)
+    ratio[ok] = (u_prev[ok] + spread[ok]) / u_prev[ok]
+
+    # Per-segment cumulative factor, anchored so the most recent segment = 1
+    # (kept at actual prices). Walk rolls back-to-front, compounding each ratio.
+    roll_ratios = ratio[roll].to_numpy()
+    n_seg = int(seg.iloc[-1])
+    factors = np.ones(n_seg + 1)
+    for s in range(n_seg - 1, -1, -1):
+        factors[s] = factors[s + 1] * roll_ratios[s]
+    factor = pd.Series(factors[seg.to_numpy()], index=idx)
+
+    out = U.copy()
+    for c in _OHLC:
+        if c in out.columns:
+            out[c] = out[c] * factor
+    return out
 
 
 def get_prices(symbol: str, adjustment: str = "backadj",
                start: Optional[str] = None,
                volume: str = "front") -> pd.DataFrame:
     """Daily bars for `symbol`.
 
-    adjustment: 'backadj' (signals + stops — settlement close, gap-free rolls,
-    shape-preserving) or 'unadj' (absolute price / point-value sizing).
+    adjustment:
+      'backadj' : additive (arithmetic) back-adjustment — settlement close,
+                  gap-free rolls, shape-preserving. Preserves absolute daily
+                  price *changes*. Default for signals + stops.
+      'unadj'   : raw front-month prices (absolute price / point-value sizing).
+      'propadj' : proportional (ratio) back-adjustment, DERIVED on read from
+                  unadj + backadj (see `_ratio_adjust`). Preserves daily *percent*
+                  returns and stays strictly positive — use it for low-priced,
+                  long-history contracts (e.g. DC / Class III Milk) where additive
+                  back-adjustment accumulates roll gaps below zero and breaks
+                  price-based stops and R-multiples.
 
     volume: which series the `Volume` column carries —
       'front'         : continuous front-month volume as Norgate reports it
@@ -34,7 +124,7 @@ def get_prices(symbol: str, adjustment: str = "backadj",
     if volume not in ("front", "reconstructed"):
         raise ValueError(f"volume must be 'front' or 'reconstructed', got {volume!r}")
 
-    df = store.read_prices(symbol, adjustment)
+    df = _ratio_adjust(symbol) if adjustment == "propadj" else store.read_prices(symbol, adjustment)
     if df.empty:
         return df
     df.index = pd.to_datetime(df.index).tz_localize(None).normalize()

tests/test_propadj.py

@@ -0,0 +1,142 @@
+"""Proportional (ratio) back-adjustment derived on read — get_prices(..., 'propadj').
+
+Models the DC / Class III Milk failure mode: a low-priced contract whose ADDITIVE
+back-adjustment (Norgate _CCB) accumulates roll gaps below zero, which breaks
+price-based stops and R-multiples. propadj must recover a strictly-positive series
+that preserves percentage returns and stays sign-identical to backadj at rolls.
+"""
+import numpy as np
+import pandas as pd
+import pytest
+
+
+@pytest.fixture()
+def store_env(tmp_path, monkeypatch):
+    monkeypatch.setenv("COTDATA_STORE", str(tmp_path))
+    return tmp_path
+
+
+def _milk_like(with_delivery_month=True):
+    """Three contract segments (two rolls) of a ~$2–4 low-priced contract.
+
+    Offsets O = B − U are piecewise-constant (−3.0, −1.5, 0), so the additive
+    close B goes negative in the oldest segment while the unadjusted close U is
+    always positive — exactly DC's situation, in miniature.
+    """
+    idx = pd.date_range("2020-01-01", periods=9, freq="D", name="Date")
+    u_close = np.array([2.0, 2.1, 2.2, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2])
+    offset = np.array([-3.0] * 3 + [-1.5] * 3 + [0.0] * 3)
+    dm = ["202003"] * 3 + ["202006"] * 3 + ["202009"] * 3
+
+    U = pd.DataFrame({
+        "Open": u_close - 0.05, "High": u_close + 0.10,
+        "Low": u_close - 0.10, "Close": u_close,
+        "Volume": [10] * 9, "Open Interest": [100] * 9,
+    }, index=idx)
+    B = U.copy()
+    for c in ("Open", "High", "Low", "Close"):
+        B[c] = U[c] + offset
+    if with_delivery_month:
+        U["Delivery Month"] = dm
+        B["Delivery Month"] = dm
+    return U, B
+
+
+def _write(sym="DC", **kw):
+    from cotdata import store
+    U, B = _milk_like(**kw)
+    store.write_prices(sym, "unadj", U, source="test")
+    store.write_prices(sym, "backadj", B, source="test")
+    return U, B
+
+
+def test_backadj_goes_negative_but_propadj_is_strictly_positive(store_env):
+    from cotdata import get_prices
+    _write()
+    assert (get_prices("DC", "backadj")["Close"] <= 0).any()      # the problem
+    p = get_prices("DC", "propadj")
+    assert (p[["Open", "High", "Low", "Close"]] > 0).all().all()  # the fix
+
+
+def test_anchored_to_actual_recent_price(store_env):
+    """Most-recent segment keeps actual (unadjusted) prices — factor == 1."""
+    from cotdata import get_prices
+    U, _ = _write()
+    p = get_prices("DC", "propadj")
+    assert p["Close"].iloc[-3:].tolist() == pytest.approx(U["Close"].iloc[-3:].tolist())
+
+
+def test_preserves_within_segment_pct_returns(store_env):
+    from cotdata import get_prices
+    U, _ = _write()
+    p = get_prices("DC", "propadj")
+    dm = p["Delivery Month"]
+    non_roll = ~(dm.ne(dm.shift()) & dm.shift().notna())
+    err = (p["Close"].pct_change() - U["Close"].pct_change())[non_roll].abs()
+    assert err.max() < 1e-12
+
+
+def test_sign_identical_to_backadj_including_rolls(store_env):
+    """Ratio- and additive-adjustment remove the same roll gaps, so every daily
+    move — including across rolls — must agree in direction."""
+    from cotdata import get_prices
+    _write()
+    p = get_prices("DC", "propadj")["Close"].diff()
+    b = get_prices("DC", "backadj")["Close"].diff()
+    both = p.notna() & b.notna() & (b.abs() > 1e-12)
+    assert (np.sign(p[both]) == np.sign(b[both])).all()
+
+
+def test_hand_computed_factors(store_env):
+    """Regression guard on the exact ratio construction.
+
+    spread(roll) = O[r−1] − O[r]; k = (U[r−1] + spread) / U[r−1]; segment factor =
+    product of k for rolls at/after it, most-recent segment anchored to 1.
+      roll@day3: spread = −3.0−(−1.5) = −1.5, U_prev = 2.2 → k1 = 0.7/2.2
+      roll@day6: spread = −1.5−0    = −1.5, U_prev = 3.2 → k2 = 1.7/3.2
+      factor[seg2]=1, factor[seg1]=k2, factor[seg0]=k1*k2
+    """
+    from cotdata import get_prices
+    _write()
+    c = get_prices("DC", "propadj")["Close"]
+    k1, k2 = 0.7 / 2.2, 1.7 / 3.2
+    assert c.iloc[0] == pytest.approx(2.0 * k1 * k2)   # oldest segment
+    assert c.iloc[3] == pytest.approx(3.0 * k2)        # middle segment
+    assert c.iloc[6] == pytest.approx(4.0)             # anchor segment, factor 1
+
+
+def test_ohlc_ordering_preserved(store_env):
+    from cotdata import get_prices
+    _write()
+    p = get_prices("DC", "propadj")
+    assert (p["High"] >= p["Close"]).all() and (p["Close"] >= p["Low"]).all()
+    assert (p["High"] >= p["Open"]).all() and (p["Open"] >= p["Low"]).all()
+
+
+def test_falls_back_to_offset_jumps_without_delivery_month(store_env):
+    """Rolls are still detected from offset steps when Delivery Month is absent."""
+    from cotdata import get_prices
+    _write(with_delivery_month=False)
+    p = get_prices("DC", "propadj")
+    assert (p["Close"] > 0).all()
+    assert p["Close"].iloc[0] == pytest.approx(2.0 * (0.7 / 2.2) * (1.7 / 3.2))
+
+
+def test_empty_when_either_series_missing(store_env):
+    from cotdata import store, get_prices
+    U, _ = _milk_like()
+    store.write_prices("DC", "unadj", U, source="test")  # no backadj written
+    assert get_prices("DC", "propadj").empty
+
+
+def test_reconstructed_volume_view_still_works_on_propadj(store_env):
+    """The volume view composes with the derived adjustment."""
+    from cotdata import store, get_prices
+    U, B = _milk_like()
+    U["Volume_Reconstructed"] = [15, 14, 10, 18, 20, 22, 25, 24, 26]
+    U["Volume_Source"] = ["reconstructed"] * 9
+    store.write_prices("DC", "unadj", U, source="test")
+    store.write_prices("DC", "backadj", B, source="test")
+    p = get_prices("DC", "propadj", volume="reconstructed")
+    assert p["Volume"].tolist() == [15, 14, 10, 18, 20, 22, 25, 24, 26]
+    assert "Volume_Source" in p.columns