diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..c58832a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,19 @@ +# Changelog + +All notable changes to this project are documented here. The format is based on +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres +to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [Unreleased] + +### Added +- **`propadj` price adjustment** — a proportional (ratio) back-adjusted view + derived on read from the stored `unadj` + `backadj` series via + `get_prices(symbol, adjustment="propadj")`. It preserves daily percentage + returns and stays strictly positive, unlike Norgate's additive `backadj`. + Motivated by **DC (Class III Milk)**: additive back-adjustment drove 46.7% of + `DC_backadj` closes ≤ 0 (range −9.83 to 23.09), making price-based stops and + R-multiples unusable; `propadj` yields a strictly-positive DC series + (4.68–25.01) over the full 1997–2026 history. Recommended for low-priced, + long-history contracts. Derived from already-stored series — no producer + re-run or schema change required. ([#23](https://github.com/mspinola/cotdata/pull/23)) diff --git a/README.md b/README.md index 1af7ffc..37d8c19 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ The **store is the API boundary** — not Python imports. Producers write Parque The store layout: -- `prices/{symbol}_{adjustment}.parquet` — Open/High/Low/Close/Volume/Open Interest, tz-naive `Date` index. `adjustment` ∈ {`backadj`, `unadj`}. Close = exchange settlement. +- `prices/{symbol}_{adjustment}.parquet` — Open/High/Low/Close/Volume/Open Interest, tz-naive `Date` index. `adjustment` ∈ {`backadj`, `unadj`} on disk; `propadj` is a third view **derived on read** (not stored). Close = exchange settlement. - `cot_legacy/{symbol}_{code}.parquet` — weekly CFTC Legacy positioning. - `cot_disagg/{symbol}_{code}.parquet` — weekly CFTC Disaggregated positioning. - `cot_tff/{symbol}_{code}.parquet` — weekly CFTC Traders in Financial Futures positioning. @@ -94,6 +94,7 @@ import cotdata # Prices — pick the adjustment that matches your use: signals = cotdata.get_prices("ES", adjustment="backadj") # signals + stops (gap-free rolls) sizing = cotdata.get_prices("ES", adjustment="unadj") # position sizing (true dollar prices) +milk = cotdata.get_prices("DC", adjustment="propadj") # ratio-adjusted: strictly positive, %-return preserving # COT — three CFTC report families: legacy = cotdata.get_cot("ES", report="legacy") # Commercial / Non-Commercial @@ -257,10 +258,17 @@ COT tables are stored per code as **`{symbol}_{code}`** (e.g. `RTY_23977A`), so ### Back-adjusted vs unadjusted prices -Futures contracts expire, forcing traders to "roll" into the next contract, which usually trades at a slightly different price. Simply stitching contracts together creates artificial price gaps, so cotdata stores two series: +Futures contracts expire, forcing traders to "roll" into the next contract, which usually trades at a slightly different price. Simply stitching contracts together creates artificial price gaps, so cotdata stores two series and derives a third: -- **`backadj` (signals & stops).** Gap-free arithmetic rolls shift historical prices to align with the new contract, preserving the *true shape* and percentage moves. Use this for indicators, signals, and stop-losses to avoid false triggers on rollover gaps. +- **`backadj` (signals & stops).** Gap-free *arithmetic* (additive) rolls shift historical prices to align with the new contract, preserving *absolute* daily point moves. Use this for indicators, signals, and stop-losses to avoid false triggers on rollover gaps. - **`unadj` (position sizing).** Back-adjustment shifts historical prices (sometimes negative), so you can't use it for dollar values. Use `unadj` (raw, real-life prices) for that day to compute true dollar risk and contract counts. +- **`propadj` (proportional / ratio adjustment — strictly positive).** Derived on read from `unadj` + `backadj`; preserves daily *percentage* returns and never goes non-positive. Use it for **low-priced, long-history contracts where additive back-adjustment accumulates roll gaps below zero** and breaks price-based stops and R-multiples. See *Class III Milk (DC)* below. + +#### Why `propadj` exists — Class III Milk (DC) + +Norgate publishes continuous futures in only two forms: unadjusted and **additive** back-adjusted (`_CCB`) — there is no native ratio-adjusted series. Additive adjustment subtracts each roll's calendar spread from all prior history, and for a low-priced, seasonal, ~29-year contract like **DC (Class III Milk, ~$15–20/cwt)** those gaps accumulate past zero: **46.7% of `DC_backadj` closes are ≤ 0** (range −9.83 to 23.09). A price-based stop, an R-multiple, or a percentage return is meaningless on a non-positive series, so CMR cannot use DC's `backadj` at all — even though DC is the flagship *new-asset-class* (Dairy) held-out generalization market. + +`propadj` salvages it. Because the additive series `B` and unadjusted series `U` differ by an offset `O = B − U` that steps only at rolls, each roll's calendar spread is recoverable (`s = O[r−1] − O[r]`) and convertible to a multiplicative roll ratio `k = (U[r−1] + s)/U[r−1]`. Scaling each historical segment by the cumulative product of `k` (most-recent segment anchored to actual prices) yields a series that is **strictly positive over the full 1997–2026 history** (DC range 4.68–25.01), preserves within-segment percentage returns exactly, and is sign-identical to `backadj` on every day including rolls. It is a pure function of two already-stored series, so it needs no producer re-run — `get_prices("DC", adjustment="propadj")` works today. Recommendation: **CMR reads DC (and any similarly low-priced contract) with `adjustment="propadj"`.** Restricting DC to its positive-price era (2011→present, ~15y) or dropping it were the fallbacks; neither is needed. ### Providers & authentication diff --git a/src/cotdata/prices.py b/src/cotdata/prices.py index 8bf9140..fcd8e99 100644 --- a/src/cotdata/prices.py +++ b/src/cotdata/prices.py @@ -2,11 +2,92 @@ expects (the old fetch_daily_ohlc contract). No network, cross-platform.""" from typing import Optional +import numpy as np import pandas as pd from . import store _COLS = ["Open", "High", "Low", "Close", "Volume", "Open Interest"] +_OHLC = ["Open", "High", "Low", "Close"] + + +def _ratio_adjust(symbol: str) -> pd.DataFrame: + """Derive a proportional (ratio) back-adjusted OHLC series from the stored + unadjusted + additive-back-adjusted series. Empty if either is missing. + + WHY. Norgate only publishes ADDITIVE (arithmetic) back-adjustment — the + ``_CCB`` continuous (see providers/norgate.py). For a low-priced, long-history + contract like DC (Class III Milk, ~$15–20/cwt over ~29y) the additive + accumulation of roll gaps drives ~47% of back-adjusted closes ≤ 0 (down to + −9.83). A close-based stop, an R-multiple, or a % return is meaningless on a + non-positive series, so CMR cannot use DC's ``backadj`` at all. A ratio-adjusted + series preserves percentage returns and stays strictly positive. + + This is a consumer-side derivation (no network, no Windows/norgatedata), in the + same spirit as the reconstructed-volume view: the transform lives in the data + layer so consumers ask for what they want. It is a pure function of two series + already in the store, so it needs no producer re-run or schema bump. + + METHOD. The additive series ``B`` and the unadjusted series ``U`` differ by an + offset ``O = B − U`` that is piecewise-constant and steps only at rolls (each + roll's step is Norgate's stitched calendar spread; verified on DC — every step + >$0.0001 lands on a Delivery-Month change). At roll ``r`` the recovered spread, + measured on the last day the OLD contract is front (day ``r−1``), is + ``s = O[r−1] − O[r] = F_new(r−1) − F_old(r−1)``; the roll ratio is + ``k = (U[r−1] + s) / U[r−1] = F_new/F_old``. Each historical segment is scaled + by the cumulative product of ``k`` for all rolls at/after it, anchoring the + most-recent segment to 1 (actual prices). Result: identical daily % returns to + ``U`` within a segment, gap-free true returns across each roll (sign-identical + to ``B``), strictly positive, anchored to the real current price. O/H/L/C are + scaled by the per-row segment factor; Volume/Open Interest/Delivery Month and + the reconstruction columns pass through from the unadjusted frame unchanged. + """ + U = store.read_prices(symbol, "unadj") + B = store.read_prices(symbol, "backadj") + if U.empty or B.empty or "Close" not in U or "Close" not in B: + return pd.DataFrame() + + U = U.copy() + U.index = pd.to_datetime(U.index).tz_localize(None).normalize() + B.index = pd.to_datetime(B.index).tz_localize(None).normalize() + U = U.sort_index() + idx = U.index.intersection(B.index.sort_values()) + if len(idx) == 0: + return pd.DataFrame() + U = U.loc[idx] + b_close = B["Close"].reindex(idx) + + # Roll boundaries: prefer the semantic Delivery-Month change (matches + # roll_dates); fall back to material offset steps when it is absent. + offset = b_close - U["Close"] + if "Delivery Month" in U.columns: + dm = U["Delivery Month"] + roll = dm.ne(dm.shift()) & dm.shift().notna() + else: + roll = offset.diff().abs() > 1e-4 + roll.iloc[0] = False + + seg = roll.cumsum() # segment id, increments at each roll + spread = (-offset.diff()).where(roll, 0.0) # F_new(r−1) − F_old(r−1) + u_prev = U["Close"].shift(1) + ratio = pd.Series(1.0, index=idx) + ok = roll & (u_prev > 0) + ratio[ok] = (u_prev[ok] + spread[ok]) / u_prev[ok] + + # Per-segment cumulative factor, anchored so the most recent segment = 1 + # (kept at actual prices). Walk rolls back-to-front, compounding each ratio. + roll_ratios = ratio[roll].to_numpy() + n_seg = int(seg.iloc[-1]) + factors = np.ones(n_seg + 1) + for s in range(n_seg - 1, -1, -1): + factors[s] = factors[s + 1] * roll_ratios[s] + factor = pd.Series(factors[seg.to_numpy()], index=idx) + + out = U.copy() + for c in _OHLC: + if c in out.columns: + out[c] = out[c] * factor + return out def get_prices(symbol: str, adjustment: str = "backadj", @@ -14,8 +95,17 @@ def get_prices(symbol: str, adjustment: str = "backadj", volume: str = "front") -> pd.DataFrame: """Daily bars for `symbol`. - adjustment: 'backadj' (signals + stops — settlement close, gap-free rolls, - shape-preserving) or 'unadj' (absolute price / point-value sizing). + adjustment: + 'backadj' : additive (arithmetic) back-adjustment — settlement close, + gap-free rolls, shape-preserving. Preserves absolute daily + price *changes*. Default for signals + stops. + 'unadj' : raw front-month prices (absolute price / point-value sizing). + 'propadj' : proportional (ratio) back-adjustment, DERIVED on read from + unadj + backadj (see `_ratio_adjust`). Preserves daily *percent* + returns and stays strictly positive — use it for low-priced, + long-history contracts (e.g. DC / Class III Milk) where additive + back-adjustment accumulates roll gaps below zero and breaks + price-based stops and R-multiples. volume: which series the `Volume` column carries — 'front' : continuous front-month volume as Norgate reports it @@ -34,7 +124,7 @@ def get_prices(symbol: str, adjustment: str = "backadj", if volume not in ("front", "reconstructed"): raise ValueError(f"volume must be 'front' or 'reconstructed', got {volume!r}") - df = store.read_prices(symbol, adjustment) + df = _ratio_adjust(symbol) if adjustment == "propadj" else store.read_prices(symbol, adjustment) if df.empty: return df df.index = pd.to_datetime(df.index).tz_localize(None).normalize() diff --git a/tests/test_propadj.py b/tests/test_propadj.py new file mode 100644 index 0000000..94a48f7 --- /dev/null +++ b/tests/test_propadj.py @@ -0,0 +1,142 @@ +"""Proportional (ratio) back-adjustment derived on read — get_prices(..., 'propadj'). + +Models the DC / Class III Milk failure mode: a low-priced contract whose ADDITIVE +back-adjustment (Norgate _CCB) accumulates roll gaps below zero, which breaks +price-based stops and R-multiples. propadj must recover a strictly-positive series +that preserves percentage returns and stays sign-identical to backadj at rolls. +""" +import numpy as np +import pandas as pd +import pytest + + +@pytest.fixture() +def store_env(tmp_path, monkeypatch): + monkeypatch.setenv("COTDATA_STORE", str(tmp_path)) + return tmp_path + + +def _milk_like(with_delivery_month=True): + """Three contract segments (two rolls) of a ~$2–4 low-priced contract. + + Offsets O = B − U are piecewise-constant (−3.0, −1.5, 0), so the additive + close B goes negative in the oldest segment while the unadjusted close U is + always positive — exactly DC's situation, in miniature. + """ + idx = pd.date_range("2020-01-01", periods=9, freq="D", name="Date") + u_close = np.array([2.0, 2.1, 2.2, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2]) + offset = np.array([-3.0] * 3 + [-1.5] * 3 + [0.0] * 3) + dm = ["202003"] * 3 + ["202006"] * 3 + ["202009"] * 3 + + U = pd.DataFrame({ + "Open": u_close - 0.05, "High": u_close + 0.10, + "Low": u_close - 0.10, "Close": u_close, + "Volume": [10] * 9, "Open Interest": [100] * 9, + }, index=idx) + B = U.copy() + for c in ("Open", "High", "Low", "Close"): + B[c] = U[c] + offset + if with_delivery_month: + U["Delivery Month"] = dm + B["Delivery Month"] = dm + return U, B + + +def _write(sym="DC", **kw): + from cotdata import store + U, B = _milk_like(**kw) + store.write_prices(sym, "unadj", U, source="test") + store.write_prices(sym, "backadj", B, source="test") + return U, B + + +def test_backadj_goes_negative_but_propadj_is_strictly_positive(store_env): + from cotdata import get_prices + _write() + assert (get_prices("DC", "backadj")["Close"] <= 0).any() # the problem + p = get_prices("DC", "propadj") + assert (p[["Open", "High", "Low", "Close"]] > 0).all().all() # the fix + + +def test_anchored_to_actual_recent_price(store_env): + """Most-recent segment keeps actual (unadjusted) prices — factor == 1.""" + from cotdata import get_prices + U, _ = _write() + p = get_prices("DC", "propadj") + assert p["Close"].iloc[-3:].tolist() == pytest.approx(U["Close"].iloc[-3:].tolist()) + + +def test_preserves_within_segment_pct_returns(store_env): + from cotdata import get_prices + U, _ = _write() + p = get_prices("DC", "propadj") + dm = p["Delivery Month"] + non_roll = ~(dm.ne(dm.shift()) & dm.shift().notna()) + err = (p["Close"].pct_change() - U["Close"].pct_change())[non_roll].abs() + assert err.max() < 1e-12 + + +def test_sign_identical_to_backadj_including_rolls(store_env): + """Ratio- and additive-adjustment remove the same roll gaps, so every daily + move — including across rolls — must agree in direction.""" + from cotdata import get_prices + _write() + p = get_prices("DC", "propadj")["Close"].diff() + b = get_prices("DC", "backadj")["Close"].diff() + both = p.notna() & b.notna() & (b.abs() > 1e-12) + assert (np.sign(p[both]) == np.sign(b[both])).all() + + +def test_hand_computed_factors(store_env): + """Regression guard on the exact ratio construction. + + spread(roll) = O[r−1] − O[r]; k = (U[r−1] + spread) / U[r−1]; segment factor = + product of k for rolls at/after it, most-recent segment anchored to 1. + roll@day3: spread = −3.0−(−1.5) = −1.5, U_prev = 2.2 → k1 = 0.7/2.2 + roll@day6: spread = −1.5−0 = −1.5, U_prev = 3.2 → k2 = 1.7/3.2 + factor[seg2]=1, factor[seg1]=k2, factor[seg0]=k1*k2 + """ + from cotdata import get_prices + _write() + c = get_prices("DC", "propadj")["Close"] + k1, k2 = 0.7 / 2.2, 1.7 / 3.2 + assert c.iloc[0] == pytest.approx(2.0 * k1 * k2) # oldest segment + assert c.iloc[3] == pytest.approx(3.0 * k2) # middle segment + assert c.iloc[6] == pytest.approx(4.0) # anchor segment, factor 1 + + +def test_ohlc_ordering_preserved(store_env): + from cotdata import get_prices + _write() + p = get_prices("DC", "propadj") + assert (p["High"] >= p["Close"]).all() and (p["Close"] >= p["Low"]).all() + assert (p["High"] >= p["Open"]).all() and (p["Open"] >= p["Low"]).all() + + +def test_falls_back_to_offset_jumps_without_delivery_month(store_env): + """Rolls are still detected from offset steps when Delivery Month is absent.""" + from cotdata import get_prices + _write(with_delivery_month=False) + p = get_prices("DC", "propadj") + assert (p["Close"] > 0).all() + assert p["Close"].iloc[0] == pytest.approx(2.0 * (0.7 / 2.2) * (1.7 / 3.2)) + + +def test_empty_when_either_series_missing(store_env): + from cotdata import store, get_prices + U, _ = _milk_like() + store.write_prices("DC", "unadj", U, source="test") # no backadj written + assert get_prices("DC", "propadj").empty + + +def test_reconstructed_volume_view_still_works_on_propadj(store_env): + """The volume view composes with the derived adjustment.""" + from cotdata import store, get_prices + U, B = _milk_like() + U["Volume_Reconstructed"] = [15, 14, 10, 18, 20, 22, 25, 24, 26] + U["Volume_Source"] = ["reconstructed"] * 9 + store.write_prices("DC", "unadj", U, source="test") + store.write_prices("DC", "backadj", B, source="test") + p = get_prices("DC", "propadj", volume="reconstructed") + assert p["Volume"].tolist() == [15, 14, 10, 18, 20, 22, 25, 24, 26] + assert "Volume_Source" in p.columns