Skip to content

Implement start_date in databento.fetch_daily_ohlc (was silently dead)#30

Merged
mspinola merged 2 commits into
mainfrom
fix/databento-fetch-daily-ohlc-start-date
Jul 23, 2026
Merged

Implement start_date in databento.fetch_daily_ohlc (was silently dead)#30
mspinola merged 2 commits into
mainfrom
fix/databento-fetch-daily-ohlc-start-date

Conversation

@mspinola

@mspinola mspinola commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Summary

  • fetch_daily_ohlc's start_date parameter was never used on any code path: a cold cache always backfilled from 2000-01-01 (clamped to the GLBX.MDP3 floor 2010-06-06), and a warm cache always resumed from last_date + 1 day, regardless of what was passed. A caller trying to bound a fetch got a silent full-history pull at full Databento API cost instead.
  • Implemented properly (revised after review -- the first version of this PR removed the parameter; that just relocates the problem to the next caller who wants a bounded fetch, and the mock-client test infrastructure to verify it correctly already exists in this file):
    • Cold cache (symbol's first-ever fetch): start_date now narrows the fetch floor (still clamped to the GLBX.MDP3 floor), so a narrow first-time query is actually cheap -- this is the real fix for the cost bug.
    • Warm cache (symbol already has history): start_date does not narrow the incremental fetch, which always resumes from the cache's own last_date + 1. The cache is a single shared, from-inception series across every caller; letting a later, narrower start_date shrink what gets fetched would silently truncate a cache other callers rely on being complete. start_date still filters what's returned in this case, just not what's fetched or persisted to disk.
  • Checked every caller in the workspace before starting: the function's only real internal caller (update_all_daily_prices) never passes start_date, neither existing test passes it, and the other fetch_daily_ohlc in the tree (npf.ml.labels) is an unrelated function that correctly forwards its own start_date into cotdata.get_prices -- not this one.

Found while building a bounded, one-off Databento divergence-check script in a sibling repo (livebook), which had to write its own bounded fetch specifically because this function couldn't do it.

Test plan

  • pytest tests/test_databento_provider.py -- 8 passed (2 existing + 6 new covering: cold-cache floor narrowing, GLBX-floor clamping still applies, return-value filtering, the None default is unchanged, and the warm-cache case -- pre-seeded 3-day cache, narrower start_date requested, asserts the fetch call still starts from last_date+1 while the returned frame excludes the pre-start row and the on-disk cache keeps the full series)
  • pytest tests/ (full suite, COTDATA_STORE set) -- 86 passed
  • Confirmed the 5 test_norgate_provider.py failures seen without COTDATA_STORE set are pre-existing and unrelated (reproduced identically with this change git stashed out)
  • grep -rn "fetch_daily_ohlc(" across the workspace -- no current caller passes start_date to this function, so this is purely additive

🤖 Generated with Claude Code

mspinola and others added 2 commits July 22, 2026 21:58
Every code path ignored it: a cold cache always backfilled from 2000-01-01
(clamped to the GLBX.MDP3 floor 2010-06-06), and a warm cache always resumed
from last_date+1, regardless of what was passed. A caller trying to bound a
fetch (e.g. 'just the last 3 months') silently got a full-history pull instead,
at full Databento API cost, with no error or warning.

Checked every caller before deciding how to fix it. grep across the workspace
shows no one relies on it working: the function's only real internal caller
(update_all_daily_prices) never passes it, both tests never pass it, and every
OTHER fetch_daily_ohlc in the tree (npf.ml.labels) is an unrelated function that
correctly forwards its own start_date into cotdata.get_prices — not this one.

Removed rather than wired in. An explicit TypeError for a future caller beats
another silent full-history surprise, and there is no current caller to write a
correctness test against if the parameter were implemented instead. A genuinely
bounded fetch, if ever needed, should be a new function with its own test
proving the bound is honored.

Found while building a bounded, one-off Databento divergence-check script in
livebook, which deliberately avoided reusing this function for exactly this
reason. 81 tests pass (5 apparent failures in test_norgate_provider.py are
pre-existing and environment-dependent — COTDATA_STORE unset — confirmed via
git stash, unrelated to this change).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correctly pointed out that removal just relocates the problem — a future caller
(including my own livebook divergence-check script, which had to write a
separate bounded fetch specifically because this function couldn't do it) still
needs a bounded fetch, and the test infrastructure to verify it correctly
already exists in this file (the mock Historical client pattern).

Cold cache (symbol's first-ever fetch): start_date now narrows the fetch floor
(still clamped to the GLBX.MDP3 2010-06-06 floor), so a narrow first-time query
is actually cheap — this is the real fix for the original API-cost bug.

Warm cache (symbol already has history): start_date deliberately does NOT
narrow the incremental fetch, which always resumes from the cache's own
last_date+1. The cache is a single shared, from-inception series across every
caller; letting a later, narrower start_date shrink what gets fetched would
silently truncate a cache other callers rely on being complete. start_date
still filters what's RETURNED in this case, just not what's fetched or
persisted to disk.

Both branches are tested against the mock Historical client, including the
warm-cache case specifically (pre-seeded 3-day cache, narrower start_date
requested, asserts the fetch call still starts from last_date+1 while the
returned frame excludes the pre-start row AND the on-disk cache keeps the full
series). 6 new tests, 8/8 in this file, 86/86 full suite (COTDATA_STORE set).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mspinola mspinola changed the title Remove dead start_date param from databento.fetch_daily_ohlc Implement start_date in databento.fetch_daily_ohlc (was silently dead) Jul 23, 2026
@mspinola
mspinola merged commit 458a093 into main Jul 23, 2026
5 checks passed
@mspinola
mspinola deleted the fix/databento-fetch-daily-ohlc-start-date branch July 23, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant