feat(leakfree): auto-resolve CPCV/horizon embargo to the label horizon (L4488a)#230
Merged
Conversation
…n (L4488a) WF_EMBARGO_DAYS defaulted to 0, which left the CPCV after-test boundary open: the excluded zone around a test group [a,b] is [a-forward_days, b+embargo_days], so with embargo=0 train rows at b+1..b+forward_days leak via OVERLAPPING LABELS with the test's late obs. That inflated every leak-free read, and MORE at longer horizons (60d needs a ~60-row embargo vs ~21 for 21d) -- a real confound for the 60d-vs-21d horizon call. The raise was bundled into W1.4, which challenger-first superseded (CF4), so it got orphaned. Fix: WF_EMBARGO_DAYS default None = auto = the label horizon (the correct overlapping-label embargo that makes the CPCV exclusion symmetric). Resolved inside leakfree_meta_oos_ic + cpcv_meta_oos_ic + expanding_wf_folds (None -> forward_days); the horizon curve inherits per-horizon h since it passes forward_days=h through. Explicit int (incl. 0) still overrides. Prod predictor .yaml does not set embargo_days -> auto applies; sample yaml updated to document the auto behavior (was a misleading explicit 0). Decouples the embargo fix from the dead W1.4. First item of the model-rotation scaffolding arc (L4488) -- scoring-honesty prerequisite for the leaderboard + the horizon decision. Tests: +1 (None auto-resolves to horizon; explicit 0 honored; cpcv too). Suite 1362 -> 1363. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
L4488a — first item of the model-rotation scaffolding arc (L4488); scoring-honesty prerequisite.
The bug
WF_EMBARGO_DAYSdefaulted to 0, leaving the CPCV after-test boundary open. The excluded zone around a test group[a,b]is[a − forward_days, b + embargo_days](leakfree_meta_ic.py:53), so withembargo=0train rows atb+1 … b+forward_daysleak via overlapping labels with the test's late observations. That inflated every leak-free read — and more at longer horizons (60d needs a ~60-row embargo vs ~21 for 21d), making it a real confound for the 60d-vs-21d horizon call. The planned raise was bundled into W1.4, which challenger-first superseded (CF4), so it got orphaned.The fix
WF_EMBARGO_DAYSdefault →None= auto = the label horizon (the correct overlapping-label embargo; makes the CPCV exclusion symmetric). Resolved insideleakfree_meta_oos_ic+cpcv_meta_oos_ic+expanding_wf_folds(None → forward_days); the horizon curve inherits per-horizonhautomatically since it passesforward_days=hthrough. An explicit int (incl.0) still overrides.predictor.yamldoesn't setembargo_days→ auto applies in production.predictor.sample.yamlupdated to document the auto behavior (was a misleading explicit0).forward_days, not the plan's old~0.01·T(that's only the residual serial-correlation buffer, undersized here).Effect
The leaderboard's leak-free CPCV/horizon reads become honest — the prerequisite for L4488b (net-of-cost scoring) and L4488d (the horizon call). The current ~0.18 CPCV is expected to drop somewhat (the honest number), and the 60d>21d gap will shrink once both are properly embargoed.
Tests
+1 (
Noneauto-resolves to horizon; explicit0honored; cpcv too). Suite 1362→1363.