Skip to content

fix: make three-period ITS test fixtures order-independent#960

Draft
anevolbap wants to merge 1 commit into
mainfrom
issue-959-flaky-persistence-test
Draft

fix: make three-period ITS test fixtures order-independent#960
anevolbap wants to merge 1 commit into
mainfrom
issue-959-flaky-persistence-test

Conversation

@anevolbap

Copy link
Copy Markdown
Collaborator

Summary

Fixes #959.

datetime_data and integer_data in test_three_period_its.py drew from the session-scoped rng fixture, so the generated data depended on how many random draws earlier tests consumed. Any PR that changed test collection (like #851, which removed one parametrize case in an unrelated file) shifted the stream and could flip test_analyze_persistence_sklearn, whose persistence_ratio >= 0 assertion only holds for some draws.

Changes

  • Both fixtures now use their own seeded generator instead of the shared session rng.
  • Dropped the persistence_ratio >= 0 assertions in the pymc and sklearn persistence tests. A negative ratio is a legitimate outcome when the counterfactual sits above the observed post-period.

Testing

The datetime_data and integer_data fixtures drew from the session-scoped
rng fixture, so the generated data depended on how many draws earlier
tests consumed. Any change to test collection order could flip the
persistence_ratio >= 0 assertion, which is also wrong in general: a
negative ratio is a legitimate outcome when the counterfactual sits
above the observed post-period.

Seed both fixtures with their own generator and drop the sign assertion.

Fixes #959
@codecov

codecov Bot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.16%. Comparing base (984163e) to head (c9d7d35).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #960   +/-   ##
=======================================
  Coverage   95.16%   95.16%           
=======================================
  Files          93       93           
  Lines       15030    15030           
  Branches      896      896           
=======================================
  Hits        14304    14304           
  Misses        507      507           
  Partials      219      219           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anevolbap anevolbap self-assigned this Jun 10, 2026
@anevolbap anevolbap marked this pull request as draft June 10, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_analyze_persistence_sklearn is order-dependent through the session-scoped rng fixture

1 participant