Skip to content

Test suite has incorrect assertions — passing tests are asserting wrong expected values #14

@bradsmithmba

Description

@bradsmithmba

Summary

Several tests in tests/backtesting/ pass with green checkmarks but contain provably wrong assertions. These tests do not catch regressions; they provide false confidence. At least two cases confirmed:

Case 1 — Wrong win rate expected value

tests/backtesting/test_performance.py, test_win_rate_calculation:

trades = [100, -50, 200, 150, -75, 300, -25, 180, -100, 250]
expected_win_rate = 0.7   # WRONG

Counting: 6 positive values (100, 200, 150, 300, 180, 250) out of 10 trades = 0.6, not 0.7. The test asserts a value that doesn't match the data it defines. If the implementation ever computes the correct answer (0.6), this test will fail. Currently it passes only if the implementation also has a counting error.

Case 2 — Annualized return assertion in wrong units

tests/backtesting/test_performance.py, test_sharpe_ratio_calculation:

assert annualized_return > 10   # asserts >1000%

An annualized_return of 10 means 1000%. This assertion passes only because annualized_return uses 365.25 as the denominator instead of trading_days_held (see Issue #12), producing inflated nonsense values. If the annualized_return bug is fixed, this test will fail. The intended assertion was almost certainly > 0.10 (10%).

Broader concern

The combination of:

  • Green tests with wrong assertions
  • INTERNALERROR crashes masking other tests
  • conftest.py injecting torch, F, np, pd as builtins

...means the test suite's green/red signals cannot be trusted at face value. Before treating any test pass as a correctness guarantee, each test's expected values should be independently verified against the documented spec.

Recommended fix

  1. Fix test_win_rate_calculation: change expected_win_rate = 0.7 to 0.6
  2. Fix test_sharpe_ratio_calculation: change assert annualized_return > 10 to assert annualized_return > 0.10
  3. Audit remaining backtesting tests for similar oracle errors — particularly any assertion that uses a round number without a derivation comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions