I'm trying to use the lib for the first time, for a sequential test. I'm getting an exception "ValueError: Information ratio must be monotonically increasing", but I don't know what it means. I've tried digesting the code, but this is buried more levels deep than I understand. Can you educate me?
Here's the snipped exception:
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py:135](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py#line=134), in compute_sequential_adjusted_alpha.<locals>.adjusted_alphas_for_group(grp)
133 def adjusted_alphas_for_group(grp: DataFrame) -> Series:
134 return (
--> 135 sequential_bounds(
136 t=grp["sample_size_proportions"].values,
137 alpha=grp[ALPHA].values[0] [/](https://escott-dsc3099.dev.de.gcp.rokulabs.net/) n_comparisons,
138 sides=2 if (grp[PREFERENCE_TEST] == TWO_SIDED).all() else 1,
139 )
140 .df.set_index(grp.index)
141 .assign(
142 **{
143 ADJUSTED_ALPHA: lambda df: df.apply(
144 lambda row: 2 * (1 - st.norm.cdf(row["zb"]))
145 if (grp[PREFERENCE_TEST] == TWO_SIDED).all()
146 else 1 - st.norm.cdf(row["zb"]),
147 axis=1,
148 )
149 }
150 )
151 )[["zb", ADJUSTED_ALPHA]]
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py:53](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py#line=52), in sequential_bounds(t, alpha, sides, state)
52 def sequential_bounds(t: np.array, alpha: float, sides: int, state: DataFrame = None):
---> 53 return bounds(t, alpha, rho=2, ztrun=8, sides=sides, max_nints=1000, state=state)
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py:343](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py#line=342), in bounds(t, alpha, rho, ztrun, sides, state, max_nints)
341 raise ValueError(f"Information ratio must must not be zero, {get_input_str()}")
342 if any(t[i] > t[i + 1] for i in range(len(t) - 1)):
--> 343 raise ValueError(f"Information ratio must be monotonically increasing, {get_input_str()}")
344 if not (sides == 1 or sides == 2):
345 raise ValueError(f"sides must either be one a zero, {get_input_str()}")
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py:337](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py#line=336), in bounds.<locals>.get_input_str()
334 def get_input_str():
335 return (
336 f"input params: t={t}, alpha={alpha}, sides={sides}, rho={rho}, ztrun={ztrun},"
--> 337 f"state_df={state.df.to_json()}, state_fcab={state.last_fcab}, max_nints={max_nints}"
338 )
AttributeError: 'NoneType' object has no attribute 'df'
And here's the setup I did, trying to follow the example notebook:
test = conf.ZTest(
metrics_df,
numerator_column='metric1',
numerator_sum_squares_column='metric1_sumsq',
denominator_column='user_cnt',
categorical_group_columns=['bucket'],
ordinal_group_column='date'
)
# This call goes boom!
result = test.multiple_difference(
level='control',
groupby=['date'],
level_as_reference=True,
final_expected_sample_size_column='expected_users',
non_inferiority_margins = (None, 'increase'),
absolute=False
)
I'm trying to use the lib for the first time, for a sequential test. I'm getting an exception "ValueError: Information ratio must be monotonically increasing", but I don't know what it means. I've tried digesting the code, but this is buried more levels deep than I understand. Can you educate me?
Here's the snipped exception:
And here's the setup I did, trying to follow the example notebook: