Skip to content

fix brittle test loo by modifying test dataset - #110

Merged
aloctavodia merged 1 commit into
bambinos:mainfrom
Cab14bacc:fix/brittle_test_loo
Jun 5, 2026
Merged

fix brittle test loo by modifying test dataset#110
aloctavodia merged 1 commit into
bambinos:mainfrom
Cab14bacc:fix/brittle_test_loo

Conversation

@Cab14bacc

@Cab14bacc Cab14bacc commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

closes #103

The original my_data dataset:
image

We modify by making y the clearly better predictor:

 data["z"] = data["y"] + 0.1 * np.random.normal(size=len(data))
image

Tested with this:

NUM_DRAWS, NUM_CHAINS = 100, 4

df_cpy = bmb.load_data("my_data") 
df_cpy["z"] = 2.0 * df_cpy["y"] + np.random.normal(0, scale=0.1, size=len(df_cpy))
data = df_cpy.copy()

result = []
model = bmb.Model("z ~ x + y", data)  

for i in range(100):
    idata = model.fit(
        draws=NUM_DRAWS, chains=NUM_CHAINS, random_seed=i, idata_kwargs={"log_likelihood": True},
        progressbar=False
    )

    ppi = kpt.ProjectionPredictive(model, idata)

    ppi.project()
    cmp_df = ppi.compare()
    result.append(all(cmp_df.index == ["reference", "x", "y", "Intercept"]))
    print("Iteration", i, ":", cmp_df.index)

@aloctavodia
aloctavodia merged commit 81be125 into bambinos:main Jun 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_loo is brittle

2 participants