robustness example using vectorized lira

[rickardbrannvall]

Description

Summary of changes

Three notebooks are added to give example of robustness calculations

• B1 notebook trains target and shadow models and stores the logits and masks
• B2 benchmarks the vectorized and iterative versions of the lira attack
• B3 calculates ROC curve confience bands and compares performance for different

The notebooks relies on the vectorised lira update which is merged with this pull request:

• Moved the iterative LiRA implementation into a separate function outside the main mia_class as requested so it can easily be directly imported and used if you have saved logits and masks.
• Added a modularized vectorization of LiRA as a separate function outside the main mia_class as requested so it can easily be directly importedand used if you have saved logits and masks.

The notebooks also imports custom utility functions for bootstrap sampling and for loading cached data

Additional changes

• Minor fix in model_handler.py so that cache directory is no longer hard coded
• Minor fix in shadow_model_handler.py that make logging less verbose
• Clean up of (some) variable names in lira.py, e.g., shadow_inmask preferred over out_indices

Resolved Issues

• The vectorised lira version is orders of magnitude faster. This is necessary for bootstrap sampling.
• Path of the cache directory was previosly hard coded to ./leakpro_output/attack_cache
• Data indices are no longer dumped to log in shadow_model_handler.py

How Has This Been Tested?

The code has been tested several times using varying amounts of shadow models and overall configurations to make sure it matches the original implementations output. It can also be easily tested using the provided example notebook.

Related Pull Requests

• Merges and superseds PR LiRA vectorized and modularized #349 LiRA vectorized and modularized

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB