Official implementation for:
SpurAudio: A Benchmark for Studying Shortcut Learning in Few-Shot Audio Classification
Authors: Anonymous (under review)
This repository extends LibFewShot for few-shot audio classification experiments on SpurAudio, with support for:
- IID and OOD episode evaluation.
- Multiple few-shot paradigms (fine-tuning, meta-learning, metric-learning, transductive).
- Backbone and classifier modular configurations.
LibFewShot/
├── SpurAudio_dataset/ # per-class spectrogram tensors (.npy)
│ ├── air_conditioner/
│ ├── blender/
│ ├── cat/
│ ├── dog+dog_bark/
│ ├── … # 38 class folders total
│ └── laughter/
├── Auxiliary/ # normalization stats + class-split definition
│ ├── Clean_Mean_Std.npy
│ ├── Spurious_Mean_Std.npy
│ └── SpurAudio_paper_splits.npy
├── data/
│ └── fewshot/
├── config/ # YAML configs
├── libfewshot_core/ # framework source
└── results/ # training outputs
Each .npy inside a class folder is a spectrogram tensor for one sample. Filenames encode the mixing of two classes:
SpurAudio_dataset/cat/cat-crying_baby_alpha=0.00013126751582603902_loop=1_id=37.npy
│ │ │ │ │
│ │ │ │ └── sample id
│ │ │ └── loop index
│ │ └── mixing coefficient (α) between foreground and background
│ └── background class
└── foreground class
The directory the file lives in is the foreground class (the label); the second class in the filename is the background spurious co-occurrence.
Auxiliary/SpurAudio_paper_splits.npy is a NumPy array (loaded with np.load(..., allow_pickle=True)) whose three top-level entries are the class-name lists for each split:
- index
0→ list of train class names - index
1→ list of validation class names - index
2→ list of test class names
Each entry is a list of strings matching subfolder names under SpurAudio_dataset/ (e.g. "cat", "dog+dog_bark"). To replicate the paper splits, leave this file untouched; to define your own splits, write a 3-element object array in the same [train, val, test] order and point class_per_split in your YAML at it.
In the meantime, you can download the SpurAudio dataset (with its splits) from the anonymous Hugging Face repository: spuraudioNips/SpurAudio-neurips-anonym.
To run the pipeline on it:
- Unpack all audio files into a single flat directory, e.g.
all_audios_unpacked/. - Run the ESC-50 full-stack script on that folder to produce the
SpurAudio_dataset/spectrogram layout shown above.
The SpurAudio spectrograms are produced from ESC-50 by the ESC-50 full-stack script. If you want to use a different audio dataset, run that same script on your dataset to produce a SpurAudio_dataset/-shaped folder — i.e. one subfolder per foreground class, with .npy filenames following the {foreground}-{background}_alpha={float}_loop={int}_id={int}.npy schema shown above. The rest of the pipeline (configs, dataloaders, normalization) expects exactly this format.
- MAML (ICML 2017)
- R2D2 (ICLR 2019)
- LEO (ICLR 2019)
- ANIL (ICLR 2020)
- BOIL (ICLR 2021)
- MeTAL (ICCV 2021)
- ProtoNet (NeurIPS 2017)
- RelationNet (CVPR 2018)
- DN4 (CVPR 2019)
- ATL-Net (IJCAI 2020)
- ADM (IJCAI 2020)
- DeepBDC (CVPR 2022)
- MCL (CVPR 2022)
cd /path/to/LibFewShot
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txtSpurAudio can be constructed manually using the merging scripts, follow the ReadMe in the Mixer directory.
SpurAudio can be downloaded from HuggingFace (link will go public soon)
When downloaded from hugging face, unpack all train, validation and test directories into a single directory in order to work with this repo, the splits npy file will take the nescessary files for the task.
Make sure these assets exist and match your local paths:
SpurAudio_dataset/(or your configured data root).Auxiliary/Spurious_Mean_Std.npy(Should be in the Github repo).Auxiliary/SpurAudio_paper_splits.npy(Should be in the Github repo).
If your machine paths differ, update the relevant YAML entries such as data_root, mean_std_file, and class_per_split.
Configs are under config/ (examples: config/proto_5shot_iid.yaml, config/maml_5shot_ood.yaml).
Run training with:
python run_trainer.py --yaml_path ./config/proto_5shot_iid.yaml # or any other yaml fileFor IID/OOD experiments, set the YAML key ood explicitly:
ood: False→ IID evaluationood: True→ OOD evaluation
Example:
ood: False # IID
# ood: True # OODNote: the key in this codebase is lowercase
ood.
run_test.py currently loads a result directory via the PATH variable inside the script.
- Open
run_test.py. - Set
PATHto your trained run directory underresults/. - Run:
python run_test.py --yaml_path ./config/proto_5shot_iid.yaml # or any other yaml filepython run_trainer.py --yaml_path ./config/proto_5shot_iid.yamlpython run_trainer.py --yaml_path ./config/proto_5shot_ood.yamlThis project is licensed under the MIT License. See LICENSE for details.
LibFewShot is an open-source framework for few-shot learning research. Contributions, feedback, and extensions are welcome.
If you use this repository in your research, please cite the corresponding SpurAudio/LibFewShot paper(s).