Code for "Efficient Re-ranking with Cross-encoders via Early Exit"

This repository contains the code and materials for our work presented at the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '25), titled "Efficient Re-ranking with Cross-encoders via Early Exit".

Environment

All the experiments were conducted using a docker container built in the following way:

docker build --build-arg UID=$(id -u) --build-arg UNAME=$(id -un) -t see:v0.1 - < Dockerfile

and run with the following command:

docker run  --runtime nvidia  \
            --user $(id -un) \
            --gpus '"device=0"' \
            --name see \
            --rm \
            -i \
            -v .:/code/ \
            -v ~/.ir_datasets:/ir_datasets/ \
            -v ~/.cache/huggingface:/hf/  \
            -v ~/.cache/torch:/home/$(id -un)/.cache/torch  \
            -v ../see-data:/data  \
            --shm-size 100G \
            -t see:v0.1 bash

Please not that four different volumes are mounted in the container:

• -v .:/code/ mounts the current directory to /code in the container. This is where the code is located.
• -v ~/.ir_datasets:/ir_datasets/ mounts the local directory ~/.ir_datasets to /ir_datasets in the container. This is where the datasets are stored.
• -v ~/.cache/huggingface:/hf/ mounts the local directory ~/.cache/huggingface to /hf in the container. This is where the Hugging Face cache is stored.
• -v ~/.cache/torch:/home/$(id -un)/.cache/torch mounts the local directory ~/.cache/torch to /home/$(id -un)/.cache/torch in the container. This is where the PyTorch cache is stored.
• -v ../see-data:/data mounts the local directory ../see-data to /data in the container. This is where the data is stored.

Running the experiments

In a brief, to recompute the results of the paper, you should run the following commands:

./rq1.sh  /data/rq1 <batch_size>
./rq2.sh  
./rq3_run_all.sh
python src/04_bm25_beir_runs.py
./rq4_run_all.sh
./r5_run_all.sh

where <batch_size> is the batch size used in the experiments, <device> is the device used in the experiments using the torch string format, and <base_dir_eemb_data> is the directory where the models generate by eemb are stored.

The above scripts produce the following results:

• rq1.sh produces the results for Section 4.2 of the related paper and stores the results in the directory ../see-data/rq1.
• rq2.sh produces the results for Section 4.3 of the related paper and stores the results in the directory ../see-data/rq2.
• rq3_run_all.sh produces the results for the in-domain comparison (see paragraph In-domain Evaluation) and stores the results in the directory ../see-data/rq3.
• python src/04_bm25_beir_runs.py produces the results for the BM25 runs using bm25s and stores the results in ../see-data/beir-bm25-runs.
• rq4_run_all.sh produces the results for the out-of-domain comparison and different truncation levels (see paragraph Out-of-domain Evaluation and Early Exit and Ranked List Truncation.) and stores the results in the directory ../see-data/rq4.
• r5_run_all.sh produces the results for Section 4.5 and stores the results in the directory ../see-data/rq5.

Creating the figures and tables

The figures and tables of the paper can be generated by running the jupyter notebook replicate-paper.ipynb.

Models and data

The models are available using the transformers library, in particular the models used in this paper are:

• veneres/monobert
• veneres/monobert-asnq
• veneres/monoelectra-msmarco
• cross-encoder/ms-marco-MiniLM-L-12-v2
• naver/trecdl22-crossencoder-electra

All the other models and CSV files are instead available at this link.

Citation License

Besides the license of the code, this code is also subject to a citation license if used in other research projects:

By downloading and using this software, you agree to cite the under-noted paper in any kind of material you produce where it was used to conduct a search or experimentation, whether be it a research paper, dissertation, article, poster, presentation, or documentation. By using this software, you have agreed to the citation license.

SIGIR 2025

@inproceedings{busolin_2025_efficient,
  author    = {Busolin, Francesco and Lucchese, Claudio and Nardini, Franco Maria and Orlando, Salvatore and Perego, Raffaele and Trani, Salvatore and Veneri, Alberto},
  title     = {Efficient Re-ranking with Cross-encoders via Early Exit},
  booktitle = {Proceedings of the 48th International {ACM} {SIGIR} {C}onference on Research and Development in Information Retrieval ({SIGIR})},
  pages     = {N/A},
  publisher = {{ACM}},
  year      = {2025},
  url       = {https://doi.org/10.1145/3726302.3729962},
  doi       = {10.1145/3726302.3729962}
}