embedding_dimension_limit

Overview

This repository contains my reproduction of the paper On the Theoretical Limitations of Embedding-Based Retrieval, completed as the final Information Retrieval assignment for the first year of the Data Analysis and Modelling master's program.

The results were presented at a course seminar and at a university ML research group seminar.

Structure

Notebooks: Located in ./notebooks. Each notebook has its own dependencies, installable via:

pip install -r ./notebooks/<notebook_name>.requirements.txt

Python Scripts:

• bm25_benchmark.py: BM25 benchmark; requires a Milvus Lite local instance.

• colbert_benchmark.py: lightonai/GTE-ModernColBERT-v1 benchmark; requires pylate for index creation.

• Install script dependencies via:

  uv sync

Usage

The accompanying presentation.pdf presentation explains the results, the theory behind the paper, and the reproduced experiments. Additional documentation and examples will be added here soon.

Results Overview

View details

The core experiments were reproduced on smaller configurations and exhibited similarities to the results reported in the paper.

Fine-tuned models from the domain shift experiment are not released due to instability.

Experimental Setup

View details

Experiments were conducted on:

• NVIDIA T4 and P100 GPUs (Kaggle)
• NVIDIA GeForce GTX 1650 (local machine)

Models

View details

Due to hardware constraints, larger models in the 7-8B parameter range were quantized to 4-bit precision. These quantized variants were created specifically for this project and are available on Hugging Face:

• intfloat/e5-mistral-7b-instruct (4-bit quantized)
• GritLM/GritLM-7B (4-bit quantized)
• Qwen/Qwen3-Embedding-8B (4-bit quantized)

Datasets

View details

Since the original qrel datasets were not publicly available and the full 50K-document setup was infeasible, smaller datasets were constructed by instantiating qrel binary patterns with natural language while preserving the name and attribute distributions of the LIMIT dataset. The missing train split was regenerated using LLM-based iterative prompting.

Available datasets:

• LIMIT-small-dense - Dense binary relevance: each query has multiple overlapping relevant documents.
• LIMIT-small-disjoint - Disjoint relevance: queries have non-overlapping relevant documents.
• LIMIT-small-cyclic - Cyclic relevance pattern: relevance relationships form a cycle across queries.
• LIMIT-small-random - Random relevance assignments: queries have randomly selected relevant documents.
• LIMIT-train - Self-produced train split of the LIMIT dataset.