Bias Measurement in LLM2Vec embeddings

Abstract

This repository contains the source code for the diploma thesis: "Μέτρηση προκατάληψης σε LLM2Vec αναπαραστάσεις" (Bias Measurement in LLM2Vec embeddings).

The research explores the application of LLM2Vec to transform Causal Language Models (Decoder-only LLMs) like LLaMA, Mistral, and Sheared-LLaMA into powerful text encoders. The primary objective is to evaluate how this transformation—and its specific training stages (MNTP, Unsupervised SimCSE, Supervised SimCSE)—affects the geometric structure and encoding of gender bias. By transitioning from traditional static models (Word2Vec) and early contextual models (BERT) to modern architectures, the thesis demonstrates that LLM2Vec enables selective encoding. These modern embeddings overcome structural limitations by learning to separate societal stereotypes from semantic utility, applying gender rationally based on context rather than polarizing neutral concepts.

Key Findings

1. The Principle of Selective Encoding: Unlike static models (Word2Vec) that blindly polarize neutral professions, LLM2Vec models achieve optimal selective encoding. They successfully place neutral professions at the center of the geometric space while preserving a robust gender dimension only where semantically necessary (e.g., Patriarch, Firewoman).
2. Supervised SimCSE as the Optimal Strategy: The Supervised SimCSE training stage systematically achieves the greatest reduction in Direct Bias for neutral professions while maintaining clear semantic separation for inherently gendered words, outperforming both MNTP and Unsupervised approaches.
3. Robustness of PCA over Single Pairs: Traditional bias metrics relying on a single definitional pair (e.g., he-she) are unstable and insufficient for contextual models. Distributional methods—specifically Principal Component Analysis (PCA) and Mean Difference—are highly robust and exhibit a high degree of convergence, with PCA explaining over 55% of the variance in the Supervised stage.
4. The Scaling Paradox & Pruning as Denoising: Increasing model parameters provides capacity but does not automatically reduce bias; training data and alignment are far more critical (e.g., Llama-3.1 presents a much more balanced space). Interestingly, model pruning (as seen in the 1.3B Sheared-LLaMA) acts as a drastic denoising mechanism, eliminating weak stereotypical parameters while preserving strong semantic gender structures.
5. Systematic Asymmetry (The "Female" Deviation): Across all evaluated architectures, there is a consistent geometric asymmetry: representations of the female gender systematically record higher Direct and Directional Bias compared to the male gender, meaning female terms deviate further from the neutral semantic axis.

Repository Structure

• src/: Core Python modules and tools.
  • llm2vec/: Execution notebooks utilizing LLM2Vec (encode methodology) for sentences and words.
  • w2v/: Baseline execution notebooks utilizing the traditional Word2Vec model.
  • bert/: Baseline execution using the Masked Language Model BERT.
  • tools/: Utility scripts for calculating bias, plotting results, and manipulating models (e.g., get_llm2vec_model.py, find_gender_axis.py).
• datasets/: The datasets containing protected attributes (gender seeds) and lists of neutral/gender-specific professions.
• models/: Directory for caching downloaded transformer models (optional/local).
• Results/: Generated logs, tables, and visualization plots.

Installation

Ensure you have Python 3.8+ installed. You can install all necessary dependencies using the provided requirements.txt:

pip install -r requirements.txt

Make sure your environment is configured for PyTorch with CUDA support if you intend to run the larger 7B/8B models efficiently.

Usage

You can reproduce the experimental findings by running the Jupyter notebooks located in the src/ subdirectories.

For instance, to run the LLM2Vec encoding pipeline on contextual sentences:

jupyter notebook src/llm2vec/llm2vec_run_all_models_sentences.ipynb

To run baseline Word2Vec experiments:

jupyter notebook src/w2v/w2v_run_sents.ipynb

Citation

If you find this work useful, please refer to the thesis:

@mastersthesis{Mylonakis2026,
  author  = {Alexandros Mylonakis},
  title   = {Bias Measurement in LLM2Vec embeddings},
  school  = {University of Ioannina, Department of Computer Science and Engineering},
  year    = {2026},
  type    = {Diploma Thesis}
}