Tabular Data Contamination in LLMs

Evaluation framework to measure whether LLMs encode latent knowledge of tabular datasets through two complementary analyses:

• Contamination: Can the model recover or recognize original rows from multiple-choice probes?
• Memorization: can the model reconstruct dataset structure/content (headers, rows, features, values).

Repository Structure

• src/: core evaluation pipelines.
  • src/contamination.py: row-level contamination benchmark runner.
  • src/memorization.py: memorization benchmark runner.
  • src/baseline.py, src/plots.py: baseline and analysis/visualization utilities.
  • src/utils/: model adapters, mappings, and helper functions.
• data/: input datasets and contamination probes.
• results/: generated outputs from contamination/memorization runs.
• script/: batch orchestration shell scripts for large experiment grids.
• memorization_query.py: single-test entrypoint using tabmemcheck APIs.
• contamination_query.py: prompt-and-parse sandbox for contamination probes.

contamination_query.py (How to use)

contamination_query.py is a local probe playground for contamination prompts:

• builds the multiple-choice prompt format used in contamination evaluation,
• sends one query to the selected backend,
• extracts the final option index from [FINAL_ANSWER_start] ... [FINAL_ANSWER_end].

Use it to quickly validate prompt behavior and output parsing before full benchmark runs.

What it does in practice:

• loads one probe record (question, options, answer_index) from a contamination .jsonl file;
• formats options as numbered candidates exactly like the benchmark prompt style;
• calls the selected backend (local vLLM model or Groq endpoint, based on model mapping);
• prints the prompt, raw model response, parsed choice index, and expected answer for quick sanity checks.

Run:

python contamination_query.py

Notes:

• model selection is currently set in-file via model_name;
• it reads an example probe from data/contamination_probes/...;
• you can change probes_path to test a specific dataset variant (real, like, obfuscated, swapped) and coverage level.

memorization_query.py (How to use)

memorization_query.py is a targeted debug/inspection runner for Tabular Memorization Checker tests.
It configures the LLM backend (Gemini, Groq, or vLLM), loads one dataset CSV from data/<dataset>/<dataset>.csv, and runs one selected memorization test.

Example:

python memorization_query.py \
  --dataset adult \
  --test row_completion \
  --model-name gemini-2.5-flash-lite \
  --num-queries 25 \
  --temperature 0.0

Common tests: header, row_completion, feature_completion, first_token, feature_names, feature_values, sample, dataset_name, all.

script/ Directory (Batch execution)

The script/ folder contains reproducible batch launchers:

• script/contamination.sh: runs src/contamination.py across model, dataset, temperature, and coverage grids (with optional dataset-tag filtering).
• script/memorization.sh: runs src/memorization.py across model, dataset, and temperature grids with selected memorization tests.

Run from repository root:

bash script/contamination.sh
bash script/memorization.sh

Both scripts:

• print structured progress logs,
• optionally activate conda env llm if available,
• exit non-zero when one or more runs fail,
• let you select models/datasets by uncommenting or commenting entries in the MODELS and DATASETS arrays.

Dataset variant selection (contamination pipeline):

• script/contamination.sh accepts dataset variants as positional arguments.
• if no variants are passed, it uses all defaults: real like obfuscated swapped.

Example (evaluate only real and like):

bash script/contamination.sh real like