QuantumBench Knowledge Distillation

QuantumBench is a focused experiment for distilling quantum information science knowledge from a larger teacher language model into a smaller student model.

Dataset (Updated May 22, 2026)

Complete dataset: data/train/quantumbench_full.jsonl (107 examples) + data/eval/quantumbench_full.jsonl (26 examples)

• Total: 133 expert-authored Q&A pairs on quantum physics and quantum information science
• Source: Kaggle Measuring AGI hackathon submission — executive function track
• Split: 80% train (107), 20% eval (26) — stratified by difficulty tier
• Difficulty tiers: 40 foundational, 63 intermediate, 30 graduate
• Format: JSONL with instruction (question with prefix) and response (expert answer)
• Characteristics: Zero newlines in all questions and answers (flat strings); answers range from 144–2813 characters; graduate questions integrate 4 sub-requirements from different quantum information sub-domains

The canonical project lane is:

• Teacher: models/teacher-model
• Teacher base: models/gpt2-large
• Student base: models/gpt2
• Student output: models/student-gpt2-distilled
• Train data: data/train/quantumbench_full.jsonl (new)
• Eval data: data/eval/quantumbench_full.jsonl (new)

The current supervised recovery lane adds:

• Curriculum builder: scripts/build_quantumbench_curriculum.py
• Canonical supervised data: data/train/quantumbench_train_canonical.jsonl
• Stage 1 curriculum: data/train/quantumbench_train_curriculum_stage1.jsonl
• Stage 2 curriculum: data/train/quantumbench_train_curriculum_stage2.jsonl
• Semantic eval companion: data/eval/quantumbench_eval_semantic.jsonl
• Semantic rubric evaluator: scripts/semantic_eval_quantumbench.py

Goal

Train a compact GPT-2 student to preserve as much quantum information science behavior as possible from a GPT-2 Large LoRA teacher, then measure whether the transfer holds on held-out quantum questions.

Models

Teacher

models/teacher-model

• Architecture: GPT-2 Large causal language model
• Base: models/gpt2-large
• Adaptation: PEFT LoRA
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target module: c_attn
• Role: quantum knowledge source for distillation

Student

models/student-gpt2-distilled

• Architecture: GPT-2 causal language model
• Base: models/gpt2
• Approximate base parameters: 124M
• Role: distilled compact quantum model

The student path is a target output. It should not be treated as valid until a clean dry run, full training run, and held-out evaluation are documented.

Data

Each dataset line is:

{"instruction": "...", "response": "..."}

• data/train/quantumbench_train.jsonl: training and distillation data
• data/eval/quantumbench_eval.jsonl: held-out quantum information science evaluation data

Do not train on the eval file.

Quick Start

Set up the environment:

bash setup.sh
source venv/bin/activate

Check the repo and dependencies:

python scripts/system_check.py
python scripts/test_dataset.py --data-path data/train/quantumbench_train.jsonl
python scripts/build_quantumbench_curriculum.py
python scripts/test_dataset.py --data-path data/train/quantumbench_train_canonical.jsonl

Run a short distillation dry run:

python scripts/distill_quantumbench.py \
  --teacher-model models/teacher-model \
  --student-model-name gpt2 \
  --data-path data/train/quantumbench_train.jsonl \
  --output-dir models/student-gpt2-distilled \
  --epochs 1 \
  --batch-size 2 \
  --learning-rate 1e-5 \
  --temperature 1.5 \
  --alpha 0.1 \
  --max-length 256 \
  --dry-run

Run the canonical full distillation:

python scripts/distill_quantumbench.py \
  --teacher-model models/teacher-model \
  --student-model-name gpt2 \
  --data-path data/train/quantumbench_train.jsonl \
  --output-dir models/student-gpt2-distilled \
  --epochs 5 \
  --batch-size 2 \
  --learning-rate 1e-5 \
  --temperature 1.5 \
  --alpha 0.1 \
  --max-length 256

Evaluate teacher and student:

python scripts/evaluate_quantumbench.py \
  --teacher-model models/teacher-model \
  --student-model models/student-gpt2-distilled \
  --data-path data/eval/quantumbench_eval.jsonl \
  --max-new-tokens 256 \
  --output-predictions results/predictions/teacher_student_eval.json

Compare distillation behavior:

python scripts/compare_teacher_student.py \
  --teacher-model models/teacher-model \
  --student-model models/student-gpt2-distilled \
  --data-path data/eval/quantumbench_eval.jsonl \
  --output results/metrics/teacher_student_comparison.json

Run a qualitative generation gate:

python scripts/generation_quality_gate.py \
  --model-path models/student-gpt2-supervised-sanity \
  --data-path data/eval/quantumbench_eval.jsonl \
  --output results/predictions/sanity_quality_gate_eval.json \
  --sample-size 5

Run semantic rubric evaluation:

python scripts/semantic_eval_quantumbench.py \
  --model-path models/student-gpt2-supervised-v2 \
  --data-path data/eval/quantumbench_eval_semantic.jsonl \
  --output results/predictions/student_supervised_v2_semantic_eval.json

Documentation

• PROJECT_STATUS.md: current project state and next actions
• docs/TRAINING_PROGRESS.md: chronological training runs
• docs/EVALUATION_PROTOCOL.md: quantum knowledge evaluation rubric
• docs/MODEL_CARD_TEACHER.md: canonical teacher details
• docs/MODEL_CARD_STUDENT.md: target student details
• docs/EXPERIMENT_LOG.md: canonical vs archived artifacts

Repository Shape

data/       train/eval JSONL files
configs/    canonical distillation and evaluation settings
scripts/    focused training, evaluation, comparison, and visualization tools
docs/       current documentation
results/    generated metrics, predictions, plots, and runs
models/     canonical base models, teacher adapter, and distilled student target
archive/    stale docs, logs, scripts, and experimental model variants

Current Caveat

Earlier student runs were exploratory and showed instability, including NaN evaluation loss and repetition-related fixes. The canonical student should be regenerated and evaluated under the protocol in docs/EVALUATION_PROTOCOL.md.