BERT Bi-Encoder Depth Ablation: Seeds, Training Budget, and Zero-Shot Transfer

Finding: Across five controlled experiments on nfcorpus, scifact, fiqa:

1. At a moderate training budget (20K steps on 500K MS MARCO triples), BERT bi-encoder depth from L3 to L12 is invariant within seed noise (three seeds; L12 σ = 0.0069 vs L3/L6/L9 σ ≤ 0.001).
2. Extended training (80K steps, 4× longer) degrades zero-shot BEIR transfer for every depth, with the degradation concentrated on fiqa (finance): L12 loses 45% of fiqa NDCG@10, L3/L6 lose ~28%. This is DRAGON-consistent: single-source overfitting hurts, and more capacity overfits harder.
3. L6 is the single most robust operating point across both training budgets — tied-best at 20K and clearly best at 80K, while encoding ~20% faster than L12.

See report.md for the full analysis and RESULTS.md for every metric.

The five experiments

Experiment	Batch	Steps	Seed	Purpose
A	256	20K	42	Large-batch baseline
B	32	20K	42	Standard batch-32 run
C	32	20K	43	Seed variance
D	32	20K	44	Seed variance
E	32	80K	42	Training saturation

Headline numbers

Mean NDCG@10 across the three BEIR benchmarks:

variant	20K seed avg (B,C,D)	±2σ	80K (E)	p50 latency (canonical)
L12	0.2400	±0.0138	0.2143	16.2 ms
L9	0.2385	±0.0006	0.2109	15.8 ms
L6	0.2438	±0.0020	0.2254	13.0 ms
L3	0.2422	±0.0018	0.2203	12.7 ms

At 20K the mean NDCG@10 spread across depths is 0.0053 — smaller than L12's single-seed variance. At 80K, L6 opens a 0.005 lead over L3 and 0.011 over L12.

What this does not claim

• Not a claim about BERT bi-encoders in general. The claim is bounded to this recipe (MultipleNegativesRankingLoss, 500K MS MARCO triples, top-N layer truncation) and the three BEIR tasks evaluated.
• Not a claim that 80K training always hurts. It hurts zero-shot BEIR transfer from MS MARCO; it might help on in-domain MS MARCO dev (not measured here).
• Not a claim that other ablation strategies (bottom-drop, alternating-drop, structured pruning) would agree. Only top-N truncation was tested.
• Not a claim that the precise 80K degradation magnitudes would hold across seeds — the 80K run is single-seed. The effect size is large enough that seed noise is unlikely to flip the direction, but seed replication would tighten the bound.

See "Limitations" and "Open Questions" in report.md for the full list.

Figures

Headline: depth × training budget. Seed-averaged NDCG@10 at 20K (±2σ band) with the 80K single-seed line overlaid. Every depth loses at 80K; L6 is the most robust operating point across both budgets.

Where the 80K degradation lives. Per-benchmark NDCG@10 change from 20K (seed-avg) to 80K. nfcorpus and scifact are roughly flat-to-positive across depths; fiqa crashes for every depth, worst for L12 (−45%).

Seed variance is depth-dependent. Per-seed NDCG@10 at 20K for each depth. L12's σ is 7–23× larger than L3/L6/L9 — a fingerprint of undertraining at deep capacity, not a real depth signal.

Pareto: accuracy vs. latency. Seed-averaged 20K points and single-seed 80K points on the canonical batch=32 latency ladder. L6 sits on the Pareto frontier at both budgets.

A note on latency profiling

Of five latency profiling runs on the same H100 machine, only two (Experiments B and E) produced a physically coherent p50 ladder: p50 ≈ 12.3 + 0.33 × n_layers. These two runs reproduce each other to the decimal. The other three profiling runs (Experiments A, C, D) under-report shallow-model p50 by 2–5× for reasons we could not isolate (likely candidates: CUDA kernel autotuner state, per-process kernel cache warm-up, Python timing noise at sub-10-ms encodes). we report the canonical B/E ladder in all headline tables and flag the anomalous runs in RESULTS.md. The retrieval finding is architecture-determined and latency-independent.

Reproduce

# Experiment A — batch=256, seed=42, 20K steps
bash scripts/run_all.sh --config config.yaml

# Experiment B — batch=32, seed=42, 20K steps
bash scripts/run_b32.sh --config config_b32.yaml

# Experiment C — batch=32, seed=43
bash scripts/run_b32.sh --config config_b32_s43.yaml

# Experiment D — batch=32, seed=44
bash scripts/run_b32.sh --config config_b32_s44.yaml

# Experiment E — batch=32, 80K steps, seed=42
bash scripts/run_b32.sh --config config_b32_80k.yaml

Checkpoints are cached and skipped on re-run. All scripts share the same downloaded data.

Individual stages:

# Train one variant
python -m src.train --config config_b32.yaml --n-layers 6

# Evaluate one variant
python -m src.evaluate --config config_b32.yaml --n-layers 6 --benchmark scifact

# Profile latency
python -m src.profile_latency --config config_b32.yaml --n-layers 6

# Regenerate all figures (depth_vs_budget, per_benchmark_degradation,
# seed_variance, pareto_updated)
python make_plots.py

Hardware and runtime

• GPU: H100 (Azure VM)
• Training: ~1 hr per variant at batch=256; ~4 hrs per variant at batch=32 / 20K; ~16 hrs per variant at batch=32 / 80K (all bf16, seq=128)
• Eval: ~10–15 min per variant across 3 benchmarks
• Latency profiling: ~5 min total (all variants)
• Total across all five experiments: ~75 hours