Data-Centric Adaptation and Failure Analysis of Code LLMs

This repository studies parameter-efficient adaptation of a 7B code language model under unit-test-based evaluation. The central question is whether data quality and failure-guided augmentation can improve code-generation reliability without changing the base model or increasing the training scale.

The experimental pipeline uses QLoRA supervised fine-tuning (SFT) in a restricted NVIDIA A40 environment. It compares naive preprocessing, indentation-preserving preprocessing, contract-aware augmentation, and unit-test-feedback augmentation. Physical GPUs 0--3 are excluded from all project workloads.

Reading Guide

Recommended reading order for the completed study:

1. PROJECT_FINAL_SUMMARY.md for the conclusions and validity boundaries.
2. experiments/final_evaluation/FINAL_EVALUATION.md for the five-run deterministic comparison and failure transitions.
3. REPORT_MATERIALS.md for report-ready tables, figures, and wording.
4. data_sources.md, data_schema.md, and eval_protocol.md for data and evaluation provenance.
5. experiments/exp003_qlora_sft_targeted_contracts/ and experiments/exp004_unit_test_feedback/ for the best run and negative ablation records.

Earlier exp001 and exp002 quick-evaluation directories are retained as historical process records. Their time-specific next-step recommendations are superseded by the completed final matrix and do not describe pending work.

Research Questions

• RQ1: Data quality. How strongly does code-format preservation affect SFT performance?
• RQ2: Failure transition. Do failures move from syntax and structure errors to interface-contract and boundary-condition errors as adaptation improves?
• RQ3: Failure-guided augmentation. Can examples derived from observed unit test failures improve pass rate without causing regressions?

Experimental Matrix

Run	Training strategy	Added targeted rows	Status
Base	Qwen2.5-Coder-7B without fine-tuning	0	Complete
exp001	Naive QLoRA SFT	0	Complete
exp002	Preserve-indent data repair	0	Complete
exp003	Contract-aware augmentation	96	Complete
exp004	Unit-test-feedback augmentation	100	Complete; negative ablation

The primary final test set contains 73 tasks whose reference solutions pass the associated tests. It is disjoint from the development set used to construct the exp004 feedback examples. Generation is deterministic: greedy decoding, seed 42, a 1,024-token limit, and termination after the first complete fenced code block.

Final Deterministic Results

Run	Passed	Pass rate	Syntax	Structural	Contract	Semantic	Boundary
Base	6/73	8.22%	12	4	51	0	0
exp001	15/73	20.55%	0	55	1	0	2
exp002	62/73	84.93%	0	0	7	0	4
exp003	64/73	87.67%	0	0	6	0	3
exp004	61/73	83.56%	0	0	9	0	3

Repairing code formatting produced the largest gain: exp002 added 47 passed tasks over exp001 and removed all 55 structural failures observed in exp001. Contract-aware exp003 added two passes with no regression and is the best run. Unit-test-feedback exp004 added no passes and regressed on three interface contracts, reducing performance to 61/73. The negative result is retained as a controlled ablation rather than replaced by a selected rerun.

Evaluation records are stored under experiments/final_evaluation/.

Method

The project follows a closed failure-analysis loop:

Code LLM
  -> deterministic generation
  -> isolated Python unit tests
  -> failure classification
  -> targeted training examples
  -> QLoRA SFT
  -> repeated evaluation

The automatic taxonomy contains five mutually exclusive categories:

1. syntax: invalid Python syntax or incomplete generated code.
2. structural: indentation or code-structure corruption.
3. interface_contract: missing names, incompatible signatures, or incorrect externally visible interfaces.
4. semantic: valid interfaces with incorrect core behavior.
5. boundary: failures on empty, singleton, extreme, or similarly constrained inputs.

Data

Training data is based mainly on Python subsets of OpenCodeInstruct and CommitPackFT. The experiment remains at approximately 10k SFT examples; no 20k/100k expansion is part of the study.

Path	Purpose
data/processed/sft_exp001_v1/	Initial SFT data used by exp001
data/processed/sft_exp001_v2_preserve_indent/	Format-preserving data used by exp002
data/processed/sft_exp003_targeted_contracts/	Contract-augmented data used by exp003
data/processed/sft_exp004_unit_test_feedback/	Feedback-augmented data used by exp004
data/eval/final_eval_reference_clean_73.jsonl	Primary reference-valid final test set
data/eval/feedback_dev_100.jsonl	Disjoint development set used for feedback mining

The exp004 preparation process identified 25 unique, reference-valid failed development examples. Repeating them four times adds 100 rows to the exp003 training set. Source-identifier overlap with the final test set and original training set is zero.

Data provenance, schema, and build records are documented in data_sources.md, data_schema.md, data/README.md, and data/DATASET_BUILD_LOG.md.

Training Configuration

• Base model: Qwen2.5-Coder-7B
• Adaptation: QLoRA SFT
• Quantization: 4-bit NF4
• LoRA rank: 64
• LoRA alpha: 16
• LoRA dropout: 0.1
• Target modules: all linear layers
• Loss mask: assistant responses only
• Allowed devices: physical GPU 4 and GPU 5 only
• Controlled effective batch: 96

exp001--exp003 used physical GPUs 4 and 5. GPU4 was occupied by an independent process during exp004, so the formal exp004 run used physical GPU5 and doubled gradient accumulation from 24 to 48. All other hyperparameters remained fixed.

Full training scripts:

bash scripts/train_exp002_qlora_sft_preserve_indent.sh
bash scripts/train_exp003_qlora_sft_targeted_contracts.sh
bash scripts/train_exp004_unit_test_feedback.sh
bash scripts/train_exp004_unit_test_feedback_gpu5.sh

Smoke-test scripts:

bash scripts/train_exp002_smoke_preserve_indent.sh
bash scripts/train_exp003_smoke_targeted_contracts.sh
bash scripts/train_exp004_smoke_unit_test_feedback.sh

The exp004 launchers validate selected physical GPUs and refuse to share a GPU with a process using more than 1 GiB of compute memory. Model caches, temporary files, logs, adapters, and evaluation artifacts remain inside the project directory.

Evaluation

Run the deterministic evaluation pipeline with one allowed physical GPU:

GPU=5 bash scripts/run_deterministic_eval.sh \
  exp003 \
  experiments/exp003_qlora_sft_targeted_contracts/output/adapter-final

The pipeline performs generation, isolated unit-test execution, first-failure diagnosis, and taxonomy classification. Individual stages are also available:

python scripts/generate_responses_incremental.py --help
python scripts/evaluate_python_unit_tests.py --help
python scripts/classify_failures.py --help
python scripts/analyze_experiment_matrix.py --help

Run the focused pipeline tests with:

python -m unittest discover -s tests -v

Repository Layout

.
|-- configs/                       # QLoRA and Accelerate configurations
|-- data/                          # evaluation and processed training data
|-- experiments/                   # plans, logs, outputs, and result summaries
|-- notes/                         # reading notes for the paper collection
|-- papers/                        # 11 source papers
|-- scripts/                       # data, training, generation, and evaluation tools
|-- tests/                         # focused experiment-pipeline tests
|-- ENVIRONMENT.md                 # hardware and software environment record
|-- RESEARCH_PLAN.md               # research scope and experiment design
|-- PROJECT_FINAL_SUMMARY.md       # project-level findings
|-- REPORT_MATERIALS.md            # tables and concise reporting material
|-- data_schema.md                 # data contracts
|-- data_sources.md                # data provenance
`-- eval_protocol.md               # evaluation rules

Paper Collection

The repository includes papers and one-page notes covering QLoRA, LoRA, Code Llama, StarCoder2 and The Stack v2, OctoPack, InstructCoder, OpenCodeInstruct, Self-Instruct, LIMA, FLAN, and domain-adaptive continued pretraining. The reading notes connect each paper to the experiment design while keeping continued pretraining outside the implemented scope.

Artifact Policy

Adapter output directories and runtime logs are excluded from Git because of their size. The tracked repository contains configurations, scripts, manifests, hashes, data records, evaluation outputs, tables, and figures required to identify and reproduce each run.

deep-research-report.md is intentionally ignored and remains untracked.

Scope

This repository is limited to QLoRA SFT, deterministic unit-test evaluation, failure analysis, and targeted data augmentation. Continued pretraining, CPT+SFT comparisons, scratch pretraining, model scaling, and large-scale data expansion are not part of the experimental claims.