Does Feedback Alignment Work for LLM Fine-Tuning?

This repository contains the official implementation accompanying Does Feedback Alignment Work for LLM Fine-Tuning? It provides the training code for comparing backpropagation (BP), feedback alignment with pretrained connectors (FA-pretrained), and feedback alignment with random connectors (FA-random) during full-parameter supervised fine-tuning of Qwen2.5-3B and Qwen2.5-7B.

At each selected linear operation, FA preserves the forward pass and local parameter gradient but replaces the input-gradient connector:

BP:             grad_input = grad_output @ current_weight
FA-pretrained:  grad_input = grad_output @ pretrained_weight
FA-random:      grad_input = grad_output @ fixed_random_weight

FA is applied to q_proj, k_proj, v_proj, and o_proj in attention and gate_proj, up_proj, and down_proj in the MLP. All feedback connectors remain fixed throughout training.

Setup

The paper configurations use two bfloat16-capable GPUs with PyTorch FSDP full sharding. The reference software environment uses Python 3.12, PyTorch 2.8, Transformers 4.57.1, and CUDA 12.8. Create an environment and install the pinned dependencies:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Model weights and datasets are downloaded from Hugging Face automatically and cached under .cache/huggingface. The training data come from openai/gsm8k, configuration main, and knkarthick/samsum.

Training

Each launch script accepts --model 3b|7b, --method bp|fa-pretrained|fa-random, and one of the paper seeds (1234, 2345, or 3456). For example:

bash scripts/run_gsm8k.sh --model 3b --method fa-pretrained --seed 1234
bash scripts/run_samsum.sh --model 7b --method fa-random --seed 2345

The scripts load the public training split, apply the paper's prompt-response formatting and response-only loss mask, and save the scheduled checkpoints under outputs/. No separate data preparation step is required.

Structure

feedback.py   Fixed feedback connectors and Qwen projection replacement
data.py       GSM8K/SAMSum loading, formatting, tokenization, and masking
train.py      Unified FSDP training entry point
configs/      Exact model-task configurations used in the paper
scripts/      GSM8K and SAMSum launch commands
tests/        Focused CPU checks for the connector, data path, and configs

Run the CPU checks with:

python3 -m pytest -q

This repository intentionally contains training code and configurations only; it does not include evaluation pipelines, generated checkpoints, or result artifacts.

License

Released under the MIT License.