Skip to content

Add UniLora tuner to PEFT#3257

Open
KaiyangLi1992 wants to merge 1 commit into
huggingface:mainfrom
KaiyangLi1992:unilora-submit-v2
Open

Add UniLora tuner to PEFT#3257
KaiyangLi1992 wants to merge 1 commit into
huggingface:mainfrom
KaiyangLi1992:unilora-submit-v2

Conversation

@KaiyangLi1992

Copy link
Copy Markdown

Motivation

This PR adds UniLora, a LoRA-style parameter-efficient fine-tuning method that reconstructs LoRA parameters from a shared low-dimensional vector.

This is a fresh PR replacing #2968, because GitHub could not reopen the old closed PR after the head branch was updated.

What changed

  • Add the UniLora tuner implementation, config, layer, and model integration.
  • Register PeftType.UNILORA and expose UniLoraConfig / UniLoraModel.
  • Add save/load handling for UniLora shared weights.
  • Add optional save_indices support for generated index and scale tensors.
  • Add package reference documentation.
  • Add a quick UniLora fine-tuning example.
  • Add a MetaMathQA experiment config.
  • Add UniLora coverage in custom model, decoder, encoder-decoder, and dedicated UniLora tests.

Notes for review

This branch is rebased on the latest main and incorporates the remaining review feedback from #2968, including naming, docs, tests, example, MetaMathQA config, deterministic index generation documentation, init_weights, and optional index saving.

Tests

  • make style
  • make quality
  • PYTHONPATH=src python -m pytest tests/test_unilora.py
  • PYTHONPATH=src python -m pytest --no-cov tests/test_custom_models.py -k UniLora
  • PYTHONPATH=src python -m pytest --no-cov tests/test_decoder_models.py -k UniLora
  • PYTHONPATH=src python -m pytest --no-cov tests/test_encoder_decoder_models.py -k UniLora

@KaiyangLi1992 KaiyangLi1992 mentioned this pull request May 23, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant