Skip to content

docs(prepare_model_for_kbit_training): note the ~0.5–1 GB CUDA reserved overhead (#3265) #3267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Anai-Guo wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(prepare_model_for_kbit_training): note the ~0.5–1 GB CUDA reserved overhead (#3265) #3267

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions src/peft/utils/other.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,15 @@ def prepare_model_for_kbit_training(model, use_gradient_checkpointing=True, grad
1- Cast the layernorm in fp32 2- making output embedding layer require grads 3- Add the upcasting of the lm
head to fp32 4- Freezing the base model layers to ensure they are not updated during training

<Tip warning={true}>

Memory note: the layernorm/RMSNorm fp32 upcast performed for non-quantized and bnb 4bit/8bit paths can
grow CUDA reserved memory by roughly 0.5–1 GB for 7B-class models in addition to the allocated bytes,
because the CUDA caching allocator holds the transient buffers from the upcast op. On 8 GB consumer or
edge accelerators (Jetson Orin Nano, Apple Silicon unified memory, RTX 4060 8 GB, …) this overhead can
be the difference between a recipe that fits and one that OOMs — budget for it explicitly.

</Tip>

Args:
model (`transformers.PreTrainedModel`):
Expand Down