Training with my own finetuned sdxl

[vikas784]

🚨 Issue: LoRA Not Updating Custom Fine-Tuned SDXL Model in SPO Training

I am training a model in two stages:

1️⃣ Stage 1 (Step-Aware Preference Model Training)

• Trained on LAION dataset, outputs sdxl_step-aware_preference_model.bin.

2️⃣ Stage 2 (SPO Training)

• Uses:
  • Base Model = Fine-tuned SDXL.
  • Reference Model = sdxl_step-aware_preference_model.bin.
• Outputs LoRA adapters.

---

Issue 1: Model Weights Size Mismatch

• My first-stage trained model (final_ckpts.bin) has two 5GB FP32 weight shards (total 10GB).
• The Hugging Face repo (sdxl_step-aware_preference_model.bin) is only 5GB in FP16.
• How is the Hugging Face model so much smaller?

---

Issue 2: LoRA Not Updating Fine-Tuned Model

• When I apply LoRA to my fine-tuned SDXL, it says "no modules changed".
• But when I apply it to Hugging Face’s SDXL, it works.
• Why does it not work on my fine-tuned model?

---

Issue 3: UNet Source in Hugging Face Repo

• The Hugging Face repo contains a unet file, but I cannot find where it came from.
• Could you clarify how it was generated?

---

Reproducibility Steps

1. Train Stage 1 model → Get final_ckpts.bin (10GB).
2. Convert to FP16 → Expected size should be 5GB, but mismatch with Hugging Face repo.
3. Train Stage 2 using sdxl_step-aware_preference_model.bin.
4. Apply LoRA to my fine-tuned SDXL → No changes.
5. Apply LoRA to Hugging Face SDXL → Works fine.

---

Expected Behavior

• LoRA should update the fine-tuned SDXL model correctly.
• Model size should match expectations.
• Source of the UNet in Hugging Face repo should be clear.

Additional Context

Any insights on these issues would be helpful. Thanks!

[RockeyCoss]

Issue 1:
The checkpoint size should be about 5 GB. Could you please share the filename of each shard?

Issue 2:
How did you apply the LoRA weights to your model? Would you mind sharing your code?

Issue 3:
Are you referring to this UNet?
https://huggingface.co/SPO-Diffusion-Models/SPO-SDXL_4k-p_10ep/tree/main/unet
If so, that’s the SDXL UNet with SPO’s LoRA weights already merged in.

Thanks!