Lora rank problem

[Whitney-9]

Thank you very much for this excellent work. I have some questions about the paper, why is the rank of LORA set to 512? Usually, people would choose a smaller rank. Wouldn't using 512 in this work lead to overfitting of the model? Another question is that the 100k dataset was only used for training for 5k times, with a batch size of 16. The data was not fully utilized. Why can we be sure that the current training is sufficient?

[nenhang]

For the LoRA rank, our initial assumption was that this task is relatively demanding, since it involves multiple model-side modifications. So we used r=512 from the beginning and kept it as the main setting. We also trained r=256 versions during development. They were still usable and the gap was not huge, but r=512 generally worked a bit better in our experience. With smaller rank, the control could sometimes become weaker or slightly more rigid. We did not run a very thorough ablation specifically on LoRA rank or overfitting. From the generated images, overfitting did not seem very obvious.

For the dataset / training-length part, the short answer is that your concern is reasonable. We discussed this in more detail in issue #5. In short, the 5k-step version was not necessarily the best in terms of generalization, and both more training and a larger data pool were still helpful in our experience.