About GPU memory requirements for model training

[jiaxiaojunQAQ]

Thanks for your excellent work! I would like to ask how much memory is needed for training stage?

[chs20]

Hi, we train the ViT-L/14 on 4x NVIDIA A100 40GB at total batch size 128