About pretraining data

[Haoyu-Hu]

Hi @csuhan!

Thanks for your great work. Now I am trying to reproduce the pretraining process of OneLLM and hope to know how I can find the .csv file used in the pretraining. Currently I can only find files for intruction tuning and evaluation on huggingface. Many thanks for your help.

[csuhan]

Hi @Haoyu-Hu , the pretraining data is a collection of large-scale webdatasets. You can download them from their links: https://github.com/csuhan/OneLLM/blob/main/docs/Data.md#download-links