Data Preparation for Tokenizer Training

[zprzpr121-design]

Hello,

Thank you for open-sourcing this project.

I have a question regarding the data preparation step mentioned in the README. In the Data & Checkpoint Preparation section, it states:

Requires ImageNet in webdataset format for tokenizer training.

However, I am not sure where to obtain the ImageNet dataset in WebDataset format. I would like to clarify:

Should we convert the original ImageNet dataset (ILSVRC2012) into WebDataset format ourselves?

If so, is there any recommended script or tool for this conversion?

Alternatively, is there a preprocessed WebDataset version of ImageNet that you recommend downloading?

Any guidance on the expected dataset structure or preparation pipeline would be greatly appreciated.

Thank you for your help!

Best regards

[kdwonn]

Hi, thanks for reaching out!

Yes, you'll need to package ImageNet into webdataset format yourself, since it's originally distributed as a simple per-class folder structure. That said, using webdataset is entirely optional — it's specified in the README only because the current dataloader assumes that format. Feel free to implement a standard PyTorch dataloader if you prefer.

Here's the script I used for webdataset repackaging (it's essentially just tar files): https://github.com/bytedance/1d-tokenizer/blob/main/data/convert_imagenet_to_wds.py

I acknowledge the current README could be more friendly. I'll work on improving it soon. Thanks!

[zprzpr121-design]

Hi, thanks for reaching out!

Yes, you'll need to package ImageNet into webdataset format yourself, since it's originally distributed as a simple per-class folder structure. That said, using webdataset is entirely optional — it's specified in the README only because the current dataloader assumes that format. Feel free to implement a standard PyTorch dataloader if you prefer.

Here's the script I used for webdataset repackaging (it's essentially just tar files): https://github.com/bytedance/1d-tokenizer/blob/main/data/convert_imagenet_to_wds.py

I acknowledge the current README could be more friendly. I'll work on improving it soon. Thanks!

Hi, thank you for the detailed and helpful clarification!

That makes things much clearer — I appreciate you sharing the conversion script and the additional context regarding the data format.

Also, I’m really looking forward to the release of the pretrained checkpoints, as they would be very helpful for reproducing and building upon your work.

Thanks again for your support!