Learning Resource
Watch this video: link
This is the core algorithm for BPE, talks about the motivation behind it and how the general algorithm works.
I recommend following along on a Google Colab. The first part of the video (up until 1:11) is VERY IMPORTANT. The remaining part is up to you if you want to watch.
Next Steps
After this, we'll try to implement this tokenizer in C++ and replicate the result that we got in Python.
Learning Resource
Watch this video: link
This is the core algorithm for BPE, talks about the motivation behind it and how the general algorithm works.
I recommend following along on a Google Colab. The first part of the video (up until 1:11) is VERY IMPORTANT. The remaining part is up to you if you want to watch.
Next Steps
After this, we'll try to implement this tokenizer in C++ and replicate the result that we got in Python.