My trial at implementing a transformer architecture, starting from Build a Large Language Model (From Scratch) by Sebastian Raschka.
With zero dependencies.
At the moment of writing, the interface of each tokenizer is dscribed in the docstring of AbstractTokenizer.
See src/tokenizers/abstract.jl.
One example of GPT-2-like tokenizer is the BPETokenizer in src/tokenizers/bpe.jl.
Just run julia --project=docs docs/make.jl.
If the documentation is building, it means that all the use cases appearing in
jldoctest environments are running.
This is a good compromise between sketching and showcasing examples of use-cases.
- LLMs-from-scratch;
- minbpe;
- Julia Base.Unicode, which seems to be adapted from utf8 proc.