A minimal implementation of a Generative Pretrained Transformer (GPT) built from scratch in PyTorch, based on the seminal paper "Attention is All You Need" and inspired by Andrej Karpathy's educational lecture series.
This project trains a character-level Transformer model on a tiny version of Shakespeare's text to predict and generate new text in a GPT-style fashion.
- Character-level language modeling
- Custom multi-head self-attention from scratch
- Transformer blocks with residual connections & layer normalization
- Text generation with sampling
- Based on core concepts from the GPT family of models
git clone https://github.com/yourusername/transformer.git
cd gpt-from-scratchwget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txtpip install torchpython main.py- Reads the dataset (
input.txt) and builds a character-level vocabulary. - Each character is assigned a unique integer ID.
- The entire text is encoded as a tensor of integers.
- Token Embeddings: Maps input character IDs to dense vectors.
- Positional Embeddings: Injects token position information.
- Transformer Blocks: Multiple stacked blocks with:
- Multi-Head Self-Attention
- Feedforward layers
- Residual connections & LayerNorm
- Language Modeling Head: Outputs logits over vocabulary to predict the next character.
- Trains on batches of sequences using cross-entropy loss.
- Periodically evaluates on a validation set.
- After training, the model can generate Shakespeare-like text.
This project is heavily inspired by:
MIT License