GPT2-style Transformer (chatgpt.py)

Overview

This repository contains a compact TensorFlow/Keras implementation of a GPT2-style Transformer model defined in chatgpt.py.

Key components

• MultiHeadAttention — multi-head self-attention layer.
• FeedForwardNetwork — two-layer feed-forward block with GELU activation.
• TransformerBlock — attention + feed-forward with residual connections and layer normalization.
• GPT2 — stacks transformer blocks, token & positional embeddings, output projection.

Requirements

• Python 3.8–3.12
• TensorFlow (tested with TF 2.10+)

Install with pip:

pip install tensorflow

If you're on an Apple Silicon Mac (M1/M2/M3) and want native performance, consider:

pip install tensorflow-macos

Usage

Run the provided file to build and inspect the model:

python chatgpt.py

This will:

• Build a GPT2-style model with VOCAB_SIZE = 50257 and MAX_LENGTH = 1024.
• Print the model summary.

Example (inside chatgpt.py)

The script demonstrates how to:

• Create tf.keras.Input with shape (MAX_LENGTH,).
• Instantiate the GPT2 model and call .build().
• Print gpt2.summary().

Notes & Caveats

• This implementation is educational and minimal — it’s not a drop-in replacement for production GPT-2 libraries.
• The model uses a simple causal mask; training this model from scratch requires careful optimization, large datasets, and significant compute.
• For serious training or inference, consider using Hugging Face Transformers or the official OpenAI checkpoints where applicable.

File list

• chatgpt.py — the model implementation.
• chatgpt drawing.svg — architecture/diagram image (embedded above).

License

MIT License — feel free to adapt and use for learning and research.

---

If you want, I can:

• Export this README as a standalone README.md file in the workspace.
• Update the image path or rename the image file for cleaner links.
• Add a usage example showing how to run a single forward pass or save/load weights.