Implement character-level GPT model from scratch

[Copilot]

Full PyTorch implementation of a character-level GPT transformer for text generation.

Architecture

• model.py: Transformer components

  • Multi-head self-attention with causal masking
  • Residual blocks with LayerNorm
  • Position + token embeddings
  • Temperature and top-k sampling for generation

• train.py: Training infrastructure

  • Character tokenizer (vocab built from input text)
  • AdamW optimizer with train/val split
  • Batch generation with configurable context window

• example.py: Usage demonstrations

Usage

from train import train, generate_text

model, tokenizer = train(
    text,
    n_embd=384,
    num_heads=6,
    num_layers=6,
    block_size=256,
    max_iters=5000
)

generated = generate_text(
    model, tokenizer,
    prompt="Once upon",
    max_new_tokens=500,
    temperature=0.8,
    top_k=10
)

Configuration

Default hyperparameters suitable for ~10K character texts. Scale n_embd, num_layers, and block_size for larger datasets.

Original prompt

A character-level GPT model built from scratch.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.