Makemore — Character-Level Language Models

Chapters 2–6 of Andrej Karpathy's Neural Networks: Zero to Hero curriculum. A progressive series building character-level language models that generate novel names. Starts from the simplest possible bigram count model and incrementally adds: learnable embeddings (MLP), BatchNorm and initialization diagnostics, manual backpropagation, and finally a WaveNet-style hierarchical architecture. All models consume names.txt (~32k words, 27-char vocab with .=0) and are evaluated by average negative log-likelihood (NLL), with ln(27) ≈ 3.3 as the random baseline.

---

Lecture Reference

• Videos: "Building makemore" series (Andrej Karpathy, Neural Networks: Zero to Hero — Lectures 2–6):
  1. The spelled-out intro to language modeling — building makemore
  2. Building makemore Part 2: MLP
  3. Building makemore Part 3: Activations & Gradients, BatchNorm
  4. Becoming a backprop ninja
  5. Building makemore Part 5: Building a WaveNet

---

Directory Contents

File	Type	Description	Status
names.txt	Data	~32k names (the training corpus).	Complete
build_makemore_part1_bigrams.ipynb	Notebook (569 KB)	Bigram count model + single-layer NN; NLL framing.	Complete
build_makemore_part2_mlp.ipynb	Notebook (388 KB)	Bengio 2003 MLP; 3-char context; embeddings; train/dev/test.	Complete
build_makemore_part3_bn.ipynb	Notebook (1.25 MB)	Initialization diagnostics; BatchNorm from scratch; PyTorchification.	Complete*
build_makemore_part4_backprop.ipynb	Notebook (59 KB)	Manual backprop through MLP+BN; 4 exercises (all solved).	Complete
build_makemore_part5_CNN.ipynb	Notebook (59 KB)	WaveNet-style hierarchical MLP; 8-char context; FlattenConsecutive.	Complete
docs/	Folder	Per-notebook chapter READMEs (see below).	Complete

* Part 3's deep-net training loop is intentionally short-circuited at 1000 steps (# AFTER_DEBUG).

---

Core Concepts Demonstrated

1. Bigram counting and normalization — the simplest possible character-level model (part 1).
2. Likelihood → log-likelihood → NLL — the standard probabilistic loss framing used throughout.
3. Softmax as exp + normalize — the differentiable version of the count model (part 1 onward).
4. Learnable embeddings — C lookup table, integer-array indexing, view semantics (part 2).
5. Mini-batch SGD with F.cross_entropy — the standard PyTorch training loop (part 2 onward).
6. Train/dev/test splits — 80/10/10 with random.seed(42) shuffle (part 2 onward).
7. Learning-rate range search and decay — lrs = 10**linspace(-3, 0, 1000), then 0.1 → 0.01 at step 100k (part 2 onward).
8. Initialization diagnostics — tanh saturation heatmaps, activation/gradient histograms, grad:data ratios, ud update-to-data ratio (part 3).
9. Kaiming init — W * (5/3) / fan_in**0.5 for tanh networks (part 3).
10. BatchNorm from scratch — bngain/bnbias, running mean/std EMA, the redundancy of b1 under BN (part 3).
11. PyTorchification — Linear/BatchNorm1d/Tanh module classes with an nn.Module-like API (part 3, extended in part 5).
12. Manual backpropagation — chunked forward pass, closed-form cross-entropy and BatchNorm backward, training under torch.no_grad() (part 4).
13. Hierarchical architecture — FlattenConsecutive as a convolution-equivalent; the WaveNet tree structure (part 5).

---

Chapter Walkthrough Index

#	Notebook	Topic	Status
1	docs/build_makemore_part1_bigrams.md	Bigram count model + single-layer NN; NLL framing.	Complete
2	docs/build_makemore_part2_mlp.md	Bengio 2003 MLP; 3-char context; embeddings; train/dev/test.	Complete
3	docs/build_makemore_part3_bn.md	Initialization diagnostics; BatchNorm from scratch; PyTorchification.	Complete*
4	docs/build_makemore_part4_backprop.md	Manual backprop through MLP+BN; 4 exercises (all solved).	Complete
5	docs/build_makemore_part5_CNN.md	WaveNet-style hierarchical MLP; 8-char context.	Complete

* Part 3's deep-net training loop is intentionally short-circuited at 1000 steps.

---

Cross-Notebook Results Summary

Part	Model	Params	Train loss	Val loss	Exercises
1 — bigrams	27×27 count matrix / single-layer NN	729	~2.45 (count) / ~2.48 (NN)	—	No
2 — MLP	Bengio 2003 MLP, 3-char context	~12k	2.1115	2.1567	No
3 — BN	MLP + BatchNorm; 6-layer deep net	~12k / ~46k	2.0674 (1-layer) / 2.4003 (deep*)	2.1057 / 2.3982	No
4 — backprop	MLP + BatchNorm, manual backward	~12k	2.0705	2.1087	Yes — all 4 solved
5 — CNN	WaveNet-style hierarchical MLP, 8-char context	~76k	1.7870	1.9926	No

* Part 3's deep net is intentionally short-circuited at 1000 steps (# AFTER_DEBUG: would take out obviously).

Common patterns across all notebooks

• Same dataset (names.txt), same stoi/itos vocab (27 chars, . = 0).
• Same build_dataset(words) with a sliding block_size context and context = context[1:] + [ix] update.
• Same 80/10/10 train/dev/test split with random.seed(42) (parts 2–5).
• Same manual training loop structure: minibatch → forward → F.cross_entropy → p.grad = None → loss.backward() → p.data += -lr * p.grad, with LR decay 0.1 → 0.01 at step 100k (parts 2–5).
• Same final sampling routine: seed context with all 0s, autoregressively sample until . is produced.
• The loss metric throughout is average negative log-likelihood, with ln(27) ≈ 3.3 as the random baseline.

---

Prerequisites & Core Dependencies

• torch — the primary framework (tensors, autograd, F.cross_entropy, nn.Module).
• matplotlib — for training curves, heatmaps, activation histograms, and embedding visualizations.
• names.txt — the training corpus (must be in the working directory).

---

How to Run

Run a notebook

cd makemore
jupyter notebook build_makemore_part1_bigrams.ipynb
# or headless:
jupyter nbconvert --to notebook --execute build_makemore_part1_bigrams.ipynb --output build_makemore_part1_bigrams.ipynb

Expected runtimes (approximate, on a modern laptop CPU)

Notebook	Runtime
Part 1	~1 minute
Part 2	~10 minutes (200k steps)
Part 3	~5 minutes (1-layer); ~30 seconds (deep, 1000 steps)
Part 4	~10 minutes (Exercise 4 training loop)
Part 5	~10 minutes (200k steps)

---

Status & Caveats

• All notebooks are complete and self-consistent. All training loops have been executed with stored outputs.
• Part 3's deep-net training loop (cell 16) is intentionally short-circuited at 1000 steps (# AFTER_DEBUG: would take out obviously). To get a properly-trained deep model, remove this break and re-run.
• Part 4 is the only notebook containing Karpathy's numbered exercises, and all four are fully solved with bit-exact gradient verification against PyTorch autograd.
• The codebase is a clean, faithful study implementation — not extended beyond Karpathy's originals yet.