Playing with gradient descent

A collection of notebooks leveraging gradient descent to fit a variety of simple datasets. From scratch, for fun.

They require Python >= 3.5.

Note: Some of the notebooks use a homemade implementation of a multilayer perceptron (in lib/mlp.py). The online book Neural Networks and Deep Learning has been of tremendous help to unravel the equations of the backpropagation algorithm.

Notebooks

The following notebooks of increasing difficulty are available.

two_points_sgd.ipynb

Fit a line through 2 points using stochastic gradient descent.

regression_sgd.ipynb

Perform regression using stochastic gradient descent.

Weights search

classification_sgd.ipynb

Perform classification using a single layer perceptron.

Weights search

xor_mlp.ipynb

Separate the XOR problem using a multilayer perceptron.

three_donuts_mlp.ipynb and six_donuts_mlp.ipynb

Separate donuts using a multilayer perceptron.

Three donuts

Six donuts

mnist_mlp.ipynb

Recognize handwritten digits using a multilayer perceptron with 97.5% accuracy.

Using MNIST dataset.

Correct predictions that were the most certain for each class

Correct predictions that were the least certain

Incorrect predictions that were the most certain

Incorrect predictions that were the least certain

Installation notes

The required python packages to install to run the notebooks are listed in requirements.txt. Use Python >= 3.5.

If you want to run the cells using Keras, make sure to configure its backend appropriately.