Linear Regression from Scratch

I built this to get practical experience with machine learning instead of only reading about it. Linear regression is one of the most used algorithms, and it is the base for more complex ones like logistic regression, so I wanted to understand it properly. I wrote every part myself with NumPy rather than calling scikit-learn, because I wanted to see the maths and the code that a library normally hides. I did this out of interest, not for a course.

What it does

It fits a straight line y = wX + b to noisy data using gradient descent. I generate the data from the true line y = 3x + 2 and add random noise. The model starts at w=0, b=0 and learns the slope and intercept back.

The maths sits in four functions in model.py:

• predict() computes y = wX + b.
• loss() measures the error with mean squared error.
• gradients() computes the slope of the loss for w and for b.
• update() moves w and b a small step in the direction that lowers the loss.

How to run

pip install numpy matplotlib
python main.py

It trains the model and saves two plots to results/: the fitted line and the loss curve.

Results

After 10,000 epochs:

• Learned: w = 3.02, b = 1.81
• Truth: w = 3.00, b = 2.00
• Loss fell from 401.84 at epoch 0 to 0.81 at the end.

The small gap in b is not a bug. It comes from the noise in the data, which means the original parameters cannot be recovered exactly.

Fit

Loss curve

What surprised me

w learned much faster than b. By epoch 1000 w was already near 3, while b was still around 1.3 and only crept up over the next 9000 epochs. This confused me for a while.

The reason is in the gradients. The gradient for w is multiplied by X, and my X values run from 1 to 10 (average near 5.5), so the w gradient is much larger than the b gradient for the same error. A larger gradient means a larger step, so w moves faster. The b gradient has no X term, so b creeps.

This is the practical reason people scale or normalise their features. If I had scaled X to a smaller range, w and b would have learned at closer rates.

What I learned

• Gradient descent moves parameters in the direction that lowers the loss fastest.
• The learning rate sets the step size. I tested 0.001 (slow), 0.005 (stable), and 0.1 (diverged, loss went to inf then nan).
• Feature scale changes how fast each parameter learns, because the gradient for w depends on X.
• Noisy data sets a floor on how close you can get to the true parameters.

Files

main.py    training loop and plots
model.py   predict, loss, gradients, update
utils.py   plotting helpers
results/   saved plots