Neural Networks From Scratch With Buzz Lightyear (Part 1: Calculus)

Warning: This is not a comprehensive article on neural networks. While it does discuss the mathematical concepts behind them and expects at least some programming experience, many aspects are glossed over. The idea is to create one from scratch with only the bare essentials, leaving the option open for further study. For more theory-heavy resources I recommend the YouTube series by Andrew Ng and sentdex.

Neural networks are a type of machine-learning algorithm vaguely inspired by the human brain. They take an input and are trained to predict a desired outcome, anything from house prices to pictures of dogs. They are constructed from very simple analogues to neurons, called perceptrons, which are basically linear functions we optimize. It’s okay if you don’t understand all that — that’s what this article is here for. All you need to know is that a neural network takes in some numbers and tries to guess the best ones to spit out.

This article is split into three parts:

Part 1: Calculus — an introduction to the differential calculus that drives neural networks (with a bunch of interactive animations)
Part 2: Perceptron — using calculus to code a perceptron, the smallest unit of a neural network
Part 3: Multi-Layer Perceptron (MLP) — stringing perceptrons end to end and stacking them into more powerful models

Calculus has a reputation for being terrifying. It’s actually very simple. A good way to see differential calculus is as the slopes of curves. And we’ll see it through the best ambassador for slopes, Buzz Lightyear.

We all know his catchphrase “To Infinity and Beyond”. While it’s very cool to say, mathematics tells us that he can never actually reach infinity, let alone go beyond it. Math is cruel like that. But we can say Buzz is approaching infinity the more he goes forward and approaching negative infinity the more he goes backward. We can measure how fast he is approaching infinity by how steeply he is ascending. In math, we call that the slope or gradient of the line.

In the simple equation $y = x$ , the gradient is 1. We can set his inclination that way to get a natural-looking flight path.

if (Math.abs(x) >= 2.5)
    speed *= -1

x += speed

drawBuzz(x, x, 1) // x, y, and inclination/slope

It sounds obvious when we’re talking about a line, but let’s imagine other functions, like a parabola $y = x^{2}$ . Then it gets harder to talk about the gradient because it keeps changing — sometimes Buzz is going up and other times he’s going down. That is where derivatives come in.

A derivative is a formula for a function’s gradient based on $x$ . The derivative of any power function $y = x^{n} + c$ is $y = n \cdot x^{n - 1}$ (any constant is ignored). So the derivative of $y = x^{2}$ is $2 x$ . We can see this working when we make Buzz’s inclination equal $2 x$ :

If the function has a coefficient we keep it too. The derivative of $y = m \cdot x^{n} + c$ is $y = m \cdot n \cdot x^{n - 1}$ .

Derivatives aren’t limited to power functions. The derivative of $sin x$ is $cos x$ .

You can get the derivatives of any combination of functions, but you need to follow the chain rule: for a function within a function, calculate the derivatives in layers and multiply them. For $y = sin 3 x$ the derivatives are $cos 3 x$ for the outer and $3$ for the inner, giving us $3 \cdot cos 3 x$ for the combined function.

Addition of functions is easy to differentiate — just add the derivatives. For $sin 3 x + x$ , we get $3 \cdot cos 3 x + 1$ .

Now you know all the calculus you need to code a basic perceptron. How? By combining it with gradient descent. Gradient descent is the core algorithm behind neural networks. It uses the gradient of a function to find where its lowest value sits. Since an increasing function has a positive gradient and a decreasing one a negative gradient, going the opposite direction of the gradient takes you toward the minimum.

In the example below, four Buzz Lightyears are scattered on random points and they all try to descend to the minimum. Use the restart button to make them go again. Some get stuck on the upper “rung” — caught in a local minimum, not the absolute one. Others disappear off the left edge because it keeps going down forever — they are even closer to the best minimum.

for (var i = 0; i < xs.length; i++) {
    var derivative = 3*Math.cos(xs[i]*3) + 1

    // subtract the gradient (0.01 is a step-size constant)
    xs[i] -= 0.01 * derivative

    // recalculate the derivative because x changed
    derivative = 3*Math.cos(xs[i]*3) + 1
}

There is also a variant called gradient ascent. Same idea, but it tries to find the maximum by going along the gradient instead of against it. Notice that there are local maxima here.

for (var i = 0; i < xs.length; i++) {
    var derivative = 3*Math.cos(xs[i]*3) + 1

    // only difference is the plus
    xs[i] += 0.01 * derivative

    derivative = 3*Math.cos(xs[i]*3) + 1
}

That’s all there is to it. In the next part of this series, we’ll see how to use gradient descent to make a perceptron — the smallest unit of a neural network — and use it to find the best-fit line for arbitrary data.