Neural Networks From Scratch With Buzz Lightyear (Part 2: Perceptrons)

In the last article, Buzz showed us the basic ideas of calculus and derivatives. In this article we’ll use those concepts to make a perceptron.

Before we start, a clarification on what a perceptron (and a neural network in general) is: it’s an object vaguely inspired by the biological neuron that takes an input and maps it to a certain output. Basically a function. There are two types: ones that output continuous values (linear regression models) and ones that output discrete values (logistic regression models). Let’s start with the former.

In a linear regression model we try to optimize a perceptron into making accurate predictions on a given dataset. The simplest and most common function this perceptron can be is a line, $o u tp u t = w e i g h t \cdot in p u t + bia s$ , weight and bias being the slope and y-intercept. Since we don’t know what weight and bias to use, we set them to random numbers. Below you can see what a randomly initialized perceptron looks like and how close it comes to describing the real data — in this case a straight line. The blue lines show the deviation from the correct answer. We call them the cost, loss or error. A convenient way to compute cost is as the square of the difference between output and desired output (its derivative is easy to take).

var perceptron = [random(-1,1), random(-1,1)]

function calculatePerceptron(x) {
    return perceptron[0]*x + perceptron[1]
}
function cost(x, y) {
    return (calculatePerceptron(x) - y) ** 2
}
function cost_deriv(x, y) {
    return 2 * (calculatePerceptron(x) - y)
}

Suppose the data is a map of stars Buzz has to fly to. He knows the first few; after that he’s on his own. The perceptron needs to fit the known stars and predict the rest. Since a randomly initialized one is useless, we need to find a way to optimize it — find the weight and bias where loss is at a minimum. That’s gradient descent again.

If the cost is $((in \cdot w + b) - o u t)^{2}$ , the derivative in terms of the bias is $2 \cdot ((in \cdot w + b) - o u t)$ . For the weight, the chain rule gives us an extra factor of $in$ : $2 \cdot ((in \cdot w + b) - o u t) \cdot in$ .

function update() {
    // variable to sum up the derivatives
    var adj = [0, 0]
    for (var i = 0; i < points.length; i++) {
        var p = points[i]
        adj[0] -= cost_deriv(p.x, p.y) * p.x
        adj[1] -= cost_deriv(p.x, p.y)
    }
    perceptron[0] += 0.05 * adj[0]
    perceptron[1] += 0.007 * adj[1]
}

Now that the perceptron is optimized, it fits the data perfectly and Buzz can find all the other stars he wants by extending the line.

A straight-line perceptron can only get a cost of zero when the data is in a straight line. If the data has slight deviations the cost will settle just above zero. And if the data goes up and down, a single perceptron is basically worthless — it can only model one direction. You can see the best fits of different types of data on a single perceptron below.

In practice we solve this by using perceptrons in chains and layers — that’s the next article. But here, in the spirit of experimentation, we can also change the type of perceptron to fit the data. We can make it quadratic, cubic, or sine. If we get the derivatives right, each is a valid perceptron that can sometimes outperform a simple line.

Messing around with the function/data pairs surfaces a lot of intuition. The cubic function fits almost everything — which makes sense because a quadratic is just a cubic with one coefficient zeroed, and a linear is one with two. You might wonder why we don’t prefer them in neural networks. The answer is that they tend to overfit and have exploding gradients (both explained next article). The simplified version: higher-degree polynomials have very sensitive coefficients — a small change reshapes the graph drastically. Even when stabilized it costs speed, which matters when your network has millions of perceptrons.

A less obvious thing: the sine function sometimes can’t fit quadratic data, settling for a horizontal line. The perceptron found a local minimum instead of the absolute one. Sometimes a perceptron only converges if you initialize it just right.

All this applies to logistic regression too. While linear regression makes perceptrons predict data, logistic regression makes them classify it. Say space is either being invaded or experiencing a pandemic — Buzz doesn’t need to know where the spaceships or planets are, he needs to know what they are when he sees them. We have a dataset of green (safe) and orange (dangerous) points, and Buzz wants to figure out which group a new point belongs to.

How do we make a perceptron output discrete values? We can’t. But we can make it output probabilities. Linear perceptrons are not useful for probability — 1000% green or -12% orange make no sense. We need it to output something from 0% to 100%. This is where the sigmoid activation function comes in: $\frac{1}{1 + e ^{- x}}$ squashes a linear value between 0 and 1. Now 1000 maps to ~100% and -12 to ~0%. Sigmoid has a convenient derivative: $s i g m o i d (x) \cdot (1 - s i g m o i d (x))$ .

With logistic regression we also change the cost function. Difference squared still works but gives slow results — the maximum possible cost is only 1. We use the negative log functions $- lo g (1 - o u tp u t)$ and $- lo g (o u tp u t)$ , depending on whether we want 0 or 1, to give a more exaggerated cost.

Combining activation, cost, and a simple linear perceptron gives a more complex derivative because of the nesting. The chain rule still gets us there: cost is $lo g (s i g (p er ce pt r o n))$ , so derivative is $l o g D er i v (s i g (p er ce pt r o n)) \cdot s i g D er i v (p er ce pt r o n) \cdot p er ce pt r o n D er i v$ . Here it is in code form, separating green and orange points:

function sigmoid(x)       { return 1 / (1 + Math.exp(-x)) }
function sigmoid_deriv(x) { return sigmoid(x) * (1 - sigmoid(x)) }

function calculatePerceptron(data) {
    return perceptron[0]*data.x + perceptron[1]*data.y + perceptron[2]
}
function cost(data) {
    var x = sigmoid(calculatePerceptron(data))
    if (data.class == 1) return -Math.log(x)
    return -Math.log(1 - x)
}
function cost_deriv(data) {
    var x = sigmoid(calculatePerceptron(data))
    if (data.class == 1) return -1 / x
    return  1 / (1 - x)
    // not negative because the inner derivative cancels it out
}
function update() {
    var adj = [0, 0, 0]
    for (var i = 0; i < points.length; i++) {
        var p = points[i]
        adj[0] -= cost_deriv(p) * sigmoid_deriv(calculatePerceptron(p)) * p.x
        adj[1] -= cost_deriv(p) * sigmoid_deriv(calculatePerceptron(p)) * p.y
        adj[2] -= cost_deriv(p) * sigmoid_deriv(calculatePerceptron(p))
    }
    perceptron[0] += 0.3 * adj[0]
    perceptron[1] += 0.3 * adj[1]
    perceptron[2] += 0.3 * adj[2]
}

To find the class of any point, check the sigmoid output. 0.5 is right on the line, <0.5 is the 0 class, >0.5 is the 1 class. We can visualize this by trying the perceptron on the whole plane:

Now Buzz can tell whether any section of space is safe or dangerous based on the little data he has. Let’s see how this works on different datasets — random clusters, nested rings, etc.

Like the linear example, sometimes a simple perceptron isn’t complex enough — especially for closed curves like circles, which can never be modeled by a line. In practice we fix this with multi-layer networks, but for the sake of experimentation here are exotic perceptron types tried on the same data:

More complex functions tend to fit more complex data, but even the most complex can’t perfectly separate data that is thoroughly mixed. A common neural-net failure mode: given bad or patternless data, the network gives back confusing predictions full of false positives and negatives.

The quadratic function can model circles and ellipses because we’re using the general formula $a \cdot x^{2} + b \cdot x + c \cdot y^{2} + d \cdot y + e$ , which can shape parabolic and hyperbolic curves too. Plugging that into a sine adds another layer — concentric rings, expanding stars.

That’s it for perceptrons. Read the next article to see how we combine perceptrons to create much more powerful and expressive neural networks.