How does backpropagation work
In the last lesson we saw a neural network flow signals forward to produce an answer. But what if that answer is wrong? A network has countless weights, and to fix it you need to know which of them are responsible for the error, and by how much. The method is to trace the error backward, from the output toward the input, working out each weight's share of the blame. Much like following fallen dominoes backward to find where the first push came from, this method is called backpropagation.
First, predict forward
When a neural network produces an answer, it flows signals forward. Starting from the input, the signal passes through the layers in order, and a prediction comes out at the output. It is exactly the flow we saw in the last lesson.
A freshly made network has its weights set more or less at random, so its prediction can be far from the right answer. Let's look at this forward flow again first.
Tap the button to send the signal forward one layer at a time, from input to output. A prediction comes out at the end, but it does not match the answer yet.
A prediction came out, but it differs from the answer. That difference is the error, and measuring how wrong it is as a number gives the loss. A large loss means a big miss; a loss close to 0 means a good fit.
The goal is to reduce the loss. But a network has countless weights. Which of them should be changed, and by how much, to make the loss go down?
Who is responsible for the error?
The output was wrong. But the output was not wrong on its own: it gave its answer from the signals it received from the layer just before it, and that layer calculated from the signals of the layer before that.
So the responsibility for the error does not lie with one weight alone; it is spread in small portions across every weight that played a part in the prediction. First, let's check how big the error is at the output.
Compare the output's prediction with the answer. Tap the button to mark the error (the loss), and you can see that we must trace backward to find where this error came from.
We have checked the size of the error at the output. Now that error has to be traced back upstream. If several connections lead into the output, some will have sent a strong signal and others a weak one.
The stronger the signal a connection sent, the more it contributed to the wrong output, and so the bigger its share of the blame. Put another way, changing that connection's weight a little changes the loss a lot. How much the loss changes when a weight changes is exactly the slope we saw in gradient descent.
Sending the error backward
Now comes backpropagation proper. The error is sent backward, one layer at a time, from the output toward the input, the exact opposite of the forward direction. It is like following fallen dominoes in reverse to find where they started to fall.
Each time you step back a layer, how much that layer's connections contributed to the error is settled. An inner layer's share is found by multiplying the outer layer's share by the effect of the connections in between. This rule of calculating backward by multiplying as you go is called the chain rule of differentiation.
Tap the button to send the error backward one layer at a time, from output toward input. With each step back, every connection gets its share of the blame; the thicker it is, the more blame it carries.
We have gone all the way back. Now every weight in the network has received its own share of the blame. Some weights contributed a lot to the error, and some have almost no responsibility.
What is remarkable is that, starting from the single error at the output, the shares of even the deep inner weights were all calculated in one pass. Since there is no need to test each weight separately, the calculation stays manageable even in networks with millions of weights.
Adjusting weights by their share
Now that we know who is responsible, it is time to fix things. Think back to gradient descent, the method of stepping down a little at a time in the direction that reduces the loss.
The share found by backpropagation is exactly each weight's slope, so each weight is adjusted by its slope multiplied by the learning rate. Weights with more blame change a lot, and weights with little blame change only a little. The important thing is to fix things a little at a time rather than all at once.
Tap adjust by share to fix the weights one step at a time. With each tap the loss bar shrinks, and the weights with more blame move more.
The more we adjusted, the smaller the loss. But one round of fixing is not the end. Predict forward again, measure the error, trace back to find the shares, and adjust by that much once more.
Repeat this round countless times, and the network's predictions get closer and closer to the right answers. Almost every neural network today learns this way.
Summary
To sum up: first, flow the signal forward to make a prediction, and compare it with the answer to measure the error as a loss. Then send that loss backward from the output toward the input, finding each weight's share of the blame, that is, its slope.
Finally, adjust the weights by that share with gradient descent. Repeating predict forward, trace back, and adjust brings the predictions closer to the answers.
Tap the key points in order to review. (predict forward → measure the error as a loss → trace back to find the shares → adjust by the share)
Now you know how a neural network fixes its mistakes. Thanks to this idea of tracing backward, even large networks with countless weights became able to learn on their own.
But is it good to keep reducing the loss on the training data forever? Just as a student who memorized only the answers to the practice problems gets lost on an unfamiliar test, a neural network can fit its training data too closely. The next lesson, on overfitting, takes up this problem.