Tracing the error backward: backpropagation

First, predict forward

To make an answer, a network flows signals forward.
Starting from the input,
it passes through each layer in turn,
and a prediction comes out at the output.
It is the forward flow we saw last lesson.
But a freshly built network
has its weights set somewhat at random,
so the prediction can be far from the right answer.
Let's look at this forward flow first.

Tap to flow the signal forward one layer at a time, from input to output. A prediction comes out at the end, but it still differs from the answer.

A prediction came out, but it differs from the answer.
This gap is exactly the error.
Measuring how wrong it is as a number
is what we call the loss.
A big loss means very wrong,
and a loss near zero means a good match.
Our goal is to shrink this loss.
But a network has countless weights.
Which one should we fix to lower the loss?

Who is to blame for the error?

The output is wrong.
But the output is not wrong alone.
The output made its answer from the signal of the layer just before,
and that layer made its answer from the one before it.
So the blame for the error
does not sit on one weight,
it is split a little
across every weight that took part in the prediction.
First let's check the size of the error at the output.

Compare the output's prediction with the answer. Tap to mark the error (loss), and it becomes clear we must trace backward to find where this error came from.

We saw the size of the error at the output.
Now we must travel back up this error.
Think about it.
When several connections feed the output,
some sent a large signal
and some sent a small one.
The larger a connection's signal,
the more it contributed to the wrong output.
So that connection's blame is bigger too.

Send the error back, in reverse

Now comes the real backpropagation.
We send the error backward,
from output toward input, one layer at a time.
The exact opposite direction from going forward.
It is like following the fallen dominoes in reverse
to find who tipped the very first one.
Each time we step back one layer,
every connection in that layer
gets its share of how much it caused the error.

Tap to send the error backward one layer at a time, from output toward input. As it travels back, each connection gets a blame share. Thicker means more blame.

We traced all the way back.
Now every weight in the network
has received its own share of blame.
Some weights contributed greatly to the error,
and some carry almost no blame.
The striking part is that from one error at the output,
the shares of even the deep inner weights
were all worked out.
Thanks to following the dominoes in reverse.

Fix each weight by its share

Now that we know the blame, it is time to fix.
Recall gradient descent from the earlier lesson.
That method of stepping down a little,
in the direction where loss decreases.
We nudge each weight by its share.
Weights with a big share change a lot,
weights with a small share change just a little.
We do not fix it all at once,
fixing a little at a time is what matters.

Tap adjust to fix each weight by its share, one step per tap. Each tap shrinks the loss bar, and predicting forward again lands closer to the answer.

The more we adjusted, the smaller the loss.
But fixing once is not the end.
We predict forward again,
measure the error again,
trace back for the shares again,
and adjust by that much again.
Repeat this one round countless times
and the network's prediction creeps toward the answer.
This is how a network gets better on its own.

Let's wrap up

Gathered on one line, it is this.
First flow the signal forward to make a prediction.
Compare with the answer and measure the error as loss.
Send that loss from output toward input,
backward, to find each weight's share.
Then adjust by that share with gradient descent.
Repeat this predict forward, trace backward, adjust
countless times, and the prediction nears the answer.
It is following the dominoes in reverse to find the culprit.

Tap the key points in order to review. (predict forward -> measure the error as loss -> trace back for shares -> adjust by the share)

Now you know how a neural network fixes its errors.
It flows forward to make an answer,
and when wrong it traces the error backward,
figures out who is to blame,
and fixes each weight a little by that much.
Thanks to this simple idea of tracing in reverse,
even a large network with countless weights
can learn on its own.
This is exactly the secret behind how today's clever machines learn.