How does gradient descent work

Going down a foggy hill

In the last lesson we said the loss is a number that measures how wrong a model is, and that smaller is better. So how do you reduce it? Plot one of the model's values on the horizontal axis and the loss at that value on the vertical axis, and you get a valley shape that dips in the middle.
The bottom of the valley is where the loss is smallest. You just need to head down toward it, but with a real model you cannot see this whole picture in advance. All you can know is the slope right where you are standing.

Tap step down. The ball rolls toward the lower side that the slope underfoot points to, and the loss drops.

With every step down, the ball's height, that is, the loss, went down. It does not matter that you cannot see the bottom: as long as you know which way the ground slopes, you always know which way is down.
If the downhill side is to the right, increase the value; if it is to the left, decrease it. Now that you know the direction, the next question is how much to actually change the value.

Adjusting the value little by little

The value here is like a knob you can turn. The slope and intercept of the line we saw earlier are exactly this kind of knob. Turning the knob makes the loss go up or down, and the slope underfoot tells you which way to turn it to make the loss smaller.
You change the value just a little in that direction. Put into words, the new value is the current value minus the slope multiplied by a small number, which means moving opposite to the slope, downhill. Repeating this is the heart of it.

Tap to adjust the value a bit. The value shifts a little in the direction the slope points, and the loss drops by that much.

With each step, the value moved closer to the middle of the valley and the loss bar grew shorter. Where the slope is steep, it moves a lot at once; nearer the bottom, the slope flattens and the steps get smaller.
That is because the size of each correction is proportional to the slope. But one thing still needs deciding: that small number the slope is multiplied by, the size of a single step. How big should it be?

Too big a stride overshoots

The number that sets the size of each step is called the learning rate. A large learning rate seems like it would get you down faster, but there is a trap. Near the bottom, if the step is too big, you pass right over the bottom and climb up the slope on the other side.
Then you may swing back and forth, and the loss can even grow. On the other hand, a learning rate that is too small is safe but takes ages to reach the bottom.

Pick a small stride or a big stride and walk. A small one goes steadily down to the bottom; one that is too big jumps over the bottom and bounces to the other side.

The small stride went steadily down, while the too-big stride overshot the bottom and bounced from side to side, getting farther away. So the learning rate has to be neither too big nor too small.
In practice, people try several values to choose one, and they may also shrink the learning rate little by little as training goes on. Now let's walk all the way down with a suitable stride.

Repeating steps to reach the bottom

Now the method is complete: use the slope to choose the direction, take a step of a suitable size, look at the slope again, and take another step. Keep repeating this, and the value approaches the bottom of the valley.
The closer it gets to the bottom, the flatter the slope becomes, so the steps shrink on their own, and at the end it barely moves at all. Settling into one place like this is called converging.

Keep tapping one step. Nearer the bottom the steps get smaller, and the value converges on the point where the loss is smallest.

Step after step, the ball settled at the bottom of the valley, and the loss bar became as short as it gets. Even in the fog, trusting only the slope underfoot, it came down one step at a time and reached the bottom.
The loss of a real model is often not shaped like this picture, with just one valley. If there are several valleys, it may stop in a shallow valley nearby instead of the deepest bottom. And because there are millions of knobs, the slope is measured separately for each knob and all of them are adjusted at once. Measuring the slope on small batches of the data at a time is also widely used.

Summary

To sum up: plot the loss against a value and you get a valley. You cannot see the bottom, but you can measure the slope underfoot, so changing the value a little toward the downhill side reduces the loss. Repeat this step and you approach the bottom.
The learning rate, the size of each step, overshoots the bottom if it is too big and is slow if it is too small. Measuring the slope and going down one step at a time: that is gradient descent.

Tap the key points in order to review. (the loss is a valley → the slope gives the direction → adjust step by step → the stride matters → converge to the bottom)

Now you know the most basic way to reduce the loss. This method of measuring the slope and correcting things one step at a time is the secret of how machines learn on their own.
So where are these knobs actually attached? The next lesson looks at the neuron, the smallest unit that makes up a neural network, and at how a neuron learns by adjusting its knobs, called weights and a bias.