Fixing little by little: gradient descent

Walk down a foggy hill

Earlier we learned about loss.
Loss is the sum of being wrong,
and the smaller it is the better.
So how do we lower the loss?
If we draw loss against the value,
it looks like a valley dipping in the middle.
The valley floor is where loss is smallest.
We just head down toward that floor,
but fog hides the floor from view.
All we can see is the slope under our feet.

Tap to send the ball down one step at a time. It rolls toward the lower side the slope points to, and the loss drops.

Each step down
lowered the ball's height, that is, the loss.
It is fine that the floor is unseen.
If we only know which way the ground tilts,
we always know the downhill direction.
Once we know the direction, next is
how much we actually change the value.
Shall we see how that one step is taken?

Fix the value little by little

Here the value is like a knob
we are allowed to turn.
Turn this knob and the loss rises or falls.
Looking at the slope under our feet
tells us which way to turn the knob
so that the loss goes down.
We change the value a little that way.
Not a big change at once, but little by little.
Then the loss drops a notch.
Repeating this again and again is the key.

Tap the one-step button. The value shifts a little along the slope direction, and the loss drops by that much.

As we take more steps
the value moves toward the valley's middle
and the loss bar grows shorter.
Where the slope is steep
the step looks naturally larger.
Nearer the floor the slope eases,
so the steps grow gentle.
But there is one thing we must decide here.
How large should one step be?

Too big a stride overshoots

The size of one step is called the stride.
In learning we call it the learning rate.
A big stride seems to go down fast,
but there is a trap.
Near the floor, if the stride is too big,
it simply skips over the bottom
and bounces up the opposite slope.
Then it sways left and right
and the loss can even grow.
The other way, too small a stride
is safe but takes ages to go down.

Choose a small or big stride and walk. Small creeps to the floor; too big overshoots and bounces to the far side.

A small stride goes step by step,
while too big a stride overshoots the floor
and bounces side to side, as we saw.
So the stride must be chosen
neither too big nor too small.
A sensible stride is fast
and still settles well at the floor.
Now, with the right stride,
shall we walk all the way to the bottom?

Repeat steps to reach the bottom

Now all the pieces are gathered.
Set the direction by the slope,
take one step with a sensible stride,
look at the slope again, one more step.
Keep repeating this
and the value nears the valley floor.
Nearer the floor the slope flattens almost level,
so the steps shrink on their own.
So in the end it eases to a stop.
That stopping spot is where loss is smallest.

Keep tapping one step at a time. As it nears the floor the steps shrink, converging to the point of smallest loss.

After step upon step
the ball settled right at the valley floor.
The loss bar grew shortest too.
Even in fog, trusting only the slope underfoot,
it walked down one step at a time and reached the bottom.
Without looking far ahead,
by fixing little by little toward the lowest side
from right where you stand, you arrive in the end.
Now let's wrap the whole thing up.

Let's wrap up

Gathered on one line, it is this.
Loss is shaped like a valley low in the middle.
Fog hides the floor,
but the slope underfoot can be felt.
Fix the value a little toward the downhill side
and the loss drops a notch.
Repeat this step and you reach the bottom.
The stride, the size of a step,
overshoots if too big and crawls if too small.
Feel the slope and step down little by little; that is gradient descent.

Tap the key points in order to review. (loss is a valley -> direction by slope -> fix step by step -> stride matters -> converge to the floor)

Now you know the most basic way
to lower the loss.
This knack of feeling the slope and fixing step by step
is the secret of a machine learning on its own.
Nudging countless knobs the same way,
a little at a time, the machine grows smart.
Next, let's keep exploring together
ways to take this step even better.