What is a loss function
Earlier we drew a prediction line through points. But how do you know whether it is a good line? If it follows the points closely, it is good; if it misses them by a lot, it is bad. Instead of judging that by feel, you can measure it with a single number and agree that the smaller the number, the better the line. This number, which measures how wrong a model is, is called the loss.
How far off is one point?
We have drawn a prediction line. The points are the real answers, and the line is the model's prediction. Pick one point, and at that point's horizontal position, the height the line predicts and the point's actual height may differ.
However far apart the two are, that is how wrong the model is. Let's first check this mismatch with our own eyes.
Tap a point. The vertical gap between the point (the answer) and the line (the prediction) is shown.
Each point had a different gap. Some points sit almost on the line, while others are far from it. A point close to the line was predicted well; a far one was missed by a lot.
But just by looking, it is hard to tell how wrong the whole thing is. The gaps need to be turned into numbers.
Measuring the gap as a distance
Turning a gap into a number is simple: measure the vertical distance from the point to the line, as it is. It is the very difference we called the residual earlier.
A long distance means a big error, and a short one means a small error. Whether the point is above the line or below it, it is wrong all the same, so the distance is always measured as a positive number.
Tap a point. The distance from the point to the line is measured as a bar, and its length appears as a number.
Each point's distance came out as a number: a small number for a close point, a big one for a far point. Now you can tell clearly how wrong each point is.
But there is more than one or two points. With a separate number for each, it is hard to see at a glance how wrong the whole is. Let's gather these numbers into one.
Adding up all the gaps
To see how wrong the whole thing is, add up the distance measured at every point, leaving none out. Small gaps and big gaps all go into one pile. The scattered numbers then combine into a single number, and that number is the loss.
In practice it is often divided by the number of points to take an average, so that the loss of data with ten points can be compared with that of data with a thousand. Instead of adding the distances as they are, squaring them before adding is also widely used. This is called the mean squared error, and it gives a heavier penalty to points that miss by a lot.
Tap the points in turn. The gaps pile up one by one and add into a single number called the loss.
The scattered gaps have gathered into a single number, the loss. Now the model is easy to evaluate: a big loss means the line is far off the points, and a small loss means it fits well.
Problems that answer with a group, such as classification, use a different loss, one that grows larger the lower the probability given to the correct group. Whatever is measured, the job is clear: make this number small.
The goal is a small loss
The loss is not a fixed number; it changes when the line moves. Fit the line closer to the points and the gaps shrink, making the loss smaller; move it away from the points and the loss grows.
So you move the line around and find the spot where the loss is smallest. The line in that spot is the best line.
Move the line with the up and down buttons. The gaps and the loss change together, and you are told when the line reaches the spot where the loss is smallest.
Every time the line moved, the loss went up and down. The better the line fit the points, the smaller the number became, and the smallest spot was the best-fitting line.
This reveals something important: finding a good model is the same as making the loss small. What machine learning calls learning is exactly this process of reducing the loss.
Summary
To sum up: at each point there is a mismatch between the prediction and the answer, that is, a vertical distance. The single number you get by adding or averaging these distances is the loss, which tells you how wrong the model is.
The smaller the loss, the better the model. Moving the line to reduce the loss is learning.
Tap the key points in order to review. (a gap at each point → measure it as a distance → add them all into the loss → reducing the loss is learning)
Now you know what the loss is and why we try to reduce it. Capturing how wrong a model is in a single number is the first step in teaching a model.
But to reduce the loss, which way should the line move? Here we only moved it up and down with two buttons, but a real model has millions of numbers to adjust. The next lesson, on gradient descent, covers how to find the direction in which the loss decreases and step down it little by little.