How does a neuron learn
Say several switches are wired to a single light bulb. Some switches affect the bulb strongly, others only weakly. Add up all their effects, and if the total is big enough, the bulb lights up; if not, it stays off. An artificial neuron works exactly like this: it multiplies each input by a strength, adds them up, and switches on when the sum reaches a set level.
Multiply inputs by weights and add
In the last lesson we saw how to reduce the loss by turning knobs a little at a time. Now it is time to see where those knobs are attached. An artificial neuron is a small unit of calculation that takes several inputs and produces one value, an idea borrowed from the nerve cells of the brain.
A neuron does not simply add its inputs. It sets a strength for each input, multiplies by that strength, and then adds. This strength is called a weight. Important inputs get large weights, and less important ones get small weights.
Tap the inputs to turn them on and off. Only the inputs that are on are multiplied by their weights and added to the sum.
Each time an input was turned on, the sum grew by that input's weight. An input with a big weight moves the sum a lot, while one with a small weight moves it only slightly. The sum reflects not the inputs alone but the inputs and the weights together.
But doesn't this shape, multiplying inputs by numbers and adding, look familiar?
The weight is the slope from regression
Recall how we wrote a straight line as ax+b in linear regression: we multiplied the input x by the slope a and added b. The bigger the slope, the more the result changed for the same input.
A neuron's weight is exactly this slope. Multiplying an input by a weight is the same as multiplying it by a slope, so raising the weight makes that input affect the result more strongly, and lowering it makes the effect weaker. A neuron also has a value corresponding to the b of regression, called the bias.
Raise or lower the weight. Like a slope, it multiplies the input, and the output changes along with it.
As the weight moved, the output followed. Just as changing the slope tilts the line, changing a weight changes how the neuron responds. So the key to making a neuron answer well is, in the end, setting the weights well.
But what does the neuron do after it has the sum? It does not pass the sum straight out; it makes one more decision.
It switches on when the threshold is reached
After working out the sum, the neuron compares it with a reference value called the threshold. If the sum reaches or passes the threshold, the neuron switches on, and the answer it gives is 1. If the sum falls short, the neuron stays off, and the answer is 0.
Just as a bulb lights only when it is bright enough, the neuron switches on only when the sum is big enough. Comparing with the threshold can also be rewritten as adding the bias to the sum and checking whether the result is negative or not.
Raise and lower the sum. When it reaches the threshold the neuron switches on (1); when it falls short, it is off (0).
The moment the sum touched the threshold, the neuron clicked on. The answer is one of two: on or off. In this way a neuron takes inputs and decides whether to switch on or stay off.
But it does not get things right from the start. With clumsy weights, it stays off when it should switch on and switches on when it should stay off. That is a mistake. So how does a neuron improve when it is wrong?
When wrong, adjust the weights a little
Think back to gradient descent, the method of reading how wrong you are and stepping down a little at a time in the right direction. A neuron learns with the same idea. If it should have switched on but stayed off, it raises its weights a little so the sum grows; if it should have stayed off but switched on, it lowers them a little.
It does not change things drastically in one go; it moves one step at a time. Repeat this many times, and it gets better and better. A device that learns this way with a single on-off neuron is called a perceptron.
Look at the wrong answer and tap adjust a bit. The weight grows one step at a time toward the correct answer until it finally gets it right.
The more it was adjusted, the fewer its mistakes. As long as the direction is right, even step by step it ends up near the correct answer. No person sets the weights one by one; the neuron looks at its mistakes and refines its weights itself.
Real neural networks, though, use functions that change smoothly instead of snapping between on and off. With a sudden snap, the slope is flat almost everywhere, which makes it hard for gradient descent to find a direction to adjust in. Functions like these are called activation functions.
Summary
To sum up: a neuron multiplies each input by a weight and adds them up. This has the same shape as ax+b in regression, and the slope a is the weight. If the sum reaches the threshold, it switches on (1); if it falls short, it stays off (0).
When it is wrong, it adjusts its weights a little at a time, as in gradient descent, and by repeating this it gets better and better. Multiply and add, switch on by a threshold, and correct when wrong: that is a neuron.
Tap the key points in order to review. (the sum of inputs × weights → the weight is the slope → past the threshold it switches on → adjust when wrong)
Now you know a small calculating unit modeled on the nerve cells of the brain. This simple unit, which multiplies and adds, switches on and off, and corrects itself when wrong, is the starting point of learning.
A single neuron, however, can only draw a boundary as a straight line. So neurons are connected in layers to let them learn far more complicated things, and the next lesson, on neural networks, looks at what that looks like.