How does a neural network work

Connecting neurons layer upon layer

A single neuron gathers its inputs with weights, adds them up, and sends out a signal depending on the sum. Being simple, its judgment is simple too; splitting things with one line is about all it can do.
But connect one neuron's output to another neuron's input, and things change. Passing signals to each other, the neurons build up together a bigger picture than any one of them could see.

Tap the neurons one at a time to switch them on. Signal lines connect the neurons that are on, and simple neurons form a network.

Each neuron is still simple. What has changed is that they are connected. Since one neuron's output becomes the next neuron's input, several of them together begin to see what none could see alone.
Rather than connecting them any old way, though, lining them up makes the calculation much cleaner. These lines are called layers.

Input layer, hidden layers, output layer

Line neurons up and you get a layer. The first layer, which receives values from outside, is the input layer, and the last layer, which gives the final answer, is the output layer. The layers in between cannot be seen directly from outside, so they are called hidden layers.
Usually each neuron in a layer is connected to every neuron in the layer before it. The signal starts at the input layer and flows only forward, through the layers in order, to the output layer.

Tap the button to send the signal forward one layer at a time. It lights up in order, from the input layer through the hidden layer to the output layer.

The signal flowed only forward. Each layer takes the previous layer's output, calculates again, and passes the result to the next layer. In a network that recognizes photos, the early layers look at small pieces such as lines and edges, and the later layers put those pieces together to recognize bigger shapes such as eyes or ears.
The deeper the layer, the more abstract the features it picks up. Learning with deep networks that have many hidden layers is called deep learning. But there is one thing that must not be left out when stacking layers.

A bending step is needed in between

It seems that stacking layers should be enough, but there is a trap. If each layer only multiplies and adds, it is a straight-line transformation. However many straight transformations you chain together, the result is just another straight transformation, so even a hundred stacked layers are no different from one.
So a step that bends the signal must be placed between layers. The activation function from the last lesson is exactly this step. A common choice today is a simple function that turns any negative input into zero and passes positive inputs through unchanged.

Turn the bending step on and off to compare. With it off, stacked layers merge into one straight line; with it on, the bent lines survive and the shapes become richer.

The difference was plain. With the bending step off, however many layers you stack, it is still a single straight line, able to do no more than one layer. With the bending step on, the signal bends a little at each layer, producing shapes that straight lines could never draw.
This bending step is what makes stacking layers worthwhile.

More layers, more complex patterns

Let's add more layers that have bending steps. With one layer, the network can only split the world in two with one straight line. Add layers and neurons, and the bent lines combine to draw boundaries that kink and curve.
Stack more still, and it learns ever more complex patterns, up to circles and winding boundaries. It is known that with enough neurons, a neural network can imitate a boundary of almost any shape. That is why neural networks are so powerful.

Tap add a layer. With few layers the boundary is a straight line; the more you add, the more it curves to wrap around complex groups.

Each added layer made the boundary freer, changing from a straight line to a kinked line and then a smooth curve. With simple neurons forming layers and bending steps placed in between, they learn together patterns none could dream of alone.
There is a cost, of course. The more layers and neurons there are, the more weights there are to adjust, and the more data and computation training needs.

Summary

To sum up: simple neurons are stacked and connected in layers, and the signal flows from the input layer to the output layer. Each layer takes the previous layer's output and picks out bigger features.
There must be a bending step between layers, because otherwise everything merges into a single straight line. Stacking layers and bending in between to learn complex patterns: that is a neural network.

Tap the key points in order to review. (stack neurons in layers → the signal flows forward → a bending step is needed in between → more layers, more complex patterns)

Now you know how simple neurons come together to learn complex patterns. Neurons that could only draw one straight line each, once formed into layers with bending in between, learned even curved boundaries.
With many layers, though, a problem appears. When the output is wrong, how do you know where among all those weights the blame lies, and how much? The next lesson, on backpropagation, covers how to share out the error by working backward from the output.