How does a CNN work

A photo is a grid of numbers

To us it is a picture, but to hand a photo to a computer we have to turn it into numbers. Look very closely and you will see tiny dots called pixels, packed in rows like the squares on graph paper. In a black-and-white photo each dot stores its brightness as one number: a bright dot gets a big number, a dark dot a small one.
A color photo stores three numbers per dot, for the strength of red, green and blue. Either way, a photo is a grid with a number in every cell.

Press the zoom button. What looked like a handwritten digit turns, up close, into a brightness number in every cell, and the photo shows itself as a grid of numbers.

Zoomed in, every cell showed a number. You can tell at a glance which digit it is, but the computer receives nothing except a table crowded with numbers. How can it find features like edges or curves in that pile?
There is too much to take in at once, so it starts with small patches and works through them one by one.

Finding a feature with a small filter

The tool for finding a feature is called a filter. A filter is a small panel of numbers, much smaller than the image, and using it is a bit like holding a magnifying glass over one spot. A filter for vertical edges, for example, responds to a pattern that is dark on the left and bright on the right.
You lay the filter over one patch of the image and compute a score for how closely the numbers there match its pattern. In this widget the score is the brightness of the right two cells minus the brightness of the left two. A good match gives a high score, a poor match a low one.

Pick a spot on the grid to place the filter. Where the patch is dark on the left and bright on the right, like the filter, the score is high, which means a vertical edge runs through that spot.

Every spot gave a different score. The highest scores came where dark cells met bright ones, and that is exactly where the vertical edge lies. Where the brightness was even, the score stayed close to zero.
One point matters a lot here. The same filter looks for the same feature wherever you place it, so it catches an edge on the left side of the picture just as well as one on the right. There is no need to stop at one spot, then; the filter can sweep the whole grid.

Sliding the filter to build a feature map

Now the filter moves one cell at a time. It starts at the top left and steps to the right; at the end of a row it drops down one row and starts again from the left. Write each score in the matching position, and the scores form a new, smaller grid.
This grid is called a feature map. One look at it tells you where the feature is strong.

Press the slide button to move the filter step by step. Each score fills the same position on the feature map, and when the sweep is done, the spots with a strong vertical edge stand out brightly.

Once the filter has swept the whole grid, the feature map is complete. A five-by-five grid produced a four-by-four map, so it is a little smaller than the original, yet the column where the vertical edge runs lights up clearly. A single filter finds only one kind of feature.
That is why a real CNN uses tens or hundreds of filters at once, each with its own pattern: vertical edges, horizontal edges, curves and more. And no person writes the numbers inside those filters. The network learns them with the gradient descent and backpropagation we met earlier, nudging them little by little to shrink the error. Picking features, which used to be a person's job, becomes something the network does for itself.

Higher layers, bigger features

Sweep filters over the feature map again and small features combine into bigger ones. The first layer catches small pieces such as short edges, the next layer combines edges into parts like an eye or a nose, and a layer above that puts the parts together into a whole face.
It is the same idea as stacking layers of neurons, except that every layer of a CNN scans with filters. The higher the layer, the larger the area of the original photo that each cell stands for, so the features it catches grow more abstract.

Press the button to go up one layer at a time. What the network sees grows from edge pieces to parts like an eye and a nose, and then to a whole face.

With every layer, what the network saw grew larger. Low layers look at small, simple pieces; high layers weave those pieces into shapes that mean something. Seeing lines, then recognizing parts, then grasping the whole is much like the way people recognize things.
When researchers look inside a trained CNN, they often find that filters in the lower layers respond to edges and color boundaries, while filters in the higher layers respond to parts such as eyes or wheels. The face in the widget is a simplified picture of that progression.

Wrapping up

In short, a photo is a grid with a brightness number in every cell. A CNN places a small filter on one spot and scores whether the feature it looks for is there. Sweeping the same filter across the grid produces a feature map that finds the feature wherever it is.
Stack several such layers and the features grow more abstract, from edges to eyes and noses to faces. The numbers inside the filters are learned from data.

Tap the key points in order to review. (a photo is a grid of brightness numbers → a small filter scores a feature → sweeping builds a feature map → higher layers, more abstract features)

Now you know how a computer sees a picture. It turns the photo into a grid of numbers, sweeps small filters across it to find features, and stacks layers to catch ever larger ones. Because the same filter covers the whole photo, it still finds the cat after the cat has moved to one side.
Words and sentences, though, do not arrive as a grid of numbers the way photos do. The next lesson covers how to turn words into numbers: embeddings.