Eyes for images: the CNN

A photo is really a grid of numbers

To our eyes it just looks like a picture.
But to hand a photo to a computer
we have to turn it into numbers.
Peer very closely at a photo
and you see tiny dots, pixels, packed in a grid.
Each dot holds its brightness as one number.
A bright dot is a big number, a dark dot a small one.
So a single photo
is really a grid with a brightness number in each cell.

Tap the picture to zoom in. Up close, each small dot holds a brightness number, so you can see the photo is really a grid of numbers.

Zoom in and the numbers showed up.
Bright cells are big numbers, dark cells small.
A person sees the shape at a glance,
but to a computer it is just a table full of numbers.
So among all these numbers,
how can it find a feature
like an edge or a curve?
Looking at everything at once is too much,
so it studies small parts one at a time.

Find a feature with a small filter

The tool that finds a feature is the filter.
A filter is a small pane, far smaller than the grid.
It is like setting a magnifier on one spot.
For example, a filter that finds a vertical edge
likes a pattern dark on the left, bright on the right.
Place the filter on one window of the image
and check how well it matches the numbers there.
If the pattern fits well, the score is high,
and if it does not fit, the score is low.

Place the filter on different windows and watch the score. Where the patch resembles the filter (dark left, bright right), the score is high, meaning a vertical edge sits there.

Each spot you tried gave a different score.
Where the pattern fit, the score was high.
That means a vertical edge is right there.
One thing matters most.
Wherever you place the same filter,
it looks for the same feature.
So whether the edge is on the left of the picture
or on the right, it can catch it.
Then instead of one spot, let's scan them all.

Slide the filter to make a feature map

Now move the filter one step at a time.
Start at the top left,
one cell right, then one cell more.
Reach the end, drop down a row,
and scan again from the left.
Write the score at each spot,
and the scores gather into a new small grid.
This is the feature map.
It shows at a glance where the feature is strong.

Tap to slide the filter one step at a time. Each spot's score fills the feature map on the right. When the filter finishes, the map reveals where the feature is strong.

The filter swept across the whole grid.
The scores gathered into a feature map.
It is smaller than the original photo,
yet it shows clearly where the feature lies.
One filter finds only one kind of feature.
So in practice we use many filters together.
Vertical edges, horizontal edges, curves,
finding different features all at once.
But these are still only small features.

Higher layers, bigger features

Run a filter over the feature map once more
and small features gather into bigger ones.
The front layer sees small pieces like dots and edges,
the layer above gathers edges into an eye or a nose,
and a higher layer gathers eyes and nose into a whole face.
It is just like stacking layers in a neural network.
The difference is that here each layer scans with filters.
So the higher you climb,
the more abstract the features it catches.

Tap the button to climb one layer at a time. From edges to parts like an eye and a nose, then to a whole face, abstraction rises step by step.

Each layer up, what it sees grew bigger.
From edges to eyes and nose, then to a face.
Low layers see small, simple pieces,
and high layers gather them into meaningful shapes.
It is much like how a person recognizes things.
See lines, recognize parts, know the whole.
Scanning with small filters to find features
and stacking those layers to raise abstraction
is exactly the skill of a CNN that looks at images.

Let's wrap up

Gathered on one line, it is this.
A photo is really a grid filled with brightness numbers.
A CNN places a small filter, a magnifier, on one window
and scores whether that feature is there.
Sliding the same filter across the grid
makes a feature map that finds it wherever it sits.
Stacking many such layers,
from edges to eyes and nose, then to a face,
it catches ever more abstract features. That is a CNN.

Tap the key points in order to review. (a photo is a number grid -> a small filter scores a feature -> scan to make a feature map -> higher layers, more abstract features)

Now you know how a computer sees a picture.
Turn the photo into a grid of numbers,
scan it with small filters to find features,
and stack layers to raise abstraction step by step.
A cat that shifted is still caught by the same filter.
This simple idea, scan with a magnifier and gather features,
is the foundation of clever machines that recognize images.
Next, let's keep exploring how this way of seeing
applies to text and sound as well.