How does CPU cache work

The desk and the bookshelf

Think about studying.
Only a few books fit on the desk within reach, but you can grab them instantly.
The bookshelf holds many more, but you have to get up and walk over.
Memory is the same: close means fast and small, far means big and slow.

Near and far: what you gain and what you give up.

You can't have both.
Memory that's both fast and big is too expensive or too hard to make.
So you have to choose: keep a little that's fast, or a lot that's slow.

Stack it in order of closeness

That's why a computer doesn't use just one kind of memory.
It stacks layers, from tiny, fast memory closest to the CPU to big, slow memory far away.
The higher you go, the faster and smaller; the lower you go, the bigger and slower.

Fast and small at the top, big and slow at the bottom.

The memory we saw in Lesson 9 is one of these layers.
Above it sit the faster cache and registers, and below it is the bigger, slower storage.
The idea is to keep things you use often up high and things you use less down low.

Grab it from the desk, or walk to the shelf?

When the CPU needs a value and it's in the nearby cache, it gets it in the blink of an eye.
If it has to go all the way to memory, it takes that much longer.
The same value can take very different amounts of time depending on where it is.

Near vs far: how long it takes to fetch.

So a computer moves values it just used, or will probably use again soon, into the nearby cache.
That way it doesn't have to go far the next time it needs them.
A little preparation saves a lot of time.

On the desk or not?

If the value the CPU wants is already in the nearby cache, that's a "hit," and it's fetched right away.
If not, it's a "miss," and the CPU has to make a trip all the way to memory.
That's why keeping frequently used values in the cache matters.

The path splits depending on whether the value is in the cache.

The more hits, the faster the computer.
That's why "keep what you use often close by" is the key rule for handling memory.
It's about using a small cache cleverly.

Why not make it all fast?

Fill it all with fast memory and it's too expensive and can't hold much.
Fill it all with big memory and it's too slow.
Layers are the compromise between the two.
Pick each of the three options below and you'll see why.

All fast · all big · layers.

Things you use often go in the small, fast upper layers; things you use less go in the big, slow lower ones.
Thanks to this split, a computer is fast and still remembers a lot.
In the next lesson, we'll see how all these parts connect into one computer.