When one machine is not enough, split it up

One machine stops being enough

Earlier we learned a database
is a giant warehouse.
It stored safely and found fast.
But information piles up without stopping.
People grow, records grow,
photos and videos pour in too.
No matter how big one warehouse is,
someday it fills up and can hold no more.
A moment comes when one machine is not enough.

Tap to keep adding data. Once it passes one machine's capacity, no more fits.

One machine got completely full.
You want to add more but there is no room.
Swap in a bigger computer
and even that fills up again someday.
Growing one machine endlessly
has a clear limit.
So we should change our approach.
Instead of growing one machine,
what if we grow into several?

We split it across several machines

The way is simpler than you'd think.
Cut the data into pieces
and scatter them across several machines.
Instead of one machine groaning under it all,
each machine takes only a part.
For example, names A to M on machine 1,
N to Z on machine 2, that kind of split.
Chopping the load and sharing it out
is called distribution.
It is also called sharding.

Tap a piece to split it across the machines. Each machine holds only part of the whole.

The load scattered across machines.
The burden one machine carried got lighter.
When data grows more,
you just bolt on one more machine.
Then the new pieces go to the new machine.
Since you can keep growing like this,
you can hold even very big data.
But one worry creeps in.
What if that one machine suddenly dies?

We keep a copy of the same thing

Splitting has one weak point.
If the machine holding a piece dies,
that whole piece vanishes.
So we copy the same data
onto several machines.
Even if one machine dies,
another machine with a copy takes over.
So the data does not break.
Keeping the same thing copied across machines
is called replication.

Tap copy to place the same data on several machines. Turn one off and another stays alive, so it does not break.

Turning one off, it stayed fine.
Thanks to the copy standing by.
Here we must tell two things apart.
Distribution, which splits, chops pieces
and shares them to different machines;
replication, which copies, keeps the same thing
on several machines, identically.
Distribution is splitting; replication is copying.
They look alike but are wholly different jobs.

A looser container, NoSQL

But when data is enormous
and must rush in and out very fast,
a strict table gets in the way instead.
The rule of lining slots up in order
holds the speed back.
So we use a container with loosened rules.
Instead of a table, hang one value on one key
and put in and take out straight.
It is the hash we saw earlier, grown big.
Such a loose container is called NoSQL.

Tap to compare a strict table with a loose key-value container. The loose one suits huge and fast handling.

The loose container,
with fewer rules, is free and fast.
No need to align slots like a table;
with just a key you put in and take out a value.
In return, the tidiness of lined-up rows
you let go of a little.
What to gain and what to let go,
you choose to fit the use.
For huge and fast data,
this loose container fits well.

Let's wrap up the data area

We've walked a long way to here.
We began with a container for values,
saw the database that gathers those containers,
and learned to find the row you want
fast in a giant table.
The index that builds a shortcut,
the relations that link tables,
and handling that stays safe if it cuts off midway.
And today we saw, when one machine is short,
how to split it up and keep copies.

Tap the big threads of the data area in order to close it. (container -> database -> fast finding -> index -> relations -> safe handling -> split and copy)

Gathered on one line, it is this.
When data is too much for one machine,
you cut it and split across several (distribution),
and copy across several so it does not break (replication).
Splitting and copying are different jobs.
For huge and fast handling,
you use NoSQL, a container that loosens the table.
This is the hash grown big,
so it is the bridge leading to the enormous data
of the artificial intelligence we meet next.

Now the big picture of handling data
comes into view at a glance.
From one small container we came
all the way to a vast operation
that splits and copies across machines.
Storing information safely,
finding it fast, and growing it big,
this craft becomes the ground
that holds up the next area, artificial intelligence.
Where data gathers,
a thinking machine grows.