Words into numbers: embeddings

Give each word a coordinate

Inside a computer there are no letters.
Only numbers.
Yet we speak and think in words.
So how do we put a word into a computer?
The way is to give each word a coordinate.
Think of it as a single point on a map.
The word cat goes here,
the word dog goes there,
so every word gets its own spot.

Tap a word. The word that was just letters lands as a single point in meaning space, that is, a number coordinate.

You tapped a word and a point appeared.
The position of that point is the coordinate.
A coordinate is a bunch of numbers, like x and y.
Now that the word is a number,
the computer can finally handle it.
But if we drop points just anywhere,
the coordinates would mean nothing.
So we add one more promise.
Words with similar meanings sit close together.

Similar meanings gather close

The key promise of embedding is simple.
Words with similar meanings sit close.
cat and dog, both animals, sit near,
king and queen, both royalty, sit near.
But cat and car,
though they sound alike, mean different things, so they sit far.
So in this meaning space
distance is the likeness of meaning.
Close means similar, far means different.

Tap the arrange button. The scattered words gather by meaning group, close within a group and far from other groups.

You tapped and the words snapped together.
Animals with animals, royalty with royalty.
Close within a group,
far from the other groups.
The striking part is, the computer has no idea
from the letters that cat is an animal.
It only looks at distance between coordinates.
Yet if the coordinates are set well,
that distance alone reveals similarity.

Add and subtract coordinates

Now that words are coordinates, something fun appears.
Coordinates are numbers, so you can add and subtract.
And that math carries meaning.
From king, subtract the man direction.
What is left is the feeling of royalty.
Add the woman direction to that,
and amazingly you arrive near queen.
From king take away man and add woman, and you get queen,
navigation on coordinates becomes navigation of meaning.

Tap the step buttons in order. Follow the path of king minus man plus woman, and the arrow lands near queen.

The arrow really reached near queen.
The computation was just coordinate add and subtract.
Yet the result fits the meaning exactly.
That is because the gap between king and queen
resembles the gap between man and woman.
If the coordinates are learned well,
such directions come to hold meaning.
Of course it is not always perfectly exact.
Still, just landing nearby is amazing.

Not only words become coordinates

Want to see why this idea is so powerful?
Turning things into coordinates is not just for words.
A single photo can become a coordinate too.
Similar photos gather close.
A voice or a song becomes a coordinate as well.
Even a person's taste can be placed as a coordinate.
Then similar tastes draw near each other.
Turn anything into coordinates and measure closeness,
and a computer can handle similarity.

Tap to switch the kind. Words, images, sounds, or users all become points the same way, and similar ones gather close.

Even switching kinds, the picture stayed the same.
Points appear, and similar ones gather close.
That is why a music app suggests similar songs,
and a photo app finds lookalike pictures.
Underneath, it is all measuring distance between coordinates.
Put different kinds in the same space
and you can even find a photo from a word.
With one shared language of coordinates,
all sorts of things are handled together.

Let's wrap up

Gathered on one line, it is this.
Since a computer knows only numbers,
it turns a word into a coordinate, a bunch of numbers.
That is embedding.
The key promise is to place similar meanings close.
So distance becomes the likeness of meaning.
Because they are coordinates, you can add and subtract,
and that math oddly carries meaning.
And not just words, but images, sounds, and users work the same way.

Tap the key points in order to review. (words into coordinates -> similar meanings stay close -> coordinate math carries meaning -> images and sounds work the same)

Now you know how a computer begins
to handle the meaning of a word.
Turning letters into coordinates
and gathering similar ones close, a simple idea,
actually carries us very far.
Search, translation, recommendation, linking pictures and text,
look underneath and this embedding lies beneath them all.
That single first step of turning words into numbers
became the foundation of today's clever machines.