What is an embedding in AI
In the last lesson, a CNN took a photo as a grid of numbers and found features in it. A photo is made of brightness numbers from the start, but words are different. Stare at the letters c-a-t as long as you like and you will find no number that stands for what the word means. To hand a computer the meaning of a word, we give each word a numeric coordinate, arranged so that words with similar meanings land near each other. Turning words into meaning coordinates like this is called embedding.
Give each word a coordinate
A computer calculates only with numbers. We speak and think in words, though, so words have to become numbers somehow. The simplest way is to number the words, but a numbered list cannot capture the fact that a cat is more like a dog than like a car.
So each word gets a coordinate, like a point on a map: cat over here, dog over there, every word with a place of its own.
Tap the words one at a time. Each word, until now just letters, becomes a point in meaning space, and its position appears as a pair of numbers.
Tap a word, and a point appears with two numbers beside it: its horizontal and vertical position. That bundle of numbers is the coordinate. Now that the word is a set of numbers, the computer can work with it.
The widget uses two numbers so it is easy to draw, but real embeddings give each word hundreds of numbers. Points dropped just anywhere would mean nothing, though, so we add one more rule: words with similar meanings sit close together.
Similar meanings gather close
Under this rule, cat and dog sit close because both are animals, and king and queen sit close because both are royalty. Cat and car, on the other hand, differ by a single letter but mean completely different things, so they sit far apart.
In this meaning space, distance measures how alike two meanings are. Close means similar; far means different.
Tap the arrange button. The scattered words gather into groups of animals, royalty and fruit, and the groups move away from one another.
The words gathered by group: close within a group, far from the others. So who decides these coordinates? Not a person setting them word by word. The computer has no idea from the letters that a cat is an animal; it only sees how words are used across a huge amount of text.
Cat and dog keep turning up next to similar words, such as feed, fur and vet. Train the coordinates so that words appearing in similar contexts move closer, and distance alone becomes a measure of how alike their meanings are.
Calculating with coordinates moves meaning
Once words are coordinates, something remarkable happens. Coordinates are numbers, so you can add and subtract them, and the arithmetic carries meaning. Start at king and subtract the man direction, and what is left is a sense of royalty; add the woman direction, and you arrive, surprisingly, near queen.
King minus man plus woman is roughly queen. Finding your way across the coordinates becomes finding your way through meaning.
Tap the step buttons in order. Follow the path of king minus man plus woman, and the arrow lands near queen.
The arrow really did land near queen. All we did was add and subtract coordinates, yet the result matches the meaning. That works because the arrow from man to woman points in nearly the same direction, and has nearly the same length, as the arrow from king to queen.
In a well-trained embedding, directions like these hold relationships such as gender, or a country and its capital. The match is not always exact, and the result often lands next to the right word rather than on it. Still, it is striking that relationships nobody taught show up in the coordinates on their own.
Not only words become coordinates
Coordinates are not just for words. A photo can be turned into a coordinate too, so that similar photos gather close. A voice or a song can become a coordinate, and even a person's taste, judged by the songs they play most, can be written as one.
Turn anything into coordinates, measure closeness, and a computer can handle similarity. The features a CNN pulls from a photo, which we met in the last lesson, are used as coordinates in just this way.
Tap to switch the kind. Words, images, sounds or users, each becomes a point in the same way, and similar ones gather close.
Switching kinds did not change the picture: points appear, and similar ones gather close. When a music app recommends similar songs or a photo app finds lookalike pictures, it is measuring distances between coordinates underneath.
Train different kinds to share one space, and you can even search for photos with words: take the coordinate of the text "a puppy on the beach" and pick the photos closest to it.
Wrapping up
In short, a computer handles only numbers, so it turns each word into a coordinate, a bundle of numbers. That is embedding. The key rule is that words with similar meanings sit close together, and the coordinates are learned from how words are used in large amounts of text.
So distance becomes likeness of meaning, and adding and subtracting coordinates carries meaning too. Images, sounds and users are handled the same way as words.
Tap the key points in order to review. (turn words into coordinates → similar meanings stay close → coordinate math carries meaning → images and sounds work the same way)
Now you know how a computer starts to handle the meaning of words. The simple idea of turning words into coordinates and gathering similar ones together supports search, translation, recommendation and the linking of pictures with text.
One coordinate per word runs into trouble, though. Apple can be a fruit or the name of a company. The next lesson covers attention, a way of looking at the other words in a sentence to settle what a word means in that spot.