How does attention work in AI

Focus more on the words that matter

Reading a long sentence, we do not spend the same effort on every word. We pick out the words that help us understand the one we are on and look at them longer. To know what 'sat' is about, for example, you need 'cat' to know who sat and 'mat' to know where.
Small words such as articles and prepositions add little to that question. Focusing more on the words that matter is the first step of attention.

Tap a word in the sentence. The more closely a word is tied to it, the darker it appears, showing where the current word focuses.

The dark words changed depending on which word you tapped. Dark words matter to the current word; faint ones barely relate to it. People read the same way.
But how does a machine know what matters? Every word has a coordinate, so the machine compares the current word's coordinate with each other word's coordinate and gives a relevance score. How to compare is learned from data, too. What matters is decided not by a word on its own but by its relationship to the words around it.

Context decides the meaning

Sometimes the same word means different things, and what decides is the surrounding words: the context. Take the English word 'bank'. With 'river' next to it, it means the land along the edge of a river; with 'money', it means a bank that holds money.
The word itself has not changed, but different surroundings give a different reading. Attention looks at exactly these surroundings to settle the meaning.

Keep the middle word as it is and tap to swap the context word beside it. The same word switches from a riverbank to a bank that holds money.

Changing just one neighboring word changed the whole reading. A word does not carry all its meaning alone; its meaning becomes clear only alongside the words around it.
So depending on which words the current word focuses on, the same word can be read in different senses. This shows most clearly in words with several meanings.

A word with several meanings moves with context

In the last lesson we saw that each word is a coordinate in meaning space. A word with several meanings has a coordinate caught between two neighborhoods. 'Apple' belongs both to the fruit neighborhood and to the company neighborhood, so on its own you cannot tell which side it is on.
Put 'eat' beside it, though, and the coordinate is pulled toward fruit; put 'stock' beside it, and it is pulled toward the company. Context settles the blurry coordinate on one side.

Tap 'eat' or 'stock' to attach it beside 'Apple'. The meaning coordinate moves between the fruit neighborhood and the company neighborhood.

The same 'Apple' changed place sharply because of one neighboring word. That is the heart of what attention does. It builds a new coordinate for the current word by blending the coordinates of the words around it: a lot of the words it focuses on strongly, only a little of the words it barely notices.
So how is the focus divided? Not equally among all the neighbors; more goes to the more relevant ones.

Turning relevance into weights

The relevance scores are not used as they are. They are turned into shares that add up to exactly 1, and these shares are called weights. The more relevant a word, the larger its share.
Because the total is fixed, when one word takes more, the others take that much less. There is only so much focus to go around, and the weights decide where it goes.

Tap a word's bar to give it more focus. When one word takes more, the rest shrink, so all the weights together always add up to 1.

When you piled focus onto one word, the others gave up the same amount, and the total stayed at 1 throughout. These weights decide which words to look at and how much. A word with a large weight pulls the current word's new coordinate strongly toward itself.
That is why 'Apple' moves toward the neighbor with the larger weight. A real language model does this for every word in the sentence at once, and runs several attentions side by side, each looking from a different angle.

Wrapping up

In short, when we read a sentence we do not look at every word equally; we focus more on the words that matter to the current one. Relevance scores are turned into weights that add up to 1, and the coordinates of the surrounding words are blended by those weights to make the current word's new coordinate.
That is how context decides meaning, and how the coordinate of a word like 'Apple' moves toward fruit or toward the company. This is attention.

Tap the key points in order to review. (focus on what matters → context decides meaning → a word's coordinate moves with context → weights always add up to 1)

Now you know what attention is. Thanks to the simple idea of picking out the words that matter instead of treating every word alike, the same word can be read differently depending on its neighbors. The machine has learned to read words in context.
A structure that stacks many layers of attention is called a transformer. Today's large language models, or LLMs, are machines built on transformers that guess the next word. The next lesson looks at how an LLM produces text.