Embeddings

A tokenizer turns text into token ids: just numbers in a list, like 403 or 34711. The id says nothing about meaning; 403 isn't "closer" to 404 in any useful way. So the first thing a language model does with each id is look up its embedding: a list of numbers, a vector, that stands for that token.

token "cat"  → id 9246 → [0.12, -0.83, 0.40, …]   (hundreds or thousands of numbers)

Close means related

The vectors aren't chosen by hand. They start random and are learned during training, nudged a little at a time so the model predicts text better. Because tokens that appear in similar places in text get used in similar ways, their vectors end up pointing in similar directions: "cat" near "kitten" and "dog", far from "truck".

Measuring closeness: cosine similarity

Two vectors are compared by the angle between them, not their length. The cosine similarity is

where $a\cdot b$ multiplies the vectors number by number and adds the results, and $\Vert a\Vert$ is a vector's length. It's 1 when they point the same way, 0 when they're at right angles, and −1 when they point opposite ways.

Many dimensions

Real embeddings have hundreds or thousands of numbers, not two. That's room for many kinds of relatedness at once: topic, grammar, tone. No single number means one human idea like "animal" or "royalty"; meanings are spread across many of them, which is why pictures of embeddings squash them down to 2D and lose a lot.

Where they're used

• Inside every language model: embeddings are the model's first layer, and its later layers keep working with vectors like these.
• Search by meaning: turn documents and a question into embeddings, then find the documents whose vectors are most similar, even when they share no words with the question.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown

Three tiny vectors

Real embeddings are long, but the sums are the same in 2D. Compare $A=(3,4)$, $B=(4,3)$ and $C=(-4,3)$ with cosine similarity.

1. Step 1

   Lengths first: $\Vert A\Vert =\sqrt{3^2+4^2}=5$, and $B$ and $C$ are also length 5.

2. Step 2

   $A\cdot B=3\times 4+4\times 3=24$, so $\cos =24/(5\times 5)=0.96$: nearly the same direction, very similar.

3. Step 3

   $A\cdot C=3\times (-4)+4\times 3=0$, so $\cos =0$: at right angles, unrelated.

4. Step 4

   $B\cdot C=4\times (-4)+3\times 3=-7$, so $\cos =-7/25=-0.28$: pointing somewhat apart. Of the three, $A$ and $B$ are the closest pair.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown

The embedding is the token id

Both are numbers attached to a token, so they're easy to mix up. The id is one whole number that only says which row to look in, and nearby ids mean nothing. The embedding is the long list of numbers stored in that row, learned so that similar tokens point in similar directions.

Fix: id = which row; embedding = what's in the row.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown

Each number in an embedding means one idea

Popular explanations draw embeddings with labelled axes, like "animal" or "royalty", so it's natural to think each number has a meaning you could name. In real models, nobody chooses what the numbers mean; training spreads every kind of relatedness across many of them at once. Directions in the whole space can line up with ideas, but single numbers usually don't.

Fix: think of similarity between whole vectors (their angle), not the meaning of any one number.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown