---
format: 1
concept: "embeddings"
kind: "explanation"
title: "Meaning as a direction"
labels: {"lang":"en","level":"beginner"}
changes: []
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":["code execution"],"sources":["https://en.wikipedia.org/wiki/Word_embedding","https://en.wikipedia.org/wiki/Cosine_similarity"]}
---
A tokenizer turns text into token ids: just numbers in a list, like 403 or 34711. The id says nothing about meaning; 403 isn't "closer" to 404 in any useful way. So the first thing a language model does with each id is look up its **embedding**: a list of numbers, a **vector**, that stands for that token.

```
token "cat"  → id 9246 → [0.12, -0.83, 0.40, …]   (hundreds or thousands of numbers)
```

## Close means related

The vectors aren't chosen by hand. They start random and are **learned** during training, nudged a little at a time so the model predicts text better. Because tokens that appear in similar places in text get used in similar ways, their vectors end up pointing in similar directions: "cat" near "kitten" and "dog", far from "truck".

## Measuring closeness: cosine similarity

Two vectors are compared by the **angle** between them, not their length. The **cosine similarity** is

$$\cos\theta = \frac{a \cdot b}{\lVert a \rVert\,\lVert b \rVert}$$

where $a \cdot b$ multiplies the vectors number by number and adds the results, and $\lVert a \rVert$ is a vector's length. It's **1** when they point the same way, **0** when they're at right angles, and **−1** when they point opposite ways.

## Many dimensions

Real embeddings have hundreds or thousands of numbers, not two. That's room for many kinds of relatedness at once: topic, grammar, tone. No single number means one human idea like "animal" or "royalty"; meanings are spread across many of them, which is why pictures of embeddings squash them down to 2D and lose a lot.

## Where they're used

- **Inside every language model:** embeddings are the model's first layer, and its later layers keep working with vectors like these.
- **Search by meaning:** turn documents and a question into embeddings, then find the documents whose vectors are most similar, even when they share no words with the question.