---
format: 1
concept: "tokenization"
kind: "explanation"
title: "The pieces a model reads"
labels: {"lang":"en","level":"beginner"}
changes: []
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":["code execution"],"sources":["https://arxiv.org/abs/1508.07909","https://en.wikipedia.org/wiki/Byte_pair_encoding"]}
---
A language model doesn't read letters or words. It reads **tokens**: pieces of text from a fixed list, its **vocabulary**, each with a number, its **token id**. Before a model sees your sentence, a **tokenizer** cuts it into tokens and turns each one into its id. The model only ever works with those numbers.

```
"unbelievable results"  →  ["un", "believ", "able", " results"]  →  [403, 34711, 481, 3135]
```

(The pieces and numbers here are made up; every model's tokenizer has its own.)

## Why not whole words?

A list of every word in every language would be enormous and would still miss new words, names, typos and code. Whole words can't cover everything.

## Why not single letters?

Letters cover everything, but sentences become very long sequences, and the model has to learn from scratch that "c-a-t" means cat. That costs a lot of computation for every sentence.

## Parts of words: the middle way

Modern tokenizers use **subwords**. Common words get a single token ("the", " and"), and rarer words are built from common pieces ("un" + "believ" + "able"). Any text can be written with the vocabulary, because single letters (or bytes) are in it too, as a last resort. Vocabularies usually hold somewhere between tens of thousands and a few hundred thousand tokens.

## Where the pieces come from

One common way to choose them is **byte pair encoding** (BPE). Start with single characters, find the pair of neighbours that appears most often in a large body of text, and merge it into a new token. Repeat until the vocabulary is the size you want. Frequent words end up as one token; rare words stay in pieces.

## Why it matters to you

- **Limits and prices are counted in tokens.** A model's context window ("128k tokens") and API prices are per token, not per word. In English, a token is often about four characters, or three quarters of a word, as a rough guide.
- **Languages aren't equal.** A tokenizer trained mostly on English text has fewer ready-made pieces for other scripts, so the same meaning in Hindi or Tamil can take several times as many tokens: more cost, and less room in the context window.
- **Odd mistakes have a cause.** A model that miscounts the letters in a word may never have seen the letters separately: the word reached it as one or two tokens.