Tokenization

A language model doesn't read letters or words. It reads tokens: pieces of text from a fixed list, its vocabulary, each with a number, its token id. Before a model sees your sentence, a tokenizer cuts it into tokens and turns each one into its id. The model only ever works with those numbers.

"unbelievable results"  →  ["un", "believ", "able", " results"]  →  [403, 34711, 481, 3135]

(The pieces and numbers here are made up; every model's tokenizer has its own.)

Why not whole words?

A list of every word in every language would be enormous and would still miss new words, names, typos and code. Whole words can't cover everything.

Why not single letters?

Letters cover everything, but sentences become very long sequences, and the model has to learn from scratch that "c-a-t" means cat. That costs a lot of computation for every sentence.

Parts of words: the middle way

Modern tokenizers use subwords. Common words get a single token ("the", " and"), and rarer words are built from common pieces ("un" + "believ" + "able"). Any text can be written with the vocabulary, because single letters (or bytes) are in it too, as a last resort. Vocabularies usually hold somewhere between tens of thousands and a few hundred thousand tokens.

Where the pieces come from

One common way to choose them is byte pair encoding (BPE). Start with single characters, find the pair of neighbours that appears most often in a large body of text, and merge it into a new token. Repeat until the vocabulary is the size you want. Frequent words end up as one token; rare words stay in pieces.

Why it matters to you

• Limits and prices are counted in tokens. A model's context window ("128k tokens") and API prices are per token, not per word. In English, a token is often about four characters, or three quarters of a word, as a rough guide.
• Languages aren't equal. A tokenizer trained mostly on English text has fewer ready-made pieces for other scripts, so the same meaning in Hindi or Tamil can take several times as many tokens: more cost, and less room in the context window.
• Odd mistakes have a cause. A model that miscounts the letters in a word may never have seen the letters separately: the word reached it as one or two tokens.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown

Five BPE merges by hand

Train BPE on a tiny corpus: "low" appears 5 times, "lower" 2, "newest" 6 and "widest" 3. Each word starts as single characters, plus an end-of-word marker </w> so the tokenizer knows where words end.

1. Step 1

   Start: l o w </w> ×5, l o w e r </w> ×2, n e w e s t </w> ×6, w i d e s t </w> ×3. Count every pair of neighbours, weighted by how often the word appears. e s appears 6 + 3 = 9 times, tied with s t and t </w>. Merge e s into a new token es (ties go to the first found).

2. Step 2

   Now n e w es t </w> and w i d es t </w>. The top pair is es t (9): merge it into est.

3. Step 3

   The top pair is est </w> (9): merge it into est</w>, a token meaning "est at the end of a word".

4. Step 4

   Now l o and o w are tied at 7 (5 from "low" + 2 from "lower"). Merge l o into lo, then the next round merges lo w into low.

5. Step 5

   After five merges the words are low </w>, low e r </w>, n e w est</w> and w i d est</w>. "newest" is now 4 tokens instead of 7 characters, and the vocabulary has 5 new tokens: es, est, est</w>, lo, low.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown

A token is a word

"Tokens" sounds like "words", and short common words often are single tokens. But many words are several tokens ("un" + "believ" + "able"), some tokens are bits of punctuation or a space plus a word (" the"), and code or rare names can break into many small pieces. Counting words is a rough guess at counting tokens, not the same thing.

Fix: think of tokens as the pieces in the tokenizer's vocabulary. To know how many a text takes, run it through that model's tokenizer.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown

Every language takes the same number of tokens

It's natural to assume a sentence costs the same to a model whatever language it's in. But a tokenizer's vocabulary is learned from its training text, which is often mostly English. Languages with less training text, or other scripts, get fewer ready-made pieces, so the same meaning can take several times as many tokens: higher cost, and less room in the context window.

Fix: measure with the model's own tokenizer, and expect differences between languages and between tokenizers.

@si / anthropic (says) / claude-opus 5.5 (says) / high (says) · 2026-10-14 · new, not checked yet · Markdown