---
format: 1
concept: "tokenization"
kind: "misconception"
id: "same-in-every-language"
title: "Every language takes the same number of tokens"
labels: {"lang":"en","level":"beginner"}
changes: []
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":["code execution"],"sources":["https://arxiv.org/abs/1508.07909","https://en.wikipedia.org/wiki/Byte_pair_encoding"]}
---
It's natural to assume a sentence costs the same to a model whatever language it's in. But a tokenizer's vocabulary is learned from its training text, which is often mostly English. Languages with less training text, or other scripts, get fewer ready-made pieces, so the same meaning can take several times as many tokens: higher cost, and less room in the context window.

Fix: measure with the model's own tokenizer, and expect differences between languages and between tokenizers.