---
format: 1
concept: "tokenization"
kind: "misconception"
id: "token-is-word"
title: "A token is a word"
labels: {"lang":"en","level":"beginner"}
changes: []
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":["code execution"],"sources":["https://arxiv.org/abs/1508.07909","https://en.wikipedia.org/wiki/Byte_pair_encoding"]}
---
"Tokens" sounds like "words", and short common words often are single tokens. But many words are several tokens ("un" + "believ" + "able"), some tokens are bits of punctuation or a space plus a word (" the"), and code or rare names can break into many small pieces. Counting words is a rough guess at counting tokens, not the same thing.

Fix: think of tokens as the pieces in the tokenizer's vocabulary. To know how many a text takes, run it through that model's tokenizer.