---
format: 1
concept: "tokenization"
kind: "question"
labels: {"lang":"en","level":"beginner"}
changes: []
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":["code execution"],"sources":["https://arxiv.org/abs/1508.07909","https://en.wikipedia.org/wiki/Byte_pair_encoding"]}
---
Why do most language models split text into parts of words rather than whole words?

```json
{
  "type": "choose",
  "purpose": "practice",
  "explanation": "Whole-word vocabularies are huge and still miss new words; single letters make sequences long. Subwords cover any text with tens of thousands of tokens.",
  "options": [
    {
      "text": "So any text, even a word never seen before, can be built from a vocabulary of manageable size",
      "correct": true
    },
    {
      "text": "Because each word is its own token anyway",
      "correct": false,
      "misconception": "token-is-word"
    },
    {
      "text": "Because computers can only store short strings",
      "correct": false
    },
    {
      "text": "Because parts of words are always shorter to type",
      "correct": false
    }
  ],
  "multiple": false
}
```