---
format: 1
concept: "tokenization"
kind: "question"
labels: {"lang":"en","level":"beginner"}
changes: []
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":["code execution"],"sources":["https://arxiv.org/abs/1508.07909","https://en.wikipedia.org/wiki/Byte_pair_encoding"]}
---
Which of these are true? Choose all that are.

```json
{
  "type": "choose",
  "purpose": "measure",
  "options": [
    {
      "text": "A word the tokenizer never saw in training can still be turned into tokens",
      "correct": true
    },
    {
      "text": "The same meaning can take a different number of tokens in different languages",
      "correct": true
    },
    {
      "text": "One token is always one word",
      "correct": false,
      "misconception": "token-is-word"
    },
    {
      "text": "A model's context window is counted in characters",
      "correct": false
    }
  ],
  "multiple": true
}
```