---
format: 1
concept: "tokenization"
kind: "worked-example"
title: "Five BPE merges by hand"
labels: {"lang":"en","level":"beginner"}
changes: []
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":["code execution"],"sources":["https://arxiv.org/abs/1508.07909","https://en.wikipedia.org/wiki/Byte_pair_encoding"]}
---
Train BPE on a tiny corpus: "low" appears 5 times, "lower" 2, "newest" 6 and "widest" 3. Each word starts as single characters, plus an end-of-word marker `</w>` so the tokenizer knows where words end.

```json
{
  "steps": [
    "Start: `l o w </w>` ×5, `l o w e r </w>` ×2, `n e w e s t </w>` ×6, `w i d e s t </w>` ×3. Count every pair of neighbours, weighted by how often the word appears. `e s` appears 6 + 3 = **9** times, tied with `s t` and `t </w>`. Merge `e s` into a new token `es` (ties go to the first found).",
    "Now `n e w es t </w>` and `w i d es t </w>`. The top pair is `es t` (9): merge it into `est`.",
    "The top pair is `est </w>` (9): merge it into `est</w>`, a token meaning \"est at the end of a word\".",
    "Now `l o` and `o w` are tied at 7 (5 from \"low\" + 2 from \"lower\"). Merge `l o` into `lo`, then the next round merges `lo w` into `low`.",
    "After five merges the words are `low </w>`, `low e r </w>`, `n e w est</w>` and `w i d est</w>`. \"newest\" is now 4 tokens instead of 7 characters, and the vocabulary has 5 new tokens: `es`, `est`, `est</w>`, `lo`, `low`."
  ]
}
```