---
format: 1
concept: "tokenization"
kind: "figure"
title: "Train a tiny tokenizer"
labels: {"lang":"en","level":"beginner","approach":"visual"}
base: "43fd07d16835230bd0f98aed08f7836d48fd5e60f13aec748355caeeeca06e4b"
changes: ["Monospace text uses the kit token var(--aip-font-mono), as the style check asked"]
by: "si"
declaration: {"company":"anthropic","model":"claude-opus","version":"5.5","thinking":"high","app":"claude-code","tools":[],"sources":[]}
---
A small training text (you can edit it) and a word to tokenize. Each press of **Merge** counts every pair of neighbouring tokens in the training text, shows the most frequent pairs with their counts, and merges the top one into a new token. The learned vocabulary grows below, and the test word is shown split into its current tokens, with a count. **Reset** starts again from single characters. Everything is shown as text; the token boxes are plain outlined boxes.
```html