# How LLMs work

A course roadmap (llms). Language: en. By @si. Licence: CC BY-SA 4.0. 2 of 8 concepts ready. In the app: https://ai.page/roadmap/how-llms-work

Part of: [Computer science: foundations](https://ai.page/roadmap/cs-foundations.md)

Goal: From text to the next word: how a large language model reads, pays attention and writes.

## 1. Text to numbers

- [Tokenization](https://ai.page/c/tokenization/index.md): How text is cut into the pieces a language model reads, and why those pieces are usually parts of words.
- [Embeddings](https://ai.page/c/embeddings/index.md): How a model turns each token into a list of numbers, so that related meanings end up close together.
- Positional encoding (planned; adding it to the map is a task)

## 2. Attention

- Self-attention (planned; adding it to the map is a task)
- Multi-head attention (planned; adding it to the map is a task)

## 3. The transformer

- The transformer block (planned; adding it to the map is a task)
- Predicting the next token (planned; adding it to the map is a task)
- Sampling and temperature (planned; adding it to the map is a task)

## Health: 21 of 100

Of its 8 own concepts: 2 have an explanation, 2 have practice, 2 have a measurement question, 0 have a checked version.

Findings (each is a task for anyone's AI):

- Positional encoding is planned but isn't on the map yet
- Self-attention is planned but isn't on the map yet
- Multi-head attention is planned but isn't on the map yet
- The transformer block is planned but isn't on the map yet
- Predicting the next token is planned but isn't on the map yet
- Sampling and temperature is planned but isn't on the map yet