Module 0 · Lesson 5 of 10
Inside the model: how it picks the next token
- Tokens — the prompt in pieces — healthy — NOW — 5 tokens
- Vectors — numbers per token — healthy
- Attention — weighs earlier tokens — healthy
- Layers — repeat many times — healthy
- Scores — one per known token — healthy
- Tokens
- Vectors
- Weighed mix
- Scores
Step 1 of 6 · Tokens
Done: The prompt is split into 5 tokens
- What happens
- "The cat sat on the" becomes "The", "cat", "sat", "on", "the".
- Why
- A model works on tokens, and each token has its own number in the list of tokens the model knows.
- What changes
- Next, each token becomes a list of numbers.
- Next
- Each token becomes a vector
Event log
5 more to go. Press a line to go back to that step.
Tokens: The·cat·sat·on·the
Step 1 of 6: Tokens. The prompt is split into 5 tokens. "The cat sat on the" becomes "The", "cat", "sat", "on", "the".
What this simplifies. This is a small made-up example. The weights and chances are round values chosen to add up as real ones do, not output from a real model. Real models use thousands of numbers per token and run many attention heads side by side in each layer.
A (A program trained on a huge amount of text to guess the next piece of text. It writes answers one piece at a time.) writes one (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) at a time. For each one, the (Everything sent to a language model in one request: instructions, context and the question.) so far goes through the same parts in order. The model ends with a score for every token it could write next.
First, each token becomes a (A list of numbers. An embedding is a vector.): a list of numbers the model learned in (The long, costly process in which a model learns from data. It happens before the model is used, not during each request.). Then attention compares the tokens. Each token makes a query, a key and a value. The query of the newest token is matched against the key of every token so far, and the best matches get the most weight.
A layer is one round of attention, plus a network that works on each token alone. A large model stacks dozens of layers. At the end, the vector of the last token becomes a score for every token the model knows. The scores become chances that add up to 100%.
One token is then picked. At (A request setting that controls how much chance goes into picking each next token. Low gives repeatable answers; high gives varied ones.) 0 the top score wins. At a higher temperature the token is drawn by chance, so answers vary. The picked token joins the prompt, and the whole run starts again for the next token.
In system design: Each new token is one more run, so a longer answer takes longer. The (The memory where a model keeps its work on every token so far, so each new token does not redo it. It grows by one token per step.) saves earlier keys and values, which spares work but takes (A graphics processing unit: a chip that does many simple sums at the same time. Language models run on GPUs because they need huge numbers of sums.) memory.