Module 7 · Lesson 1 of 4
One token at a time
Four tokens, four runs
A model writes an answer by running once per token, each run reading everything so far and adding one piece.
Scene 1 of 5 · Run 1: "It"
Run 1 reads the question and writes one token, "It", which goes straight to your screen.
- Language model — healthy
- Chat app — healthy
- Your screen — healthy — "It"
| Run | Reads | Writes |
|---|---|---|
| Run 1 (highlighted) | the question | "It" |
- Runs so far
- not shown yet
- Time at the lab's pace
- not shown yet
- Run 1: "It". Run 1 reads the question and writes one token, "It", which goes straight to your screen.
- Run 2: "left". Run 2 reads the question plus "It", then writes "left".
- Run 3: "the". Run 3 reads everything so far and adds "the"; your screen now shows three tokens.
- Run 4: "depot". Run 4 adds "depot": four tokens took four runs, one after another.
- Why text streams in. At the serving lab's 28 ms a run for one request, these four runs take about 112 ms.
- Now you try. Set the answer length, and see how many runs it takes. For example: 10 tokens take 10 runs, about 280 ms one after another. Each run needs the token before it, so the runs cannot happen at the same time.
Scene 1 of 5: Run 1: "It". Run 1 reads the question and writes one token, "It", which goes straight to your screen.
Key points
- Each run of the model writes exactly one token.
- Each run reads the question plus every token written so far.
- The runs happen one after another, never all at once.
- Apps show each token the moment it is written, so text streams in.
A model does not write a whole answer in one go. It picks one (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.), a small piece of text (see AI basics), and adds it to everything written so far. Then it runs again to pick the next token.
Each run looks back over all the text so far, and produces exactly one new token. Lesson 3 shows how it avoids redoing that earlier work. A ten-token answer still needs ten separate runs through the model, one after another, not a single long write.
This is why an answer streams in a piece at a time on screen. The app shows each token the moment the model produces it, instead of waiting for the whole answer to finish.
In system design: Streaming an answer means sending each token to the user as it is produced, instead of waiting for the whole answer. This is why chat apps show text appearing gradually.