Module 7 · Lesson 2 of 4
Two stages: prefill and decode
18 prompt tokens: prefill takes 72 units. The first token arrives after 97 units, and each later token after 25 more.
- Application — sends the prompt — healthy — NOW — prompt: 18 tokens
- Language model — on a GPU — healthy
- The prompt
- One answer token
Step 1 of 9 · Application
Done: The application has the prompt ready
- What happens
- The prompt is 18 tokens long. A token is a word or a piece of a word.
- Why
- The model has not seen the prompt yet, so it has written nothing.
- What changes
- Nothing is saved, and no time has been used.
- Next
- Prefill: the model reads the whole prompt
Event log
8 more to go. Press a line to go back to that step.
Prompt:
Answer so far:
Time so far: prefill 4 units for each of 18 prompt tokens, then 25 units per answer token. 0 of 247 units used so far.
KV cache rows: 0
Step 1 of 9: Application. The application has the prompt ready. The prompt is 18 tokens long. A token is a word or a piece of a word.
What this simplifies. Real models do this for thousands of tokens, with huge sums run on a GPU. The rows here stand for much larger blocks of numbers, and the time units are made up. Here every decode step takes the same time; in a real model it grows a little as the KV cache grows.
The full prompt for this example: You are a support agent. Answer from the policy. Where is my parcel?
Writing an answer has two stages. The first is (The first stage of an answer, where the model reads the whole prompt in one go and saves its work for later steps.): the model reads the whole (Everything sent to a language model in one request: instructions, context and the question.), all its (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) side by side, and saves its work for the next steps.
The second stage is (The second stage of an answer, where the model writes one new token per step, using the work it saved.): the model writes one new token per step, using the work it saved. Prefill handles many tokens together; decode handles one token per request, per step.
This split matters for speed. A longer prompt still makes prefill take longer, since there is more to read, but it does not add one step per prompt token. Decode adds one step per answer token, run one after another.
In system design: Long prompts slow prefill; long answers add decode steps. A design that only shortens prompts misses half the cost, since decode time grows with answer length too.