Module 7 · Lesson 4 of 4
Why the first token is slow
A slow start, then a stream
The first token waits for the queue and for the whole prompt to be read; every token after it comes at a steady pace.
Scene 1 of 4 · Wait in the queue
The request waits 0.8 s in a busy queue before the GPU can start on it.
Request
- Request sent at 0 s, sent
- Prompt read starts at 0.8 s, reading starts
Answer
- Time to first token
- not shown yet
- Tokens a second after that
- not shown yet
- Wait in the queue. The request waits 0.8 s in a busy queue before the GPU can start on it.
- Read the prompt. Prefill then reads the 3,000-token prompt in one pass, which takes about 0.12 s.
- The first token. One 49 ms step later, the first token arrives at 0.97 s: the time to first token.
- Then a steady stream. After that, a token arrives every 49 ms, using the saved work, so token 21 lands at 1.95 s.
- Now you try. Change how long the request waits in the queue. For example: First token at 1.77 s; token 21 at 2.75 s, about 20 tokens a second either way. The queue only delays the start; once writing begins, tokens arrive at the same steady pace.
Scene 1 of 4: Wait in the queue. The request waits 0.8 s in a busy queue before the GPU can start on it.
Key points
- The first token waits for the queue, then for prefill.
- This wait is the time to first token.
- Later tokens each wait for one short step only.
- A busy queue delays the start, not the pace after it.
The first (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) of an answer is often the slowest. It waits in the (A waiting line for work. One part adds messages, and another takes them off and handles them later, at its own pace.) behind other requests, then waits for (The first stage of an answer, where the model reads the whole prompt in one go and saves its work for later steps.), the pass that reads the whole (Everything sent to a language model in one request: instructions, context and the question.) before any token is written.
Every later token is much faster. Each one only waits for a single (The second stage of an answer, where the model writes one new token per step, using the work it saved.) step, since the prompt was already read and its saved work is already in the (The memory where a model keeps its work on every token so far, so each new token does not redo it. It grows by one token per step.).
Two numbers describe this. (How long a user waits from sending a request until the first piece of the answer appears.) is the wait for the first piece of the answer. Tokens per second is the speed of every token after that.
In system design: Time to first token and tokens per second are the two numbers users feel. One is the first wait; the other is how fast the rest streams in.