Module 11 · Lesson 1 of 5
A GPU works on a batch
One step, many requests
A GPU writes one token for every request in its batch at each step, so more requests share each step.
Scene 1 of 4 · One request alone
With one request, each step takes 28 ms and writes one token for it.
- Waiting requests — healthy
- GPU — healthy — 1 in the batch
- One request alone. With one request, each step takes 28 ms and writes one token for it.
- 8 at once. With 8, a step takes 49 ms, far less than 8 separate steps of 28 ms.
- More work done. Together the 8 get about 163 tokens a second, against 36 for one alone.
- Each user slower. Each user's own speed drops, here from 36 to 20 tokens a second, since each step is longer.
- Now you try. Set how many requests share the batch. For example: 4 in the batch: 37 ms a step, 108 tokens a second in all, 27 for each user. More requests share each step, so the total rises while each user's own speed drops.
Scene 1 of 4: One request alone. With one request, each step takes 28 ms and writes one token for it.
Key points
- Each step writes one token for every request in the batch.
- A bigger batch makes each step a little longer.
- A bigger batch makes more tokens a second in total.
- But each user gets their own tokens more slowly.
A (A graphics processing unit: a chip that does many simple sums at the same time. Language models run on GPUs because they need huge numbers of sums.) is the chip that runs a model's sums (see How a model writes). It does not have to work on one request at a time. A model server gives it a whole group of requests to work on together, in one step. This group is called a batch.
A bigger batch uses the GPU better, since more requests share the same step instead of each waiting for its own. But each step still takes a little longer, because the GPU does a little more work for every extra request in it.
This happens at every step of the batch, each time it writes one more (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) for every request still running.
In system design: A server batches requests to use the GPU well. Bigger batches raise the work done each step, but each request waits slightly longer for its own step to finish.