Module 11 · Lesson 2 of 5
Memory decides how many fit
Memory sets the batch size
Every running request keeps its saved work in GPU memory, so request size decides how many run at once.
Scene 1 of 5 · Two limits
This GPU has room for 8,000 tokens of saved work, and its batch has 8 seats.
- Queue — healthy
- GPU — healthy — 0 of 8,000 used
- Running
- not shown yet
- Waiting
- not shown yet
- Two limits. This GPU has room for 8,000 tokens of saved work, and its batch has 8 seats.
- Short requests. 12 requests arrive; each grows to 800 tokens: a 700-token prompt plus a 100-token answer.
- Seats run out first. Memory could hold 10 of them, but the batch has 8 seats, so 4 wait in the queue.
- Long prompts. At 4,000 tokens each, memory holds only 2, so 10 of the 12 requests wait.
- Or one is paused. Preemption: if running answers outgrow memory, the server pauses one and later redoes its prompt and answer so far.
- Now you try. Set the saved tokens per request, and see how many of 12 can run. For example: 5 run at once, and 7 of 12 wait in the queue. Memory is the limit here: bigger requests leave room for fewer at once.
Scene 1 of 5: Two limits. This GPU has room for 8,000 tokens of saved work, and its batch has 8 seats.
Key points
- Saved work grows by one token at every step of the answer.
- When memory or the batch's seats run out, new requests wait.
- Long prompts mean far fewer requests run together.
- If answers outgrow memory, a server may pause one and redo its work later.
Each request's saved work, the (The memory where a model keeps its work on every token so far, so each new token does not redo it. It grows by one token per step.) (see How a model writes), takes up memory on the (A graphics processing unit: a chip that does many simple sums at the same time. Language models run on GPUs because they need huge numbers of sums.). A longer (Everything sent to a language model in one request: instructions, context and the question.) or a longer answer so far means more saved work, so it takes more memory.
The GPU has a fixed amount of this memory. Once it is full, a new request cannot start: it waits in the (A waiting line for work. One part adds messages, and another takes them off and handles them later, at its own pace.) until enough memory frees up.
Sometimes a running request is paused instead, called a (Pausing a running request and freeing its memory so others can continue. It must redo its saved work when it returns.). Its saved work is dropped to free memory for others. It must redo that work, its (The first stage of an answer, where the model reads the whole prompt in one go and saves its work for later steps.), once it starts again (see How a model writes).
Try it: the lab's long-prompts experiment shows requests getting paused this way.
In system design: Memory sets how many requests run together. Long prompts and answers use more of it, so fewer requests fit, and the rest wait or get paused.