Module 11 · Lesson 4 of 5
Static and continuous batching
Wait for all, or join at once
Static batching makes a new request wait for the whole batch; continuous batching gives it the first free seat.
Scene 1 of 5 · A batch of 4
4 requests start together; the longest, A, needs 12 steps, and D needs only 2.
- Queue — healthy
- GPU batch — healthy — 4 of 4 seats used
Static batching
- Seat 1: A, running, steps 1 to 12.
- Seat 2: B, running, steps 1 to 3.
- Seat 3: C, running, steps 1 to 4.
- Seat 4: D, running, steps 1 to 2.
Continuous batching
- Seat 1: A, running, steps 1 to 12.
- Seat 2: B, running, steps 1 to 3.
- Seat 3: C, running, steps 1 to 4.
- Seat 4: D, running, steps 1 to 2.
- E waits, static
- not shown yet
- E waits, continuous
- not shown yet
- A batch of 4. 4 requests start together; the longest, A, needs 12 steps, and D needs only 2.
- A short question. Request E arrives after step 2, and its answer needs only 2 steps.
- Static: wait for all. With static batching, E waits until step 12, while the seats of B, C and D sit idle.
- Continuous: join early. With continuous batching, D leaves after step 2, and E takes its seat on the very next step.
- The difference. E waits 10 steps, about 370 ms, with static batching, and none with continuous.
- Now you try. Change how many steps the long answer A needs. For example: Static: E waits 18 steps. Continuous: E starts on the very next step. Static batching ties E to the slowest answer in the batch; continuous batching needs only one free seat.
Scene 1 of 5: A batch of 4. 4 requests start together; the longest, A, needs 12 steps, and D needs only 2.
Key points
- Static batching starts a new batch only when every request in it is done.
- A short request can wait behind one long answer.
- Continuous batching lets requests leave and join at every step.
- So a short request starts as soon as any seat frees up.
A model server can batch requests two ways. (Running a fixed group of requests together until the slowest one finishes, before any new request can start.) runs a fixed group together. A new batch starts only once every request in that group has finished, even one that finished long ago.
Static (Grouping several items, such as messages or requests, and handling them together instead of one at a time.) also reserves memory for the longest answer any request might give, since it cannot see the real length in advance. That memory sits reserved even for a request that finishes early.
(Letting requests join and leave a batch at every step, instead of waiting for the whole batch to finish.) lets requests join and leave the batch at every step instead, not only at the end. A finished request leaves right away, and a new one can join on the very next step.
Try it: switch batching modes in the lab to see the difference.
In system design: Static batching is simpler but slower for a short request stuck behind a long one. Continuous batching adds complexity but keeps the batch full and responsive.