Module 4 · Lesson 1 of 5
A timeout on every model call
Stop waiting, start plan B
A first-token timeout cancels a call that has sent nothing, so the user gets a fallback instead of a spinner.
Scene 1 of 4 · A healthy call
A healthy call sends its first words after 0.6 s, so the user sees text almost at once.
Healthy call
- Request sent at 0 s
- Words reach the user at 0.6 s, first words
- First-token limit
- not shown yet
- Whole-answer limit
- not shown yet
- A healthy call. A healthy call sends its first words after 0.6 s, so the user sees text almost at once.
- An overloaded provider. When the provider is overloaded, nothing comes back; with no timeout, the user could wait 45 s.
- Cancel at 3 s. With a 3 s first-token limit, the stuck call is cancelled and the small model answers by 3.4 s.
- A longer whole limit. The whole answer gets a longer limit, 20 s, since a stream that started well can still stall.
- Now you try. Set the first-token limit, and see what happens to an overloaded call and to a slow but healthy one. For example: The overloaded call's user gets the fallback at 1.4 s; the slowest healthy call (2.5 s) is cancelled too. Too short a limit cancels calls that were working, wasting them and adding load from fallbacks.
Scene 1 of 4: A healthy call. A healthy call sends its first words after 0.6 s, so the user sees text almost at once.
Key points
- A healthy call usually starts sending words within a second or two.
- With no timeout, an overloaded provider can keep the user waiting a long time.
- A short first-token limit catches that early and hands over to plan B.
- A longer limit on the whole answer catches a stream that stalls.
A model call can take seconds even when all is well, and much longer when a provider is overloaded. Without a (The time a caller waits before giving up on a call and treating it as failed.), users wait as long as the slowest call.
Pick a timeout from what users will put up with, such as a few seconds for a chat reply. A call that runs past it counts as failed.
Streaming helps: set a short limit for the first (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.), since a fast start usually means the call is healthy. Keep a longer limit for the whole answer, since a stream can stall.
In system design: Every model call needs a timeout, chosen from the user's patience, not from how long the model might need.