Module 4 · Lesson 4 of 5
Falling back when the model fails
Predict first: which model or saved answer replies, and how long does the user wait? Step through to check.
- User — healthy — NOW
- Application — timer and fallbacks — healthy
- Large model — best answers — healthy
- Small model — faster and simpler — healthy
- Answer cache — answers saved earlier — healthy
- Request
- Error or timeout
- Fallback
- Streamed answer
Step 1 of 8 · User
Done: The user sends a chat message
- What happens
- A user sends a message and waits for the reply.
- Why
- Every second of waiting counts from here, whatever happens behind the application.
- What changes
- The application has the message. The clock for this reply starts at 0 s.
- Next
- Build the request and start a 3 s timer
Event log
7 more to go. Press a line to go back to that step.
Step 1 of 8: User. The user sends a chat message. A user sends a message and waits for the reply.
The whole chain at a glance
- Large model timed out, 3 s
- Small model answered, 0.4 s
- Cached answer not needed
- Try-again message not needed
The small model answers after about 3.4 s in all, so the user gets a shorter, simpler answer.
The large model gets a 3 s timeout. What this simplifies. The timings are made up to show the idea. Real timeouts depend on what users will wait for. A circuit breaker would skip a provider that keeps failing, so later users reach the fallback faster. On a "too many requests" error this app moves on at once. Another app might wait and try the same model again.
A (A simpler answer sent instead of the real one, used when a call is skipped or fails.) is a plan B: when the main model times out or fails, the app answers another way instead of showing an error.
A common chain is a smaller model, then a cached answer from earlier, then an honest message asking the user to try again.
When a provider fails again and again, a (A switch that stops calls to a failing service for a while, so callers fail fast instead of waiting.) stops calling it for a while, so users go straight to the fallback. The Circuit breaker lab shows this.
Break a dependency and watch the breaker move through closed, open and half-open while protecting both services.
Open the lab →In system design: Design the fallback chain on purpose, and tell users when they get a simpler or older answer.