Module 9 · Lesson 4 of 6
Sending easy questions to a small model
- One request, large model
- $0.0105
- One request, small model
- $0.0009
- A month with this mix
- $1,129
- A month, all on the large model
- $3,150
- Saved by routing
- $2,021 (64%)
Example prices per million tokens: large model $3 in and $15 out; small model $0.25 in and $1.25 out.
What this simplifies. The prices are examples, not any provider's real prices, which vary and change often. The small model is assumed to answer its share well; in practice you check that with a test set. Cached input tokens, retries and failed calls also change the real bill.
Where one request's time goes
Adds 150 ms. The text it finds is part of the prompt tokens above.
- First words on screen after
- 229 ms
- Whole answer after
- 14.9 s
How it is worked out: network 100 ms; each prompt token takes 0.04 ms to read; each answer token takes 49 ms to write. So a longer prompt delays the first words, and a longer answer delays the end.
What this simplifies. These times are worked out from example speeds. They are not measurements. The large model uses the Serving lab's speeds for a full batch of 8 requests. The small model is assumed to be three times as fast. Network and search are fixed example times. Real times change with load, hardware and the provider.
Small models usually cost far less per (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) and answer faster, but they make more mistakes on hard questions.
(Sending each request to the cheapest model likely to answer it well, such as easy questions to a small model.) sends each question to the cheapest model likely to answer it well, such as simple lookups to a small model.
The (A network device that passes each packet on, one step at a time, towards the machine it is addressed to.) can be simple rules or a small model that sorts questions into easy and hard. Hard or failed cases can go up to the large model.
In system design: Measure answer quality on each route with a test set, so the savings do not quietly cost you correct answers.