Module 9 · Lesson 3 of 6
Cost and speed go together
Fewer tokens, faster and cheaper
Trimming the prompt cuts both the bill and the wait, while streaming only changes how soon the first words appear.
Scene 1 of 4 · Eight chunks
With 8 retrieved chunks, the prompt is 3,450 tokens, and the chunks are most of it.
- Prompt tokens
- 3,450
- Input cost (example prices)
- $0.0104
- Prefill time
- 138 ms
- First words, streamed
- not shown yet
- Whole answer
- not shown yet
- Eight chunks. With 8 retrieved chunks, the prompt is 3,450 tokens, and the chunks are most of it.
- Four chunks. Sending the best 4 instead makes it 1,850 tokens, about half.
- Cheaper and faster. Input cost falls from $0.0104 to $0.00555, and prefill from 138 to 74 ms.
- Streaming helps the feel. Streaming costs the same, but the first words show after 123 ms, not after the whole 14.8 s answer.
- Now you try. Change how many retrieved chunks go into the prompt. For example: 5,050 prompt tokens: $0.0152 of input and 202 ms of prefill per request. Each chunk adds tokens you pay for and wait for; the 300-token answer still takes about 14.9 s, so stream it.
Scene 1 of 4: Eight chunks. With 8 retrieved chunks, the prompt is 3,450 tokens, and the chunks are most of it.
Key points
- Retrieved chunks are often most of the prompt.
- Fewer prompt tokens cost less and read faster.
- Streaming does not cut the cost at all.
- It does show the first words long before the answer is done.
A longer (Everything sent to a language model in one request: instructions, context and the question.) takes longer to read, and a longer answer takes longer to write. So trimming (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) saves time as well as money.
Streaming the answer does not make it cheaper. But users see the first words sooner, which makes the wait feel shorter.
In system design: Give each part of the prompt a token budget: the instructions, the retrieved chunks and the conversation so far.