Module 8 · Lesson 2 of 4
Reusing a shared prompt start
Read the shared start once
A prefix cache keeps the saved work for a prompt's fixed start, so prefill reads only the new part at the end.
Scene 1 of 4 · The same long start
Every request starts with the same 3,000 tokens of instructions, then a 50-token question.
- Prefill time
- not shown yet
- Input cost (example prices)
- not shown yet
- The same long start. Every request starts with the same 3,000 tokens of instructions, then a 50-token question.
- No prefix cache. Without a prefix cache, prefill reads all 3,050 tokens every time, about 122 ms at the lab's rate.
- Reuse the start. A prefix cache keeps the saved work for the start, so prefill reads only 50 new tokens.
- Often cheaper too. If cached input costs 10% of the price, input drops from $0.00915 to $0.00105 a request.
- Now you try. Lay out the prompt, and see how much of it the cache can reuse. For example: 0 tokens reused, 3,060 read fresh: about 122 ms of prefill. Something that changes sits at the top, so nothing after it can be reused.
Scene 1 of 4: The same long start. Every request starts with the same 3,000 tokens of instructions, then a 50-token question.
Key points
- Most prompts start with the same long instructions.
- A prefix cache keeps the saved work for that shared start.
- Prefill then reads only the new tokens at the end.
- Anything that changes must come last, or nothing after it can be reused.
Many (Everything sent to a language model in one request: instructions, context and the question.) start with the same long instructions and differ only at the end, in the user's question.
A (A cache that keeps a model's saved work for the shared start of many prompts, so only the new part has to be read.) keeps the model's saved work for that shared start, its (The memory where a model keeps its work on every token so far, so each new token does not redo it. It grows by one token per step.) (see How a model writes). Then (The first stage of an answer, where the model reads the whole prompt in one go and saves its work for later steps.) only has to read the new part.
This saves time, and often money, since many providers charge less for input (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) read from such a (A small, fast store that keeps copies of popular data, so most reads skip the slower database.). The start must match exactly, and the saved work expires, often after a few minutes.
In system design: Put the fixed parts of a prompt first and the changing parts last, so the shared start stays identical and can be reused.