Module 7 · Lesson 3 of 4
The KV cache: saved work
Saved work, step by step
The KV cache saves the model's work on every token, so each new step computes only the newest token and reads the rest.
Scene 1 of 5 · Read the prompt once
Step 1 reads the 6-token question and saves its work on each token: this is the KV cache.
| Step | Work, nothing saved | Work, with the KV cache |
|---|---|---|
| Step 1: "It" (highlighted) | ? | ? |
| Step 2: "left" | ? | ? |
| Step 3: "the" | ? | ? |
| Step 4: "depot" | ? | ? |
- Total work, nothing saved
- not shown yet
- Total work, with the cache
- not shown yet
- Requests that fit in 8,000 tokens
- not shown yet
- Read the prompt once. Step 1 reads the 6-token question and saves its work on each token: this is the KV cache.
- Nothing saved. With nothing saved, step 4 would redo all 9 tokens so far, to add only one more.
- With saved work. With the KV cache, each later step computes only the newest token and reads the rest from memory.
- The totals. For this 4-token answer: 30 tokens of work with nothing saved, and 9 with the cache.
- It takes memory. It sits in the memory of the GPU, the model's chip: 8,000 tokens hold 4 requests of 2,000.
- Now you try. Set the answer length, and compare the total work with and without saved work. For example: A 20-token answer: 310 tokens of work with nothing saved, 25 with the KV cache. Without saved work, each step redoes all the text so far, so the gap grows with every token.
Scene 1 of 5: Read the prompt once. Step 1 reads the 6-token question and saves its work on each token: this is the KV cache.
Key points
- Prefill reads the prompt once and saves its work on each token.
- With nothing saved, every step would redo all the text so far.
- With the KV cache, each later step computes one new token and reads the saved work.
- The saved work takes memory on the chip running the model, limiting how many requests fit.
Redoing all the earlier work at every step would be slow. Instead, the model saves its work on each (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) as it reads or writes it. So a later step computes only the one new token. It still reads the saved work for every earlier token, and that reading grows as the text grows.
For each token, the model works out two lists of numbers, called a key and a value. A new token compares itself with the keys of earlier tokens to decide which ones matter, then mixes in their values. This step is called attention.
The keys and values are the saved work, so it is called the (The memory where a model keeps its work on every token so far, so each new token does not redo it. It grows by one token per step.): K for key, V for value. It grows by one entry per token: one per (Everything sent to a language model in one request: instructions, context and the question.) token after (The first stage of an answer, where the model reads the whole prompt in one go and saves its work for later steps.), then one more after every (The second stage of an answer, where the model writes one new token per step, using the work it saved.) step.
The KV (A small, fast store that keeps copies of popular data, so most reads skip the slower database.) lives in the memory of the (A graphics processing unit: a chip that does many simple sums at the same time. Language models run on GPUs because they need huge numbers of sums.), the chip that runs the model's sums. Bigger prompts and longer answers use more of that memory, one entry at a time.
In system design: Memory for the KV cache limits how many requests run together. With fixed GPU memory, longer prompts and answers mean fewer requests can run at once.