Module 3 · Lesson 5 of 7
Measuring search quality
Did search find the right chunk?
Mark the chunk that answers each saved question, then count how often search puts it in the top k.
Scene 1 of 5 · Mark the right chunk
Each of 8 saved questions is marked with the one chunk that holds its answer.
- Saved questions — healthy — 8 marked questions
- Search — healthy
- Right chunk in top k? — healthy
| Saved question | Right chunk ranked | In top k |
|---|---|---|
| Refund window for shoes? | 1st | – |
| Store credit after 45 days? | 2nd | – |
| Refunds on sale items? | 1st | – |
| Return with no receipt? | 4th | – |
| Swap for another size? | 1st | – |
| Send returns from Canada? | Not in top 10 | – |
| Who pays return postage? | 3rd | – |
| Refund to a gift card? | 1st | – |
- Mark the right chunk. Each of 8 saved questions is marked with the one chunk that holds its answer.
- Note each rank. For every question, record where the search placed its marked chunk in the results.
- Recall at 1. Only 4 of 8 questions have their chunk in first place.
- Recall at 4. Taking the top 4 finds 7 of 8, at the price of a longer prompt.
- Never found at all. "Send returns from Canada?" is never found, whatever k is, so the search itself needs fixing.
- Now you try. Set k, the number of chunks kept, and count the questions whose right chunk is among them. For example: Recall at 2: 5 of 8 questions. A bigger k finds more right chunks, but each one adds tokens to every prompt.
Scene 1 of 5: Mark the right chunk. Each of 8 saved questions is marked with the one chunk that holds its answer.
Key points
- Each saved question names the one chunk that holds its answer.
- Recall at k counts how often that chunk lands in the top k.
- A bigger k finds more, but every extra chunk lengthens the prompt.
- A chunk that is never found needs better chunking, not a bigger k.
A wrong answer has two possible causes: the search missed the right text, or the model misused it. Measure the search on its own, so you know which part to fix.
Take saved questions and mark, for each one, the (A short passage cut from a longer document, so it can be turned into an embedding and found on its own.) that holds its answer. Run the search, and note where that chunk ranks.
(The share of test questions whose right chunk appears in the top k search results.) is the share of questions whose right chunk lands in the top k results. A higher k finds more, but every extra chunk makes the (Everything sent to a language model in one request: instructions, context and the question.) longer and costlier.
A question whose chunk is never found needs a different fix: better chunking, a missing page, or word search for an exact term.
In system design: Track recall at k beside answer quality, and measure both again whenever chunking, the (A list of numbers that stands for the meaning of a piece of text. Texts with similar meanings get similar lists.) model or the index settings change.