Module 2 · Lesson 1 of 4
Building a test set
A few by hand can fool you
A fixed set of real questions, each with a check, shows what a change really does, which a few tries by hand can miss.
Scene 1 of 4 · Three tries by hand
You change the prompt and try three questions by hand: all three look better.
| Question | Before | After |
|---|---|---|
| Can I return shoes after 40 days? (highlighted) | fail | pass |
| Is the blue jacket in stock? (highlighted) | fail | pass |
| Why was I charged twice? (highlighted) | fail | pass |
- Pass rate before
- not shown yet
- Pass rate after
- not shown yet
- Three tries by hand. You change the prompt and try three questions by hand: all three look better.
- A fixed set. This small test set keeps 10 real questions, and each one has its own clear check.
- Run all of them. Run on every change, the set shows 60% passing before and 80% after.
- The one that broke. But "What is your phone number?" now fails, and none of your three hand tries found it.
- Now you try. Pick the questions you would try by hand, and compare with the full set. For example, tapping "How do I reset my password" and "Do you match lower prices" gives this: By hand: 0 better, 0 worse, 2 the same. The full set: 60% to 80%, with 1 broken. Your sample missed the broken question, so only the full set shows the whole picture.
Scene 1 of 4: Three tries by hand. You change the prompt and try three questions by hand: all three look better.
Key points
- Three questions tried by hand can all look better.
- A fixed test set gives each question a clear check.
- Run the whole set on every change, like code tests.
- Add every question that goes wrong in real use.
You cannot tell whether a (Everything sent to a language model in one request: instructions, context and the question.) change helps by trying two or three questions by hand. You need a fixed set of real questions, with a clear check for each.
A check can be exact, such as "the answer mentions the 30-day limit", or a short note on what a good answer must do.
Run the whole set on every change, the way code runs its tests, and keep adding questions that went wrong in real use.
In system design: Treat the test set as part of the code: keep it with the code, and run it before every prompt or model change.