Module 2 · Lesson 2 of 4
Fixes that break other things
v1 passes 6 of 10 (60%); v2 passes 8 of 10 (80%). v2 fixed 3 and broke 1.
| Question | Check | v1 | v2 |
|---|---|---|---|
| 1. Can I return shoes after 40 days?fixed by v2 | Says no, since the limit is 30 days | fail | pass |
| 2. Do you ship to Canada? | Says yes, with the 5 to 7 day wait | pass | pass |
| 3. How do I reset my password? | Gives the reset-link steps | pass | pass |
| 4. Is the blue jacket in stock?fixed by v2 | Says it cannot check live stock | fail | pass |
| 5. What is your phone number?broken by v2 | Gives the support phone number | pass | fail |
| 6. Can I pay with a gift card? | Says yes | pass | pass |
| 7. Why was I charged twice?fixed by v2 | Offers the refund steps, without blame | fail | pass |
| 8. Can I change my order after it ships? | Says no, and offers a return | pass | pass |
| 9. Do you match lower prices? | Says there is no price matching | fail | fail |
| 10. Where is order 123? | Asks the user to sign in first | pass | pass |
Read one question's answers, and how each was scored
5. “What is your phone number?”
Scoring rule. The answer passes when it contains "555 0100".
“You can call support on 0800 555 0100.”
“You can reach us any time through our friendly help page, where our team is always happy to help.”
A model grader against a person, on v2's answers
They agree on 8 of 10, but the grader caught 1 of the 2 answers the person failed. They disagree on:
- “Do you ship to Canada?” The answer: “Yes. 5 to 7 days.” The grader says fail; the person says pass, since the check is: says yes, with the 5 to 7 day wait.
- “What is your phone number?” The answer: “You can reach us any time through our friendly help page, where our team is always happy to help.” The grader says pass; the person says fail, since the check is: gives the support phone number.
What this simplifies. The questions, answers and checks are made up to show the idea. Each answer is scored by a rule that looks for exact words. Such a rule is cheap and easy to read, but a right answer in other words would fail it. A real test set has hundreds of questions, and the grader here stands for a second model asked to judge each answer.
A change that fixes some answers often breaks others. Looking only at the overall score can hide this.
So compare each question between the old version and the new one, and look at what got fixed and what got broken.
In system design: Read every question a change breaks before shipping it, even when the overall score goes up.