Module 2 · Lesson 3 of 4
When a model grades a model
Check the checker
A second model can grade answers fast, but it must first be compared with a person, because it can be fooled.
Scene 1 of 4 · A person marks them
A person checks 10 answers against their checks: 8 pass and 2 fail.
| Question | Person | Grader |
|---|---|---|
| Can I return shoes after 40 days? | pass | ? |
| Do you ship to Canada? | pass | ? |
| How do I reset my password? | pass | ? |
| Is the blue jacket in stock? | pass | ? |
| What is your phone number? | fail | ? |
| Can I pay with a gift card? | pass | ? |
| Why was I charged twice? | pass | ? |
| Can I change my order after it ships? | pass | ? |
| Do you match lower prices? | fail | ? |
| Where is order 123? | pass | ? |
- Marks that agree
- not shown yet
- Bad answers the grader caught
- not shown yet
- A person marks them. A person checks 10 answers against their checks: 8 pass and 2 fail.
- A model marks them. A second model marks the same answers in seconds, far faster than a person can.
- Where they differ. They agree on 8 of 10, but the grader catches only 1 of the 2 bad answers.
- Fooled by a warm tone. It passed a friendly answer that never gave the phone number: tone won over the actual check.
- Now you try. Pick an answer, and compare the person's mark with the grader's. For example: Person: pass. Grader: fail. The answer was "Yes. 5 to 7 days." The answer was right but short, and the grader marked it down: it tends to favour longer answers.
Scene 1 of 4: A person marks them. A person checks 10 answers against their checks: 8 pass and 2 fail.
Key points
- A model grader checks many answers fast and cheaply.
- It can favour answers that sound sure, warm or long.
- Compare its marks with a person's on a sample.
- Rely on it only where its marks match people's.
Checking hundreds of answers by hand is slow, so teams often ask a second model to grade each answer against its check.
A (A second model asked to judge whether an answer passes its check. It is fast and cheap, but can be fooled.) is fast and cheap, but it can be fooled. It tends to favour answers that sound confident, longer answers, and answers in its own style.
So compare its verdicts with a person's on a sample, and trust it only for checks where it agrees with people.
In system design: Use a model grader to check more answers, but first measure how often it agrees with people.