Module 2 · Lesson 4 of 4
Watching quality in real use
Watch real chats, not only tests
Signals from real traffic, such as how often a person must take over, show a quality drop that a test set missed.
Scene 1 of 4 · A normal week
For five days, about 5% of chats are handed to a person to finish.
- Two days before
- not shown yet
- Two days from
- not shown yet
- A normal week. For five days, about 5% of chats are handed to a person to finish.
- A new release. On day 6, release v2.4 ships, with a new prompt and a new model.
- The jump. From that day, about 12% of chats need a person, up from 5%: check that release first.
- Logs to find out why. Each logged request keeps its prompt, chunks, answer and model, with private fields hidden.
- Now you try. Pick a release, and compare handoffs in the two days before it and the two days from it. For example: Before: 11.5%. After: 12.5%. No real change around this release, so it is probably not the cause.
Scene 1 of 4: A normal week. For five days, about 5% of chats are handed to a person to finish.
Key points
- A test set cannot cover everything real users ask.
- Watch thumbs down, repeated questions and handoffs to a person.
- A sudden jump points at whatever changed that day.
- Logs need limits: they hold prompts with personal data.
A test set cannot cover everything users ask, so also watch signals from real traffic.
Useful signals include thumbs up or down, how often users ask the same thing again, and how often a person has to take over.
Log each request with its (Everything sent to a language model in one request: instructions, context and the question.), the (A short passage cut from a longer document, so it can be turned into an embedding and found on its own.) it used, the answer and the model. Prompts often hold personal data, so limit who can read the logs, hide private fields and delete old logs.
In system design: Watch quality signals next to (How long one request takes, from sending it to getting the answer.) and cost, and alert on sudden changes, as you would for errors.