Design an LLM Evaluation Platform
A shared service that tests every prompt or model change before release. It runs each team's test set, grades the answers, shows which cases got fixed or broken, and watches a sample of live answers.
Like a school exam board. The same questions go to every class, the answers are marked against a key, and a second teacher checks a sample of the marking.
Where this analogy stops working: A pupil gives the same answer twice. A model may not, so the platform runs a changed case again before it calls the case broken.
- Clients
- Developer
- Live traffic
- Reviewers
- Entry
- Eval API
- Services
- Run service
- Eval workers
- Sampler
- Queues, caches
- Case queue
- Data, outside
- Test sets
- Results store
- Model under test
- Grader model
1. Requirements and scope
- Teams keep test sets of real questions, each with a check, next to their code.
- Every prompt or model change runs its test set. The result shows each case fixed or broken, not only a score.
- Graders score the answers: exact checks in code where possible, and a grader model for the rest.
- A small share of live answers is graded too, and a drop in quality raises an alert.
- A run of 500 cases finishes in about 10 minutes, so a developer can wait for it before merging.
- Runs share the provider's rate limit fairly, so one team's large run cannot block the others.
- Every result records the exact model, prompt, grader and settings, so two runs can be compared.
- Live samples have personal data hidden, and they are deleted after 30 days.
2. Back-of-the-envelope numbers
- Writes: 3.5 results saved a second, on average (38%)
- Reads: 5.8 model calls a second, on average (62%)
Balanced: reads and writes are close. Both paths need care, so neither can be slow.
3. API
4. Data model and storage
CASE(set_id, case_id, input, check, must_pass) · RUN(id, team_id, set_version, prompt_version, model, grader_version, settings, status) · RESULT(run_id, case_id, answer, passed, grader_reason, tokens) · REVIEW(result_id, reviewer, passed)
5. The design, in three versions
Version 1: A script on a laptop
A developer runs a script that asks the model 20 questions and reads the answers by eye.
LLM Evaluation Platform: Version 1: A script on a laptop Version 1 of 3
The picture follows one request through the parts of this version, in order. Under it, switch a part off to see what the request does without it.
One request, step by step
Each numbered arrow is one step. Press "Simulate Request Flow" to follow them one at a time.
- The order of the steps
- A request or its reply
The parts: switch one off
1. Clients
2. Services
3. Data, outside
The steps in words0 of 3 steps shown
Developer to Test script
A developer runs a script with 20 questions.
Test script to Model under test
The script asks the model each question.
Test script to Developer
The developer reads the answers and decides by eye.
Write path, read path and failures
Each path follows one request through the final design (version 3), one hop at a time.
Write pathGrade a sample of live answers
- Live traffic
- Sampler
- Case queue
- Eval workers
- Grader model
- Results store
Live traffic to Sampler
One in 100 answers is copied, with its question and the model that wrote it.
Sampler to Case queue
Names, emails and card numbers are hidden before the sample is stored.
Eval workers to Grader model
The grader scores the answer against the checks for that kind of question.
Eval workers to Results store
The grade is saved. If the pass rate drops in an hour, an alert fires.
Read pathCheck a prompt change before release
- Developer
- Eval API
- Run service
- Case queue
- Eval workers
- Model under test
- Grader model
Developer to Eval API
A change to prompt v7 starts a run of the 500-case support test set.
Run service to Case queue
The run becomes 500 jobs, one per case.
Eval workers to Model under test
Each case is answered with prompt v7, a few at a time for this team.
Eval workers to Grader model
Exact checks run in code. The grader model scores the rest.
Eval API to Developer
Each case is compared with the last release: 13 fixed, 3 broken. A broken must-pass case blocks the change.
What breaks, and what happens then
A run hits the provider's rate limit
- What users see
- Calls are refused, and the run slows down.
- What the design does
- Workers wait and retry with backoff. A cap for each team keeps one large run from blocking the others.
The provider changes the grader model
- What users see
- Every score shifts, so old and new runs cannot be compared.
- What the design does
- Fix the grader version. Check a new grader against people, and grade the last release again with it, before switching.
A case passes on one run and fails on the next
- What users see
- A change looks like it broke a case it did not touch.
- What the design does
- A changed case runs 3 more times. One that does not fail steadily is marked unstable, for a person to read.
Live samples hold personal data
- What users see
- Test results become a store of private messages.
- What the design does
- The sampler hides personal fields first, few people can read samples, and they are deleted after 30 days.
6. Deep dives
7. Follow-up questions
- The provider retires the grader model. What happens to old scores? (They can no longer be compared. Grade the last release's answers with the new grader, and check it against people, before switching.)
- A team wants to try a new model, not a new prompt. What changes? (Nothing in the flow: the run records the model, and the comparison shows cases fixed and broken. Show cost and latency next to the pass rate.)
- Thumbs-down rise after a release, but the test set still passes. What do you do? (Read the failing live answers, add them to the test set as new cases, then fix the prompt.)
8. How answers are judged
A passing answer at Mid level. A complete design with numbers: about 5.8 model calls a second on average, and about 33 when runs bunch in working hours. It adds a grader model and goes deep on one part.
| What is judged | Junior | Mid | Senior | Staff |
|---|---|---|---|---|
| BreadthHow much of the system does the answer cover? | Names the core parts: clients, servers, a database and a cache. | Covers the whole main flow, from the request to the stored data and back. | Builds a complete design without being led through it. | Covers the system, the systems around it and who owns each part. |
| DepthHow far does it go into the hard parts? | Explains what each part does and why it is there. | Goes deep on one part, such as the database or the cache. | Goes deep on the two or three riskiest parts. | Goes deepest where the risk is largest, and explains why the risk is there. |
| DrivingWho leads the conversation? | Follows the interviewer, step by step. | Moves forward alone, with an occasional nudge. | Leads the conversation from start to finish. | Changes the problem when the question asked is not the right one. |
| Trade-offsAre choices compared and justified? | Names more than one option. | Says why one option was picked over another. | Compares options in detail, with what each one costs. | Ties each choice to cost, speed and risk for the business. |
| Unclear questionsWhat happens when the question is vague? | Answers the questions that are asked. | Asks questions to make the problem clear. | Sets the scope and says what is left out. | Questions the assumptions behind the problem itself. |
| ScopeWhat does it consider beyond the normal path? | Covers the normal path, when everything works. | Adds what happens when a part fails. | Adds growth: what breaks at ten times the load, and the fix. | Adds abuse, cost and the daily work of running the system. |
Requirements and scope
Before I design, I want to know who changes what. Do teams change prompts, models or both? Must a run block a release, or only report? May we store live answers, given they hold user data?
The platform runs each team's questions through the model and shows a score. Teams decide what to do with it.
Shows: Unclear questionsThe difference: Whether a run blocks a release decides how fast and how exact it must be. The rules for live data decide what may be stored at all.
Estimates
400 runs a day of 500 cases need 400,000 calls, at 2 a case. Live grading adds 100,000, so about 5.8 calls a second. But 20 runs at once, each done in 10 minutes, is about 33 a second.
About 6 calls a second is tiny, so the runner can send every case of a run at once.
Shows: DepthThe difference: Runs arrive together in working hours, and a run sent all at once hits the provider rate limit. The peak shapes the design, not the daily average.
Core design
A run service splits each run into one job per case on a queue. Workers call the model, then grade: exact checks in code first, a grader model for the rest. Each result is saved with its model, prompt and grader versions.
For each change, a script runs the test set and prints the pass rate, and the team ships if it went up.
Shows: DepthThe difference: A pass rate that rises can still hide cases that broke. Results saved per case, with their versions, let two runs be compared case by case.
Deep dive
The hard part is trusting a change. I compare each case with the last release, so a rise from 84% to 86% shows 13 cases fixed and 3 broken. A changed case runs 3 more times, so chance is not counted as a break.
I compare the two pass rates. If the new one is higher, the change is better.
Shows: DepthThe difference: A higher score can still break cases that matter, and one run of a case can differ by chance. Comparing each case, and running changed ones again, shows both.
Failures and trade-offs
When a run hits the provider's rate limit, workers wait and retry with backoff, under a cap for each team. The grader's version is fixed, and a new grader is checked against people first. Live samples hide personal data.
If the provider refuses calls, the run fails and the developer starts it again.
Shows: Trade-offsThe difference: Starting again sends every call again and hits the same limit. Waiting, retrying only the failed calls and capping each team keeps all runs moving.
We wrote this guide in our own words, informed by public interview guides such as AlgoMaster and Hello Interview. We are not connected to either site. Interviewers differ. The guide and the example answers show the direction each level takes, not a checklist.
10. Check your design with your own AI
Write your design first. Then copy this prompt into the AI chat you already use, and add your design at the end. It asks for a score in each area, one fix for each, and two follow-up questions.
The site sends nothing anywhere. An AI review can be wrong, so check what it says against sections 8 and 9.
See the prompt
You are a system design interviewer. Review my design for the interview problem below. Problem: Design an LLM Evaluation Platform A shared service that tests every prompt or model change before release. It runs each team's test set, grades the answers, shows which cases got fixed or broken, and watches a sample of live answers. What users must be able to do: - Teams keep test sets of real questions, each with a check, next to their code. - Every prompt or model change runs its test set. The result shows each case fixed or broken, not only a score. - Graders score the answers: exact checks in code where possible, and a grader model for the rest. - A small share of live answers is graded too, and a drop in quality raises an alert. How well it must work: - A run of 500 cases finishes in about 10 minutes, so a developer can wait for it before merging. - Runs share the provider's rate limit fairly, so one team's large run cannot block the others. - Every result records the exact model, prompt, grader and settings, so two runs can be compared. - Live samples have personal data hidden, and they are deleted after 30 days. Numbers to assume: - 40 teams, each making about 10 changes a day, with test sets of about 500 cases. - The product answers 10 million questions a day, and 1 in 100 answers is graded. Judge my design at the Mid level. A passing answer at this level: A complete design with numbers: about 5.8 model calls a second on average, and about 33 when runs bunch in working hours. It adds a grader model and goes deep on one part. For each area below, say whether my design is strong, fine or weak. Give one reason and one fix for each: 1. Requirements and scope 2. Estimates 3. Core design 4. Deep dive 5. Failures and trade-offs Then ask me the two follow-up questions an interviewer would ask next. Do not write a full model answer for me. My design: [Describe your parts, how one request moves through them, and why you chose each one.]