Design an LLM Gateway
One service that every team's app calls to reach several model providers. It checks keys and budgets, limits rates, falls back when a provider fails, and reports the cost of each team.
Like a company's travel desk. Every team books through one desk, which knows each team's budget, the approved airlines, and what to do when a flight is cancelled.
Where this analogy stops working: A travel desk has hours to decide. A gateway decides in a few milliseconds for every call, and it must never become the slowest or least reliable part of the system.
- Clients
- Team apps
- Entry
- Load balancer
- Services
- Gateway server
- Queues, caches
- Limits store
- Usage queue
- Data, outside
- Provider A
- Provider B
- Warehouse
1. Requirements and scope
- Give every team one API for chat and embedding calls, whichever provider serves them.
- Check each team's key, rate limit and monthly token budget before a call goes out.
- When a provider is slow or down, send the call to a backup model, and say which model answered.
- Record tokens, cost and time for every call, so each team sees its own bill.
- Adds little delay: an example target is under 20 ms of gateway time per call.
- Stays up when a provider does not, and is never a single point of failure itself.
- Streams each answer through as it is written, without waiting for the end.
- Provider keys stay in the gateway, so no app holds them.
2. Back-of-the-envelope numbers
3. API
4. Data model and storage
TEAM(id, monthly_token_budget, rate_limit) · ROUTE(name, primary_model, fallback_model, timeout_ms) · USAGE(call_id, team_id, model, tokens_in, tokens_out, cost, latency_ms, status, at)
5. The design, in three versions
Version 1: Every app calls the provider
Each team puts a provider key in its own app and calls the model directly.
LLM Gateway: Version 1: Every app calls the provider Version 1 of 3
The picture follows one request through the parts of this version, in order. Under it, switch a part off to see what the request does without it.
One request, step by step
Each numbered arrow is one step. Press "Simulate Request Flow" to follow them one at a time.
- The order of the steps
- A request or its reply
The parts: switch one off
1. Clients
2. Data, outside
The steps in words0 of 3 steps shown
Team apps to Provider A
A team app sends a chat call with its own copy of the provider key.
Provider A to Team apps
The answer streams back to the app.
Another team app to Provider A
Another team does the same, so the key sits in many apps and nobody sees the total bill.
Write path, read path and failures
Each path follows one request through the final design (version 3), one hop at a time.
Write pathRecord what a call cost
- Gateway server
- Limits store
- Usage queue
- Warehouse
Gateway server to Limits store
The answer has ended. The gateway gives back the tokens it held but did not use.
Gateway server to Usage queue
It puts one record on the queue: team, model, tokens, cost and time.
Usage queue to Warehouse
A loader writes the records in batches, and each team sees its bill.
Read pathAnswer one chat call
- Team apps
- Gateway server
- Limits store
- Provider A
Team apps to Gateway server
An app sends a chat call with its team key and the route name chat-default.
Gateway server to Limits store
The gateway checks the team rate limit and holds max_tokens from its budget.
Gateway server to Provider A
The route sends the call to the main model, with a time limit for the first token.
Provider A to Gateway server
Tokens stream back as the model writes them.
Gateway server to Team apps
The app shows each token at once, with the name of the model that wrote it.
What breaks, and what happens then
Provider A is down
- What users see
- Calls on its routes get no first token.
- What the design does
- Its circuit breaker opens, and calls go to the fallback model on Provider B until trial calls succeed.
The limits store cannot be reached
- What users see
- The gateway cannot check rate limits or budgets.
- What the design does
- Choose in advance: refuse calls, or allow them for a short time under a rough limit kept in each server.
One team's script sends calls in a loop
- What users see
- It uses up the shared provider limit, and other teams' calls fail.
- What the design does
- A rate limit for each team, and a cap on its calls in flight, stop it at the gateway.
A gateway server stops part way through a stream
- What users see
- The answers it was streaming end early.
- What the design does
- New calls go to the other servers, and the app shows an error so the user can ask again. Held tokens are given back after a time limit.
6. Deep dives
7. Follow-up questions
- A team's prompts contain customer data. What should the gateway log? (Tokens, cost and model always. Prompts only where the team chose it, with access controls and a short retention.)
- Provider A raises its price. How do you move traffic without changing every app? (Change the model behind the route name, test the new model on that route first, and move a small share of traffic at a time.)
- How do you stop one team's runaway script from using up the shared provider limit? (A per-team rate limit set below the provider's, and a cap on each team's calls in flight.)
8. How answers are judged
A passing answer at Mid level. A complete design with numbers: about 230 calls a second on average and 1,000 at peak. It adds rate limits for each team and goes deep on one part, such as fallback.
| What is judged | Junior | Mid | Senior | Staff |
|---|---|---|---|---|
| BreadthHow much of the system does the answer cover? | Names the core parts: clients, servers, a database and a cache. | Covers the whole main flow, from the request to the stored data and back. | Builds a complete design without being led through it. | Covers the system, the systems around it and who owns each part. |
| DepthHow far does it go into the hard parts? | Explains what each part does and why it is there. | Goes deep on one part, such as the database or the cache. | Goes deep on the two or three riskiest parts. | Goes deepest where the risk is largest, and explains why the risk is there. |
| DrivingWho leads the conversation? | Follows the interviewer, step by step. | Moves forward alone, with an occasional nudge. | Leads the conversation from start to finish. | Changes the problem when the question asked is not the right one. |
| Trade-offsAre choices compared and justified? | Names more than one option. | Says why one option was picked over another. | Compares options in detail, with what each one costs. | Ties each choice to cost, speed and risk for the business. |
| Unclear questionsWhat happens when the question is vague? | Answers the questions that are asked. | Asks questions to make the problem clear. | Sets the scope and says what is left out. | Questions the assumptions behind the problem itself. |
| ScopeWhat does it consider beyond the normal path? | Covers the normal path, when everything works. | Adds what happens when a part fails. | Adds growth: what breaks at ten times the load, and the fix. | Adds abuse, cost and the daily work of running the system. |
Requirements and scope
Before I design, I want to check the scope. Do teams call only chat models, or embeddings too? Must budgets be hard limits or warnings? May the gateway store prompts? I will assume streamed answers and under 20 milliseconds of added delay.
The gateway forwards each call to the provider and returns the answer. I will add the other features once the basic path works.
Shows: Unclear questionsThe difference: Budgets, fallback and logging rules change the design itself, such as a shared store for limits. Leaving them out designs a proxy, not a gateway.
Estimates
20 million calls a day is about 230 a second, and the stated peak is 1,000. Each answer streams for about 10 seconds, so at peak about 10,000 calls are open at once. I size the servers for those open connections.
1,000 calls a second is light work for a proxy, so two small servers are enough.
Shows: DepthThe difference: Each call stays open for about 10 seconds while it streams. So the gateway must hold about 10,000 connections at once, not 1,000.
Core design
Gateway servers keep no state and sit behind a load balancer. Each call checks the team key, then its rate limit and budget in Redis. A route name maps to a main model and a fallback, and usage records go through a queue to a warehouse.
Each gateway server keeps its own count of each team's calls and tokens in memory, so the check is fast.
Shows: DepthThe difference: With several servers, each one sees only part of a team's traffic, so a team can use several times its limit. The counts must live in one shared store.
Deep dive
The hard part is the budget, since the cost is known only when the answer ends. Before each call, I hold max_tokens from the team's budget. When the answer ends, I record the real count and give back the rest.
I add each call's tokens to the team's total when it finishes, and refuse new calls once the total passes the budget.
Shows: DepthThe difference: Hundreds of calls can be in flight before the first one finishes and counts. Holding room for each call before it starts keeps the team inside its budget.
Failures and trade-offs
Each provider gets a time limit for the first token and a circuit breaker. Before the first token, a failed call goes to the fallback model. After it, I end the stream with an error, since joining two models' answers would confuse the user.
If the provider fails part way through an answer, the gateway sends the call to the fallback model and keeps streaming.
Shows: Trade-offsThe difference: The user has already seen the start of one answer, and the fallback model writes a different one. Falling back is safe only before the first token.
We wrote this guide in our own words, informed by public interview guides such as AlgoMaster and Hello Interview. We are not connected to either site. Interviewers differ. The guide and the example answers show the direction each level takes, not a checklist.
10. Check your design with your own AI
Write your design first. Then copy this prompt into the AI chat you already use, and add your design at the end. It asks for a score in each area, one fix for each, and two follow-up questions.
The site sends nothing anywhere. An AI review can be wrong, so check what it says against sections 8 and 9.
See the prompt
You are a system design interviewer. Review my design for the interview problem below. Problem: Design an LLM Gateway One service that every team's app calls to reach several model providers. It checks keys and budgets, limits rates, falls back when a provider fails, and reports the cost of each team. What users must be able to do: - Give every team one API for chat and embedding calls, whichever provider serves them. - Check each team's key, rate limit and monthly token budget before a call goes out. - When a provider is slow or down, send the call to a backup model, and say which model answered. - Record tokens, cost and time for every call, so each team sees its own bill. How well it must work: - Adds little delay: an example target is under 20 ms of gateway time per call. - Stays up when a provider does not, and is never a single point of failure itself. - Streams each answer through as it is written, without waiting for the end. - Provider keys stay in the gateway, so no app holds them. Numbers to assume: - 50 teams and about 20 million model calls a day, with peaks of about 1,000 a second. - An average call sends 1,500 tokens and gets 300 back, streamed over about 10 seconds. Judge my design at the Mid level. A passing answer at this level: A complete design with numbers: about 230 calls a second on average and 1,000 at peak. It adds rate limits for each team and goes deep on one part, such as fallback. For each area below, say whether my design is strong, fine or weak. Give one reason and one fix for each: 1. Requirements and scope 2. Estimates 3. Core design 4. Deep dive 5. Failures and trade-offs Then ask me the two follow-up questions an interviewer would ask next. Do not write a full model answer for me. My design: [Describe your parts, how one request moves through them, and why you chose each one.]