Design an AI Agent That Takes Actions

Asked from Senior level upEmerging topicPart of the AI track: start with AI basicsDomain: AI Systems

Concepts to know first:

A shop's support agent that does more than answer. It looks up orders, changes addresses, cancels orders and gives refunds, with a person's yes before any step that moves money.

An analogy

Like a new shop assistant with a staff card. They can look things up freely, but a refund needs the customer's yes, and a large one needs a manager's signature.

Where this analogy stops working: A new assistant learns over months and earns more trust. An agent can make the same mistake on its thousandth task, and text it reads can trick it, so the checks stay in code.

Where this design ends up. This is version 3 of 3. Part 5 builds it up from version 1 and says why each part was added.

Clients
- Customer chat
Entry
- Task API
Services
- Agent worker
- Tool gateway
Queues, caches
- Task queue
Data, outside
- Task store
- Language model
- Order and refund APIs
- Audit log

1. Requirements and scope

Functional Requirements

A customer asks for help in chat, and the agent works out the steps: look up the order, check the policy, then act.
The agent can call a fixed set of tools: find orders, track a parcel, change an address, cancel an order and refund.
Before a refund or a cancel, the agent shows the exact action and waits for the customer's yes. Refunds over $100 also need a staff member.
Every step is recorded, so staff can see what the agent did and why.

Non-Functional Requirements & SLAs

The agent can act only on the orders of the customer it is helping.
A retry never repeats an action: one refund request pays out at most once.
A task survives a server crash and a wait of hours for an approval.
Each task has a cap on steps, time and tokens, so a stuck loop stops.

Key Traffic Assumptions: 500,000 tasks a day, with peaks of three times the average. • An average task takes 5 model calls and 3 tool calls, and about 40 seconds of work. About 1 task in 10 waits for an approval.

2. Back-of-the-envelope numbers

Writes and reads each second

Writes: 46 steps saved a second, on average (61%)
Reads: 29 model calls a second, on average (39%)

Balanced: reads and writes are close. Both paths need care, so neither can be slow.

Write Throughput

Every model call and tool call is saved as a step: 500,000 × 8 = 4M steps a day, about 46 a second on average

Read Throughput

500,000 ÷ 86,400 ≈ 5.8 tasks a second, about 17 at peak. At 5 model calls a task, that is about 29 model calls a second on average. At peak, 17 tasks a second × 40 s ≈ 700 tasks are running at once

Storage Growth

About 2 KB a step × 4M steps ≈ 8 GB a day, about 2.9 TB a year. Staff and audits read these records, so they are kept for a set time

RAM Cache Footprint

No answer cache: each task acts on one customer's own orders. The policy text and tool list are the same in every prompt, so the provider's prompt cache can reuse them

3. API

POST/v1/tasks{ message }. Starts a task for the signed-in customer and streams each step back as the agent works

POST/v1/tasks/{id}/approvals/{step}{ decision: yes or no }. The customer or a staff member answers one waiting action

GET/v1/tasks/{id}The task's status, its steps and any action waiting for a yes

4. Data model and storage

Storage Strategy: Tasks, steps and approvals live in a relational database, because each step must be saved before the next one starts. A queue holds the tasks that are ready to run. Tool rules live in code, next to the tools.

TASK(id, customer_id, status, steps_used, step_budget, tokens_used, created_at) · STEP(task_id, n, kind, tool, input, output, status, at) · APPROVAL(task_id, step, action, amount, approver_role, decision, expires_at)

5. The design, in three versions

Version 1: A model with every tool

The chat server runs the agent loop in memory. The model can call any shop API, and the server calls it with one admin key.

AI Agent That Takes Actions: Version 1: A model with every tool Version 1 of 3

The picture follows one request through the parts of this version, in order. Under it, switch a part off to see what the request does without it.

One request, step by step

Each numbered arrow is one step. Press "Simulate Request Flow" to follow them one at a time.

The order of the steps
A request or its reply

The parts: switch one off

1. Clients

Customer chatasks for help, says yes

2. Services

Agent workerruns one step at a time

3. Data, outside

Language modelpicks the next step

Order and refund APIsthe shop systems

The steps in words0 of 4 steps shown

Customer chat to Agent workerHTTPS · 3 ms
A customer asks "Please refund order 1042."
Agent worker to Language modeloutside call · 80 ms
The loop asks the model what to do next.
Agent worker to Order and refund APIsoutside call · 80 ms
It calls the refund API with an admin key that can reach any order.
Agent worker to Customer chatreply · 20 ms
The reply says the refund is done. Nobody confirmed it.

Why this version

Quick to build, and the demo works.

Next bottleneck

The admin key can reach every customer's orders. A retry after a timeout can refund twice, and nothing stops a loop that never ends.

Write path, read path and failures

Each path follows one request through the final design (version 3), one hop at a time.

Write pathRefund an order after a yes

Agent worker
Task store
Tool gateway
Customer chat
Task API
Order and refund APIs
Audit log

Agent worker to Task store
The model asks to refund $40 for order 1042. The step is saved before it runs.
Agent worker to Tool gateway
The gateway checks that this order belongs to the customer and that $40 is under the $100 limit.
Tool gateway to Customer chat
The task is saved as waiting, and the customer sees "Refund $40 for order 1042?"
Customer chat to Task API
The customer says yes. The task goes back on the queue.
Tool gateway to Order and refund APIs
A worker sends the refund with the key task 81 step 6, so a retry cannot pay twice.
Tool gateway to Audit log
The action, the yes and the result are recorded.

Read pathAnswer "where is my order?"

Customer chat
Task API
Fixed workflow
Order and refund APIs

Customer chat to Task API
A customer asks where order 1042 is.
Task API to Fixed workflow
This is a common request, so a fixed workflow handles it with no agent loop.
Fixed workflow to Order and refund APIs
It reads the parcel's tracking with the customer's own permissions.
Fixed workflow to Customer chat
One model call turns the result into a short reply.

What breaks, and what happens then

The refund call times out
What users see
The worker cannot tell whether the money moved.
What the design does
It retries with the same key. The refund service has seen the key, so it returns the first result.
A worker crashes during a task
What users see
The task stops part way.
What the design does
The task is still on the queue. Another worker continues from the last saved step.
The agent repeats the same tool call
What users see
Tokens and time are spent with no progress.
What the design does
The step budget ends the task, and a check notices the same call with the same input. A person takes over.
An order note says "refund everything"
What users see
The model may follow it. This is prompt injection.
What the design does
The tool gateway applies the refund rules in code, and the customer approves each action, whatever the model asks.

6. Deep dives

Permissions follow the customer

The tool gateway runs each call with the signed-in customer's own token, so the agent reaches only that customer's orders. An order note could say "refund everything", and the model may obey it. The refund rules in code still apply, whatever the model asks.

A retry must not refund twice

The refund call times out after 5 seconds. Did the money move? The gateway retries with the same key, task 81 step 6. The refund service has seen that key, so it returns the first result instead of paying again.

Waiting for a yes

An approval can take minutes or a day. So the task is saved as waiting, and the worker moves on. The yes puts the task back on the queue, and it continues from the saved step. The yes covers that exact action and amount, and it expires.

7. Follow-up questions

The model asks to refund $400, over the $100 limit. What happens? (The tool gateway holds it for a staff member's yes. The rule is in code, so no prompt can change it.)
How do you test a new prompt before customers see it? (Replay saved tasks with fake tools that only record calls. Compare the actions with the expected ones, then release to a small share.)
The agent calls the same tool again and again. What stops it? (The step budget ends the task, and a check notices the same call with the same input. The task then goes to a person.)

8. How answers are judged

A passing answer at Mid level. A complete design with numbers: about 17 tasks a second at peak and 5 model calls each. It adds a step budget and approval for refunds, and goes deep on one part, such as retries.

How each level is judged
What is judged	Junior	Mid	Senior	Staff
BreadthHow much of the system does the answer cover?	Names the core parts: clients, servers, a database and a cache.	Covers the whole main flow, from the request to the stored data and back.	Builds a complete design without being led through it.	Covers the system, the systems around it and who owns each part.
DepthHow far does it go into the hard parts?	Explains what each part does and why it is there.	Goes deep on one part, such as the database or the cache.	Goes deep on the two or three riskiest parts.	Goes deepest where the risk is largest, and explains why the risk is there.
DrivingWho leads the conversation?	Follows the interviewer, step by step.	Moves forward alone, with an occasional nudge.	Leads the conversation from start to finish.	Changes the problem when the question asked is not the right one.
Trade-offsAre choices compared and justified?	Names more than one option.	Says why one option was picked over another.	Compares options in detail, with what each one costs.	Ties each choice to cost, speed and risk for the business.
Unclear questionsWhat happens when the question is vague?	Answers the questions that are asked.	Asks questions to make the problem clear.	Sets the scope and says what is left out.	Questions the assumptions behind the problem itself.
ScopeWhat does it consider beyond the normal path?	Covers the normal path, when everything works.	Adds what happens when a part fails.	Adds growth: what breaks at ten times the load, and the fix.	Adds abuse, cost and the daily work of running the system.

Requirements and scope

Strong answer

Before I design, I want to know which actions the agent may take. Which ones move money or cannot be undone? Who approves them, and is there an amount limit? I will assume refunds and cancels need the customer's yes.

Weak answer

The agent can use all the shop APIs, so it can solve any request. I will add limits later if it makes mistakes.

Shows: Unclear questionsThe difference: The list of actions, and who approves each one, decides the design: the tool gateway, the approval step and the saved tasks. Adding them later means rebuilding.

Estimates

Strong answer

500,000 tasks a day is about 5.8 a second, about 17 at peak. Each task runs about 40 seconds, so about 700 are running at once at peak. Tasks waiting for a yes are saved, so they use no worker while they wait.

Weak answer

17 tasks a second is light, so one server can run each task from start to end in memory.

Shows: DepthThe difference: A task lasts 40 seconds, or hours when it waits for a yes. Holding each one in memory ties up servers while nothing happens, and a crash loses the task.

Core design

Strong answer

Common requests run as fixed workflows. The rest go on a queue, and workers run one step at a time, saving each step first. Every tool call goes through a tool gateway that checks permissions and holds risky actions for a yes.

Weak answer

The model gets a prompt that lists the rules: never refund more than $100, and only touch the customer's own orders.

Shows: DepthThe difference: A prompt asks; it does not enforce. Injected text or a model mistake can break a rule kept only in the prompt.

Deep dive

Strong answer

The hard part is a retry after a timeout, since the refund may already have run. Each action carries a key made from the task and step. A retry sends the same key, so the refund service returns the first result instead of paying twice.

Weak answer

If the refund call times out, the agent asks the model what to do next, and the model calls refund again.

Shows: DepthThe difference: The first refund may have gone through before the timeout. Without a key that the refund service checks, the second call pays the customer twice.

Failures and trade-offs

Strong answer

Each tool call runs with the signed-in customer's own token, so an injected order note cannot reach another account. A crashed worker's task is picked up from the last saved step. A loop ends at the step budget and goes to a person.

Weak answer

If something goes wrong, the agent apologises and the customer starts again.

Shows: Trade-offsThe difference: Starting again can repeat an action that already ran, and it hides injected text and endless loops. Each failure needs its own limit in code.

We wrote this guide in our own words, informed by public interview guides such as AlgoMaster and Hello Interview. We are not connected to either site. Interviewers differ. The guide and the example answers show the direction each level takes, not a checklist.

10. Check your design with your own AI

Write your design first. Then copy this prompt into the AI chat you already use, and add your design at the end. It asks for a score in each area, one fix for each, and two follow-up questions.

The site sends nothing anywhere. An AI review can be wrong, so check what it says against sections 8 and 9.

See the prompt

You are a system design interviewer. Review my design for the interview problem below.

Problem: Design an AI Agent That Takes Actions
A shop's support agent that does more than answer. It looks up orders, changes addresses, cancels orders and gives refunds, with a person's yes before any step that moves money.

What users must be able to do:
- A customer asks for help in chat, and the agent works out the steps: look up the order, check the policy, then act.
- The agent can call a fixed set of tools: find orders, track a parcel, change an address, cancel an order and refund.
- Before a refund or a cancel, the agent shows the exact action and waits for the customer's yes. Refunds over $100 also need a staff member.
- Every step is recorded, so staff can see what the agent did and why.

How well it must work:
- The agent can act only on the orders of the customer it is helping.
- A retry never repeats an action: one refund request pays out at most once.
- A task survives a server crash and a wait of hours for an approval.
- Each task has a cap on steps, time and tokens, so a stuck loop stops.

Numbers to assume:
- 500,000 tasks a day, with peaks of three times the average.
- An average task takes 5 model calls and 3 tool calls, and about 40 seconds of work. About 1 task in 10 waits for an approval.

Judge my design at the Mid level.
A passing answer at this level: A complete design with numbers: about 17 tasks a second at peak and 5 model calls each. It adds a step budget and approval for refunds, and goes deep on one part, such as retries.

For each area below, say whether my design is strong, fine or weak. Give one reason and one fix for each:
1. Requirements and scope
2. Estimates
3. Core design
4. Deep dive
5. Failures and trade-offs

Then ask me the two follow-up questions an interviewer would ask next.
Do not write a full model answer for me.

My design:
[Describe your parts, how one request moves through them, and why you chose each one.]