Module 9 · Lesson 5 of 6

Capping spend per customer

A budget checked on every call

Each customer gets a token budget, checked before every model call, with a warning before the limit and a clear refusal at it.

Scene 1 of 5 · A monthly budget

The plan allows 2,000,000 tokens a month, checked in a fast store before every model call.

CustomerCustomerToken budgetToken budgetLanguage modelLanguage model
  • Customer — healthy
  • Token budget — healthy
  • Language model — healthy
This month's plan: 2,000,000 tokens
  • Used0 tokens
  • Held for a call0 tokens
  • Left (highlighted)2,000,000 tokens
Requests this month
0
  1. A monthly budget. The plan allows 2,000,000 tokens a month, checked in a fast store before every model call.
  2. Hold the most needed. Before the call, it holds 1,700 prompt tokens plus the 1,000-token longest answer: 2,700 in all.
  3. Record the real count. The answer used 300 tokens, so the hold is released and the real 2,000 is recorded.
  4. Warn near the end. After 800 requests of 2,000 tokens, 80% is used, and the customer gets a warning.
  5. Request 1,000 refused. After 999 requests, 2,000 tokens are left: too few to hold 2,700, so request 1,000 is refused.
  6. Now you try. Change the tokens each request really uses, and see how far the plan goes. For example: 499 requests run, with a warning after request 400; request 500 is refused. The budget counts tokens, not requests, so longer requests use it up faster.

Scene 1 of 5: A monthly budget. The plan allows 2,000,000 tokens a month, checked in a fast store before every model call.

Key points

  1. Count tokens per customer, the way a rate limiter counts requests.
  2. Hold room for the longest answer allowed, then record the real count.
  3. Warn customers well before they reach the limit.
  4. At the limit, refuse with a clear message, not a surprise bill.

One customer, or one runaway script, can send thousands of long requests and run up a large bill.

Give each customer a budget per day or month, and a per minute, checked before the model is called.

The answer length is unknown in advance, so reserve the largest answer allowed, then record the real count afterwards.

When a customer nears their limit, warn them first, then slow or refuse requests, instead of surprising them with a bill.

Example. A plan with 2 million tokens a month allows about 1,000 requests of 2,000 tokens. The first request that no longer fits gets a clear message that the limit is reached.

In system design: Count tokens per customer in a fast store, the way a rate limiter counts requests, and check that count on every call.

Check yourself

Under a token budget, why hold room for the longest allowed answer before a call?