Module 4 · Lesson 3 of 5
Provider rate limits and batch jobs
- Nightly job — 600 tickets — healthy
- Model API — 60,000 tokens a minute — healthy — NOW — 600 arrive at once
- Requests
- Refused: 429
Step 1 of 3 · Model API
Done: The job sends all 600 requests at once
- What happens
- Each request asks for one ticket summary, and all of them start together.
- Why
- 60,000 tokens a minute ÷ 2,000 tokens a request = 30 requests a minute.
- What changes
- Only 30 fit in this minute.
- Next
- 570 come back as "429 Too Many Requests"
Event log
2 more to go. Press a line to go back to that step.
Each minute: answered, and refused with 429
No minute has run yet.
Step 1 of 3: Model API. The job sends all 600 requests at once. Each request asks for one ticket summary, and all of them start together.
What this simplifies. The limit, the request size and the price ($3 per million tokens) are examples, and so is the half price for batch work. Real limits are often set per minute for both requests and tokens, and they differ by provider and account.
Model providers limit how much each account may send, usually per minute: a number of requests and a number of (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.). A call over the limit is refused with (The rules an app or browser uses to ask a server for something and get an answer, such as GET /profile/42 and then 200 OK.) status 429, "Too Many Requests", often with a Retry-After header that says how long to wait.
Retrying refused calls straight away only adds more refused calls. Instead, count your own tokens and pace the calls a little under the limit, with one limiter that every caller of the key shares. Then a 429 becomes rare, and each one waits as told.
Work nobody is waiting for, such as nightly summaries, can go to a batch API. You send one file of many requests, and the results come back within a set time, often up to 24 hours. Some providers charge much less for it.
Never batch a request a person is waiting on. Keep live chat on the normal API. Move bulk jobs out of its way, so they do not use up the limit that live users need.
In system design: Pace calls under the provider limit with one shared limiter, honour Retry-After, and send bulk work that nobody waits for to a batch API.