Two numbers describe how a model server feels to use. (How long a user waits from sending a request until the first piece of the answer appears.) (see How a model writes) is the wait for the first piece of the answer: the (A waiting line for work. One part adds messages, and another takes them off and handles them later, at its own pace.) plus reading the (Everything sent to a language model in one request: instructions, context and the question.).
(A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) per second is the speed of every token after that first one. It says how fast the rest of the answer streams in, once the model has started writing.
A bigger batch raises the (A graphics processing unit: a chip that does many simple sums at the same time. Language models run on GPUs because they need huge numbers of sums.)'s total tokens produced every second, across every request together. But since each step takes a little longer with more requests in it, each single user's own tokens per second falls.
Because of this trade-off, a design sets a target for both numbers, not for one alone. A fast start with a slow stream can still feel disappointing once the answer begins.
Example. Illustrative: a chat product might aim for the first token within about 1 second, and at least 15 tokens a second after that. These are example targets, not fixed rules.
In system design: Set a target for both numbers, and watch them together. A larger batch trades one for the other, so pick a batch size that meets both targets.