Module 0 · Lesson 6 of 10

The context window

One window for prompt and answer

The prompt and the answer share one fixed space, so a long prompt leaves little room to answer.

Scene 1 of 4 · A fixed limit

This example model reads at most 128,000 tokens per request, counting the prompt and the answer together.

One request, in tokens
  • Context window (example model) (highlighted)128,000 tokens
  • Your prompt0 tokens
  • Room left to answer128,000 tokens
Pages that fit
about 300
  1. A fixed limit. This example model reads at most 128,000 tokens per request, counting the prompt and the answer together.
  2. A long prompt. A prompt of 120,000 tokens leaves room for only 8,000 tokens of answer.
  3. A whole manual. A 400-page manual is about 170,667 tokens, more than the whole window, so it cannot go in.
  4. Send what matters. Send only the 20 pages that matter, about 8,533 tokens, and most of the window is free.
  5. Now you try. Set the prompt size, and see how much room is left for the answer. For example: Room left for the answer: 8,000 tokens. The prompt and the answer share one window, so a shorter prompt leaves more room to answer.

Scene 1 of 4: A fixed limit. This example model reads at most 128,000 tokens per request, counting the prompt and the answer together.

Key points

  1. A model reads only so many tokens per request.
  2. The prompt and the answer count against the same limit.
  3. A document bigger than the window has to be cut down.
  4. Send only the parts that matter, and room is left over.

A model cannot read an unlimited amount of text in one request. The is the most it can read at once, counting the you send and the answer it writes together.

If a document is longer than the context window, it will not fit. It must be shortened, split into pieces, or chosen from carefully, so only the most relevant part is sent.

The size of the window depends on the model. It ranges from a few thousand tokens to a million or more, and many models also set a smaller limit on the answer alone. Check the limits of the model you use.

A bigger context window lets you send more background at once. Sending more text costs more, since you pay for every token the model reads, and a long request can also take longer to answer.

Example. Take an example model with a 128,000-token context window. It holds roughly 96,000 English words, about 300 pages of a book. A user manual longer than that cannot go into the prompt whole; it must be cut down first.

In system design: Your backend keeps the prompt, and any documents it adds, under the model's context window. Going over it means shortening text or sending fewer documents.

Check yourself

A model's context window is 8,000 tokens, and your prompt is 7,900 tokens. What does that leave for the answer?