Module 1 · Lesson 5 of 6
Long chats: what to send from earlier turns
- Asha — chats with support — healthy
- Chat app — builds each prompt — healthy — NOW — turn 1: 550 tokens
- Chat history — every turn saved — healthy
- Model — window 4,000 tokens — healthy
- A message or prompt
- Saved turn
- Too long
Step 1 of 4 · Chat app
Done: Turn 1: Asha says her order is #4417
- What happens
- The app sends the model its 500-token instructions and her 50-token message: 550 tokens.
- Why
- A model keeps nothing between calls. It only knows what the app sends this time.
- What changes
- The turn is saved, so later prompts can include it.
- Next
- Each turn is saved in the chat history
Event log
3 more to go. Press a line to go back to that step.
Prompt size each turn, out of a 4,000-token window (3,500 usable)
- Turn 1550
Step 1 of 4: Chat app. Turn 1: Asha says her order is #4417. The app sends the model its 500-token instructions and her 50-token message: 550 tokens.
What this simplifies. The sizes are round examples: real turns vary, and real windows are often far larger, such as 128,000 tokens or more. A larger window delays the problem; it does not remove it, and every token sent is paid for. Real apps often mix these ways.
A model keeps nothing between calls. A chat app seems to remember because it sends the earlier turns again in every (Everything sent to a language model in one request: instructions, context and the question.). So each turn makes the next prompt longer, and every (A piece of text that a language model reads or writes, often part of a word. Prices and limits are counted in tokens.) sent is paid for.
Sooner or later the chat outgrows the (The most tokens a model can read at once, counting both the prompt and the answer.), and most APIs then refuse the call. The app must choose what to send. The simplest choice keeps only the last few turns, but then anything older is forgotten.
Two choices keep old facts without sending everything. A summary note folds older turns into a few lines, at the cost of one more small call each time. A lookup saves every turn with an (A list of numbers that stands for the meaning of a piece of text. Texts with similar meanings get similar lists.), and sends only the old turns that match the new message. That is (Looking up relevant documents first and putting them in the prompt, so the model answers from them instead of from memory alone.) over the chat itself.
Each can fail in its own way. A note can leave out a detail that matters later, and a search can miss the right turn. Many apps mix them: recent turns whole, a note for the rest, and a lookup for exact facts.
In system design: Decide up front what a chat keeps, and test it on a long conversation whose last question needs a fact from the first turn.