Module 0 · Lesson 7 of 10
Why models make things up
A wrong guess, said with confidence
With no source in the prompt, a model fills the gap with a likely guess that sounds every bit as sure as a fact.
Scene 1 of 5 · Only the question
The bot sends the model only the customer's question, with no refund policy in the prompt.
- Customer — healthy
- Support bot — healthy
- Language model — healthy
- Refund policy — healthy
| What the prompt held | Answer |
|---|---|
| The question only (highlighted) | not yet |
- Only the question. The bot sends the model only the customer's question, with no refund policy in the prompt.
- A likely number. The model writes "90 days", because that is a common refund window in its training text.
- Fluent but false. This is a hallucination: the answer sounds sure, but the real policy says 30 days.
- Add the policy. Now the bot puts the real policy in the prompt, with a rule to answer only from it.
- The true answer. The model answers 30 days and names the policy as its source, so anyone can check it.
- Now you try. Choose what the bot sends, and see what the model answers. For example: The bot answers "60 days", but the real policy says 30 days. Sources stop guessing only when they are right: an out-of-date policy gives a sure but wrong answer.
Scene 1 of 5: Only the question. The bot sends the model only the customer's question, with no refund policy in the prompt.
Key points
- The model judges what sounds likely, not what is true.
- A missing fact gets filled with a common guess.
- Real sources in the prompt, plus a rule to use them, cut this down.
- The source must be current, or the answer is wrong in a new way.
A model writes by guessing the next likely piece of text, one piece at a time. It has no built-in way to check whether a sentence is true; it only judges whether the words sound likely.
Without the right information in the (Everything sent to a language model in one request: instructions, context and the question.), the model can still write a fluent, confident sentence that is false. This is called a (A false statement a model writes as if it were true. It happens because the model guesses likely text and has no built-in fact check.): a false statement written as if it were true.
A hallucination is not a lie, since lying needs intent, and the model has none. It is a wrong guess, stated in the same confident voice the model uses for a correct answer.
Four habits cut this down. Give the model real sources in the prompt, and tell it to answer only from them. Ask it to cite which source each claim came from, and check its answers before a user sees them.
In system design: Any feature that answers from company data needs sources in the prompt and an instruction to use only them. It also needs citations and answer checks.