Module 3 · Lesson 2 of 7
Finding the nearest chunks fast
Speed or the true closest
An approximate index checks only likely chunks; checking fewer is faster, but it finds fewer of the true closest ones.
Scene 1 of 4 · Embed the question
The question is turned into an embedding first, by the same model that embedded the chunks.
- Help app — healthy
- Embedding model — healthy
- Vector database — healthy
- Embed the question. The question is turned into an embedding first, by the same model that embedded the chunks.
- Check every chunk. Checking all 10,000,000 chunks finds the true closest 10, but takes about 20 seconds.
- Low effort. Checking only 1,000 likely chunks takes 2 ms, but finds only 7 of the true closest 10.
- A middle setting. Checking 5,000 takes about 10 ms and finds 9 of 10: the usual trade.
- Now you try. Pick an index setting, and see the time against how many of the true closest 10 it finds. For example: 20,000 checks, about 40 ms, finding 10 of the true closest 10. More checks find all 10 here, at a higher cost in time per search.
Scene 1 of 4: Embed the question. The question is turned into an embedding first, by the same model that embedded the chunks.
Key points
- The question becomes an embedding, the same way the chunks did.
- Checking every chunk is exact but takes seconds at this size.
- An index setting decides how many likely chunks to check.
- Recall is the share of the true closest chunks it finds.
At question time, the question is turned into an (A list of numbers that stands for the meaning of a piece of text. Texts with similar meanings get similar lists.) too. The (A database that stores vectors and finds the ones closest to a given vector, so you can search by meaning.) then finds the (A short passage cut from a longer document, so it can be turned into an embedding and found on its own.) whose embeddings are closest to it.
Comparing the question with every chunk is too slow once there are millions. So (A list of numbers. An embedding is a vector.) databases use an (An extra structure, kept sorted or grouped, that lets a database find matching items without reading all of them.) that checks only the likely areas: (Finding the stored vectors closest to a question's vector. The approximate kind checks only part of the data. It is much faster, but now and then it misses a close match.).
Approximate means it can now and then miss a close chunk. In return it answers in milliseconds instead of seconds.
In system design: You trade a little recall, the share of truly closest chunks found, for speed. Settings on the index control that trade.