Module 3 · Lesson 6 of 7
Keeping the index up to date
When a page changes
An edited page reaches the index as an event: the indexer replaces that page's chunks, so answers follow the change within minutes.
Scene 1 of 5 · The policy changes
At 09:00 someone edits the help page: refunds now close after 14 days, not 30.
- Help pages — healthy — refunds: 14 days
- Change events — healthy
- Indexer — healthy
- Vector index — healthy — still says 30
- The policy changes. At 09:00 someone edits the help page: refunds now close after 14 days, not 30.
- Wait for the night. A rebuild at 02:00 leaves the old 30-day answer live for 17 hours.
- One event per edit. Instead, the edit puts an event on a queue, and the indexer picks up only that page.
- Replace, not add. It deletes the page's old chunks and writes the new ones, about 1 minute after the edit.
- Keep one model. Switching embedding models needs a second index, filled from every page, before searches move to it.
- Now you try. Pick a way to keep the index up to date, and see what a refund question finds at noon. For example: It finds: only the old 30-day chunk. Out of date for 17 hours. The change waits for the 02:00 rebuild, so the old answer stays live for 17 hours.
Scene 1 of 5: The policy changes. At 09:00 someone edits the help page: refunds now close after 14 days, not 30.
Key points
- A nightly rebuild leaves changed facts wrong for most of a day.
- An event per edit lets one worker re-index only that page.
- Replacing a page means deleting its old chunks, not only adding new ones.
- A new embedding model means re-embedding every page before switching.
Documents change: a price goes up, a policy changes, a page is deleted. The (An extra structure, kept sorted or grouped, that lets a database find matching items without reading all of them.) must follow, or search keeps finding the old text and the model answers from it.
Rebuilding the whole index on a timer is simple but slow to catch up. A common design sends an event for each edit through a (A waiting line for work. One part adds messages, and another takes them off and handles them later, at its own pace.), and a worker re-chunks and re-embeds only that page.
An update must replace, not add: delete the page's old (A short passage cut from a longer document, so it can be turned into an embedding and found on its own.), or both versions stay findable. Storing each chunk's page and version makes this easy.
(A list of numbers. An embedding is a vector.) from two (A list of numbers that stands for the meaning of a piece of text. Texts with similar meanings get similar lists.) models cannot be compared. Changing the model means re-embedding every page into a new index, then switching searches over.
In system design: Index updates are a data (A fixed chain of automatic steps, where each step hands its result to the next. A CI/CD pipeline builds, tests and releases every code change.): events, a queue, (Safe to repeat. Doing it twice has the same effect as doing it once.) workers and deletes. Measure how far the index lags behind its sources.