Caching by meaning

3 of 12 new questions reuse a cached answer, saving 3 model calls. 2 of those get the wrong answer. 6 could safely have reused one but did not.

• Reused, right answerNew: “How can I reset my password?”Cached: “How do I reset my password?” · similarity 0.97
• Reused, wrong answerNew: “Is shipping free to the US?”Cached: “Is shipping free to the UK?” · similarity 0.96
• Reused, wrong answerNew: “What time do you open on Sunday?”Cached: “What time do you open on Monday?” · similarity 0.95
• Asks the modelNew: “How do I change my password?”Cached: “How do I reset my password?” · similarity 0.93
• Asks the modelNew: “How do I pause my subscription?”Cached: “How do I cancel my subscription?” · similarity 0.92
• Asks the modelNew: “Where is my parcel?”Cached: “Where is my order?” · similarity 0.91
• Asks the modelNew: “How can I reach customer support?”Cached: “How do I contact support?” · similarity 0.90
• Asks the modelNew: “How do I change my billing address?”Cached: “How do I change my delivery address?” · similarity 0.89
• Asks the modelNew: “Can I buy a gift card?”Cached: “Do you sell gift cards?” · similarity 0.88
• Asks the modelNew: “Can I get my money back?”Cached: “What is your refund policy?” · similarity 0.86
• Asks the modelNew: “How do I delete my last order?”Cached: “How do I delete my account?” · similarity 0.84
• Asks the modelNew: “Can I pay by card?”Cached: “Which payment methods do you take?” · similarity 0.80

What this simplifies. The questions and their similarity scores are made up to show the idea. Real scores come from comparing embeddings. The right threshold depends on the model and the kind of question, so teams test it on real traffic.