The Similarity Score You're Trusting Blindly
Most teams pick a similarity threshold once during dev and never look at it again. That number is making decisions for you every day.
Somewhere in your RAG pipeline, there’s a number. Maybe it’s 0.75. Maybe 0.8. It was chosen during initial testing when you were trying to get the system to stop hallucinating, or maybe it was just a default in the library you copied. Either way, nobody’s touched it since.
That number is running your retrieval. And it’s probably wrong.
What the threshold is actually doing
Your similarity score — cosine similarity, dot product, whatever your vector store outputs — is a measure of geometric distance between embeddings, not a measure of relevance. These two things are correlated, but they are not the same thing.
When you set a threshold, you’re saying: “If the score is below X, discard the result.” That sounds reasonable. The problem is X was calibrated on a handful of test queries at a specific point in time, against a corpus that has since changed, using an embedding model you may have since upgraded, with user queries you hadn’t yet seen.
The threshold doesn’t know any of that. It just cuts.
How it fails in production
Teams see two failure modes, rarely at the same time:
Over-filtering: The threshold is too aggressive. Relevant documents get clipped because they’re worded differently from the query. Users get “I don’t have information on that” when the answer was in the index the whole time. Support tickets increase. The team adds more documents, wondering why the system doesn’t know things.
Under-filtering: The threshold was loosened after complaints. Now it’s too permissive. Marginally-relevant chunks get passed to the LLM. Answers become vague and hedging. The model starts blending context from multiple loosely-related sources. Hallucination risk climbs.
Most teams oscillate between these two modes without realizing the threshold is the dial they need to turn.
What to do instead
First, instrument it. Log every retrieval: query, scores, chunks returned, final answer. Without logs, you’re flying blind.
Second, build an offline eval set. Take 50–100 real queries with known correct answers. Sweep your threshold from 0.5 to 0.95 in 0.05 increments. Plot recall versus precision. Now you have a curve instead of a guess.
Third, separate your search threshold from your context assembly threshold. You can retrieve more broadly and then rerank or trim downstream — instead of cutting at the vector layer, you get smarter filtering at the content layer where you have more signal.
Finally, revisit it whenever your corpus changes significantly. A threshold calibrated on 10,000 documents will behave differently when your index hits 500,000. The score distribution shifts. Your cutoff doesn’t.
The honest version of the problem
The similarity score feels scientific because it’s a number. But a number you set once and never revisited isn’t a system — it’s a guess that got frozen in place.
Your retrieval quality is only as good as the threshold controlling it. Treat it like a parameter that needs tuning, not a setting that needs deploying.