Why Similarity Search Alone Returns Bad Results

The Similarity Assumption

Vector similarity operates on a single assumption: the best answer to a query is the stored text whose embedding vector is closest to the query's embedding vector. This works surprisingly well for small, curated knowledge bases where all content is current, verified, and non-overlapping. When you have 200 carefully written documentation pages and a user asks a question, the page with the most similar text is usually the right answer.

The assumption breaks down when the knowledge base grows organically. A customer support system that accumulates memories from real conversations inevitably stores multiple versions of the same information as products change, contradictory statements from different agents, casual remarks alongside definitive answers, and temporary workarounds alongside permanent solutions. All of these entries compete equally in similarity search because the ranking mechanism cannot see the differences between them.

The Five Failure Modes

Stale Results

The most common failure is returning outdated information. When a product changes its API from version 2 to version 3, both sets of documentation exist in the memory store. A query about "how to authenticate" matches both versions with nearly identical similarity scores because both discuss authentication using similar vocabulary. The system has no way to prefer the v3 documentation over the v2 documentation because similarity does not encode temporal validity.

This problem gets worse over time. After two years of operation, a system might have five generations of authentication documentation, each with slightly different procedures. The most similar result to a new query could be from any of those generations, and the user has no way to know whether the answer is current without verifying it independently.

Contradictory Results

When information changes, the old and new versions often contradict each other. "The rate limit is 100 RPM" and "The rate limit is 500 RPM" both have high similarity to a query about rate limits, and both might appear in the top results. Returning contradictory information to an LLM for answer generation produces confused, hedging responses ("The rate limit is either 100 or 500 RPM depending on your plan") or confidently wrong answers (it picks one at random).

Similarity search cannot detect contradictions because it operates on individual documents in isolation. It does not compare documents against each other, track which superseded which, or evaluate consistency across the result set. Cognitive scoring's confidence mechanism addresses this through contradiction detection during consolidation.

Noise from Casual Mentions

Not all stored memories are equally authoritative. A memory that says "I think the timeout is around 30 seconds" and a memory that says "The timeout is configured to 30000ms in the production config at line 247" both match a query about timeouts with similar similarity scores. But the second memory is clearly more authoritative, more specific, and more useful. Similarity cannot distinguish between a casual remark and a precise, verified statement.

This is particularly problematic for systems that store conversations or meeting notes where participants make uncertain statements, hypothesize, or speculate. Those casual mentions accumulate in the memory store and compete with definitive answers in similarity rankings.

Missing Contextual Connections

Similarity search misses connections that are obvious to a human who understands the domain. A query about "why deployments are failing on staging" might best be answered by a memory about "we updated the Kubernetes cluster version last Wednesday," but the text similarity between deployment failures and cluster version updates is low. A human would immediately connect these through domain knowledge (cluster updates can break deployments), but vector embeddings do not encode this kind of causal reasoning.

Spreading activation through a knowledge graph captures these connections. If both memories share entities related to infrastructure, Kubernetes, and the staging environment, the graph traversal boosts the cluster update memory's ranking for the deployment failure query. The entities bridge the gap that text similarity cannot cross.

Diminishing Returns at Scale

As the number of stored memories increases, the average similarity score of the top results converges. In a store with 100,000 memories, dozens of entries might score between 0.85 and 0.90 similarity for a given query. At this point, the difference between the first and twentieth result is effectively noise, and the ranking becomes arbitrary within the top cluster. The system returns plausible-looking results, but the ranking order carries almost no useful signal.

This convergence is a mathematical property of high-dimensional vector spaces. As more points are added to the space, the nearest neighbors become increasingly similar in distance, making it harder to distinguish meaningful matches from incidental ones. Adding a second scoring dimension (like activation or confidence) breaks the tie and restores meaningful ranking differentiation.

What Similarity Search Does Well

Despite its limitations, similarity search is an essential first stage. It excels at the recall problem: finding memories that are topically related to the query. Even when the ranking within the top results is unreliable, the set of candidates that similarity search returns usually contains the right answer somewhere. The problem is not that similarity fails to find relevant content; it is that similarity fails to rank that content by quality, currency, and reliability.

This is why the best retrieval architectures use similarity search for candidate retrieval and a separate mechanism for candidate ranking. The candidate retrieval stage needs to be fast and have high recall (finding all potentially relevant results). The ranking stage needs to be precise and multi-dimensional (ordering those results by actual usefulness). Trying to make similarity search do both jobs leads to the failure modes described above.

How Cognitive Scoring Addresses Each Failure