The vector store is almost never the reason retrieval is bad. Chunking, missing keyword search, and lack of reranking are the real culprits. The database is usually fine, and swapping it fixes nothing. Shaving milliseconds off retrieval while a language model takes seconds is optimizing the wrong stage. You already know that vector search is just one part of a larger pipeline.
Retrieval quality depends on more than the database. Chunking strategies destroy context when they split paragraphs mid-sentence. Embedding drift creates distance between conversational queries and formal text. Hybrid search fixes lexical misses. Reranking bridges the gap between semantic similarity and factual accuracy.
Semantic gaps and chunking errors
The most common RAG failure happens on day one. Teams pick a chunk size of 512 or 1024 tokens and apply it uniformly across the entire corpus. This approach destroys semantic context. A chunking strategy that splits documents at arbitrary boundaries separates code examples from explanations and breaks legal clauses across two chunks. When neither chunk is complete enough to be useful, retrieval accuracy drops.
Structure-aware chunking combined with parent-document retrieval improves accuracy by 25% to 40% compared to fixed-size chunking. A 50-token overlap between two 512-token chunks does not preserve the relationship between a table header and its data rows. It only duplicates a few words at the boundary.
Embedding model mismatch also causes silent failures. A user searching for "how do I cancel my subscription" might match against document chunks that say "Account termination procedures are outlined in Section 4.2." The embedding distance between a conversational query and formal document text can be large enough that the correct chunk ranks sixth instead of first. If the top-k is set to 3, the system misses the answer entirely.
The lexical failure of pure vector search
Embeddings capture meaning, not exact strings. Vector search finds passages that are semantically similar and routinely misses the exact identifiers enterprise users search for, such as policy numbers, statute references, part codes, error codes, or proper nouns. A user searching for a specific contract clause by its reference gets documents about similar clauses instead of the one they named, which makes the system look stupid to the end user regardless of how much prompt engineering you apply.
This failure is structural. Any base that relies on users searching by SKU or invoice number needs BM25 or sparse vectors sitting next to the embeddings. Weaviate handles this with an alpha parameter. A value of 1.0 is pure vector, while 0.0 is pure keyword. Qdrant allows users to combine sparse and dense vectors through a prefetch stage, which they can fuse with Reciprocal Rank Fusion or Distribution-Based Score Fusion.
The implementations vary significantly. pgvector does not fuse results for you. It provides Postgres full-text search, but the developer must write the code to combine and weight the two result sets. Pinecone supports sparse vectors but imposes a ceiling of 2,048 non-zero values per sparse vector.
Metadata filtering and permission leaks
Every real query carries a filter for tenant, department, document type, or permission scope. In a demo, the filter is an afterthought, but in production, it is the largest source of two complaints: the system returned nothing, or it returned another team’s document. Retrieval that ignores permissions is a data leak and is the failure mode most likely to end a project.
Restrictive filters cause empty results because they drop elements after the vector search runs. A narrow metadata filter over an HNSW index fragments the graph or throws away most of the result set. This is why post-filtering fails. Pre-filtering can look like a fix, but Qdrant notes that pre-filtering should not be used over large datasets because it breaks too many links in the HNSW graph and accuracy drops.
Syntax errors also cause empty returns. If a user provides an array to a $eq operator, Pinecone returns a validation error because the $eq operator expects a string, boolean, or number. For example, if the filter is {"kind": {"$eq": ["runbook"]}}, the system will fail.
Which parameter setting provides the optimal balance for a specific workload?
The cost of scaling and memory
Self-hosting an engine requires enough system RAM to hold the index. HNSW is a graph that requires memory to walk. Chroma recommends at least 2 GB of RAM, and anything under that is not recommended. Weaviate states that memory usage is roughly two times the memory footprint of all vectors. If a collection exceeds available memory, the operating system starts swapping and the system becomes unusable.
Managed services change the cost structure. Instead of provisioning memory, you pay for query volume.
| Provider | Storage Cost | Write Cost (per 1M units) | Read Cost (per 1M units) |
|---|---|---|---|
| Pinecone Serverless | $0.01 / GB | $0.50 | $0.025 |
| Weaviate Cloud | $0.095 / 1M dims | Variable | Variable |
Pinecone does not have a performance cliff, but it has a cost cliff. At 10 million vectors with the s1 pod type, the cost is $700 per month. At 50 million vectors, the cost exceeds $3,000 per month.
Reindexing and model incompatibility
Every corpus gets re-embedded eventually. A better model ships, or the dimension count changes. This is not a migration you run with an ALTER command. It is a full rebuild of the index. Changing the embedding model requires a full re-upsert of the corpus.
Dimension mismatches cause system crashes. If a user changes an embedding model from 3072 dimensions to 1024 dimensions, the system crashes with an out of range error. Even if the dimensions remain the same, such as two different 3072-dimension models, the search returns garbage results because the vectors occupy incompatible semantic spaces. Users only discover the problem when search quality degrades.
Rebuilding an index on Pinecone incurs write costs. Re-upserting a corpus costs between $4 and $4.50 per million write units on the Standard plan. If you go through the import process, there is an additional $0.25 per GB charge.
Hallucination and the reranking bottleneck
Hallucination occurs when the system retrieves chunks that are topically relevant but factually insufficient. If the top-k is too small, the LLM misses the specific number or clause it needs and fills the gap with a plausible fabrication. The hallucination rate in production RAG systems without faithfulness checking ranges from 15% to 25%.
Faithfulness checking reduces hallucination rates from 18% to under 3%. This process adds roughly $0.002 per query and 300 to 500 milliseconds of latency with GPT-4o-mini.
Reranking is the single highest-impact way to improve retrieval quality, but it is also the easiest way to blow a latency budget. Cross-encoder rerankers improve NDCG@10 by 15% to 25% in most benchmarks. However, reranking at top-100 candidates adds 400 to 800 milliseconds of latency. If a system has a total latency budget of 2 to 3 seconds, the reranker can only take 500 milliseconds.
API constraints and eventual consistency
Building an agentic RAG system requires managing the separation between the control plane and the data plane. An AI agent cannot query a vector using the Control Plane API URL. Each index has a unique host URL generated upon creation. If an agent attempts to manage infrastructure without retrieving the index host from the control plane first, the operation fails.
Rate limits are a constant pressure. Pinecone enforces specific limits based on the plan. If an agent attempts to bulk-upsert thousands of documents too rapidly, Pinecone returns an HTTP 429 Too Many Requests error. You must implement exponential backoff and retry logic to handle these errors.
Eventual consistency causes retrieval failures in automated workflows. When using the Document API, the operation is asynchronous. Documents are indexed in the background. If an agent upserts a document and immediately issues a search tool call to verify the insertion, the search returns empty. The agent will assume the insertion failed and trigger a destructive retry loop.
Maintain high retrieval accuracy by prioritizing chunking coherence, hybrid search, and faithfulness verification over raw database latency.
