Semantic caching for LLMs: how it works and why it cuts costs by 40%
Semantic caching detects similar queries and returns cached responses instantly — $0 cost, ~80ms latency, zero quality loss. For any application with repeated or similar queries (chatbots, FAQ, RAG pipelines), it’s the single highest-impact optimization you can make.
This guide explains how semantic caching works under the hood, when to use it, when to avoid it, and what kind of savings to expect in production.
Related: Token optimization for LLMs: advanced techniques | AI cost optimization: practical guide
What is semantic caching?
Traditional caching (Redis, Memcached) matches queries by exact key. If the user asks “What’s your return policy?” and then “How do I return a product?”, traditional cache misses — the strings are different.
Semantic caching matches by meaning. Both questions mean the same thing, so the second one gets the cached response from the first. No API call. No tokens consumed.
The difference in hit rates is dramatic:
| Cache type | Hit rate (typical chatbot) | Cost per cache hit |
|---|---|---|
| Exact-match (Redis) | 5-10% | $0 |
| Semantic cache | 30-50% | $0 |
That’s 3-5x more cache hits — which translates directly to 30-50% fewer API calls.
How it works under the hood
Semantic caching has three components:
1. Embedding generation
When a query arrives, it’s converted to a vector embedding — a numerical representation of its meaning. Models like text-embedding-3-small from OpenAI or open-source alternatives like all-MiniLM-L6-v2 generate these embeddings in ~5ms.
The embedding captures semantic meaning: “return policy” and “how to return” produce vectors that are close together in embedding space, while “return policy” and “quarterly revenue” are far apart.
2. Similarity search
The query embedding is compared against all cached embeddings using cosine similarity. If the similarity score exceeds a threshold (typically 0.92-0.95), it’s a cache hit.
similarity("What's your return policy?", "How do I return a product?") = 0.94 → HIT
similarity("What's your return policy?", "What are your business hours?") = 0.31 → MISSVector databases like Pinecone, Qdrant, or even pgvector handle this search in <10ms, even with millions of cached entries.
3. Response retrieval
On a hit, the cached response is returned directly. No LLM call happens. Total latency: ~80ms (embedding + similarity search + retrieval) vs. 500-2000ms for a full LLM call.
On a miss, the query goes to the LLM normally. The response is then cached with its embedding for future hits.
When semantic caching works best
The ROI depends entirely on how repetitive your queries are. Some workloads are perfect for caching. Others aren’t.
High-impact use cases
Customer support chatbots: 30-50% of questions are variations of the same 20-30 topics. “Where’s my order?”, “Track my package”, “Order status” — all semantically identical. Expected hit rate: 35-50%.
FAQ and knowledge base: Users ask the same questions in different words. A well-tuned semantic cache can handle 40-60% of queries without touching the LLM.
RAG pipelines: When multiple users query the same document corpus, cache hits on similar questions eliminate redundant retrieval + generation cycles.
Internal tools: Developers asking similar coding questions, analysts running similar data queries — patterns repeat more than you’d expect.
When to avoid it
Creative content generation: Each request is intentionally unique. A marketing team generating ad copy variations needs different outputs every time. Caching would return the same copy.
Personalized responses: If the response depends on user-specific context (account data, preferences, history), semantic similarity of the question alone isn’t enough to guarantee the cached response is correct.
Low-volume applications: If you’re making <100 calls/day, the infrastructure overhead of embedding + vector search isn’t worth the savings.
Real-world numbers
Published benchmarks from production deployments:
| Metric | Without cache | With semantic cache |
|---|---|---|
| Avg. latency | 800ms | 120ms (cache hit) / 850ms (miss) |
| Cost per 1M queries | $150-500 | $90-300 |
| Cache hit rate | 0% | 30-50% |
| Infrastructure cost | $0 | ~$20/month (embedding + vector DB) |
ProjectDiscovery’s Neo agent documented a 59% cumulative cost drop from prompt caching alone, climbing to over 90% on fully-optimized paths that combine semantic caching with prompt compression.
Semantic cache vs. exact-match cache
You don’t have to choose — they complement each other.
| Feature | Exact-match (Redis) | Semantic cache |
|---|---|---|
| Match type | Identical strings only | Similar meaning |
| Hit rate | 5-10% | 30-50% |
| Latency overhead | <1ms | ~10ms (embedding + search) |
| False positives | 0% | 1-3% (tunable via threshold) |
| Best for | Identical repeated queries | Paraphrased queries |
The optimal setup: exact-match first (fastest, zero false positives), then semantic cache for misses. This two-layer approach maximizes hit rate while minimizing latency.
Tuning the similarity threshold
The threshold controls the trade-off between hit rate and accuracy:
- 0.98+: Very strict. Almost exact matches only. Low false positive rate, but hit rate drops to ~15%.
- 0.93-0.95: Sweet spot for most applications. Good hit rate (30-40%) with <2% false positives.
- 0.88-0.92: Aggressive. Higher hit rate (40-50%) but 3-5% false positives. Fine for FAQ, risky for anything requiring precision.
Start at 0.95 and lower gradually while monitoring response quality. If users start getting irrelevant cached responses, raise the threshold.
How to implement it
Option 1: Build it yourself
You need three components:
- Embedding model — OpenAI
text-embedding-3-small($0.02/1M tokens) or self-hostedall-MiniLM-L6-v2 - Vector database — Qdrant, Pinecone, pgvector, or Weaviate
- Cache logic — Intercept LLM calls, check cache, store misses
It’s straightforward but requires maintenance: cache invalidation, embedding model updates, threshold tuning, monitoring hit rates.
Option 2: Use an AI proxy
Platforms like Veltrix integrate semantic caching as part of a broader optimization layer (alongside prompt compression and model routing). You swap the baseURL in your SDK and caching works automatically — no vector database to manage, no threshold to tune.
The advantage of the proxy approach: caching is combined with other optimizations. A query that’s a cache miss might still benefit from prompt compression (-45% tokens) and model routing (cheaper model for simple tasks).
Key takeaways
- Semantic caching cuts costs by 30-50% for repetitive workloads — with 80ms latency instead of 500ms+.
- It works by matching query meaning, not exact strings — using embeddings and cosine similarity.
- Best for: chatbots, FAQ, RAG, internal tools. Worst for: creative generation, personalized responses.
- Start with threshold 0.95, lower gradually while monitoring quality.
- Combine with exact-match cache for maximum hit rate with minimum false positives.
If your application has any degree of query repetition — and most do — semantic caching is free money on the table.
Pronto pra reduzir sua fatura de IA?
Teste grátis por 7 dias. Sem cartão. Sem refactor. Resultado mensurável no primeiro dia.
Começar agora →