Exact caching saves the same response for the same request. Semantic caching saves responses for semantically equivalent requests — catching “What is France's capital?” and “Tell me the capital of France” as the same query. In production, this cuts API calls by 30–70%.
Use when: Production: low-latency cache with persistence and TTL support
import redis
import numpy as np
from openai import OpenAI
import json, hashlib
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
embed_client = OpenAI()
SIMILARITY_THRESHOLD = 0.92 # Tune: higher = stricter matching
CACHE_TTL = 3600 # 1 hour
def embed(text: str) -> list[float]:
return embed_client.embeddings.create(
input=text, model="text-embedding-3-small"
).data[0].embedding
def cosine_similarity(a: list[float], b: list[float]) -> float:
a, b = np.array(a), np.array(b)
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def cache_lookup(query: str) -> str | None:
query_embedding = embed(query)
# Scan all cached keys (use Redis Search in production for O(log n))
for key in r.scan_iter("cache:*"):
entry = json.loads(r.get(key))
similarity = cosine_similarity(query_embedding, entry["embedding"])
if similarity >= SIMILARITY_THRESHOLD:
return entry["response"] # Cache hit
return None # Cache miss
def cache_store(query: str, response: str) -> None:
embedding = embed(query)
key = f"cache:{hashlib.md5(query.encode()).hexdigest()}"
r.setex(key, CACHE_TTL, json.dumps({
"query": query,
"embedding": embedding,
"response": response,
}))from fastapi import FastAPI
import anthropic
app = FastAPI()
client = anthropic.Anthropic()
@app.post("/chat")
async def chat(request: ChatRequest) -> ChatResponse:
# 1. Check semantic cache
cached = cache_lookup(request.message)
if cached:
return ChatResponse(answer=cached, from_cache=True)
# 2. Cache miss — call LLM
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=500,
messages=[{"role": "user", "content": request.message}],
)
answer = response.content[0].text
# 3. Cache the response (async so it doesn't slow down the response)
import asyncio
asyncio.create_task(async_cache_store(request.message, answer))
return ChatResponse(answer=answer, from_cache=False)Near-exact matches only. Low hit rate, high accuracy. Good for factual queries where small wording changes matter.
Catches paraphrases and common rewording. Best default for FAQ bots, support, and documentation search.
Caches aggressively. Risk of serving wrong cached response. Only suitable for narrow, constrained topic domains.
Log from_cache: true/false on every request. A hit rate below 20% means your queries are too diverse for semantic caching to help much. Above 50% is excellent. The embedding call costs ~$0.00002 — cache overhead pays for itself after 1 LLM call saved.
Never cache personalized responses.Semantic caching is only safe for queries whose answers don't depend on the specific user's account, session, or private data. Always check: “Could this response be wrong for a different user?”
Next: we look at async and batch inference patterns — how to serve AI at high throughput without blocking your API or burning money on idle model time.