Exact caching saves the same response for the same request. Semantic caching saves responses for semantically equivalent requests — catching “What is France's capital?” and “Tell me the capital of France” as the same query. In production, this cuts API calls by 30–70%.
Use when: Production: low-latency cache with persistence and TTL support
import redis
import numpy as np
from openai import OpenAI
import json, hashlib
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
embed_client = OpenAI()
SIMILARITY_THRESHOLD = 0.92 # Tune: higher = stricter matching
CACHE_TTL = 3600 # 1 hour
def embed(text: str) -> list[float]:
return embed_client.embeddings.create(
input=text, model="text-embedding-3-small"
).data[0].embedding
def cosine_similarity(a: list[float], b: list[float]) -> float:
a, b = np.array(a), np.array(b)
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def cache_lookup(query: str) -> str | None:
query_embedding = embed(query)
# Scan all cached keys (use Redis Search in production for O(log n))
for key in r.scan_iter("cache:*"):
entry = json.loads(r.get(key))
similarity = cosine_similarity(query_embedding, entry["embedding"])
if similarity >= SIMILARITY_THRESHOLD:
return entry["response"] # Cache hit
return None # Cache miss
def cache_store(query: str, response: str) -> None:
embedding = embed(query)
key = f"cache:{hashlib.md5(query.encode()).hexdigest()}"
r.setex(key, CACHE_TTL, json.dumps({
"query": query,
"embedding": embedding,
"response": response,
}))from fastapi import FastAPI
import anthropic
app = FastAPI()
client = anthropic.Anthropic()
@app.post("/chat")
async def chat(request: ChatRequest) -> ChatResponse:
# 1. Check semantic cache
cached = cache_lookup(request.message)
if cached:
return ChatResponse(answer=cached, from_cache=True)
# 2. Cache miss — call LLM
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=500,
messages=[{"role": "user", "content": request.message}],
)
answer = response.content[0].text
# 3. Cache the response (async so it doesn't slow down the response)
import asyncio
asyncio.create_task(async_cache_store(request.message, answer))
return ChatResponse(answer=answer, from_cache=False)Near-exact matches only. Low hit rate, high accuracy. Good for factual queries where small wording changes matter.
Catches paraphrases and common rewording. Best default for FAQ bots, support, and documentation search.
Caches aggressively. Risk of serving wrong cached response. Only suitable for narrow, constrained topic domains.
Log from_cache: true/false on every request. A hit rate below 20% means your queries are too diverse for semantic caching to help much. Above 50% is excellent. The embedding call costs ~$0.00002 — cache overhead pays for itself after 1 LLM call saved.
Never cache personalized responses.Semantic caching is only safe for queries whose answers don't depend on the specific user's account, session, or private data. Always check: “Could this response be wrong for a different user?”
Real embeddings have hundreds or thousands of dimensions, but the arithmetic is the same as in three. Cosine similarity is the dot product of two vectors divided by the product of their lengths. Take three toy query embeddings:
import numpy as np
a = np.array([0.8, 0.6, 0.0]) # "What is France's capital?"
b = np.array([0.6, 0.8, 0.0]) # "Tell me the capital of France"
c = np.array([0.0, 0.6, 0.8]) # "How do I reset my password?"
def cosine(u, v):
return float(u @ v / (np.linalg.norm(u) * np.linalg.norm(v)))
# |a| = sqrt(0.64 + 0.36) = 1, |b| = 1, |c| = sqrt(0.36 + 0.64) = 1
print(cosine(a, b)) # 0.8*0.6 + 0.6*0.8 + 0 = 0.96 -> HIT at threshold 0.92
print(cosine(a, c)) # 0*0.8 + 0.6*0.6 + 0 = 0.36 -> MISSMany embedding APIs return vectors already normalized to length 1, in which case cosine similarity is just the dot product. If yours do not, normalize before storing, or the thresholds below will not mean what you think.
With hit rate h, embedding cost e per request, and LLM cost L per miss, the average cost per request is e + (1 − h)·L. Using the lesson's embedding figure of about $0.00002 and an illustrative LLM cost of $0.0105 per request, a 35% hit rate gives 0.00002 + 0.65 × 0.0105 = 0.00002 + 0.006825 = $0.006845, which is 34.8% below the uncached $0.0105. The cache breaks even when h > e/L = 0.00002 / 0.0105 ≈ 0.2%, so cost is rarely the obstacle. The real risk is quality.
Suppose the system handles 10,000 requests a day with a 40% hit rate. That is 4,000 cached answers served daily. If 1% of those are wrong for the user who received them, that is 40 bad answers every day, and none of them raise an error. A threshold change that adds 5 points of hit rate is only a good trade if the extra hits are correct, so measure precision as well as hit rate. To do that, sample logged hits, show the new query next to the matched cached query, and label whether the cached answer is truly valid for the new one.
Comparing a query with every stored vector is linear in cache size, which is fine for thousands of entries but not for millions. Vector indexes such as HNSW trade a small amount of recall for much faster approximate nearest neighbor search; Redis, Chroma, and similar stores provide them. Because the index is approximate, it can miss the best match occasionally, which only lowers the hit rate and never serves a worse answer. Also bound the cache size with an eviction policy such as least-recently-used so memory does not grow without limit.
A semantic cache embeds each query, returns a stored answer when similarity clears a threshold, and otherwise calls the model and stores the result. The savings are easy to calculate; the work is in keeping wrong hits rare. Scope keys properly, expire and invalidate entries, tune the threshold with measured precision, and keep personalized responses out of the cache entirely.
Next: we look at async and batch inference patterns — how to serve AI at high throughput without blocking your API or burning money on idle model time.