Semantic Caching for LLMs: When It Cuts Costs and When It Lies

“Caching” in LLM applications covers three different mechanisms at three different layers, and teams often adopt the risky one before the safe one. This article explains where each fits, how to build a semantic cache that does not leak answers between users, and how to measure whether it is worth running. Broader cost controls, such as model routing and per-tenant metering, are covered in integrating OpenAI and Claude APIs into enterprise web apps.

Three caches, three layers

Start with provider prompt caching

Prompt caching should usually come first, because it cannot change an answer. The figures below are from the providers’ own documentation as of October 2026:

• Anthropic (prompt caching docs): enable it with cache_control, either automatically at the top level of the request or on specific blocks. A 5-minute cache write costs 1.25× the base input price, a 1-hour write costs 2×, and a cache read costs 0.1×, or 0.05× on Claude Opus 5.5 and Sonnet 5.5. Current models need at least 512 tokens in the prefix. Shorter prompts are simply not cached. Check cache_read_input_tokens in the response usage to confirm hits.
• OpenAI (prompt caching guide): enabled by default on supported models. The guide says cached input is “discounted up to 95%”, with exact rates depending on the model. Hits appear in usage.input_tokens_details.cached_tokens.

The design rule is the same for both: put stable content first (tools, system prompt, reference documents) and variable content last. A timestamp or user name near the top of the system prompt invalidates everything after it.

When a semantic cache is worth it

A semantic cache saves the whole call, so the upside is larger. The risk is larger too. Redis’s LangCache concepts page states the trade-off plainly: semantic caching “gives up the guarantee that a cache hit is always correct” in exchange for matching paraphrases.

The classic false positive is two questions that are close in embedding space but have different answers: “What time does the store open?” and “What time does the store close?” Embeddings encode topic well and small but decisive words poorly. That is the same weakness that makes hybrid BM25-vector search necessary for retrieval.

Scope the cache key, not just the prompt

The prompt text is only part of what determines an answer. A cached response is valid only if every other input also matches. Store these as filterable fields and require an exact match on all of them:

• Tenant or audience. Never let one customer’s answer serve another. If answers vary per user, either scope by user (which usually kills the hit rate) or do not cache.
• Prompt version and model. A new system prompt or model should start a fresh cache, not inherit answers from the old one.
• Knowledge version. For RAG answers, include the index or content version so that a content update invalidates stale answers.
• Locale. The same question in a different market may have a different correct answer.

Also keep separate caches (or a task field) per use case. A support question and a code question can sit close together in embedding space while needing nothing in common.

A minimal implementation

This Python sketch uses RedisVL’s SemanticCache in front of the Claude API, with Anthropic prompt caching on the system prompt, so both layers work together. The vectoriser runs locally; swap in whichever embedding model you use.

import anthropic
from redisvl.extensions.cache.llm import SemanticCache
from redisvl.query.filter import Tag
from redisvl.utils.vectorize import HFTextVectorizer

PROMPT_VERSION = "faq-v7"
MODEL = "claude-haiku-5-5"
SYSTEM_PROMPT = open("prompts/faq-v7.md").read()  # long, stable: worth prompt caching

cache = SemanticCache(
    name="faq_answers",
    redis_url="redis://localhost:6379",
    distance_threshold=0.1,  # Redis cosine distance, 0-2; lower is stricter
    ttl=86_400,              # backstop expiry in seconds
    vectorizer=HFTextVectorizer("redis/langcache-embed-v2"),
    filterable_fields=[
        {"name": "tenant", "type": "tag"},
        {"name": "prompt_version", "type": "tag"},
        {"name": "kb_version", "type": "tag"},
    ],
)
llm = anthropic.Anthropic()

def answer(question: str, tenant: str, kb_version: str) -> str:
    scope = (Tag("tenant") == tenant) & (Tag("prompt_version") == PROMPT_VERSION) \
        & (Tag("kb_version") == kb_version)

    hits = cache.check(prompt=question, filter_expression=scope,
                       num_results=1, return_fields=["prompt", "response"])
    if hits:
        return hits[0]["response"]

    msg = llm.messages.create(
        model=MODEL,
        max_tokens=1024,
        system=[{"type": "text", "text": SYSTEM_PROMPT,
                 "cache_control": {"type": "ephemeral"}}],  # provider prompt cache
        messages=[{"role": "user", "content": question}],
    )
    text = "".join(block.text for block in msg.content if block.type == "text")

    # Only cache complete answers. Never cache truncated or refused output.
    if msg.stop_reason == "end_turn":
        cache.store(prompt=question, response=text, filters={
            "tenant": tenant, "prompt_version": PROMPT_VERSION, "kb_version": kb_version,
        })
    return text

Two details are easy to miss. First, RedisVL’s check() refreshes the TTL of matched entries, so frequently hit answers never expire on their own. Your version fields, not the TTL, must handle invalidation. Second, the embedding lookup runs on every request, hit or miss, so a miss is slightly slower than having no cache at all. Measure that overhead against your hit rate.

Choosing the threshold

There is no correct universal threshold. LangCache defaults to a similarity of 0.85 and recommends starting between 0.8 and 0.9. RedisVL expresses the same idea as cosine distance, where lower is stricter. Both vendors give the same advice: start strict and loosen gradually while you watch the results.

1. Collect a few hundred real question pairs from your logs, labelled “same answer” or “different answer”.
2. Compute their embedding distances with the model the cache will use.
3. Choose the threshold that keeps false positives (different-answer pairs that would hit) at a level you can accept for this use case. A public FAQ can tolerate more than a billing assistant.
4. Re-run whenever you change the embedding model. Distances are not comparable across models.

Measure it like any other cache

• Hit rate by use case and tenant. A cache with a low hit rate adds latency and complexity for little return.
• False-positive rate from a regular human spot-check of sampled hits: the question asked, the question matched and the answer served.
• Savings calculated from your actual token prices and hit counts, net of embedding and Redis costs.
• Latency separately for hits and misses, so the extra lookup on a miss stays visible.

Expect hit rates to vary widely by product, so do not budget on someone else’s numbers. Run the cache in shadow mode first: look up and log what would have been served, but always call the model, and compare. That gives you a hit rate and a false-positive sample before any user sees a cached answer.

In agent systems, cache tool results that are pure lookups rather than whole agent responses. See strict tool calling and recovery loops for where those boundaries sit.

Decision guide

• Long, repeated system prompt or documents? Use prompt caching. Low risk.
• Byte-identical requests, such as batch reprocessing? Use an exact-match cache. No correctness risk.
• Many users asking the same public questions in different words? Consider a semantic cache, scoped and shadow-tested first.
• Answers depend on the user, the conversation or the clock? Do not use a semantic cache.

Questions people actually ask

What is the difference between semantic caching and prompt caching?

Prompt caching happens at the provider. The model still runs, but a repeated exact prefix is billed at a discount and processed faster. Semantic caching happens in your application and skips the model call when a similar enough question has already been answered. The first cannot change an answer; the second can.

Can I use semantic caching with conversation history?

Rarely safely. “What about the second one?” means nothing without context, and embedding the whole history makes near-matches unlikely anyway. Cache only standalone, single-turn questions, or rewrite follow-ups into standalone questions before the lookup.

How do I invalidate a semantic cache when content changes?

Include a content or knowledge-base version in the cache scope and change it when you publish. Old entries stop matching immediately and expire through their TTL. Trying to find and delete “related” entries by similarity is unreliable.