A cache is the largest cost lever an LLM gateway has: a hit costs nothing and returns in milliseconds. But an exact cache only fires when two requests match byte for byte, which real traffic rarely does. The obvious fix is to serve the answer to the nearest stored prompt instead. Measured on prompts built to test exactly that, the obvious fix is not a weaker version of the right answer — it is a mechanism for confidently answering questions nobody asked. There is a design that fixes it, and having built and measured it, my conclusion is still that most systems should not run one.
The exact cache almost never hits
The gateway pipeline I built earlier caches on a hash of the request as the client sent it: the model, the messages, the sampling parameters. It works, it is fast, and on anything resembling human traffic it is nearly useless. A trailing “please”, a reordered clause, a capital letter, a rephrased question with the same intent — all misses. The cache fires for retries, for repeated automated calls, and for the same user pressing the button twice. It does not fire for two people asking the same thing.
That is the entire motivation for semantic caching, and the reason it appears in every LLM gateway feature list. The pitch is short enough to fit in a sentence: embed the prompt, find the nearest stored one, serve its answer if it is close enough. Every part of that sentence is easy. The last four words are the problem.
The obvious fix, and why it isn’t one
To find out what “close enough” is worth, I needed prompts where I knew the right answer in advance. So I built 40 triples from a public prompt set. Each has the original prompt; a paraphrase, rewritten to mean exactly the same thing; and a near-miss, changed as little as possible so that it now has a different correct answer — one swapped number, name, date or entity. “A trip to Hawaii” becomes “a trip to Japan.” A cache should serve the paraphrase and must not serve the near-miss.
Then I embedded all 120 with Amazon Titan Text Embeddings v2 and measured the cosine similarity of each variant to its original.
The dangerous neighbors are the close ones. Changing “Hawaii” to “Japan” barely moves an embedding; rewording a sentence moves it a lot.
The distributions are not merely overlapping. They are inverted. The variants that are unsafe to serve are, on average, nearer than the ones that are safe: median 0.911 against 0.836. Thirty-three of the forty near-misses score above the median paraphrase; thirty of the forty paraphrases score below the median near-miss.
Once you see it, the mechanism is obvious, and it is a property of what embeddings are for. An embedding encodes what a sentence is about. Two sentences about booking travel to a Pacific island are about the same thing whether the island is Hawaii or Japan — that is the point of the representation, and it is exactly what makes retrieval work. But a cache does not want topical similarity. It wants answer equivalence, and the single token that decides the answer is, to an embedding, a rounding error.
This is not a resolution problem, either. Titan v2 is Matryoshka-trained, so I can take the same text at 256, 512 and 1,024 dimensions. At 1,024 the medians move to 0.802 and 0.899 — the gap stays inverted and the overlap stays total. Four times the dimensions buys nothing, because the information the cache needs was never in the embedding to begin with.
This is a known result
I went looking for prior work after the measurement, expecting to find that I had made an error. I found the opposite: the inversion is documented, and the people who documented it best are the ones shipping semantic caches.
Zhu, Zhu and Jiao put a number on the core problem in 2024. They ask directly whether embedding similarity can predict when a cached response actually answers a new prompt, and on a dataset of hard negatives an off-the-shelf embedding scores 0.51 AUC — indistinguishable from a coin flip. Distilling a purpose-trained embedding gets them to 0.81, which is progress and is still not a gate you would put in front of production traffic (arXiv:2402.01173).
GPTCache, the widely used open-source semantic cache, says so in its own paper. Its evaluation is honest about the ceiling — the authors note that the best hit rates they see “do not exceed 90% with current embedding models”, and that the resulting wrong hits would be “unacceptable in real production scenarios”. More usefully for anyone reaching for the obvious remedy: they tried second-stage rerankers — vector distance, retrieval distance, the Cohere rerank API, an SBERT cross-encoder — and report that none of them sufficiently separated good hits from bad ones (Bang, NLP-OSS 2023). A better similarity model is still a similarity model.
vCache (ICLR 2026) attacks the threshold itself, replacing one global value with a threshold learned per cached prompt, and is the first such cache to offer a user-specified error-rate bound rather than a hope. It reports up to 12.5× the hit rate and 26× lower error than static thresholds (arXiv:2502.03771). That is the statistically principled version of the argument this article makes operationally: one number cannot govern all prompts, because the distance that means “same answer” depends on the prompt.
And then there is AWS’s own benchmark of semantic caching with ElastiCache, which I recommend to anyone who thinks a strict threshold buys safety. On 63,796 real chatbot queries, accuracy of cached responses stays between 91.2% and 92.6% as the threshold falls from 0.99 all the way to 0.75, while the hit ratio climbs from 23.5% to 90.3% (AWS Database Blog, November 2025).
Read that table twice. Tightening the threshold from 0.75 to 0.99 costs you two thirds of your hit rate and buys 0.9 percentage points of accuracy. And at the strictest setting, 0.99 — as strict as a semantic cache can be while still being one — accuracy is 92.1%, which means roughly eight percent of cache hits were wrong anyway. The threshold is not doing what people believe it is doing. It is not a safety control; it is a hit-rate control that happens to be correlated with safety just enough to be misleading.
Similarity recalls; a model decides
If similarity cannot decide, stop asking it to. Give it the job it is actually good at — narrowing a large corpus to a handful of plausible candidates — and give the decision to something that can read.
Two stages with different jobs. The first is cheap and generous; the second is the only one that can end the call.
The verifier is a small model given one narrow question: do A and B have the same answer? Not “are these similar” — a model asked that will say yes to Hawaii and Japan as readily as an embedding does. The prompt that works names the failure mode explicitly, asking for YES only if any correct answer to A is also a correct answer to B, and NO if any detail that changes the answer differs. It is capped at five output tokens, because the answer is one word.
This inverts how the threshold should be set. If similarity were the gate, you would raise the threshold to buy precision. Here the threshold is only a recall knob, and a candidate that is never recalled can never be served, so it should be low. Precision is the verifier’s job, and the verifier is good at it.
Building it on DynamoDB
The interceptor already keeps its spend ledger, its request-correlation records and its cached answers in one on-demand DynamoDB table. DynamoDB now supports a vector index declared directly on a table, which means the semantic cache adds no new datastore: the vectors live beside the answers they point at, under the same TTL, the same IAM policy and the same bill.
VectorIndexes:
- IndexName: semantic-cache
VectorAttribute:
AttributeName: embedding
Dimensions: 256
DistanceFunction: COSINE
SearchSchema:
- { AttributeName: tenant, SearchSchemaElementType: HASH }
Projection:
ProjectionType: INCLUDE
NonKeyAttributes: [cache_key, prompt]Four things in that block are doing real work:
- Only items carrying
embeddingare indexed. The spend ledger and the cached answers sit in the same table and are never searched. This is why one table is enough. tenantas the HASH search-schema element partitions the index, so a search is scoped to one tenant by construction rather than by a filter applied afterward. A search cannot leak across a tenant boundary even if the code asks it to.Projection: INCLUDEreturns the stored prompt and the cache key with each match and leaves the 256 floats behind. The verifier needs the prompt text; nobody needs the vector back.- 256 dimensions, not 1,024. Since the dimension count buys no separation, take the cheap one: a quarter of the item size, and ranking is all the embedding is being asked for.
Answers expire by TTL, and the vectors expire with them. That leaves a window where a vector outlives its answer — the recall points at something already gone — which the plugin treats as a miss and moves on. The alternative, keeping the vector alive to preserve the hit rate, would mean serving an answer whose TTL has expired.
What it measures
The benchmark runs each triple as three calls against the live gateway: the original (a miss, which stores the answer and its vector), the paraphrase (a hit is correct), and the near-miss (a hit is wrong). Same 20 triples, same models, three configurations.
Turning the verifier off does not degrade the cache. It converts it into a machine that answers 90% of near-miss questions with the wrong answer, faster.
| Arm | Paraphrases served | Near-misses served | Median hit |
|---|---|---|---|
| Verify on, recall ≥ 0.65 | 90% | 5% | 937 ms |
| Verify on, recall ≥ 0.80 | 50% | 5% | 894 ms |
| Verify off, recall ≥ 0.80 | 50% | 90% | 389 ms |
Three things stand out.
The verifier is the product. At the same threshold, turning it off leaves the hit rate unchanged at 50% and takes the false-hit rate from 5% to 90%. The unverified cache is not a slightly riskier cache; on adversarial-but-entirely-realistic input it is wrong nine times out of ten when it fires.
The unverified cache is also the fast one. 389 ms against 937 ms. Anyone benchmarking hit latency and hit rate alone — the two numbers a cache normally reports — would conclude the unverified configuration is strictly better. Both of its headline metrics improve. Only the metric nobody instruments gets worse.
Lowering the recall threshold is free. Going from 0.80 to 0.65 took the hit rate from 50% to 90% with no change in false hits. This is the design working as intended: recall generously, verify strictly. A threshold tuned for precision is a threshold doing a job it is bad at.
The one false hit that survived verification, at both thresholds, was the same pair in each run — a near-miss the verifier judged equivalent. Five percent is a floor, not zero, and any design that serves stored answers to different questions has a residual error rate you have to be able to live with.
Why prompt caching works and this doesn’t
The obvious objection to everything above is that caching LLM calls demonstrably works — prompt caching is shipped by every major provider, people report large savings, and nobody warns you about wrong answers. So what is different here?
Prompt caching and semantic caching are not two settings of one idea. They differ on two axes, and the danger appears only when both go the wrong way.
| Key | What is reused | Wrong-answer risk | |
|---|---|---|---|
| Prompt caching | exact token prefix | the model’s KV state — computation | none: the model still generates |
| Exact response cache | the exact request | the final answer | none: the same question was asked |
| Semantic response cache | approximate similarity | the final answer | a 5% floor, undetectable |
Prompt caching caches computation; semantic caching caches conclusions. On a prompt-cache hit the model still runs and still writes your answer — it skips recomputing attention over a prefix it has already processed, and the output is what you would have received anyway. There is no approximation in the mechanism, so there is nothing to be wrong about. A miss costs money; it cannot cost correctness.
The deeper difference is how each one earns its hit rate. Prompt caching works by decomposition: it splits a request into the part that is genuinely identical across callers — a system prompt, a long document, a few-shot block — and the part that varies, then reuses only the identical part and generates the rest fresh. That is why its ideal case is a long shared prefix with a short unique question, and why a high hit rate costs nothing in quality. Measured through this same gateway on a shared 8,372-token prefix, 8,360 of those tokens came from cache — an exact reuse, with the answer still generated for the actual question.
Semantic caching refuses that split. It takes the whole prompt — including the varying part that determines the answer — and approximates it. And the varying part is precisely the part that decides whether a stored answer is correct. That is the mechanism behind Figure 1: swapping Hawaii for Japan changes the answer completely and the embedding barely at all, because the thing being thrown away is the thing that mattered.
So the correct summary is not “caching LLM calls is risky.” It is: two of the three ways to do it carry no correctness risk at all, and are underused. The third is the one that needs an argument.
Whether to run it at all
It is worth separating two questions that get answered together and should not be. Does the cache pay for itself? And is it safe to use? The first has a cheerful answer and the second is the one that decides.
The economics work, against an expensive model. The stage timings come from the interceptor’s own metrics, so they are in-Lambda and exclude the network:
| Stage | Median |
|---|---|
| Embed the prompt, search the index | 116 ms |
| One verification | +564 ms |
| Total on a hit | 680 ms |
One verification costs, measured rather than estimated, 148 input tokens and 4 output tokens — about **0.0000052, that is 32 times the price of the call it avoids, and the cache is a straightforward loss at any hit rate. Against the 675B model at $0.0016 a call it is roughly a tenth the price of what it avoids, and the break-even arrives almost immediately: 0.15% on cost, 1.25% on latency. Verifications you pay for and throw away are cheap enough that they barely matter.
So on price, the answer is yes. Which is the trap.
The error rate does not go away. Five percent of served hits were wrong at both thresholds — that is a floor, measured with the verifier working exactly as designed. It is not a tuning parameter. And because it applies to hits, the better the cache performs, the more wrong answers it serves:
The saving and the errors have the same cause, so you cannot buy one without the other.
At a 20% hit rate you have bought a 16% cost reduction by handing one percent of all your traffic a fluent, confident answer to a question nobody asked. At 40% it is two percent. Whether that is a bargain or a disqualification is not an engineering question, and it is certainly not one the cache should answer silently on your behalf.
And you cannot see it happening. A false hit is indistinguishable in production from a model being wrong. There is no error, no latency anomaly, no log line — the only way to know your false-hit rate is to hold ground truth you do not have, which is precisely why this article needed a constructed dataset to measure it at all. Every figure above comes from knowing the right answer in advance.
This also explains why the feature is so widely shipped and so rarely measured this way. Hit rate and hit latency are easy to instrument and both improve when you remove the verifier. The number that gets worse is the one nobody is watching.
Four things that went wrong
Declaring a vector index reserves that attribute name for the whole table. The interceptor’s RESPONSE phase receives no request, so the REQUEST phase leaves a correlation record behind — and I had it stash the embedding there as a JSON string under embedding. The moment the vector index existed, every one of those writes was rejected with Invalid type for parameter embedding, Expected: 32-bit floating point number list, including for requests that had nothing to do with caching. A vector attribute is a table-wide type declaration, not an index-local one. Renaming the correlation field fixed it.
A cross-region inference profile authorizes against whichever region it lands on. The verifier uses a us.-prefixed inference profile, which routes to the foundation model in any of several regions. The IAM policy granted bedrock:InvokeModel on arn:aws:bedrock:us-east-1::foundation-model/* — the stack’s own region, which looks like least privilege and is in fact a bug. Calls that landed in us-east-2 were denied. The model grant has to cover every region the profile spans; only the inference-profile ARN itself can be pinned to one region.
A silent recall failure is indistinguishable from an empty cache. My first version caught exceptions from the vector search and treated them as a miss — reasonable, since an empty index legitimately returns nothing. What it produced was a cache that never hit and never complained, and the “everything is fine” reading was identical to the “nothing works” reading. This is the same failure shape I hit in the previous article, where the cache never cached because the key was computed in a phase that could not see it. Both times the code was doing exactly what I wrote, and the absence of a signal was the signal. The plugin now records the exception text on the request.
The index needs a warm-up you will misdiagnose. For the first several minutes after the index was created, writes were landing but searches returned nothing, and every request logged a clean, confident miss. I convinced myself this was IAM scoping — the SearchVectors grant named the index ARN but not the table ARN — and I was wrong. I found out by removing the table grant again after things started working, at which point they kept working. In steady state a newly written vector was searchable in a median of 0.08 s across five trials, which is fast enough that the request-then-paraphrase pattern works. Had I not run the control, the article would now contain a confident, tidy, false explanation.
Limits
- Forty pairs, twenty triples, one embedding model. Enough to establish that the inversion exists and is large; not enough to characterize its shape across domains. The prompts are general-assistant style. Code, math and lookup traffic will behave differently, and a domain where the answer-determining token is also the topical token would show less inversion.
- The near-misses are adversarial by construction. They were built to change the answer with a minimal edit, which is the hard case, not the average case. A production false-hit rate would be lower — but the pairs are realistic, not contrived: “the 2024 figures” against “the 2025 figures” is a Tuesday.
- The verifier was not itself validated at scale. Its 5% residual error is measured on 20 near-misses. A serious deployment would sample hits into an offline audit.
- Single tenant, single Region, no load. Hot-partition behavior on the vector index under a busy tenant, and the cost of the index at corpus sizes far beyond a few hundred vectors, were out of scope.
- This measures a cache in front of cheap models. The economics section extrapolates to the expensive case from measured prices rather than from a second benchmark run against the 675B model.