Tag: caching
All the articles with the tag "caching".
Semantic Caching for LLMs on AWS
An exact LLM cache almost never hits, and the obvious fix — serve the answer to the nearest stored prompt — is worse than no cache: the near-misses that change the answer sit closer in embedding space than the paraphrases that don't. A verification stage fixes the ranking and leaves a 5% floor of confidently wrong answers that you cannot measure in production. Built on DynamoDB vector search, measured three ways, and mostly an argument for not running it.