Tag: llm-routing
All the articles with the tag "llm-routing".
Customizing inference on AgentCore Gateway: a plugin pipeline for routing, budgets and guardrails
Every model call through Amazon Bedrock AgentCore Gateway passes one place where you can check a budget, clamp tokens, run a guardrail, pick the model, cache the answer and meter the cost. A reference architecture that makes each of those a swappable plugin, deployed and measured, plus seven things about interceptors the documentation does not tell you.
Replicating RouteLLM on Amazon Bedrock
Prompt-only LLM routers post strong in-distribution scores that are almost entirely recognition of which group a prompt came from. A reproduction of RouteLLM on Amazon Bedrock and AgentCore Gateway, a diagnostic suite that exposes it, and a zero-cost deferral rule that works where prompt-only routing does not.