Scaling LiteLLM on AWS

A reference architecture and operating guide for running LiteLLM as an enterprise LLM gateway on AWS, with Amazon Bedrock as the primary model provider, plus the FinOps, governance and security work that scales with it, and a managed alternative on Amazon Bedrock AgentCore Gateway.

September 2026. Built on current LiteLLM, Amazon Bedrock and AWS service documentation.
I work at Amazon Web Services. The views and assessments here are my own and do not represent AWS.

1. Design decisions
2. Reference architecture
3. Compute: ECS or EKS
4. Load balancing and edge
5. The request hot path
6. Aurora PostgreSQL
7. ElastiCache
8. Bedrock capacity
9. Multi-Region
10. Platform security
11. Observability
12. Scaling stages
13. Where it breaks first
14. FinOps
15. Data governance
16. AI security
17. Responsible AI
18. Operating model
19. AgentCore Gateway
20. Reference configuration
21. Sources

Design decisions

Concern	AWS service	Why
Entry and protection	Application Load Balancer, AWS WAF, AWS Certificate Manager, Amazon Route 53	Path-based routing to the three LiteLLM tiers, long-lived streaming connections, managed TLS, coarse edge rate limiting.
Compute	Amazon ECS on AWS Fargate (default) or Amazon EKS	Stateless tasks that scale on request rate. Fargate is what LiteLLM’s official AWS module deploys.
Shared state	Amazon ElastiCache (Valkey or Redis OSS)	Cross-task rate limits, router state, response cache, spend buffer.
Durable state	Amazon Aurora PostgreSQL, optional Amazon RDS Proxy	Keys, teams, spend. IAM database authentication; proxy for connection pooling at high task counts.
Models	Amazon Bedrock	IAM-role auth with no static keys, private connectivity, cross-Region inference, Provisioned Throughput.
Secrets and keys	AWS Secrets Manager, AWS KMS	Master key, salt key, non-AWS provider keys, database credentials.
Logs and metrics	Amazon CloudWatch, Amazon S3	Alarms on throttling, connections and CPU; long-term request and spend logs.
Provisioning	Terraform (BerriAI/litellm/aws) or the AWS Solutions Library guidance	Repeatable, per-Region stacks.

Reference architecture

The architecture below follows the layout of LiteLLM’s official AWS Terraform module (gateway, backend and UI on separate services; Aurora; ElastiCache; S3; Secrets Manager; path-routed ALB) and adds the pieces a production deployment typically needs on top: WAF, HTTPS, private Bedrock connectivity, optional RDS Proxy, and alarms.

Compute: ECS or EKS

Both work well; choose based on what the platform team already operates. The gateway is the only service that needs aggressive scaling. Backend and UI can run at fixed small counts.

	ECS on Fargate	EKS
Provisioning	Official BerriAI/litellm/aws Terraform module; AWS Solutions Library guidance	LiteLLM Helm charts (monolithic or componentized); AWS Solutions Library guidance
Scale signal	Target tracking on ALBRequestCountPerTarget (module input gateway_target_requests_per_second), with CPU as a backstop	HPA on CPU or custom request metrics; Karpenter for node capacity
Bedrock credentials	ECS task role	EKS Pod Identity or IRSA
Best when	You want the least to operate	You already run EKS, or need fine-grained node and bin-packing control

Workers per task

Run one worker process per vCPU and set --num_workers explicitly to the task or pod CPU allocation. Don’t rely on $(nproc) in containers: under Kubernetes CPU limits it typically reports the node’s cores, not your limit. Recycle workers with --max_requests_before_restart together with --run_gunicorn, which LiteLLM documents as the more mature recycling path, and set LITELLM_MODE="PRODUCTION".

Prefer more, smaller tasks over fewer large ones. Tasks of 2 to 4 vCPU keep blast radius small and let target tracking react in finer steps, but remember every worker holds its own database pool (section 6).

Load balancing and edge

The ALB is the right front door for an LLM gateway. It handles long-lived streaming responses and does path-based routing to the three LiteLLM services natively. Four settings need attention before production:

Setting	Recommendation
Listener	The official module fronts services with an HTTP/80 listener by default. Add an HTTPS listener with an ACM certificate and redirect HTTP, or place the ALB behind your existing TLS edge.
Idle timeout	The ALB default is 60 seconds. Long generations and reasoning models can pause longer than that between streamed chunks; raise it (the ALB allows up to 4,000 seconds) to match your longest expected request.
Scheme	Use an internal ALB for workforce and service-to-service traffic. Only expose an internet-facing ALB when external clients genuinely need it.
AWS WAF	Attach managed rule groups and a rate-based rule per source as a coarse outer limit. It protects the gateway from floods, while LiteLLM enforces per-key budgets and limits inside.

The request hot path

Every request touches ElastiCache several times and Aurora rarely. Only the Bedrock call is slow in absolute terms, so everything around it has to stay cheap.

Aurora PostgreSQL

Aurora is usually the first AWS component to hurt, and the cause is connection multiplication rather than query load. LiteLLM’s database_connection_pool_limit applies per worker process, so total connections equal the limit times workers times tasks. Size the limit backwards from the instance’s max_connections, which on Aurora scales with instance class.

Lever	Effect
proxy_batch_write_at	Flushes spend updates on an interval (60 seconds in LiteLLM’s production example) instead of per request.
use_redis_transaction_buffer	Queues spend updates in ElastiCache so tasks don’t race each other writing to Aurora.
Amazon RDS Proxy	Pools and multiplexes connections so task count stops translating one-for-one into database connections. Validate it against LiteLLM’s Prisma client under load and watch DatabaseConnectionsCurrentlySessionPinned; heavy pinning cancels the benefit.
Aurora Serverless v2	A good fit for spiky or non-production environments. Set a minimum ACU high enough that connection limits aren’t hit during scale-up.
S3 logging callback	Moves bulky per-request logs to S3 so Aurora holds only what the proxy needs for auth and budgets.
Migrations task	Serving tasks set DISABLE_SCHEMA_UPDATE=true; a one-off ECS task applies migrations per release, as the official module does.

ElastiCache

ElastiCache carries short-lived, latency-sensitive state. The working set is small, often a few gigabytes even at high traffic, but hot. Size memory for what you choose to cache and size CPU for throughput: a Redis-protocol engine executes commands on one main thread, and in-transit TLS competes for CPU.

Guidance	Why
Node-based cluster, cluster mode enabled, Multi-AZ	Sharding spreads the counter keyspace across processes; LiteLLM’s sizing guidance prefers this over one larger node.
Serverless only for plain caching	Semantic caching on valkey-search requires a node-based cluster.
Connect with host, port and password, not redis_url	LiteLLM reports the URL form benchmarking roughly 80 RPS slower.
Engine version 7.0 or later	LiteLLM’s documented minimum.
Alarm on EngineCPUUtilization	It reflects the main engine thread, which is the real constraint, better than host CPU does.
Enable backups if you use the spend buffer	Queued spend updates are the one piece of state here you don’t want to lose.

Bedrock capacity

Bedrock quotas are set per model and per Region, measured in requests and tokens per minute and visible in Service Quotas. This is where gateway throughput is actually decided. LiteLLM helps by treating each capacity source as a separate deployment under one model_name, then retrying, cooling down and falling back across them. Build capacity in layers:

Cross-Region inference and governance. Prompts and outputs can be processed outside the source Region, so confirm data-residency requirements before enabling a profile, and prefer geographic over global profiles when residency matters. Service control policies must allow every destination Region in the profile, not just the source Region, or requests will fail. Cross-Region traffic has its own quota line items in Service Quotas, separate from on-demand.

Keep Bedrock traffic private with an interface VPC endpoint for bedrock-runtime, and authenticate with the task or pod IAM role. Nothing in the LiteLLM config needs a static AWS key. For deployments in other accounts, give each its own role to assume.

Quota mechanics that surprise teams

Bedrock does not count quota the way a gateway counts tokens. These mechanics cause most of the “we have quota headroom but we’re being throttled” incidents.

Mechanic	What happens	Mitigation
max_tokens is reserved up front	At request start Bedrock deducts input tokens plus max_tokens from TPM, then replenishes the unused part when the request ends.	Clamp max_tokens per model in a LiteLLM pre-call hook; alert when requested max_tokens far exceeds actual output.
Output burndown	For Claude 3.7 and later, each output token consumes 5 tokens of quota. Billing is based on actual tokens only.	Size quota requests in burndown terms, not billed tokens.
Cache reads	Cache read tokens are excluded from the final quota calculation.	Use prompt caching for long, stable system prompts; it buys throughput as well as cost.
Counter mismatch	LiteLLM tracks tokens roughly 1:1, while Bedrock throttles on reservations with weighted output. The router can see headroom that doesn’t exist.	Set LiteLLM tpm well below the real quota and rely on cooldowns.
Retry amplification	Client SDK, LiteLLM and botocore retries multiply load exactly when you’re throttled.	Give one layer ownership of retries; tune allowed_fails and cooldown_time.
Shared account quota	Quotas are per model, per Region, per account, so every workload in the account competes.	Dedicated accounts for gateway traffic; request increases well before launch.

Bedrock also offers four inference service tiers: Reserved, Priority, Standard and Flex. Priority is requested per call through the service_tier parameter and costs more; Flex is discounted and deprioritized under load, which suits evaluations and batch-style agent work. Model coverage varies by tier, so check current support, and confirm your LiteLLM version forwards service_tier.

Multi-Region

LiteLLM’s AWS module is single-Region by design; the documented pattern is one root configuration per Region. Put Route 53 latency or failover routing in front and treat each Region as an independent cell with its own ALB, tasks and ElastiCache cluster.

The hard problem is the database. Keys and budgets must be consistent everywhere, but spend writes from Region B have to reach the writer in Region A. Two workable options exist. You can run an Aurora Global Database and point Region B at the primary writer (or use Aurora write forwarding where your engine version supports it), accepting cross-Region write latency on batched spend flushes. Alternatively, run fully independent cells with separate key sets per Region, which is simpler and more resilient but pushes key distribution to your clients. Test the first option’s latency with your batch interval before committing.

Platform security

Control	Implementation
No static AWS keys	Bedrock via ECS task role or EKS Pod Identity; Aurora via IAM database authentication.
Secrets	Master key, LITELLM_SALT_KEY and non-AWS provider keys in Secrets Manager, injected at task start. The salt key encrypts stored credentials; set it once and never change it.
Private data path	Tasks, Aurora and ElastiCache in private subnets; interface endpoints for Bedrock, Secrets Manager, ECR and CloudWatch Logs to cut NAT dependency.
Encryption	KMS keys for Aurora, ElastiCache, S3 and Secrets Manager; TLS in transit to ElastiCache and the ALB.
Edge	AWS WAF on the ALB; internal ALB wherever possible.
Supply chain	Pull LiteLLM images into ECR, pin by digest, and scan on push. Upgrade deliberately; the gateway sits in front of every model call.

Observability

Alarm on the signals that precede user-visible failure, mapped to the bottlenecks in section 13.

Signal	Metric	What it tells you
Model throttling	AWS/Bedrock InvocationThrottles	You’ve hit quota; add a capacity layer, not tasks.
Gateway latency	TargetResponseTime (ALB)	Includes model time; compare with Bedrock InvocationLatency to isolate gateway overhead.
Gateway saturation	ECS or pod CPUUtilization	Scale-out lag or undersized tasks.
Upstream failures	HTTPCode_Target_5XX_Count	Provider errors surfacing after retries and fallbacks.
Database pressure	Aurora DatabaseConnections	Approaching max_connections; revisit the calculator.
Proxy efficiency	DatabaseConnectionsCurrentlySessionPinned	RDS Proxy multiplexing is being defeated.
Cache pressure	ElastiCache EngineCPUUtilization	Main engine thread nearing saturation; add shards.

For per-key, per-team and per-model views, export LiteLLM’s Prometheus metrics to Amazon Managed Service for Prometheus and chart them in Amazon Managed Grafana. Check which metrics your LiteLLM edition exposes first.

Scaling stages

Each move is triggered by a specific, observable symptom, not by traffic volume alone.

Where it breaks first

This ordering is my engineering assessment, not a published benchmark. It reflects where contention appears in the hot path.

FinOps

At pilot scale, one Bedrock line item on the bill is fine. At enterprise scale, finance asks three questions the platform must answer: who spent it, was it worth it, and can we stop it before it happens. LLM spend differs from classic cloud spend in two ways. It is driven by user behavior in real time rather than by provisioned capacity, and it is measured by several token meters that never quite agree.

Three token meters

Meter	What it counts	Use it for
Billing (CUR 2.0)	Actual input, output, cache-read and cache-write tokens at list or negotiated price, attributed to the caller principal, application inference profile or Bedrock project. Arrives with a lag.	Chargeback and the financial source of truth.
Quota (Bedrock TPM)	Input plus max_tokens reserved at start, then adjusted; weighted output for newer Claude models; cache reads excluded.	Capacity planning and throttling analysis. Never for cost.
Gateway	LiteLLM spend logs priced from its model price map, per key, team, user and tag. AgentCore Gateway TPM limits use a tokenizer estimate reconciled against provider-reported usage.	Real-time budgets, showback and enforcement.

Expect the gateway meter to drift from the bill once service tiers, cross-Region routing, cache pricing and negotiated discounts are involved. Override prices per deployment in LiteLLM (input_cost_per_token, output_cost_per_token) where you have private pricing, and reconcile monthly.

Attribution architecture

Attribution has to be designed in, because a gateway erases it by default. Behind any gateway, Bedrock sees the gateway’s IAM role as the caller, so native cost allocation collapses to one principal. Bedrock supports three native attribution mechanisms: IAM principal tags on the caller (recorded in CUR 2.0 across Bedrock APIs, including the OpenAI-compatible bedrock-mantle endpoint), application inference profiles on bedrock-runtime, and projects on bedrock-mantle. To keep them meaningful behind LiteLLM, route each team to its own application inference profile or assumed role, or run a gateway per business unit.

Tagging taxonomy

Tag key	Example	Where it’s applied
cost-center	CC-4410	IAM principal tags, inference profile or project tags, LiteLLM team metadata
business-unit	customer-operations	Same as above
application	ticket-summarizer	LiteLLM key metadata; one inference profile or project per application
environment	prod	All of the above plus infrastructure resources
data-class	confidential	LiteLLM team metadata; drives model and Region allowlists (section 15)
owner	team alias	All of the above

Enforce the taxonomy with AWS Organizations tag policies and make tags a precondition for issuing a key. Retrofitting attribution after six months of untagged traffic is not possible.

Budgets and enforcement

Layer	Mechanism	Nature
Key, team, user	LiteLLM max_budget with budget_duration, plus per-key tpm and rpm	Hard, real time
Request	max_tokens clamp, per-team model allowlists	Hard, real time
Account	AWS Budgets with alerts and budget actions	Soft, lagged
Anomalies	AWS Cost Anomaly Detection on Bedrock usage	Detective
Capacity	Service Quotas and InvocationThrottles alarms	Protective

Unit economics

Total spend is a poor steering metric because it rises with adoption, which is the goal. Track cost per unit of value instead: cost per resolved ticket, per generated document, per active user-day, or per successful agent task. For reference, Anthropic reports enterprise Claude Code usage averaging about $13 per developer per active day, as cited in AWS’s AgentCore usage-interceptor sample. The same source notes that agent teams in plan mode use roughly seven times more tokens than standard sessions, which is why per-outcome metrics matter more than per-request ones for agentic work.

Optimization levers

Lever	Effect	Caveat
Model tiering and routing	Send simple tasks to smaller models behind a stable alias	Needs evaluation evidence per task type
Prompt caching	Cache reads are billed far below input rates (a tenth of the input rate for Claude models in AWS’s sample price table) and don’t count toward quota	Only helps stable prefixes; cache writes cost more than input
Batch inference and Flex tier	Roughly half of Standard pricing in one AWS partner’s published example	Latency-tolerant work only; check model coverage
max_tokens hygiene	Frees quota and concurrency	Doesn’t change cost directly
Provisioned Throughput or Reserved tier	Predictable capacity and price for steady load	You pay whether used or not; size from real traffic, not forecasts
Response caching	Eliminates repeated calls	Low hit rates for conversational and agentic traffic
Context management	Trim history, tune retrieval depth	Quality trade-off; measure it

Reporting

Export CUR 2.0 with caller identity allocation data to S3, query it with Athena, and join it to LiteLLM spend-log exports on team and application tags. Publish showback weekly from the ledger and chargeback monthly from CUR. If finance uses a multi-cloud FinOps tool, AWS Data Exports can also emit the FOCUS format.

Data governance

A shared gateway becomes the largest concentration of sensitive text in the company: prompts contain customer records, contracts and source code, and responses echo them back. Data governance for LLM traffic answers four questions: which data may go to which model, where it is processed, what is retained, and who can see it.

Classification-driven routing

Data class	Allowed routing	Additional controls
Public	Any approved model, including global cross-Region profiles and approved third-party providers	Standard guardrails
Internal	AWS-hosted models; geographic cross-Region profiles	Standard guardrails
Confidential	Named models in approved Regions or geography; no third-party providers	PII masking; no shared response cache
Restricted (PII, PHI, regulated)	Named models in named Regions only	Payload logging off or separately encrypted; human review of outputs; per-application approval

Implement this in layers. LiteLLM teams carry a data-class tag and a matching model allowlist. Service control policies restrict bedrock:InvokeModel to approved Regions and model ARNs, so a misconfigured key cannot route around the policy. For workloads that call Bedrock directly, IAM conditions on bedrock:GuardrailIdentifier can require a specific guardrail on every inference call.

What gets logged, and where

Store	Contents	Guidance
LiteLLM spend logs (Aurora)	Request metadata, tokens, cost; payloads only if you enable it	Keep payloads out unless required; archive metadata after 90 days
S3 logging callback	Full requests and responses if configured	Classify at the level of the most sensitive data it may contain; KMS, lifecycle rules, Object Lock where regulation requires
Bedrock model invocation logging	Full prompts and completions	Enable deliberately, with restricted access and a named owner
ElastiCache response cache	Full responses	Short TTLs; disable for confidential and restricted classes
CloudWatch	Metrics, errors, traces	Keep payloads out of application logs

Treat every store in this table as a data store with an owner, a retention period and a deletion process, including for erasure requests. Remember that traffic sent to non-AWS providers leaves AWS and is governed by that provider’s terms, so only data classes approved for that provider should reach it.

AI security

Platform security (section 10) protects the gateway. AI security protects what flows through it. The table maps the most relevant risks from the OWASP Top 10 for LLM Applications to controls a central gateway can enforce; the rest remain application responsibilities.

Risk	Gateway-level control	Application-level control
Prompt injection (LLM01)	Prompt-attack guardrail on inputs; in AgentCore, Cedar policies that deny on prompt-attack scores	Treat retrieved content and tool output as untrusted input
Sensitive information disclosure (LLM02)	PII masking in both directions; classification-based routing	Minimize data sent; scope retrieval to the user’s permissions
Supply chain (LLM03)	Pinned gateway images; approved model and provider catalog	Pin SDK and model versions
Improper output handling (LLM05)	Output content filters	Never execute or render model output without validation
Excessive agency (LLM06)	Tool authorization outside the model, such as AgentCore Policy	Least-privilege tools; confirmation for high-impact actions
System prompt leakage (LLM07)	Prompt-leakage detection	Keep secrets and authorization logic out of prompts
Unbounded consumption (LLM10)	Budgets, per-identity token limits, max_tokens clamp, WAF rate rules	Loop limits and timeouts in agents

Identity is the control everything else hangs on. Prefer SSO-issued, short-lived credentials over long-lived shared virtual keys, so every request is attributable to a person or a workload. Check which authentication options your LiteLLM edition includes, since some SSO and JWT features are licensed separately.

Responsible AI

At scale, responsible AI stops being a review each application team passes once and becomes a set of controls that are consistent, measurable and cheap to adopt. The gateway is the natural enforcement point for controls that don’t depend on application context; the rest stay with application owners.

Dimension	Platform control	Application owner
Safety	Bedrock Guardrails content filters on by default	Tune thresholds for the domain
Privacy	PII masking; classification-based routing	Data minimization
Fairness	Bias testing as part of model onboarding	Domain-specific fairness tests
Veracity	Contextual grounding checks available centrally	Citations in RAG answers; task evaluations
Transparency	Model catalog with model and AI service cards	Tell users when they are interacting with AI
Human oversight	Approval workflows for high-impact actions	Human-in-the-loop design
Accountability	Identity-attributed audit trail; incident process	A named owner per application

Evaluation as a gate

Make evaluation a precondition for change, not an afterthought. Run offline evaluations before onboarding a model or changing a routing alias, using Amazon Bedrock evaluations or AgentCore Evaluations for agents, and sample production traffic for online evaluation. When a provider releases a new model version behind an alias, rerun the suite before switching. LLM-as-a-judge scales the work, but calibrate it against human labels periodically.

Guardrails at scale

Roll guardrails out in log-only mode first, calibrate thresholds against labeled traffic, then enforce. Budget for their latency and cost on every request, and decide explicitly how streamed outputs are moderated, because checking a complete response and checking a stream are different problems. Map each application to a risk tier using a framework such as the NIST AI RMF, ISO/IEC 42001 or the EU AI Act’s categories, and scale the required controls with the tier rather than applying the maximum everywhere.

Operating model and model lifecycle

The limiting factor after the first dozen teams is usually process, not infrastructure. These are the pieces worth standardizing early.

Area	What to standardize
Onboarding	Self-service request that creates the team, tags, budget, model allowlist and keys in one step; measure time-to-first-call in hours, not weeks
Model catalog	Approved models with owner, allowed data classes and Regions, cost tier, evaluation scores and end-of-life dates
Model lifecycle	Stable aliases (model_name) so applications don’t change when the underlying model does; track Bedrock legacy and end-of-life notices; evaluate before every swap
Change management	Gateway config and policies in version control; canary releases for images and routing changes; tested rollback
Service levels	Gateway availability excluding provider outages, gateway-added latency at p95, throttle rate, and budget-lockout rate, each with an error budget
Runbooks	Provider outage, quota exhaustion, budget lockout, guardrail false-positive spike, key compromise
Financial cadence	Weekly showback, monthly chargeback, quarterly Provisioned Throughput and tier review

Alternative: Amazon Bedrock AgentCore Gateway

AgentCore Gateway began as a way to expose APIs and Lambda functions as MCP tools. With inference targets it can also act as a managed LLM proxy: one endpoint, routing based on the model field, credential abstraction, and governance through Bedrock Guardrails and AgentCore Policy. Most of the capabilities this section relies on (inference targets, rate limits, guardrails in policy, HTTP interceptors) shipped between June and August 2026, so validate every detail below against current documentation before committing.

How inference targets work

A target is configured in one of two ways. A connector target is zero-configuration for supported providers: bedrock-mantle, openai and anthropic. A provider target gives explicit control over the endpoint, supported operations, model IDs or glob patterns, provider path rewriting and model-prefix stripping, and works for any HTTPS provider without a connector.

Clients call /inference/v1/chat/completions, /inference/v1/responses or /inference/v1/messages using the OpenAI or Anthropic SDKs, and switch models by changing the model string. Routing follows three rules. A model ID qualified as target/model goes to that target. An unqualified ID is matched against all targets, with exact matches beating globs. When several targets match equally, the Bedrock target wins if present; otherwise requests are spread across the matches (AWS’s connector and provider pages describe this as random and as round-robin respectively). Either way it is load spreading, not failover: nothing retries a throttled or failed request on another target, so retries belong to the client. Qualify model IDs whenever the destination matters.

Inbound authentication is IAM (SigV4) or JWT, and outbound authentication is either the gateway’s IAM role or an API key held in the AgentCore Identity token vault, so clients never hold provider credentials. Private access uses the com.amazonaws.region.bedrock-agentcore.gateway interface endpoint. Note that VPC endpoint policies can only filter IAM principals, so OAuth callers require a wildcard principal in the endpoint policy.

Multi-Region. The bedrock-mantle connector calls Mantle in the gateway’s own Region. To add capacity in other Regions, create one provider target per Regional endpoint (for example https://bedrock-mantle.us-west-2.api.aws), give each a distinct name, and select one by qualifying the model ID, either in the client or in a REQUEST interceptor that rewrites model to target/model. Don’t leave several Bedrock targets matching the same unqualified ID and let collision handling choose.

Permissions and discovery. A gateway role you create yourself needs bedrock-mantle:CreateInference on Mantle project resources before the connector can call Bedrock; the AmazonBedrockMantleInferenceAccess managed policy covers inference plus read access. The aggregated /inference/v1/models endpoint returns target-prefixed IDs, but a provider target defined with glob patterns has nothing concrete to enumerate, so publish your model catalog rather than relying on discovery.

Quotas. AgentCore publishes Gateway quotas in tool terms: 200 tool calls per second and 5,000 concurrent connections, each per gateway and per account, plus a 15-minute invocation timeout, all adjustable. There is no separate published quota for inference requests, and customer-defined rate limits can only tighten the service-managed ceiling, never raise it. Size expected requests per second and concurrent streams against those figures and request increases through Service Quotas before launch.

Capability comparison

Capability	LiteLLM on ECS or EKS	AgentCore Gateway
Operations	You run tasks, Aurora and ElastiCache	Fully managed and serverless
Bedrock path	bedrock-runtime, including cross-Region profiles and application inference profiles	bedrock-mantle in the gateway’s Region; other Regions need one provider target each; metrics in a separate AWS/BedrockMantle namespace
Routing and resilience	Weighted deployments, retries, cooldowns, fallbacks	Model-based routing; load spreading on collisions; no automatic retry or cross-target failover
Rate limits	Per key and team RPM and TPM	RPS, RPM, TPM and connections per user, role, target or model
Budgets	Native per key, team and user	Not native; build with interceptors (see below)
Spend tracking	Native spend logs	Not native; interceptor ledger plus Mantle project metrics
Authorization	Virtual keys and team model access	IAM or JWT plus Cedar policies
Guardrails	Bedrock Guardrails integration	Guardrails inside Policy: content filters, prompt attacks, sensitive information
Response caching	Redis, including semantic caching	None documented
Streaming	Native	SSE passed through; RESPONSE interceptors force buffering
Custom logic	In-process Python hooks and callbacks	One REQUEST and one RESPONSE Lambda interceptor per gateway
Cost model	Infrastructure plus the team that runs it	$0.005 per 1,000 Gateway invocations;$0.000025 per Policy authorization; guardrail safeguards at Bedrock Guardrails rates; rate limits free; plus interceptor Lambda

At September 2026 list prices, a million inference requests through a gateway with a policy engine attached cost about $30 in platform fees ($5 for Gateway, $25 for Policy), before model tokens, guardrail safeguards and interceptor Lambda time. AWS’s pricing examples describe Gateway invocations in tool terms (ListTools, InvokeTool, Ping), so model each inference call as one invocation until the pricing page says otherwise. Model tokens dominate at any realistic volume; the platform fees matter mainly for comparison with the cost of running LiteLLM’s infrastructure.

Custom hooks: interceptors

Interceptors are the extension point that makes AgentCore Gateway usable for enterprise LLM governance. A REQUEST interceptor runs before the target is called and can validate, rewrite or short-circuit the request. A RESPONSE interceptor runs before the response returns to the caller and can inspect or rewrite it. Inference targets use the HTTP interceptor envelope (an http key with base64-encoded bodies), not the MCP envelope. The constraints below shape every design.

Constraint	Detail	Design response
Cardinality	At most one REQUEST and one RESPONSE interceptor per gateway, and only Lambda functions	One dispatcher function per interception point, routing internally by path and model
Payload	Bodies are base64 strings; headers arrive only when passRequestHeaders is true; httpMethod is read-only	Decode defensively; never log Authorization headers
Short-circuit	Returning transformedGatewayResponse from a REQUEST interceptor returns immediately without calling the model. AWS documents that the RESPONSE interceptor then does not run; in my testing in September 2026 it was invoked anyway, with the short-circuit response	Use for budget denials and policy errors, and make the RESPONSE handler a no-op when nothing was forwarded
Response context	For inference targets, the RESPONSE interceptor receives no request or headers; request ID and gateway ARN arrive in the Lambda client context. I confirmed that both phases of one call carry the same REQUEST_ID, which the documentation does not state	Stash identity keyed by REQUEST_ID in DynamoDB with a TTL
Response headers	In my testing, headers set by a RESPONSE interceptor on inference responses did not reach the client; changes to the body did	Annotate responses in the body, as a field clients ignore, not in headers
Streaming	HTTP interceptors run in buffered mode only: the gateway buffers the entire SSE stream and passes it to the RESPONSE interceptor as raw events, not JSON. With no token rate limit configured, the stream carried no usage data unless the request asked for it. In my measurement a 300-token stream’s first byte arrived at 1,496 ms, together with its last, against 506 ms direct	Parse the body as SSE; set stream_options.include_usage in the REQUEST interceptor; streaming clients see no tokens until generation finishes, see the options below
Size	Lambda’s 6 MB synchronous payload limit applies; a payload filter can exclude RESPONSE_BODY	Excluding the body also removes the usage data you wanted; keep outputs bounded instead
Retries	The gateway may retry interceptor invocations	Make spend writes idempotent on the request ID
Latency	AWS’s sample measured +93 ms at p50 and +63 ms at p90 with provisioned concurrency; without it, p90 overhead rose to +267 ms. My own paired measurement found a median +84 ms for invoking a do-nothing interceptor on both phases, on top of 36 ms for the gateway hop	Provisioned concurrency for both functions; reserved concurrency to protect the account’s Lambda pool
Permissions	The gateway role invokes the interceptor functions	Grant only the specific function ARNs, never a wildcard

For a worked reference implementation of this — one interceptor running a chain of swappable plugins for identity, budgets, guardrails, routing, caching and metering, deployed and measured — see Customizing inference on AgentCore Gateway.

The streaming trade-off. Because interceptors are configured per gateway, attaching a RESPONSE interceptor affects every interactive stream on that gateway. Three workable options: accept buffering for batch and agent traffic; run interactive traffic on a separate gateway that has only a REQUEST interceptor and relies on native token rate limits; or record usage asynchronously from logs and metrics rather than inline. Test the user-visible effect before choosing.

Reference pattern: per-user budgets

AWS publishes a sample, sample-agentcore-gateway-usage-interceptor, that adds per-user cost attribution and hard daily and monthly budgets to an inference gateway used by Claude Code and Codex. Its flow is the template for most custom FinOps on AgentCore:

1. The client sends a JWT; the gateway’s custom JWT authorizer validates it.
2. The REQUEST interceptor reads the user and team claims, stores them against the request ID, reads the user’s spend row from DynamoDB, and short-circuits with an error if a cap is exceeded.
3. The gateway calls Bedrock through bedrock-mantle with its own role.
4. The RESPONSE interceptor parses usage from the JSON body or buffered SSE stream, prices it, emits CloudWatch embedded metrics by user, team and model, and increments the spend counters atomically.
5. A scheduled function archives counters to S3 at the configured rollover time, where Athena and QuickSight provide history and dashboards. The sketches below show the core of both functions. They are illustrative; the AWS sample is the production-grade reference.

import base64, json, os, time
from decimal import Decimal
import boto3

ddb = boto3.resource("dynamodb")
spend = ddb.Table(os.environ["SPEND_TABLE"])
context_table = ddb.Table(os.environ["CONTEXT_TABLE"])
DEFAULT_DAILY_CAP = Decimal(os.environ.get("DAILY_CAP_USD", "50"))
PASS_THROUGH = {"interceptorOutputVersion": "1.0", "http": {}}

def jwt_claims(headers):
    # The gateway's JWT authorizer has already verified the signature.
    auth = next((v for k, v in (headers or {}).items() if k.lower() == "authorization"), "")
    part = auth.split(" ")[-1].split(".")[1]
    return json.loads(base64.urlsafe_b64decode(part + "=" * (-len(part) % 4)))

def deny(status, code, message):
    body = json.dumps({"error": {"code": code, "message": message}}).encode()
    return {"interceptorOutputVersion": "1.0", "http": {"transformedGatewayResponse": {
        "statusCode": status, "contentType": "application/json",
        "body": base64.b64encode(body).decode()}}}

def lambda_handler(event, context):
    request = event["http"]["gatewayRequest"]      # requires passRequestHeaders
    request_id = context.client_context.custom["REQUEST_ID"]
    claims = jwt_claims(request.get("headers"))
    user, team = claims["sub"], claims.get("team", "unassigned")

    # RESPONSE interceptors don't receive the request, so stash identity.
    context_table.put_item(Item={"request_id": request_id, "user": user,
                                 "team": team, "ttl": int(time.time()) + 900})

    row = spend.get_item(Key={"pk": f"USER#{user}"}).get("Item", {})
    cap = row.get("daily_cap_usd", DEFAULT_DAILY_CAP)
    if row.get("daily_spend_usd", Decimal(0)) >= cap:
        return deny(429, "BUDGET_EXCEEDED", "Daily budget reached")  # short-circuit
    return PASS_THROUGH

def extract_usage(body_b64, content_type):
    raw = base64.b64decode(body_b64 or "").decode("utf-8", "replace")
    if "event-stream" not in (content_type or ""):
        return json.loads(raw or "{}").get("usage") or {}
    usage = {}  # buffered SSE: merge usage from the events that carry it
    for line in raw.splitlines():
        data = line[5:].strip() if line.startswith("data:") else ""
        if data in ("", "[DONE]"):
            continue
        event = json.loads(data)
        for src in (event, event.get("message") or {}, event.get("response") or {}):
            usage.update(src.get("usage") or {})
    return usage

def lambda_handler(event, context):
    response = event["http"]["gatewayResponse"]
    request_id = context.client_context.custom["REQUEST_ID"]
    who = context_table.get_item(Key={"request_id": request_id}).get("Item")
    usage = extract_usage(response.get("body"), response.get("contentType"))
    if not (who and usage):
        return PASS_THROUGH
    try:  # the gateway may retry interceptors, so record each request once
        context_table.update_item(
            Key={"request_id": request_id},
            UpdateExpression="SET recorded = :t",
            ConditionExpression="attribute_not_exists(recorded)",
            ExpressionAttributeValues={":t": True})
    except context_table.meta.client.exceptions.ConditionalCheckFailedException:
        return PASS_THROUGH
    cost = price(usage)  # Decimal; your price table incl. cache read/write rates
    spend.update_item(
        Key={"pk": f"USER#{who['user']}"},
        UpdateExpression="ADD daily_spend_usd :c, monthly_spend_usd :c",
        ExpressionAttributeValues={":c": cost})
    emit_metrics(who, usage, cost)  # CloudWatch EMF: user, team, model
    return PASS_THROUGH

Attach both functions at the gateway level. AWS’s sample does this with UpdateGateway, which requires resending the gateway’s existing settings:

aws bedrock-agentcore-control update-gateway \
  --gateway-identifier <gateway-id> \
  ...existing gateway settings... \
  --interceptor-configurations '[
    {"interceptor": {"lambda": {"arn": "<request-fn-arn>:live"}},
     "interceptionPoints": ["REQUEST"],
     "inputConfiguration": {"passRequestHeaders": true}},
    {"interceptor": {"lambda": {"arn": "<response-fn-arn>:live"}},
     "interceptionPoints": ["RESPONSE"],
     "inputConfiguration": {"passRequestHeaders": true}}
  ]'

The same REQUEST interceptor can also implement virtual models: AWS’s documentation shows a function that rewrites a stable alias to a concrete target/model ID before routing. That is the closest AgentCore equivalent to LiteLLM’s model aliases and a simple form of cost-based routing.

Native rate limits

Rate limits are the native control to use before reaching for interceptors. Dimension keys group traffic by targetName, toolName, qualifiedModelId, JWT claims, or the IAM principal or source identity. Each rate limit contains entries with request, token and connection rates.

aws bedrock-agentcore-control create-gateway-rate-limit \
  --gateway-identifier <gateway-id> \
  --dimension-keys '["$.context.jwt.team", "qualifiedModelId", "$.context.jwt.sub"]' \
  --description "Per-user token limits by team and model" \
  --entries '[
    {"dimensions": {"$.context.jwt.team": "research",
                    "qualifiedModelId": "*", "$.context.jwt.sub": "*"},
     "tokens": [{"rate": 80000, "period": "minute"}]},
    {"dimensions": {"$.context.jwt.team": "*",
                    "qualifiedModelId": "*", "$.context.jwt.sub": "*"},
     "tokens": [{"rate": 20000, "period": "minute"}],
     "requests": [{"rate": 30, "period": "minute"}]}
  ]'

Key behaviors to design around:

• Token limits apply to inference paths only. The gateway estimates input tokens with a general-purpose tokenizer, deducts the estimate up front, then reconciles with provider-reported usage. For streaming chat completions it adds include_usage automatically.
• Counts don’t match Bedrock quota. The gateway counts input and output tokens as reported and does not adjust for prompt caching, while Bedrock quota weights newer Claude output tokens five times. Treat gateway TPM as fairness control, not quota protection.
• Evaluation order matters. Rate limits run before Policy, so a caller who will be denied by Policy still consumes budget; add zero-rate entries for such groups. All limits must pass, more-specific limits are evaluated first, and wildcards may only appear in trailing positions.
• Always include a catch-all entry. Callers that match no entry bypass that rate limit entirely.
• Rate limiting fails open. Use it for fairness and quality of service, not as a security boundary.

Policy and guardrails

AgentCore Policy evaluates Cedar policies at the gateway boundary with default-deny and forbid-overrides-permit semantics. Guardrails can now be expressed inside policy, and they run on inference targets (POST /inference) as well as MCP and runtime targets. Supported safeguards are content filters, prompt-attack detection and sensitive-information detection, each with a confidence threshold. A forbid policy blocks a request; the new suppressOutput effect suppresses a response that violates a guardrail.

forbid (
  principal,
  action == AgentCore::Action::"<inference-action-id>",
  resource == AgentCore::Gateway::"<gateway-arn>"
) when guardrails {
  BedrockGuardrails::PromptAttack(
    ["PROMPT_INJECTION", "JAILBREAK"],
    [context.input.<field-holding-user-content>]
  ).maxConfidenceScore().greaterThan(decimal("0.4"))
};

Take the exact action ID and data paths from the schema your policy engine generates. Run the engine in LOG_ONLY mode first and calibrate thresholds against labeled traffic, as AWS recommends. The gateway role needs bedrock:InvokeGuardrailChecks. Guardrails in policy are available in a subset of Regions, including US East (N. Virginia and Ohio), US West (Oregon), Europe (London and Stockholm) and Asia Pacific (Sydney and Tokyo), and a single policy cannot mix standard Cedar conditions with guardrail conditions. Separately, AWS warns that without a token limit policy each inference request can stream an unbounded response, so configure one on every inference target.

Cost attribution on AgentCore

Bedrock records the gateway’s IAM role as the caller for all traffic, so native attribution sees one principal. IAM principal cost allocation now covers bedrock-mantle, but it still attributes everything to that role. Three ways to split it:

• A gateway per business unit, each with its own tagged role. Simple and authoritative; the default quota is 1,000 gateways per account.
• Bedrock projects, which provide tag-based attribution on bedrock-mantle. Clients normally select a project through the OpenAI SDK’s project setting. Whether a REQUEST interceptor can set that on the outbound call on behalf of users is not documented; validate before relying on it.
• An interceptor ledger, as in AWS’s sample, which gives per-user figures in near real time but is an estimate that must be reconciled against CUR.

Choosing between them

Choose LiteLLM when	Choose AgentCore Gateway when
You need cross-Region inference, application inference profiles or Provisioned Throughput on bedrock-runtime	Your models are available on bedrock-mantle, OpenAI or Anthropic, in-Region capacity is sufficient, and you want no gateway infrastructure
You need native budgets, spend logs, fallbacks and response caching today	Identity-based authorization, Cedar policy and centralized guardrails matter more than routing sophistication
Traffic is high-volume or streaming-heavy and needs inline accounting	You already use AgentCore Gateway for MCP tools and want one governed front door for tools and models
You have a platform team to operate it	You want AWS to own availability and patching

A hybrid is also possible: AgentCore Gateway as the governed front door for identity, policy, guardrails and per-user limits, with LiteLLM registered behind it as a custom provider target for routing, failover, budgets and caching. That combines strengths but doubles the moving parts, the latency hops and the token meters, so treat it as a pattern to prototype, not a default.

My assessment: for Bedrock-centric organizations with moderate volume and a mature SSO posture, AgentCore Gateway removes an entire operational tier and is worth piloting now. For high-volume, multi-Region or streaming-heavy workloads that need budgets and failover today, LiteLLM remains the more complete gateway. Revisit the comparison quarterly, because AgentCore Gateway is changing quickly.

Reference configuration

LiteLLM Terraform

module "litellm" {
  source  = "BerriAI/litellm/aws"
  version = "~> 1.89"          # pin to your tested release

  region = "us-east-1"
  azs    = ["us-east-1a", "us-east-1b", "us-east-1c"]
  tenant = "acme"
  env    = "prod"

  # Scale the gateway on ALB requests per target
  gateway_target_requests_per_second = 50   # tune from load tests
}
# One root configuration per Region. Add HTTPS, WAF and
# endpoints alongside the module; see sections 4 and 10.

LiteLLM config

model_list:
  # Baseline: Provisioned Throughput
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/<provisioned-model-arn>
      aws_region_name: us-east-1
      weight: 3

  # Burst: geographic cross-Region profile
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/us.<model-id>
      aws_region_name: us-east-1
      weight: 1

  # Independent quota: second account
  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/us.<model-id>
      aws_region_name: us-west-2
      aws_role_name: arn:aws:iam::<account>:role/litellm-bedrock
      weight: 1

  - model_name: claude-haiku
    litellm_params:
      model: bedrock/us.<smaller-model-id>
      aws_region_name: us-east-1

router_settings:
  routing_strategy: simple-shuffle
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  redis_password: os.environ/REDIS_PASSWORD
  num_retries: 2
  fallbacks: [{"claude-sonnet": ["claude-haiku"]}]

litellm_settings:
  success_callback: ["s3"]      # request logs to S3
  cache: True
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    password: os.environ/REDIS_PASSWORD

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  proxy_batch_write_at: 60
  database_connection_pool_limit: 8   # from the calculator
  use_redis_transaction_buffer: true

No AWS access keys appear anywhere: the first two deployments use the task role, and the third assumes a role in another account. Check parameter names against the LiteLLM release you pin, since the Bedrock provider options evolve quickly.

Task environment and command

# Injected from Secrets Manager
LITELLM_MASTER_KEY, LITELLM_SALT_KEY, DATABASE_URL, REDIS_PASSWORD

# Plain environment
LITELLM_MODE="PRODUCTION"
DISABLE_SCHEMA_UPDATE="true"

# Command (4 vCPU task)
--port 4000 --config ./config.yaml \
--num_workers 4 --run_gunicorn \
--max_requests_before_restart 10000

Sources