Customizing inference on AgentCore Gateway: a plugin pipeline for routing, budgets and guardrails

Every model call that goes through an LLM gateway passes a single point where you can decide something about it: who is calling, whether they can afford it, whether the prompt is acceptable, which model should answer, and what it cost. This is a reference architecture for Amazon Bedrock AgentCore Gateway that turns each of those decisions into a small, swappable plugin, deployed and measured end to end.

September 2026. Code, infrastructure and measurements: github.com/bfeeny/paper-to-aws-routing (the gateway/ package and infra/pipeline.yaml).
I work at Amazon Web Services. The views and assessments here are my own and do not represent AWS.

Why the gateway is the place

An application can enforce policy on its own model calls, and many do. It stops working the moment there is more than one application, more than one team, or a model choice that should change without a release. A gateway sees every call regardless of which client made it, so it is where organization-wide decisions belong:

• Cost control. A per-tenant budget in dollars, and hygiene such as capping max_tokens, which matters more on Bedrock than it looks. Bedrock reserves the full max_tokens against a model’s tokens-per-minute quota when a request starts, so one client asking for 32,000 “to be safe” throttles everyone else.
• Safety. Screening prompts once, centrally, whichever model ends up answering.
• Model choice. Clients ask for a stable alias; what it resolves to can change, split for an A/B test, or come from a trained router.
• Attribution. Tokens and dollars per tenant and model, from the provider’s own usage figures.

AgentCore Gateway can front Amazon Bedrock and other providers with one OpenAI- and Anthropic-compatible endpoint. Its extension point is the interceptor, and the rest of this post is about using that one extension point well.

Native first, then interceptors

Before writing Lambda code, it is worth being precise about what the gateway already does, because a native feature is cheaper to run and harder to get wrong than anything reimplemented in an interceptor.

Need	Native in AgentCore Gateway	Where an interceptor is still needed
Rate limits	Requests, tokens and connections per target, model, IAM principal or JWT claim; token limits reserve an estimate and reconcile actual usage	Limits in dollars, or across models with different prices
Guardrails	Policy guardrails on inference targets: content filters, prompt attacks, sensitive information	Pattern rules such as deny lists or regular expressions, which Policy guardrails do not offer; Regions where Policy guardrails are not available
Model routing	Route on the model field; qualify as target/model	Virtual aliases, weighted splits, learned routers
Traffic rules	Gateway rules match IAM principal or path	Anything keyed on the model, a claim or the prompt
Budgets and chargeback	Not native	Spend ledger and per-tenant metering
Response caching	Not native as a response cache. Bedrock’s own prompt caching discounts a repeated prompt prefix, but the model still runs	An exact-match response cache that skips the model entirely

Two caveats about the native column. Rate limits fail open by design, so AWS advises against treating them as a security boundary. And the documentation does not say whether native limits see the model before or after an interceptor rewrites it, so a per-model limit and an interceptor-based router should be tested together before relying on both.

The interceptor’s natural territory is therefore the right-hand column: dollar budgets, metering, custom routing, custom rules, and anything that has to combine several of those.

One interceptor, a chain of plugins

AWS documents two hard constraints that shape everything else: a gateway can have at most one REQUEST interceptor and one RESPONSE interceptor, and interceptors can only be Lambda functions. There is no native chaining. So the design is one function, attached to both phases, that runs an ordered chain of plugins in-process.

A plugin is a class with two optional hooks. The request hook may rewrite the body or return a rejection; the response hook sees the provider’s response and may record or annotate it, but cannot reject, because by then the tokens are spent.

class Plugin:
    name = "plugin"

    def on_request(self, call: Call) -> Reject | None: ...   # rewrite call.body, or refuse
    def on_response(self, call: Call) -> None: ...           # meter, settle, annotate

A Call carries the parsed body, the headers, the resolved tenant and a scratch dictionary that plugins use to hand facts forward: the router records its score, metering records the cost that the budget plugin then settles. Every plugin is timed on every call, and one CloudWatch embedded-metric record per phase carries those timings, which is how the costs later in this post were measured.

The chain itself is a JSON document in SSM Parameter Store:

{
  "fail_open": true,
  "plugins": [
    {"plugin": "tenant",       "params": {"header": "x-tenant-id"}},
    {"plugin": "max_tokens",   "params": {"default": 1024}},
    {"plugin": "router",       "params": {"strategy": "length", "threshold": 2000,
                                          "strong": "mistral.mistral-large-3-675b-instruct",
                                          "weak": "mistral.ministral-3-3b-instruct"}},
    {"plugin": "model_policy", "params": {"allow": {"*": ["*"]}}},
    {"plugin": "budget",       "params": {"daily_usd": {"*": 1}}},
    {"plugin": "guardrail",    "params": {"guardrail_id": "…", "version": "2"}},
    {"plugin": "metering",     "params": {}}
  ]
}

What the interceptor contract forces

The interceptor contract is small, and three parts of it dictate the design.

Rejection is a short-circuit. Returning transformedGatewayResponse from the REQUEST phase makes the gateway answer immediately without calling the model. The pipeline uses it for every refusal, with an OpenAI-compatible error body so existing SDKs surface it cleanly: 401 or 403 for identity and model policy, 400 for a guardrail, 429 for a budget.

The response phase does not see the request. The RESPONSE interceptor’s input carries gatewayRequest: null: no headers, no body, no tenant. What it does receive, in the Lambda client context, is the gateway’s REQUEST_ID. The documentation lists that value but does not say the two phases share it. They do; I confirmed it on the live gateway. So the request phase leaves a small record under that ID, and the response phase reads it back.

Each phase starts from nothing. This is the part that catches people, and it caught me. The RESPONSE invocation is a separate Lambda call: it gets no request, and it does not share memory with the REQUEST invocation, which may not even have run in the same container. A value computed on the way in simply does not exist on the way out. Anything the response phase needs has to be written down deliberately, and the correlation record above is where it goes. My cache plugin got this wrong first: it computed a key on the way in, looked for that key on the way out, found nothing, and stored nothing. Every request was a miss, no error was raised, and the only visible symptom was that the cache never worked.

The gateway may retry an interceptor. AWS says so and advises idempotent functions. For metering that is harmless. For a spend ledger it is a double charge waiting to happen, so settlement is a single DynamoDB transaction: a conditional write marking the request settled, and the spend increment. A retry finds the mark, the condition fails, and nothing is charged twice.

The correlation write costs a DynamoDB round trip on every call, so the handler only makes it when some plugin in the chain declares that it needs the response phase.

The plugins

Nine plugins cover the common cases. Each is a single short module; the costs are measured medians from the live stack.

Plugin	Phase	What it does	Measured cost
tenant	request	Resolves the caller; rejects unknown tenants	<0.1 ms
max_tokens	request	Caps output tokens per model, returning quota to everyone else	<0.1 ms
router	request	Resolves a virtual model: pinned, length threshold, weighted split, or learned	<0.1 ms (length)
model_policy	request	Which models each tenant may call, as glob patterns, judged after routing	<0.1 ms
budget	both	Refuses calls once today’s spend reaches the limit; settles actual cost afterward	3.4 ms, 20.7 ms
guardrail	request	Screens the prompt with the ApplyGuardrail API	190 ms
metering	both	Prices the call from the provider’s usage block; emits cost by tenant and model	0.1 ms
cache	both	Answers a repeat request from the store, without calling a model	2.3 ms (Valkey), 4.5 ms (DynamoDB)
escalate	response	Replaces a weak answer that failed with the strong model’s answer	+750 ms when it fires

The tenant plugin reads a header in this reference deployment, which is only acceptable for a demonstration. In production the tenant must come from something the caller cannot forge, such as a claim in a JWT validated by the gateway’s authorizer. The plugin is the one place that decision lives, so hardening it does not touch anything else.

Two plugins that change the outcome

Seven of the nine plugins inspect a request and let it through or stop it. The other two are more interesting, because the interceptor can also decide what the client actually receives.

A cache answers the call itself. The mechanism is the one a rejection uses: a REQUEST interceptor that returns a response short-circuits the gateway. A rejection returns an error; a cache hit returns a real answer. The model is never called, no tokens are spent, and the client cannot tell the difference except that it arrived sooner. Measured on the live gateway, a miss took 1,609 ms and hits settled at 267 ms — and that remaining quarter-second is not the cache, it is the gateway hop plus the interceptor’s own two invocations.

The key is the request as the client sent it, including the virtual model name rather than the resolved one, so an entry is “the answer we would serve for this request”. That also means an answer improved by an escalation is what gets cached, which is why the plugin sits before the router.

Exact matching is the honest limit of this plugin. Two requests hit the same entry only if they are byte for byte identical: a trailing “please”, a reordered clause or a different capital letter is a miss. That suits machine-generated traffic — evaluation suites, batch jobs, retried agent steps, a UI that sends the same canned prompt — and it suits very little human chat. The obvious repair is to match on meaning rather than bytes, and that turns out to be a harder problem than it sounds: in my own measurements, prompts edited so that their answers change were more embedding-similar to the original than honest paraphrases were. A semantic cache therefore needs more than a similarity threshold, and it is the subject of the follow-up to this article.

An escalation replaces a bad answer. A RESPONSE interceptor may rewrite the body, and nothing stops it calling a model of its own. So the cheap model answers, the interceptor looks at what came back, and if the answer stopped at the token ceiling or ran unusually long, it asks the strong model and returns that answer instead. The client sees a single call.

This is the deployable form of a result from the companion study, Replicating RouteLLM on Amazon Bedrock: a prompt-only router barely beat prompt length at predicting which requests needed the strong model, while the cheap model’s own output predicted it far better. Deciding after an answer exists is an easier problem than guessing before one does.

The price is stated plainly, because it is steep. On the live gateway an escalated request took 1,234 ms against 484 ms, and cost 31 times as much, since both calls are billed and they run in series. Escalation is for traffic where a wrong answer costs more than a slow one, and the trigger deserves tuning against real failures rather than the defaults here.

Three ways to cache, and when each one helps

“Caching” covers three different mechanisms on AWS, and they are not substitutes for one another.

	What it avoids	When it applies	Measured
Interceptor response cache	The entire model call	The identical request repeats	Hit 267 ms against a 1,609 ms miss
Bedrock prompt caching	Re-reading a shared prompt prefix	The prefix repeats, the question differs	8,360 of 8,372 input tokens served from cache
Semantic cache	The model call, for similar-not-identical requests	Paraphrases recur	Treated separately: an embedding per request, and a similarity threshold alone is not safe

The middle row deserves its own numbers, because it is native and often overlooked. Marking a long system prompt as cacheable and calling Claude through bedrock-runtime three times produced this:

no cache point   inputTokens 8372   cacheWrite     0   cacheRead     0
1st with point   inputTokens   12   cacheWrite 8360   cacheRead     0
2nd, 3rd         inputTokens   12   cacheWrite     0   cacheRead 8360

The model still runs and still generates, so latency is barely affected, but the repeated prefix stops being charged as fresh input. That is the right tool for a long system prompt, a big retrieved document, or a tool schema shared across requests, where the answers must differ and a response cache would be useless. The interceptor can set those markers on traffic that never asked for them — which is exactly the sort of policy the gateway exists to apply centrally.

Two honest caveats. I measured prompt caching directly against bedrock-runtime with a Claude model, not through the gateway, because the Claude models were not callable through this account’s bedrock-mantle connector. And the open-weight models I could reach through the gateway reported no cache fields at all, so support is per model and worth verifying before designing around it.

DynamoDB or ElastiCache for the response cache

The cache backend is a plugin parameter, so both could be compared behind the same Lambda, in the same VPC, with nothing else changing. Thirty hits each:

Backend	Plugin time, p50	What the client sees
DynamoDB on-demand	4.51 ms	267 ms
ElastiCache Serverless (Valkey)	2.26 ms	271 ms

ElastiCache is twice as fast at the thing it does, and it makes no difference whatsoever to the caller, because 84 ms of Lambda invocation sits in front of it. It would begin to matter at a request rate where a few milliseconds per call is real money, or where the same store is also carrying rate-limit counters and session state.

It is not free, either. ElastiCache lives in a VPC, so the interceptor has to join that VPC, and then everything else it calls needs a path. The stack builds that properly rather than reaching for a NAT gateway: private subnets with no internet route, interface endpoints for bedrock-mantle, bedrock-runtime, ssm and logs, and the free gateway endpoint for DynamoDB. The cost of the VPC is paid in cold starts, which went from about 1.7 s to about 4 s once the function had to attach an elastic network interface.

Order is policy

The order of the chain is not a detail. It decides what gets checked, what gets paid for, and what each plugin can know.

Four rules fall out of it:

1. Cheapest rejection first. The budget plugin refuses an over-budget call after one DynamoDB read, so a guardrail check that costs 190 ms and a per-request fee is never paid for a call that was going to be refused anyway.
2. Route before judging the model. model_policy and a reserving budget must see the model that will actually be called, not the auto alias the client sent, so they run after the router.
3. Responses unwind in reverse. Metering is last on the way in, so it is first on the way out: it prices the call before the budget plugin, earlier in the chain, settles it. One ordered list serves both phases.
4. Decide what failure means. A plugin that throws is a bug in the plugin, not a verdict on the request. The chain defaults to fail-open: record the error and continue. That is the right default for metering and the wrong one for a compliance guardrail, so fail_open: false turns any plugin error into a 503. The next section describes the failure that made this rule concrete.

Swapping behavior without a deploy

Because the chain is data, changing gateway behavior is a parameter write. Moving from the length router to a weighted split for an A/B test looks like this:

aws ssm get-parameter --name /gwpipeline/pipeline --query Parameter.Value --output text > pipeline.json
jq '.plugins |= map(if .plugin == "router" then .params = {"strategy": "weighted",
      "models": {"mistral.ministral-3-3b-instruct": 90, "mistral.mistral-large-3-675b-instruct": 10}}
    else . end)' pipeline.json > next.json
aws ssm put-parameter --name /gwpipeline/pipeline --overwrite --type String --value file://next.json

Each function reloads within 30 seconds. The weighted strategy hashes the prompt rather than drawing at random, so a retried request lands on the same arm and an A/B comparison does not double-count.

The router plugin also accepts a trained scorer, which is where the idea for this architecture started. A companion study, Replicating RouteLLM on Amazon Bedrock, tested whether a learned prompt-only router beats a simple length threshold. Mostly it did not, once evaluated on traffic groups it had not seen. That is an argument for making the router cheap to replace, not for leaving it out: the pipeline turns “which router” into a configuration choice that can be measured in production, instead of a code change.

What it costs

I measured the overhead of each layer through a live gateway, pairing every gateway call with a direct Bedrock call made immediately before it, so that model latency drift cancels out. Four conditions, 60 pairs each, with an 8-token completion from mistral.ministral-3-3b-instruct:

Condition	Median overhead	95% interval	p90
Gateway, no interceptor	36 ms	[17, 55]	287 ms
Interceptor on both phases, empty chain	120 ms	[74, 135]	270 ms
Six plugins, no guardrail	166 ms	[146, 175]	277 ms
All seven plugins	349 ms	[334, 360]	430 ms

The numbers decompose cleanly, and the interceptor’s own timings account for each step:

• Invoking the interceptor is the fixed cost. With an empty chain the Lambda does almost nothing, yet the two invocations add 84 ms: roughly 40 ms of round trip per phase. Provisioned concurrency removes the cold starts (the first call in a new container took 1.7 s) but not this.
• Shared state is cheap but not free. A budget read is 3.4 ms; the idempotent settlement transaction is about 20 ms; the correlation record makes up the rest of the 46 ms.
• The guardrail dominates. 190 ms inside the Lambda, almost all of it the ApplyGuardrail call. Screening every prompt more than doubles the pipeline’s cost.
• In-process logic is free. Tenant resolution, token capping, length routing, model policy and pricing each took under a tenth of a millisecond.

For scale, the direct call itself took about 280 ms. Against a short completion the full chain roughly doubles latency; against a long generation it disappears into the noise. The design guidance follows directly: keep work in-process where possible, pay for a network hop only when the decision needs it, and put the expensive hop after the cheap checks.

Streaming is the real trade-off. Attaching a RESPONSE interceptor makes the gateway buffer a streamed response and hand it over whole. Through the pipeline, the first byte of a 300-token stream arrived at 1,496 ms, the same instant as the last; calling Bedrock directly, the first byte arrived at 506 ms. For batch and agent traffic that is acceptable. For an interactive chat UI it is not, and the fix is architectural: put interactive traffic on a gateway with only a REQUEST interceptor, and rely on native token rate limits there.

What the documentation doesn’t tell you

Running the pipeline against a live gateway turned up seven behaviors that the documentation either leaves out or contradicts. They are observations from one account and Region in September 2026, and worth rechecking as the service evolves.

1. Both phases share the REQUEST_ID. The documentation lists the value in the client context but never says the REQUEST and RESPONSE invocations of one call carry the same one. They do, and the correlation design depends on it.
2. The RESPONSE interceptor runs after a short-circuit. The documentation says it does not, for HTTP and inference targets. When the guardrail refused a request, the interceptor was invoked again with the 400 response. The handler now treats a response with no forwarded request as a no-op.
3. Headers set by the RESPONSE interceptor are dropped; body changes are kept. An x-gateway-cost-usd header never reached the client. The same value added as a top-level field of the JSON body did. The pipeline annotates responses with an x_gateway object, which OpenAI-compatible clients ignore.
4. Streams reach the RESPONSE interceptor, buffered, as raw events. The body is a server-sent event stream, not JSON, so a handler that assumes JSON fails silently and meters nothing. It also carries no token usage unless the request asked for it, so the metering plugin sets stream_options.include_usage on streamed chat completions and reads usage from the final chunk. One side effect: clients receive that usage chunk too.
5. Guardrail strength is a confidence floor, and HIGH is aggressive. With the prompt-attack filter at HIGH, “Reply with the single word: ok.” was blocked as a low-confidence prompt attack. At MEDIUM that prompt passes and a real injection attempt, scored at high confidence, is still blocked.
6. A security group scoped to the VPC broke DynamoDB, silently. Moving the interceptor into a VPC for ElastiCache, I wrote egress rules allowing 443 only to the VPC’s own CIDR, which looks tighter and is wrong: a DynamoDB gateway endpoint routes to public AWS ranges through a prefix list, not to an address inside the VPC. Every request then hung for the full 60-second Lambda timeout. Allowing 443 to anywhere is the correct rule here and is not as broad as it reads, because the subnets have no NAT and no internet gateway, so the only reachable destinations are the endpoints themselves.
7. Fail-open can hide a bug. The state store was opened lazily, so the first request in every new Lambda container reached the budget plugin with no store, raised, and, the chain being fail-open, skipped the budget check. Nothing failed visibly; the error appeared only in the per-plugin error field of the metrics. The fix was one line, opening the store at initialization. The lesson is that a fail-open plugin needs its errors watched as closely as its rejections.

I got the fourth one wrong at first. The streaming case looked like the RESPONSE interceptor never ran, because the handler’s JSON parse failed and it passed the response through without logging a metric. The Lambda’s own invocation records showed otherwise. When an interceptor appears not to run, check the invocation count, not the application logs.

Deploying it

The whole stack is one CloudFormation template: the gateway with a bedrock-mantle inference target, the interceptor Lambda attached to both phases, the DynamoDB table, a Bedrock guardrail, and the SSM parameter holding the chain.

git clone https://github.com/bfeeny/paper-to-aws-routing && cd paper-to-aws-routing
make pipeline-up                  # deploy; cache on DynamoDB, no VPC
make pipeline-up BACKEND=valkey   # the same, with a private VPC and ElastiCache
make pipeline-bench               # measure each layer against direct Bedrock
make pipeline-down                # remove everything

Clients point any OpenAI-compatible SDK at the gateway URL plus /inference/v1, send model: "auto", and never learn which model answered unless they read the x_gateway annotation. The offline test suite, tests/test_pipeline.py, exercises every plugin, the event contract, rejection, correlation, idempotent settlement and stream metering without calling AWS.

Limits of this reference

• One account, one Region, open-weight models. The measurements used Mistral models through bedrock-mantle in us-east-1. Claude models were not callable through the gateway’s bedrock-mantle connector on this account when this was written.
• Header-based tenancy. The demonstration reads the tenant from a header. Production must derive it from the authorizer.
• Sixty pairs per condition. Enough for stable medians and tight intervals on the medians; not enough to characterize tail latency, where the p90 figures above are dominated by model variance on both paths.
• Budget overshoot is possible. Between the budget check and settlement, concurrent calls can each pass and together exceed the limit by up to one call each. The reserve option refuses any call whose worst case would cross the limit, which never overshoots at the price of refusing some calls that would have fit.
• Not load-tested. Throughput, DynamoDB hot keys for a single busy tenant, and Lambda concurrency limits were out of scope. The cache comparison in particular measures the regime where ElastiCache has least to offer.
• Prompt caching was measured outside the gateway, against bedrock-runtime with a Claude model, because Claude was not reachable through this account’s gateway connector. The interceptor-side marker injection follows from it but was not exercised end to end.
• The VPC option costs money at idle. Interface endpoints and ElastiCache Serverless both bill whether or not traffic flows, unlike the DynamoDB default. make pipeline-down removes them.