The gateway pipeline I have been building enforces a per-tenant budget, partitions a cache per tenant, partitions a vector index per tenant, and meters cost per tenant. Every one of those controls resolved the tenant by reading a request header. Any caller could set it to any value. This article replaces that header with something a caller cannot forge, adds the rate limit the earlier articles kept promising, and keeps personal data out of the model — and each of the three turned out to have a sharp edge that only showed up once it was running.
The hole in the first three articles
The pipeline resolves the tenant like this:
value = call.headers.get("x-tenant-id", "").strip()Everything downstream keys on the result. The daily budget is per tenant. The exact cache is partitioned per tenant. The semantic cache’s vector index is partitioned per tenant — by a HASH search-schema element, so the isolation is structural and enforced by DynamoDB. The metering that produces the chargeback view is per tenant.
All of it rests on a string the caller chooses. x-tenant-id: globex is not an assertion the system can check; it is a request to be treated as Globex, and the pipeline grants it. Spend another tenant’s budget, read another tenant’s cached answers, bill another tenant for your traffic — one header away, every time.
I wrote that limitation into the first article’s closing section and moved on, which is the normal and slightly dishonest thing to do: the demo works, the caveat is disclosed, the reader is trusted to fix it. The trouble is that a structural isolation built on a forgeable key isn’t weaker isolation, it’s the appearance of isolation, and the DynamoDB partitioning makes it look more solid than it is.
Tenancy that survives a lying client
AgentCore Gateway supports a CUSTOM_JWT authorizer: give it an OIDC discovery URL and a list of allowed client IDs, and it validates every inbound token — signature, issuer, expiry, client — before any of your code runs. A request that fails never reaches the interceptor.
The issuer here is an Amazon Cognito user pool with one app client per tenant using the client-credentials grant, which is the machine-to-machine case: no user, no redirect, just a service proving which client it is. Tenancy lives in a resource-server scope, gw/tenant-acme, because a Cognito M2M access token carries no user attributes at all — you can add custom claims, but only with a version-three pre-token-generation trigger on the Essentials or Plus feature plan, and a scope says the same thing without that machinery.
Same interceptor, same plugins, same state table. The only difference is who gets to say who you are.
The live check, run against the deployed gateway:
| Request | Result |
|---|---|
Token scoped gw/tenant-acme | tenant = acme |
Token scoped gw/tenant-globex | tenant = globex |
Acme’s token + header x-tenant-id: globex | tenant = acme |
| No token | 401, never reaches the interceptor |
| Malformed token | 401, never reaches the interceptor |
| Valid token, signature tampered | 403, never reaches the interceptor |
The header is still in the request. Nothing reads it any more.
Should the interceptor check the signature again?
The gateway already validated the token. Re-doing that work in the plugin looks redundant, and the argument for skipping it is real: fetch the JWKS, verify RS256, check issuer and expiry — that is work on every request to re-establish something the platform established a moment ago.
The argument against skipping it is that “the gateway validated it” is not a property of the request, it is a property of the configuration, and configuration changes. Switch the authorizer back to AWS_IAM for an afternoon of testing and a plugin that only decodes will read the tenant out of a token anyone can mint with a text editor. Nothing fails. Nothing logs. The billing is simply wrong from then on.
So I measured it, from the interceptor’s own per-plugin timings:
| median | p90 | |
|---|---|---|
| Decode claims only | 0.176 ms | 0.193 ms |
| Verify signature against cached JWKS | 0.536 ms | 0.778 ms |
| Reading a header (the old plugin) | 0.005 ms | 0.010 ms |
Verification costs 0.36 ms more than trusting, plus one JWKS fetch per container — an ~805 ms outlier visible exactly once per cold start, cached for an hour afterward with a forced refresh when an unknown kid appears, because an issuer’s routine key rotation should not become an outage some weeks later.
A third of a millisecond, against a model call that takes hundreds. Verify.
Counting the wrong thing
The earlier articles kept deferring rate limiting, and having built it I understand why it is usually shipped as “N requests per minute”: that limit is easy, and it is almost unrelated to what an LLM gateway needs to protect.
A request is not a unit of anything. One request can be eighty tokens or eighty thousand. The quantity that runs out — the model’s tokens-per-minute quota — is measured in tokens, and a request counter cannot see it. This is not a modeling preference; it is how the platform behaves. On the bedrock-mantle endpoint the input tokens plus max_tokens are checked against the input-token-per-minute quota before a request is admitted, and one that would exceed it is throttled. Omitting max_tokens does not help: the model’s own maximum is used instead.
So the plugin enforces both windows, and charges tokens the way the platform does — reserving max_tokens up front and reconciling against real usage on the way out.
Same ten requests, same 4,000-token reservation each. The request limit cannot see the 40,000 tokens; the token limit sees nothing else.
The bottom two rows are the interesting ones, and they are not a bug. Arriving together, ten reservations of 4,000 are outstanding at once and the limit stops the burst after two. Arriving in series, each reservation is reconciled down to actual usage — around 18 tokens — before the next request arrives, so the window never accumulates and all ten pass.
That is the behavior you want from a quota guard and it is unlike a rate limiter. It does not limit throughput; it limits simultaneous commitment. A client making a thousand sequential small calls is never refused, because it never threatens the quota. Ten concurrent calls that each reserve 4,000 tokens do threaten it, whatever their eventual usage, and they are refused before any of them answers — which is the only moment at which refusing is useful.
It also inherits a dependency worth naming: the reservation is only as honest as the client’s max_tokens. A client asking for 4,000 “to be safe” and using 18 consumes eighty times its share of the limit for the duration of its call. That is exactly the hygiene problem the max_tokens clamp exists for, and the two plugins want to be deployed together: clamp first, then reserve the clamped value.
The reservation nobody gave back
Here is the part I did not design, and found by reading a log.
The token arm refused requests 2 through 9 and kept refusing them well past the point where the earlier reservations should have been reconciled. The counter climbed by 4,000 per request and never came down. Tracing it:
0ms request reserved_total=4000 (fine)
272ms response reconciled 4000/18 (correct — 3,982 returned)
520ms request ended_by=cache reserved_total=4018
777ms request ended_by=rate_limit reserved_total=8018
1045ms request ended_by=rate_limit reserved_total=12018The third line is the bug. The exact cache served that request from the interceptor — no model call, no tokens spent — and because the cache short-circuits the chain, the response phase never ran, so nothing ever returned the 4,000 tokens the rate limiter had reserved a few microseconds earlier. Every cached request walked the tenant’s own window up by a full reservation. A tenant whose traffic was entirely cache hits, costing nothing and consuming no quota at all, would lock itself out within a handful of calls.
The general shape: a plugin that reserves something needs a path back out of every early exit, not just the successful one. The pipeline had two hooks, on_request and on_response, and a rejection or a cache hit used neither.
Two of the three exits were implemented. The missing one was the exit that costs nothing, which is why it looked harmless.
So the pipeline gained a third hook. When a plugin ends the call, every plugin that already ran is unwound in reverse order — newest first, like a stack — and told what happened:
def _abort(self, order, stopper, call, verdict, trace):
ran = order[:order.index(stopper)]
for plugin in reversed(ran):
try:
plugin.on_abort(call, verdict)
except Exception as exc:
trace.errors[f"{plugin.name}.abort"] = repr(exc)[:200]The plugin that ended the call is skipped, because it knows what it did. A failure inside a release is recorded and swallowed, because a broken cleanup must not turn a clean 429 into a 500. The rate limiter releases its tokens and deliberately keeps the request count — the request was real, it just never reached a model.
PII: block, redact, or put it back
Three things get called PII handling at a gateway and they are not substitutes.
Blocking refuses the request. Bedrock Guardrails does this natively and the pipeline already wires it up. It is safe and it is useless for the common case, where the personal data is the entire point of the request — “summarize this support ticket” is about a named customer.
Redacting replaces the data with a placeholder and sends that. The model never sees it, and the answer comes back discussing {NAME_0}, which is correct and unreadable.
Tokenizing redacts on the way in and substitutes the real values back on the way out. The model never sees the data; the caller never sees a placeholder. It is only possible somewhere that sees both halves of the call, which is the whole argument for doing it at a gateway rather than in each application.
The detector is the whole product
I planted 16 known entities across eight prompts and asked two detectors to find them.
| Detector | Found | Median latency | Cost per call |
|---|---|---|---|
| Regex | 7 / 16 (0.438) | 0.11 ms | none |
Amazon Comprehend DetectPiiEntities | 16 / 16 (1.000) | 38 ms | $0.0003 floor |
The regex caught every email address, phone number, SSN and IP address, which is what patterns are for. It missed every name, the street address, the postal code, an age, a bank account number and a routing number — because nothing about the string “Priya Raman” says it is a name. That is not a regex that needs improving. It is a question regexes cannot answer.
Comprehend costs 0.0000052, PII detection costs fifty-eight times the inference it protects. Against a frontier model at $0.0016 it is a rounding error. As with the semantic cache, the control is worth its price only in proportion to the call it wraps.
The cheaper API that must not be used here
Comprehend has a second PII operation, ContainsPiiEntities, which answers “is there any personal data in this document” instead of “where is it”. It costs 0.0001 — a 0.0003, fifty times cheaper. The optimization writes itself: screen every prompt with the cheap call, and pay for span detection only when the screen says there is something to find. At a realistic 5% PII rate that is 14× cheaper overall.
I implemented it, measured it, and turned it off. On five prompts carrying obvious personal data, the screen returned no labels at all for two of them:
| Prompt | ContainsPiiEntities | DetectPiiEntities |
|---|---|---|
| ”Email the invoice to priya.raman@…” | (nothing) | EMAIL, score 1.0 |
| ”Wire to account 000123456789, attn Marcus Feld.” | (nothing) | BANK_ACCOUNT_NUMBER 1.0, NAME 1.0 |
| ”Priya Raman called; call back (415) 555-0173.” | PHONE 1.0, NAME 1.0 | NAME 1.0, PHONE 1.0 |
| ”SSN 123-45-6789 on file.” | SSN 0.94 | SSN 1.0 |
| ”Ship to 1600 Amphitheatre Parkway…” | ADDRESS 0.808 | ADDRESS 1.0 |
This is not a threshold that wants lowering — there is no score to compare, the label list came back empty. The cheap call answers a weaker question and answered it wrong on an email address, which is the single most detectable category there is.
Used as a gate in front of masking, it forwards exactly the data the plugin exists to hide, and it does so silently: the prompt goes to the model unmasked, nothing is logged, and the only evidence is in a service you are no longer calling. The parameter is still in the code, defaulted off, with the measurement in the docstring — it is the right tool for deciding whether to alert, a question you are allowed to get wrong occasionally.
Where the mapping lives, and why order matters
The placeholder-to-original mapping has to reach the response phase, and the RESPONSE interceptor receives no request, so it travels in the correlation record — which means the pipeline writes a list of exactly the strings it is protecting into DynamoDB. That record is written only when restoration is on, carries a short TTL, and belongs in a table with a customer-managed key in any real deployment. Redaction without restoration needs no such record; when you can live with placeholders in the answer, that is the safer build. It is a real choice, not an oversight.
Placing the plugin before the cache then produces something better than I expected. The cache key is computed on the masked prompt, so:
- nothing personal is ever written to the cache, and
- two different people asking the same question mask to the same text, share one entry, and the hit rate goes up.
The catch is that the stored answer is full of placeholders, and a cache hit short-circuits the chain — so on_response never runs and the caller receives {EMAIL_0} verbatim. The substitution has to happen on the abort path, against the mapping built from this request, which is the correct mapping precisely because the masked prompts matched. The same hook the rate limiter needed, for a completely different reason. Verified end to end against the live gateway: five prompts, every planted entity restored, no placeholder leaked.
What the three cost
In-Lambda medians from the interceptor’s own metrics:
| Plugin | Median | Notes |
|---|---|---|
tenant (header) | 0.005 ms | what it replaces |
jwt_tenant, decode only | 0.176 ms | trusts the gateway |
jwt_tenant, verified | 0.536 ms | plus one JWKS fetch per container |
rate_limit | 6.4 ms | one DynamoDB atomic counter (p90 51 ms) |
pii, regex | 0.11 ms | 44% recall |
pii, Comprehend | 38 ms | 100% recall, $0.0003 |
Identity is free. Quota costs one round trip to DynamoDB. Keeping personal data out of the model costs more than the inference does on a small model, and is negligible on a large one.
Native first, as always
Two of these have native counterparts worth preferring where they fit.
AWS WAF attaches to an AgentCore Gateway and evaluates every inbound request before it reaches any target. Its rate-based rules count requests per aggregation key over a 60, 120, 300 or 600 second window, and can key on source IP, a header, a cookie, a query argument, a label, or ASN. That is the right tool for volumetric abuse, and it stops a flood outside your code entirely. It cannot express “this tenant may spend 50,000 tokens a minute”, and — worth knowing before you plan around it — it cannot key on a JWT claim, because WAF does not decode tokens; keying on the raw bearer token aggregates per token rather than per subject. One caution in the docs deserves repeating: a request missing the key you aggregate on is not rate limited at all, which makes a custom-header key a bypass waiting to be found. Association is not a gateway property either — you set wafConfiguration.failureMode on the gateway (FAIL_CLOSE by default) and make the association through WAF itself.
Bedrock Guardrails blocks and can anonymize PII natively, on a policy the security team can own without touching the pipeline. Use it for the block decision. What it does not do is hand back the mapping needed to restore an answer, which is the only reason the pii plugin exists rather than a guardrail configuration.
There is also a native control pointing the other way that I did not use and want to flag: the interceptor’s input configuration accepts a PayloadFilter that excludes the RESPONSE_BODY, so the gateway can decline to show your own Lambda the model’s output. For a pipeline whose job is data minimization that is an attractive guarantee — and it is incompatible with metering from the response, caching the answer, escalation, and PII restoration, all of which need the body. Worth knowing it exists; worth knowing what it costs.
Limits of this build
- One issuer, two tenants, one Region. Cognito with two app clients is enough to prove the mechanism and nothing about federation, multi-issuer trust, or token exchange.
- The scope is the tenancy claim. It works and it is conventional, but it conflates authorization scope with identity. A deployment with many tenants wants a real claim, which on Cognito M2M means a V3 pre-token-generation trigger on a paid feature plan.
- Fixed windows. The limiter’s default window admits up to twice the limit across a boundary. The
smoothoption weights the previous window at the cost of one extra read per call; it is implemented and was not benchmarked here. - Ten requests per arm. Enough to show a limit engaging and a reservation being returned; not a load test, and nothing here characterizes DynamoDB hot-partition behavior for a single very busy tenant.
- PII on eight English prompts. The entity set is planted and small, and Comprehend’s perfect recall on it should be read as “the class of thing a regex misses, it catches”, not as a recall figure for your corpus. Non-English text, code, and structured logs will all behave differently.
- The gateway validates; the plugin re-validates. Neither checks revocation. A token valid for its full lifetime remains valid after the client is disabled, which for a five-minute M2M token is usually acceptable and should be a deliberate decision rather than an assumption.