If you have already invested in an AI gateway or in a hyperscaler’s Guardrails, the honest first thing to say is: keep them. A gateway’s job is the model call — route it, cache it, balance it, budget it. Guardrails’ job is content safety on that call. Both are real and both are good at what they do.
This page used to say the two categories do not overlap at all. That is no longer accurate in either direction, and both corrections belong here:
- Several gateways now document agent-facing surfaces of their own. The sourced boundaries are in the next section; read them rather than assuming a gateway stops at the request.
- Olivares has one narrow, measured execution path. It is described below with the conditions it actually carries, not as a routing product.
TL;DR: Olivares AI is not a general AI gateway: it does not cache or load-balance model traffic, and it claims no universal provider, model or routing matrix. It does resolve routing policies into an ordered fallback chain, and it has one measured execution path — deny-closed, actuating only through a Claude Messages-compatible client that you configure, either directly or with your own gateway endpoint as the Messages base URL. Beyond that path it is the governance and evidence plane: in-process enforcement in the agent runtime, a tamper-evident ledger, non-human-identity lifecycle, and human-in-the-loop / break-glass / kill-switch over live sessions. It composes with the gateway you already run rather than replacing it.
What a gateway and Guardrails do well (use them for this)
These are commodity, well-understood capabilities, and the vendors describe them plainly:
- AI gateways are request-path managers for model calls. LiteLLM is an “OpenAI Proxy Server (LLM Gateway) to call 100+ LLMs in a unified interface & track spend, set budgets per virtual key/user” (LiteLLM); Cloudflare AI Gateway lets you “Connect to any model, dynamically route requests, and manage usage, billing, and logs from one unified gateway” (Cloudflare); Portkey “records real-time API requests, including cost” (Portkey). Routing, fallbacks, caching, virtual keys, per-key budgets, request logging — this is their lane.
- Hyperscaler Guardrails are content-safety filters. Bedrock Guardrails “provides configurable safeguards to help you build safe generative AI applications” that “detect and filter undesirable content and protect sensitive information that might be present in user inputs or model responses” — content filters, denied topics, word filters, PII redaction, contextual-grounding and automated-reasoning checks (AWS).
If your problem is “give my apps one endpoint to many models, with budgets, caching and content filtering,” that stack solves it, and you do not need a control plane to do it. We integrate with that pattern; we do not reimplement it.
What each product documents today, per product
Read on 2026-09-12 from each vendor’s own page. These are boundaries of the pages read, not statements about a whole product, and none of them is a ranking.
| Product | Agent-facing surface it documents | What the page does not cover |
|---|---|---|
| LiteLLM | The proxy documentation carries an “Agent & MCP Gateway” section, plus Guardrails, Policies, Authentication and Budgets + Rate Limits; memory is “Scoped per user and team, with built-in access control” (LiteLLM) | The page read documents the proxy surface; enforcement inside an agent runtime that never traverses the proxy is outside it |
| Portkey | Its product list names Agents, MCP Gateway, Guardrails and Security & Compliance alongside the gateway, which “records real-time API requests, including cost and guardrail violations” (Portkey) | The page read is a feature overview; it does not describe a cross-estate permitted-versus-observed access map |
| Cloudflare AI Gateway | Presented as “An intelligent control plane for your AI applications” — “Connect to any model, dynamically route requests, and manage usage, billing, and logs”, with “fallback routing, rate limiting, and safety guardrails” (Cloudflare) | The page read is a product overview; self-hosted or air-gapped operation of that plane is not described on it |
| Bedrock Guardrails | ”configurable safeguards” that “detect and filter undesirable content and protect sensitive information”, usable inline or “directly through the ApplyGuardrail API without invoking the foundation models” (AWS, AWS) | These are content-safety pages; agent identity lifecycle, session intervention and approvals are not their subject and are not documented on them |
So the old framing was wrong. “Gateways never see the agent” is not true of the pages above: LiteLLM and Portkey both document agent and MCP surfaces, and Cloudflare calls its gateway a control plane. What remains different is where enforcement sits and what kind of record comes out, which is an architectural comparison rather than an absence claim.
Where the architectures still differ
Conditional, not universal — each line holds when the condition on the left does:
| If your agents… | Then a request-path product | Then Olivares AI |
|---|---|---|
| …only ever reach models through the proxy | governs every call it sees, at the request | adds little on the request path itself |
| …also run locally and reach DBs, object stores, MCP or files directly | cannot see calls that never traverse it | enforces deny-closed in-process at the agent, before the tool runs |
| …need a record an auditor can verify off-box | emit request logs, which are mutable records | append-only, hash-chained, Ed25519-signed ledger, verifiable off-box |
| …must be stopped mid-session | is not where a live session is halted | HITL approvals, break-glass and a kill switch with dual-control re-enable |
| …must be identified over their whole life | a virtual key is a budget bucket | non-human-identity lifecycle: staleness block, offboarding cascade, dual-control rotation |
| …must stay inside your boundary | SaaS planes process that traffic in their cloud | self-hosted or air-gapped; the data plane does not leave your boundary |
The Olivares execution path, with its actual limits
Olivares does touch inference, in one measured place, and the honest description is the one its own current measurement carries:
- The route
POST /routing-policies/{id}/executeis deny-closed by construction and actuates only through a Claude Messages-compatible client, directly or with a resolved gateway endpoint as the Messages base URL — so your existing gateway can be that endpoint. - Policy resolution produces an ordered fallback chain. The module always resolves a route and acts only through an executor port; the default executor is unwired, so routing resolves with no provider call until an operator composes one.
- Two deny-closed scope gates and the kill-switch stop gate run before the FinOps budget gate, which runs before the executor.
And what it explicitly does not establish, in the same record’s words:
- no universal provider or model matrix — the Claude Messages protocol establishes one configured route, not a matrix;
- no centralized key custody — provider key fields are references;
- no execution from the console — the console resolves and tests policies and has no execute call.
This is a measurement of current sources, not a capability acceptance: the current claims record holds this and every other capability as implemented and not accepted, with no execution receipt behind it. Treat it as the shape of the path, not as a certified feature.
On Guardrails specifically: content safety is a hook, not a competitor
Bedrock Guardrails can be applied two ways — inline during a Bedrock inference
call, or “directly through the ApplyGuardrail API without invoking the
foundation models”, which works “with any foundation model whether hosted on
Amazon Bedrock or self-hosted models”
(AWS). That is genuinely useful, and
Olivares treats content safety as a detector you plug in, never a wall we ask
you to choose instead of Guardrails. Two honest, distinct facts:
- The inline inference proxy exposes a content-inspection seam — a pluggable point where a content / DLP detector returns a verdict the deny-closed decider acts on. Content safety belongs there, in the pipeline, rather than being re-implemented as a competing filter.
- Olivares reads your Guardrails’ own decisions read-first. The AWS connector
ingests Bedrock guardrail decisions from their CloudWatch / S3 logs as posture
and evidence; it deliberately does not call the paid
ApplyGuardrailruntime itself. Your content verdicts become part of the tamper-evident record.
So content safety composes with what you already run. The two Bedrock Guardrails pages cited above are content-safety pages: agent identity lifecycle, session intervention, human approvals and cost governance are not their subject and are not documented on them (read 2026-09-12; a statement about those two pages, not about AWS). Olivares complements them: it carries the identity, the session controls, the approvals and the evidence; the content filter stays where it lives.
How they compose
A healthy arrangement keeps every tool in its lane:
- Keep your gateway (LiteLLM / Portkey / Kong / Cloudflare) as the model-call plane — routing, caching, virtual keys, budgets on the request.
- Keep your Guardrails (Bedrock / Azure Content Safety) as your content-safety
detector — the Olivares PEP runs a pluggable detector at its content-inspection
seam and reads your Guardrails’ own decisions read-first as evidence; it does not
invoke
ApplyGuardrailitself. - Add Olivares beside them as the governance and evidence plane: the in-process PEP on the agents that never hit your gateway, the access map across the whole estate, the tamper-evident ledger, and the live HITL/break-glass/kill controls.
A second, separate path governs raw SDK/curl callers — API-key-only, described
in Governing subscription-authed agents.
It exists to govern traffic your other tools cannot reach, not to compete on
routing, and it never carries a subscription credential.
When your gateway is enough
Honesty cuts both ways. If your agents only ever call models through your gateway, your content-safety needs are met by Guardrails, you have no self-hosted or laptop-resident agents reaching databases / object stores / MCP directly, and you have no sovereignty or tamper-evident-evidence requirement — then your gateway plus its logs and Guardrails may be all you need, and you should not add a control plane for its own sake.
Olivares earns its place when the questions become estate-wide and adversarial: which agents exist and what each actually reached, can I stop a bad action deny-closed at the agent, who approved the risky one, and can I hand an auditor immutable proof — all without sending that picture to someone else’s cloud. For the deeper treatment of two adjacent comparisons, see vs AI control towers and vs LLM observability.