EU by default
Out of the box the router only allows EU-sovereign models: requests stay on infrastructure inside the EU. External providers (OpenAI, Anthropic, Google, Mistral) sit outside the EU and are blocked until your organisation explicitly accepts non-compliant routing. That acceptance is recorded once, with the identity and timestamp of the admin who made it, and can be revoked at any time. Even after acceptance, an external call only proceeds when the request is protected: either the PII firewall is on for that request, or the request carries a valid compliance acknowledgement. The egress guard enforces this in code and fails closed: if neither condition holds, the call is refused rather than sent.Models
The Models tab lists the model catalogue with the context window, capabilities and credit cost per 1,000 input and output tokens for each. Use it to enable or disable individual models for your organisation. A disabled model is rejected for your tenant even though it exists in the catalogue. You can sort the table by model, provider, or input/output price to compare options. The same catalogue is available over the API, so you can build a model picker or validate a prompt length without hardcoding anything:
A
null is a fact, not a gap: it means the provider publishes no value, and we would rather say so than invent one. The list contains only the models your workspace may actually call, so anything disabled for you is absent rather than present-but-forbidden.
Routing aliases
A routing alias is a name you invent that maps to one or more real models. You send the alias as themodel in your request; the router resolves it to the first target, and if that target is unavailable it falls back to the next one in order.
Aliases let you keep a stable name in your code, for example fast or default, and change what it points to here without redeploying. They also give you resilience: a primary model with an external fallback (or vice versa) keeps requests flowing if one provider has an outage.
Aliases or a models chain?
You can also send a chain inline, withmodels instead of model:
An alias is policy. Its value is late binding: when a provider has an incident at two in the morning, an admin repoints the alias and every request moves, with no deploy from anyone.
A chain is a preference. Its value is that there is nothing to set up: paste a list into the call you already have.
They compose. An alias inside
models expands in place, so the chain above tries the large Mistral model, then whatever fast currently resolves to, then GPT. That lets you lean on an admin’s curated chain and still append your own last resort.
Whichever you use, a fallback is never silent, but where the target that served is named depends on how you asked. On an ordinary response it is the model field. On a stream it is the additive served object on the finishing chunk, because every chunk of one completion carries one envelope and what will serve is not known when the first of them goes out. Do not read model on a stream to detect a fallback; read served, described under Streaming.
Provider preferences
A request can express a routing preference with aprovider object.
Read this first, because the name is borrowed and one meaning differs. Other gateways use provider.order to choose between several companies hosting the same model. Our catalogue maps one model to exactly one provider, so there is no host to choose. Here order sorts the entries of your chain by who serves them. The practical consequence: on a single-model request, provider.order does nothing at all. It only has an effect once there is more than one target.
Preferences only narrow
Every field above can remove targets. None can add one. Nothing you send in a request can enable a model your workspace has disabled, reach a provider you have no credentials for, or route data outside the EU without a recorded acknowledgement. An API key belongs to one workspace, so a key holder reorders and restricts within what an admin permitted, and never past it.residency: "eu" is the useful case: it lets a single sensitive request insist on EU-only routing without changing any configuration, even in an organisation that has accepted external routing generally.
If your preferences remove every target, the request is refused rather than quietly widened, and the reason is recorded on the audit event.
Provider keys (bring your own key)
By default, calls to external providers use the platform’s relationship with that provider. On the Provider keys tab you can instead store your own API key for a provider. When a key is present, requests to that provider are billed to your account with them rather than brokered through the platform. Keys are encrypted at rest and never shown again after you save them: only the last few characters are displayed so you can tell which key is stored.Compatibility
The Compatibility tab tunes how the router handles requests that do not map cleanly onto every provider:- Unsupported parameters. When a request uses an OpenAI parameter a target provider does not support, choose lenient (drop the unsupported parameter and proceed) or strict (refuse rather than silently ignore it). In strict mode a chain moves on to the next target instead of failing outright, so a model that cannot do tool calls simply steps aside for one that can. The request fails only when no target can honour it.
- Firewall tool content. When on, the PII firewall also scans the content of tool and function-call messages, not just ordinary message text, before anything leaves the platform.
What is additive over the OpenAI API
Everything below is extra. An unmodified OpenAI client works without any of it.modelsin place ofmodel, for an inline fallback chain. Sending both is rejected, so there is no guessing about which one you meant.- The
providerobject described above. usage.credits, a{base, services, total}breakdown on every response, so you can reconcile spend without opening the dashboard.baseis inference,servicesis everything else charged against that request, andtotalmatches what the ledger debited. Note thatservicesis an amount of credits, not a list of which services ran.- Streaming follows OpenAI’s convention: set
stream_options: {"include_usage": true}and a final chunk arrives carryingusagewith an emptychoicesarray. We also keepusageon the finishing chunk whether or not you ask, so nobody loses sight of what a request cost by not knowing the flag exists.
Streaming
Setstream: true and the answer is written as the serving provider generates it. Chunks are the provider’s own tokens, so they are variable in size and there is no fixed length to build on: read delta.content and append it. A cached answer and a provider that does not stream both arrive in even pieces instead, which is why the size is not something to depend on either way.
Tool calls stream too, in OpenAI’s incremental shape. Each chunk carries one delta.tool_calls[] entry with an index, and a call’s id and function.name arrive once at its start while function.arguments arrives as a series of fragments that are not valid JSON on their own. Group by index, concatenate the argument fragments in order, and parse when the stream finishes.
The choices of three consecutive chunks, with the envelope each of them also carries (id, object, created, model) left out:
id and model, because a client accumulating them keys on that and a second identity appearing halfway through is what such a client cannot survive. So model on a stream is the name the completion was opened under, not the target that answered: the router has not called anyone when the first chunk goes out, and an alias or a models chain does not resolve until it does. The finishing chunk carries an additive served object instead, and that is what to read when you need to know a fallback happened:
The finishing chunk, with the envelope and the usage object it also carries left out:
A cached answer is the exception, and it is the one case where it does not matter: it reached no provider, so it carries no
served object and its model is the target that produced the stored answer, exactly as on an ordinary response.
Fragments are withheld where they cannot be re-identified. A pseudonym can be split across two fragments of argument JSON, and rewriting half of one would corrupt the call your machine then executes. So on a request whose prompt was pseudonymized, the fragments are not sent at all and the finishing chunk carries the whole tool_calls array with the real values restored, entries indexed exactly as the fragments would have been. On a request with nothing masked, which is the default EU route, you get the fragments and the finishing chunk does not repeat the calls. One accumulator keyed on index handles both, so reading them the way above covers each case without knowing which one you are in.
A Guard block ends the stream rather than raising. The 200 and the headers go out with the opening chunk, so once the body has begun there is no status code left to send. When an output rule blocks the answer, the last chunk’s choice carries finish_reason: "content_filter" and, beside it, an additive content_filter.categories naming what fired; [DONE] follows. Treat content_filter as a refusal, not as a completed answer. A non-streaming request is unaffected and still receives the guard_violation 422.
A Guard redaction on a streamed answer arrives too late to apply. An output rule set to redact rewrites the answer, and on a stream that rewrite is produced after the text it removes has already been sent to you. It cannot be sent afterwards either, because you would append it to what you hold rather than replace it. So the finishing chunk keeps its ordinary finish_reason and carries an additive content_filter: {"redacted": true}, which says a rule rewrote this answer and the copy you assembled is not the rewritten one. Everything the platform stores afterwards, the cached entry and the conversation transcript, holds the redacted form. If a redaction has to reach your readers, use buffered below or do not stream.
Whether the answer streams while Guard’s output rules are still to run is a workspace setting, under Guard. Async, the default, streams and reports afterwards: a block ends the stream, a redaction is only marked. Buffered delivers nothing an output rule has not passed: where the workspace enforces and a rule scans the output, nothing streams and the whole answer arrives at once, already redacted where a rule redacted it, and a block instead ends the stream with the same content_filter chunk before a byte of the answer has been written.
Buffered holds nothing back where there is nothing to wait for, so a request still streams under it when no rule scans the output, when the enforcement mode is shadow, which is where a workspace sits until it explicitly enforces, and when the Guard service is off.
Any other mid-stream failure is an error frame inside the 200, in the same {"error": {"message", "type", "code"}} shape a failed non-streaming request returns. A provider that dies partway through an answer is the case this exists for.
Fallback stops at the first byte. The chain in Routing aliases tries the next target when one fails, but only while nothing has been delivered. Once any fragment of an answer has reached you, a later failure is reported rather than retried elsewhere: a second target would restart the answer from its first token, and you would render the fragments you already hold followed by the whole of a different answer, with nothing in the stream marking the seam.