Migrate to the gateway
Move to Orca AI Gateway from another LLM gateway, or from the earlier mcp-gateway service, without a hard cutover.
The posture
Whatever you are migrating from, the shape is the same:
- Stand the gateway up alongside the incumbent, on a different port. Nothing routes to it yet.
- Send a sample of traffic to it and compare responses and telemetry against your baseline.
- Cut over, keeping the incumbent configured as a fallback destination for the first week.
- Remove the fallback once you have a week of clean signal.
Step 3 is worth taking literally. A fallback strategy whose last target is your previous gateway
means a mistake in the new configuration degrades into the old path rather than into an outage.
routes:
- name: chat
match:
path: "/v1/chat/completions"
strategy:
mode: fallback
targets:
- destination: openai-primary
- destination: legacy-gatewayConcept mapping
Most LLM gateways express the same handful of ideas. The translation:
| Their concept | Orca AI Gateway |
|---|---|
| A configured provider or served endpoint | A destinations[] entry with the matching kind |
| Traffic split by percentage | routes[].strategy.mode: loadbalance with weights |
| Fallback chain | routes[].strategy.mode: fallback |
| Retry policy | routes[].strategy.retry |
| Virtual key or API key management | vaults[] plus credentials.vault on a destination |
| Usage tracking | plugins.usage_sinks[] |
| Policy-decision logging | plugins.audit_sinks[] |
| Guardrails and content filters | plugins.guardrails[] |
| Rate limits | plugins.rate_limiter plus rate_limits[] |
| Model aliasing | model_map on a destination |
Two differences usually need attention rather than translation. For model traffic, a config with no
matching route uses the lexicographically first compatible destination as route default; make a
catch-all explicit when you need a stable, named default. Tenancy is whatever you declare in
identity.scope_dims, so a gateway with a fixed notion of "user" or "team" maps onto a dimension
you choose.
Check the supported plugin kinds before assuming a one-to-one mapping. If
your incumbent writes usage to a warehouse, the supported delivery paths are Kafka, Postgres, the
Agent Engine registry, or stdout collection. Parquet and OTLP usage sink kinds are not wired.
From the earlier mcp-gateway service
Deployments that ran the earlier mcp-gateway service have two migration aids. Both are
deprecated.
The endpoint alias. POST /mcp is served as an alias for /v1/mcp, logging a deprecation
warning at most once a minute. This lets you point an existing client at the new gateway without
changing its URL.
The config compatibility shape. The loader accepts the original YAML verbatim under a
legacy_mcp_gateway: key and expands it into the modern shape before validation:
legacy_mcp_gateway:
server:
port: 8090
upstreams:
- name: managed-llm
url: "http://llm.svc:9000"
policies:
- allow:
workspace_id: "*"The expansion maps server.port to server.listen, each upstreams[] entry to an mcp destination
plus a route pointing /v1/mcp at it, and policies[] to a yaml_acl authorizer.
When legacy_mcp_gateway: is present, every other top-level key must be absent. The two shapes are
mutually exclusive and mixing them fails validation, so this is a migration step rather than a
permanent mode.
Rewrite into the modern shape as soon as the cutover is stable. The compat layer cannot express routing strategies, guardrails, budgets, or anything else added after the original gateway.
Verifying a migration
Before cutting over, confirm three things beyond "requests succeed":
- Credentials resolve for every destination, not just the one you tested. Run
orca-gateway admin config validate --file <path>against the running gateway, which exercises live vault resolution rather than just structure. - Rate limits land where you expect.
token_bucket_localis per replica; useredis_windowwhen a fleet must share one ceiling. See Rate limits. - Telemetry is complete. Compare usage record counts against your incumbent's for the same window. A silent sink misconfiguration looks exactly like low traffic.