AI gateway

An AI gateway is a proxy that sits between applications and large language model (LLM) providers, so every model call passes through it. Because it sees all model traffic, it can report token spend per team or model, enforce budgets, fail over between providers and keep provider API keys out of application code.

How it works

Applications send model requests to the gateway instead of the provider, often through an OpenAI-compatible API. The gateway forwards each request, records the model, tokens, latency and status, and applies routing and budget rules. If the application propagates the W3C traceparent header, the gateway's spans join the application's trace. It does not see your retrieval steps, tool logic or answer quality.

Why it matters in production

Without a gateway, model spend and provider errors are scattered across every service that calls a model. A gateway gives one place to see which team or model drove the bill, with no application code changes. It complements in-process instrumentation rather than replacing it.

See also

• LLM observability

• Rate limiting

• Multi-tenancy

• Token (LLM)

On the One2N blog

• What is AI observability? A guide for LLM and agent apps

• AI agent observability: tool calls, turns and runaway loops

• Observing AI apps: instrument, break, compare

References

1. Envoy AI Gateway documentation