Streaming makes responses feel fast, but it creates a hard recovery boundary. Learn when an LLM request can fail over safely, why emitted tokens cannot be replayed silently, and how clients should handle partial output.
Point a router alias at a candidate set with the Latency strategy, keep the automatic cross-provider failover, tune the two settings that decide how fast a bad provider is abandoned, and prove the effect on your own p95.
An average latency number describes an experience nobody had. Here is how to read LLM latency as a distribution — p50, p95, p99, time-to-first-token and total completion time — and how to find what is actually causing a bad tail.
You could run your own LLM gateway, and some teams should. We chose to run one for you instead. Here is the operational bill self-hosting hands you, the mechanism we replace it with, what our choice genuinely costs you, and the teams that should still self-host.
Checking a balance and then calling a provider is a race, and under the fan-out an LLM gateway is built for it loses. Here is the reserve, settle and release contract as you can observe it — what your balance does on success, on an upstream failure, on a timeout, and when the cost is never knowable.
A coding agent turns one task into hundreds of model calls. Here is how a gateway gives each run a hard ceiling that holds under burst, a fallback path that does not strand a half-finished edit, and a cost you can read per task.
Both a throttle and a budget block can arrive as 429, and both an empty balance and a team budget arrive as 402. Branch on the error code, not the status — with backoff, jitter, and idempotent retries.
When a provider 5xxs, overloads or times out, the request should still return. Here is how an ordered fallback chain advances only on retryable failures, bills exactly one hop, and turns a 38% error rate into 0.125%.
