Provider Fallback Chains: Surviving an OpenAI Outage

The answer: a fallback chain is an ordered list of routes the gateway tries for one logical model. It advances only on failures a different healthy provider could plausibly succeed at — 429, 529, 5xx, timeouts — and never on 400, 401, a budget refusal or a guardrail block. Credit is reserved once for the customer request and settled against the route that actually served it, so a failover costs one call, not three. x-nr-model on the response tells you which route won.

Every model provider has bad days. A regional capacity crunch that turns into 429s. A degradation that answers slowly enough to trip your timeout without ever returning an error. A genuine outage with a status page and a postmortem. None of that is unusual and none of it is your fault — but if your application calls one provider directly, their bad day is your bad day, and your users find out before you do.

A fallback chain is the difference between "OpenAI is down" being an incident and being a line in a dashboard. This post is the engineering of that: what the chain actually promises, which failures advance it and which do not, what a real degradation looks like in arithmetic, how the money works across a failover, and where the whole idea stops helping.

The request that dies at 3am

Here is the shape of the incident, as your on-call sees it:

03:12  p95 latency on /chat/completions climbs 1.8s → 9.4s
03:14  first 529 overloaded responses appear
03:19  error rate 38%; retries make it worse, not better
03:41  provider status page acknowledges "elevated error rates"
04:44  recovery begins

Ninety minutes. During which your product either degraded gracefully or did not, and that was decided months earlier by whether anything sat between your code and that one provider.

The detail that makes this hard is 03:19. The natural reaction — retry the failing call — is actively harmful when the failure is capacity. Every client retrying simultaneously against a provider that is already shedding load is a textbook thundering herd, and the Google SRE book's chapter on handling overload is largely a catalogue of ways that goes wrong. Retrying the same route multiplies the load that caused the problem. Retrying a different route does not.

Why an application retry loop is not failover

Most teams already have retry logic. It is usually a decorator, it usually retries 5xx and 429, and it usually looks like enough. Four things separate it from failover.

It retries the same place. A retry against a saturated provider is a request that will fail again, slightly later, having consumed one of that provider's rate-limit slots on the way. You have converted an error into a slower error.

It retries the wrong things. A naive wrapper retries on status code, so it happily retries a 400 three times — the request is malformed everywhere, and you have paid nothing but latency for the privilege — and, more expensively, it retries a 429 that is a budget refusal rather than a throughput one. Per-key budget blocks arrive as 429 with key_budget_exceeded, and a wrapper that cannot tell them apart loops until something else times out. The complete taxonomy is 429 vs 402 on an LLM Gateway: Which to Retry, Which to Stop.

It has no idea which providers are healthy. Your retry decorator knows about one call. It cannot know that this provider has failed 40% of the last hundred requests across your fleet while another has failed none, because it has no fleet-wide view. Health is a property of aggregate traffic, and a per-process retry loop never sees aggregate traffic.

Every service implements it differently. Three teams, three retry policies, three sets of jitter constants, and the newest service has none. Whatever guarantee you thought you had is really the weakest implementation in the estate.

The fix is to put the decision where the traffic converges. That is the same argument as Managed LLM Gateway vs Self-Hosted: Why We Carry the Pager, applied to one specific failure mode.

The mechanism: an ordered chain, advanced only on retryable failure

A fallback chain is a customer-visible contract with a small number of promises. Configure it in Router Settings; the client sends the same request it always sent.

request for model "frontier-chat"
  ├─ hop 1  openai/gpt-5.5                       → 503 service_unavailable  ✗ retryable
  ├─ hop 2  anthropic/claude-sonnet-4-5-20250929  → 200 OK                   ✓ returned to caller
  └─ hop 3  bedrock/claude-haiku-4-5-20251001    (not attempted)

The caller asked for a capability, not a provider endpoint. That indirection is what makes failover possible at all: because the client never hardcoded a provider, the gateway is free to satisfy the request from whichever route is healthy. It is the same property that makes model switching a config change rather than an SDK rewrite — One API Key Across Providers: The Integration You Stop Writing.

The promises, stated as promises rather than implementation:

1. Ordering is yours and it is deterministic. The chain is tried in the order you configured. There is no hidden reranking.
2. Advance is conditional. Only a failure that a different healthy provider could plausibly succeed at advances the chain. Everything else fails fast to the caller.
3. One reservation per customer request. Credit is reserved once and settled against the route that served it. A failover is one bill.
4. The response reports the truth. x-nr-model names what actually ran, so a run whose cost or behaviour changed has a visible reason.
5. Exhaustion is an error, not a silent stub. If every route in the chain fails, you get the last failure, plainly. Nothing fabricates a response.

Which failures advance, and which do not:

The last four rows are the ones that separate a fallback chain from a retry loop. Falling back on a policy refusal would be worse than useless: it would try the same forbidden request against every route you own, spending real money to be refused several more times.

Worked example: 90 minutes of provider degradation

Numbers, on the incident from the top. Figures are illustrative arithmetic for one traffic shape, shown so the reasoning is checkable — read your own from analytics.

window            90 minutes
requests          12,000
primary route     38% failing (529 / timeout) at peak
chain             hop 1 primary → hop 2 secondary

availability with the chain      11,985 / 12,000  =  99.875%
availability without it           7,440 / 12,000  =  62.0%

Thirty-eight points of availability, bought with a config line. Now the two costs of buying it, both real and both worth stating.

Latency. The failed hop is not free wall-clock. Say the primary returns its 529 in 240 ms and the secondary answers in 1,900 ms, against a healthy baseline of 1,850 ms:

62.0% of requests   1,850 ms                      (one hop)
37.9% of requests     240 ms + 1,900 ms = 2,140 ms (two hops)

weighted mean   0.62 × 1,850 + 0.38 × 2,140  =  1,960 ms
                                                (+110 ms vs baseline)

A hundred and ten milliseconds on the mean, and a visibly fatter tail, in exchange for not returning errors to 38% of users. That is a trade worth making — and worth measuring rather than assuming, because a chain that fires constantly is a permanent latency tax. LLM Latency: p50, p95, p99, and Time-to-First-Token is how to read the difference honestly.

Money. If the secondary route is dearer than the primary — $0.0042 against $0.0031 per request in this example — the window costs slightly more than a healthy one:

4,545 failed-over requests × ($0.0042 − $0.0031)  =  $5.00 extra
4,560 failed first hops                            =  $0.00  (nothing billed)

Five dollars for ninety minutes of not being down. The second line is the one that matters structurally: the failed hop is not billed. No provider work was completed, no cost is settled, and there is no phantom charge to reconcile later.

Credits across a failover: reserve once, settle the winner

This is the part most retry implementations get wrong, and it is worth being precise about because the wrong version is invisible until an audit.

1  reserve      credit held once, for the customer request
2  hop 1        529 → nothing settled, reservation untouched
3  hop 2        200 → settle the SERVING route's authoritative cost
4  release      any remainder of the reservation returns to available

Reserve-per-attempt would double-hold credit during exactly the moments a provider is unhealthy, which is to say it would make outages cost customers money. Settling the requested model's price rather than the serving route's would be a fiction that mis-prices every run during a degradation. Both are avoided by making the reservation a property of the customer request rather than of the network call. The general machinery is Reserve-and-Settle: Never Overspend a Credit Balance and the single-authoritative-number contract is One Authoritative Cost Per LLM Request, Across Providers.

Two consequences worth internalizing. Your per-run cost accumulator may see a different number than it expected during a failover — that is correct, and x-nr-model explains it. And if the serving route cannot be priced, the cost header is absent rather than 0, paired with x-nr-cost-status: unpriced, for the reasons in Cost Honesty: Unpriced Is Never $0 on Your LLM Bill.

Ordering the chain

A chain is ordered by a blend of three factors, in this priority:

1. Reliability first. Your most consistently available route leads, so the common case is one hop and the latency tax is zero most of the time.
2. Latency second. Among comparably reliable routes, the faster one goes earlier, because it protects the tail on the day the chain does fire.
3. Cost as a tiebreaker. When two routes are genuinely equivalent, the cheaper one leads.

The tempting mistake is putting the cheapest route first. If it is also the flakiest, you fall back constantly, you pay the extra round trip on a large share of traffic, and you have built a permanent latency regression to save fractions of a cent. Availability is what a chain buys; optimize it for that and let a deliberate cost/quality policy handle spend separately — Cost-vs-Quality LLM Routing: Which Tasks Can Go Cheap.

Keep chains short. Three hops is generous; five is a latency budget nobody agreed to. And keep the hops genuinely independent — two routes to the same provider region fail together, which makes the second hop decoration.

Edge cases we had to decide

Each is the case, the behaviour, and the reason.

1. When the chain is exhausted, the caller gets the last route's failure, not a synthesized one. Inventing a generic "service unavailable" would hide which provider failed last and make the incident harder to diagnose. You get a real error with a real x-nr-request-id, and the id is the thing to quote.

2. When a failure is ambiguous — a timeout — we advance. A timeout could mean the provider is dead or that it is slowly producing a perfectly good answer we will never see. Treating it as retryable risks a duplicate generation; treating it as fatal guarantees a failed request. We take the duplicate risk, because a request that returns is worth more than a request that is theoretically pure. For non-idempotent workloads, that is a reason to keep chains short.

3. A guardrail block never advances the chain. It is a decision, not a failure. Falling back would send the same disallowed content to another provider, which is the precise opposite of what the policy was for.

4. A budget or credit refusal never advances the chain. Both are your ceilings. Trying another route would be spending money you have already said you do not want to spend, and it would multiply the refusal rather than resolve it. The full pre-flight ordering is Credits, Budgets, Rate Limits, Guardrails: Four Pre-Flight Gates.

5. Streaming changes when failover is possible. Once the first token has been handed to your client, the response has started and the chain cannot be rewound — a mid-stream provider failure surfaces as a truncated stream, not a silent reroute. Failover protects the start of a streamed request, not its middle. Buffered requests have no such boundary.

6. A rate limit measured by the gateway does not advance the chain. If your RPM or TPM ceiling produced the 429, another provider route would not help: the ceiling is yours, not theirs. Only a provider-side capacity signal advances.

What you see from the outside

Everything is observable from the customer surface.

The operational read is simple: watch the rate at which x-nr-model differs from the model you asked for. A step change in that ratio is a provider degradation, and you will usually see it before the vendor's status page does. Wire an alert to it via Alerts, and cross-check platform state on Status.

In analytics, the same signal is a breakdown by served model over time — the secondary route's share rising is the picture of an outage. What you will not find is request or response content: bodies are never written to the audit trail, so a failover is visible as identity, cost, latency and status, never as the prompt that was in flight. A response body can live briefly in the serving cache — a few minutes, keyed to your organisation and team, so a byte-identical repeat request skips the provider — but that is a cache and not a record: it is not queryable, not exported, and it has expired long before anyone opens a postmortem. If you need the payload for a postmortem, log it in your own store deliberately — What an LLM Request Log Should Contain — and What to Leave Out.

Testing the unhappy path

An untested fallback chain is a comforting config line that may or may not work on the night you need it. Three assertions make it real, and all three are cheap:

1. It fires. Point hop 1 at a model or route you know will fail and confirm the request still returns 200, with x-nr-model naming hop 2. If it does not fire, nothing else in this post applies to you.
2. It bills once. Sum x-nr-request-cost for the test request and check the ledger for the same window. One settled cost, matching the serving route — not two, and not the price of the model you asked for. How to Read Your LLM Credit Ledger is the reconciliation guide.
3. It refuses to fire on the wrong things. Send a deliberately malformed body and confirm you get one fast 400, not a 400 per hop. Then trip a per-key budget and confirm you get 429 key_budget_exceeded immediately rather than a walk down the chain.

Run the three whenever you change the chain. A chain that has never been exercised is in the same category as a backup that has never been restored. Troubleshooting steps for each failure shape are in Troubleshooting.

Limits

Failover does not fix a bad request. If your prompt exceeds a context window or violates a policy, every route in the chain rejects it identically. The chain is for provider health, and nothing else.

Models are not interchangeable. Hop 2 is a different model with different behaviour, different formatting habits and possibly different tool-calling semantics. If your application parses output strictly, test the fallback route against your parser before the outage, not during it.

Latency is paid on every fire. A chain that fires 1% of the time is nearly free. One that fires 30% of the time is a permanent tail regression, and the correct response is to fix the primary route or reorder, not to add a fourth hop.

Mid-stream failures are not recoverable. As above: once bytes have shipped, the chain is spent for that request. Applications that must survive that need retry logic of their own, at the level where a partial answer can be discarded safely.

A chain is not multi-region high availability for your own stack. It routes around provider failure. Whatever your application does when it cannot reach anything at all is still your design problem.

Correlated failure is real. Two routes on the same underlying provider, or the same cloud region, are one route wearing two names. Independence is a property you have to choose deliberately when you build the chain — check what is actually available on Models.

Try it

Configure a two-hop chain for one model in Router Settings, then prove it works by breaking hop 1 on purpose.

curl -i https://api.nrouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $NROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5-20250929",
    "messages": [{"role":"user","content":"fallback smoke test"}]
  }'

Read x-nr-model off the response. In the healthy case it names hop 1; with hop 1 broken it names hop 2 and the status is still 200. Then check the ledger for that request id and confirm exactly one settled cost. Three minutes, and you now know the difference between having a fallback chain and believing you have one.

No account yet? Sign up — a card is required and a $5 minimum charge is taken, with a $10 first-purchase bonus for new accounts, so $5 loaded lands as $15 of credit. Questions belong in the community.

See also

• LLM Routing for AI Agents: One Model Per Step, One Key — why a mid-chain failure in an eleven-step pipeline destroys work that already cost money.
• Reserve-and-Settle: Never Overspend a Credit Balance — the credit machinery that makes one failover cost exactly one call.
• 429 vs 402 on an LLM Gateway: Which to Retry, Which to Stop — the four refusals that must never advance a chain, and how to tell them apart.
• LLM Latency: p50, p95, p99, and Time-to-First-Token — measuring the tail cost of a chain that fires, instead of guessing at it.
• Cost-vs-Quality LLM Routing: Which Tasks Can Go Cheap — the deliberate spend policy that should own cost decisions, so the chain can own availability.
• Multi-Agent Cost Tracking: Attributing Spend Across an Agent Run — reconciling a run whose cost moved because a hop changed.
• Models — which providers and routes are actually available, so your second hop is genuinely independent of your first.

Sources

Verified 2026-08-23. Request counts, failure rates, latencies and per-request costs in the worked example are illustrative arithmetic for one traffic shape, shown so the reasoning is checkable rather than quoted as a rate card. Current rates are on Models and current fees on Pricing. Corrections to hello@nrouter.ai and we will update.

• Anthropic API error reference, including 529 overloaded_error: docs.anthropic.com/en/api/errors
• OpenAI API error codes: platform.openai.com/docs/guides/error-codes
• Google SRE book, Handling Overload: sre.google/sre-book/handling-overload
• Google SRE book, Addressing Cascading Failures: sre.google/sre-book/addressing-cascading-failures
• OpenAI status page: status.openai.com
• Anthropic status page: status.anthropic.com
• nRouter router settings: nrouter.ai/docs/guides/router-settings