Your first $5 becomes $15Get started
Routing

One API. 240+ models.

Call Alibaba US (Qwen, DeepSeek — our largest catalog), OpenAI (GPT, o-series), AWS Bedrock (Claude, Llama, DeepSeek, Nova, Qwen), Azure Foundry, Google Vertex AI (Gemini), and Anthropic (Claude) live today through a single endpoint. Cost, latency, and weighted routing, deterministic A/B splits, configurable fallback chains — config, not redeploy.

Get startedSee the pipeline
routing-decision · request_id 7f2a

Live routing trace

Strategycost
Routed byx-nemo-routed-by
Primaryvertex/gemini-2.5-flash
Fallback chainanthropic → bedrock
Retries used0
Routing overhead< 1 ms
cost-routedfallback-readyconfig-not-redeploy
Routing strategies
3

Cost, latency, weighted, per org

Retries
Configurable

Per-org retry count + per-request override

A/B splits
Deterministic

Model + prompt variants, retry-safe

Fallback chains
Opt-in

On error + on rate limit, cross-provider

Strategies

Three strategies. One config field.

Pick the signal your workload cares about — cost, latency, or your own weighted split. The router does the rest, per org, no redeploy.

costprice per token

Picks the cheapest capable model in your candidate set. Optimize the bill, not the benchmark.

latencyrecent p95

Steers each request to the model with the lowest recent p95 latency for your org.

weightedyour split

Call nemo-auto and we split traffic across a basket of models by the weights you set.

How it works

A routing decision, end to end

Strategy match, primary provider, fallback chain, settle. A substitution is surfaced on the x-nemo-routed-by response header — routing itself adds < 1 ms.

Routing decision flow

  1. Request

    POST /v1/chat/completions

    Bearer sk-nemo-..., model="gemini-2.5-flash"

  2. Strategy match

    cost · candidate set

    Alias resolves to the cheapest capable model in your set.

  3. Primary provider

    Vertex AI · gemini-2.5-flash

    5xx / timeout triggers the fallback chain.

  4. Fallback chain

    Anthropic → Bedrock → OpenAI

    Ordered backup list per model; each link retried in turn.

  5. Settled

    cost + latency logged

    Strategy, primary, fallback, retries: all in the request log.

Drop-in

Two-line migration from OpenAI

OpenAI-compatible

Change the base URL and the key. That’s it

The unmodified OpenAI SDK works against nRouter on every supported language. Streaming, tool use, JSON mode, embeddings, image generation — all proxied transparently.

  • Python, Node.js, Go, Java, Ruby, PHP, cURL
  • Base URL: https://api.nrouter.ai/v1
  • Auth: Bearer sk-nemo-...
  • Drop-in for chat.completions, embeddings, and images
diff · client.py

Two-line change

- base_urlhttps://api.openai.com/v1
+ base_urlhttps://api.nrouter.ai/v1
- api_keysk-proj-...
+ api_keysk-nemo-...
Code lines changed2
OpenAI SDK7 languagesstreaming-safe
Full routing reference — strategies, fallbacks, retries, A/B
Fallback chains
Ordered backup list per model — on error and on rate limit. A 5xx or timeout triggers the next link, cross-provider (Vertex → Anthropic → Bedrock). Opt-in per org; the chain is configured, not coded.
Retries + timeouts
Per-org retry count (default 3) and request timeout, with a per-request override. Provider retry-after hints are honored on the retry.
Deterministic A/B splits
Split a model or prompt into variants and bucket traffic deterministically. The same request_id always lands on the same variant, so retries stay consistent and results are attributable.
Never depend on one model
A missing, unknown, or retired model resolves to your default (a weighted basket, your org default, or a platform default) instead of failing. The call still succeeds.
OpenAI-compatible surface
Same chat.completions, embeddings, and images endpoints. Base URL https://api.nrouter.ai/v1, auth Bearer sk-nemo-... Streaming, tool use, and JSON mode proxied transparently.
Live model catalog
Capabilities and fallback chains live in the catalog, not your repo. New models are added within hours of provider launch — public read-only catalog at /api/public/models.
FAQ

Common routing questions

What latency does routing add?

Routing decisions happen in-memory and add <1ms. The dominant factor is always LLM inference time, not our proxy overhead.

How many models are supported?

240+ models live now on Alibaba US, OpenAI, AWS Bedrock, Azure Foundry, Google Vertex AI & Anthropic; Alibaba US (Qwen, DeepSeek — our largest catalog), OpenAI (GPT, o-series), AWS Bedrock (Claude, Llama, DeepSeek, Nova, Qwen), Azure Foundry, Google Vertex AI (Gemini), and Anthropic (Claude) live today. We add new models within hours of provider launch. Check the model catalog for the full list.

Can I use custom routing strategies?

Yes. Configure per-org routing in your dashboard: cost (cheapest capable model), latency (lowest recent p95), or weighted (your own traffic split via nemo-auto). Point an alias at a candidate set and the router resolves it per request.

What happens when a provider goes down?

nRouter automatically retries with the configured fallback chain. If the primary model times out or returns an error, we transparently retry on a backup model. Your users never see the error.

One key. One bill. Every model.

Replace your homegrown failover layer in an afternoon

Sign up, paste your virtual key, change the base URL. Cost / latency / weighted routing, A/B splits, and fallback chains are unlocked on every plan.

Get startedBrowse models