Router Settings
Configure intelligent routing rules, model weights, automated failovers, and caching policies in nRouter to optimize latency, cost, and reliability goals.
Last updated
Router Settings (/router-settings) creates organization-scoped Smart Router aliases. It is writable by Owner and Organization Admin roles; other members can read the configuration.
Routing strategy
Point a router alias at a set of candidate models and choose how each request resolves to one of them:
| Strategy | Behavior |
|---|---|
| Cost | Routes to the cheapest model in the set (by your list price) |
| Latency | Orders candidates by the gateway-observed latency EWMA; falls closed to priority until every candidate has a signal |
| Weighted | Uses request-ID-seeded weighted rendezvous selection across candidates with positive weights |
Each alias is per-org and resolves at request time — change the set or the strategy in the dashboard, no redeploy. A concrete model you name directly is never re-routed (routing is opt-in).
Fallback chains
The selected candidates form an addressable deployment chain. A later candidate is attempted only when retrying is known not to risk a second model bill: an upstream admission refusal (429, 503, or Anthropic 529) or a connect-phase transport failure. Other statuses and failures after a connection may already have generated billable work, so they fail loudly instead of being replayed.
Reorder by priority (drag or arrow keys) and delete chains you no longer need. Duplicate primaries are rejected.
There is one chain per Smart Router alias, not one chain per failure type. A concrete model named directly follows its direct route and does not inherit a hidden platform fallback.
Retries & timeouts
| Setting | Scope | Default |
|---|---|---|
| Maximum provider attempts | per request | 3 total attempts, including the first |
| Cumulative retry wait budget | per request | 20 seconds |
On a retry, standard Retry-After plus the millisecond provider variants are honored when they fit inside the remaining wait budget. Request-body retry/timeout overrides are not accepted.
Per-model weights
Assign each candidate a non-negative weight; at least one must be positive. Selection is deterministic for one request ID and distributable across different request IDs. Weights are relative, so 70/30 and 7/3 describe the same ratio.
Response caching
When two requests are byte-identical, nRouter can answer the second one from a short-lived serving cache instead of calling the provider again. You save the round trip. A cache hit is still metered and billed like any other request, so caching changes latency, not what you pay.
Caching is off unless it is enabled for the deployment you call. Where it is enabled, it applies to complete buffered, successfully priced responses, and your organization can switch it off for every request with the caching toggle on the Router Settings page. Streaming requests always go to the provider.
What the cache is, precisely:
- Tenant-keyed. An entry is keyed to your organization and team, alongside the model, the exact request body, and the guardrail chain that produced it. One organization's completion is never served to another, and a response produced under one guardrail configuration is never replayed under a different one.
- Short-lived. Entries expire on their own after a few minutes. It is a serving cache, not a record: it is not queryable, not exported, and not part of your request log. Prompt and completion content is never written to the log — see Observability & Logs.
Turning it off for a request
Send nrouter_cache: false in the JSON request body:
{
"model": "gpt-5.5",
"messages": [{ "role": "user", "content": "Summarize this contract." }],
"nrouter_cache": false
}That request is not answered from the cache and does not fill it — the response never becomes an entry another call could be served from. It is an nRouter control field: the gateway removes it before forwarding, so it never reaches the provider. Most SDKs pass it through an extra_body-style escape hatch; see the snippet for your language under SDKs.
The setting is per request, so where caching is enabled you can leave it on everywhere and opt individual calls out — the ones carrying regulated or user-identifying content, say — without changing anything globally. To keep every call out, switch caching off for the organization in Router Settings instead of relying on each caller to send the field.
What it costs you. Every opted-out request pays full provider latency, every time, including the repeats a cache would have absorbed. It does not change what you are billed: a hit and a miss cost the same, so the trade is latency only.
Reading the response header
Every eligible response says what happened, so you never have to infer it:
x-nr-response-cache | Meaning |
|---|---|
hit | Served from the cache. No provider call, no provider charge. |
miss | Went to the provider. Nothing eligible was cached. |
bypass | You sent nrouter_cache: false. Not read from the cache, not written to it. |
On a hit, x-nr-response-cache-age gives the age of the entry in seconds, so you can tell a two-second-old replay from a much older one. Both headers sit in the same x-nr-* namespace as x-nr-request-cost and x-nr-request-id.
Saving changes
Creating a Smart Router writes the customer-facing configuration and its gateway deployment rows in one database transaction. Duplicate aliases are rejected rather than silently replacing a live chain. The Rust gateway resolves the database configuration on each request, so no application redeploy is required.
Next steps
- Model Catalog — The models you route between
- A/B Testing — Split traffic across variants for experiments
- Analytics & Reports — Benchmark models to inform your strategy
FAQ
Who on my team can change router settings?
Router Settings is writable by Owner and Organization Admin. Members and Viewers can review but cannot mutate it. Team roles are separate and do not create organization-wide routing authority.
Do I have to change my application code to use routing?
No. Routing is opt-in: point a router alias at a set of candidate models and call that alias. A concrete model you name directly is never re-routed. You change the model set or strategy in the dashboard and it takes effect at request time — no redeploy and no SDK change.
What's the difference between the Cost, Latency, and Weighted strategies?
Cost orders candidates by their direct catalog input/output prices. Latency uses the gateway-observed latency EWMA and falls closed to priority until every candidate has data. Weighted uses deterministic request-ID-seeded weighted rendezvous selection.
What happens when a model fails mid-request?
Failover requires a configured Smart Router. On that alias, an upstream 429, 503, Anthropic 529, or connect-phase failure can advance to another eligible deployment. Generic 5xx, read timeouts, and other post-connect failures are not replayed because the provider may already have generated and billed for work. A direct model request has no hidden fallback chain. Use the shared x-nr-request-id to correlate all attempts in routing decisions and support logs.
How many times will a request retry, and can I change it?
The gateway permits 3 total attempts, including the first, and caps cumulative retry waiting at 20 seconds. These money-safety limits are not request overrides. A provider Retry-After value is honored only when it fits inside the remaining budget.
How do the per-model weights work?
Weights apply to a Smart Router alias. They are relative non-negative values, at least one must be positive, and selection is deterministically seeded by the request ID.
Does picking the Cost strategy actually lower my bill?
The Cost strategy sends each request to the cheapest eligible model by list price, so you're billed for whichever model actually runs, reported on the x-nr-request-cost response header. Router Settings only decides which model handles a request — your spend ceilings live separately under Budget Controls and are enforced independently.
Is response caching enabled, and how do I turn it off?
Not unless it is enabled for the deployment you call — caching is off otherwise. Where it is enabled, it covers eligible buffered requests; your organization can switch it off for every request in Router Settings, and a single call can skip it by putting "nrouter_cache": false in the JSON request body. That field is an nRouter control field and never reaches the provider. An opted-out call is neither answered from the cache nor added to it, and the response comes back with x-nr-response-cache: bypass so you can confirm it.
How do I tell whether a response came from the cache?
Read the x-nr-response-cache response header: hit means it was served from the cache with no provider call (it is still metered and billed like any other request), miss means it went to the provider, and bypass means the request was opted out, by nrouter_cache: false or by your organization's caching toggle. On a hit, x-nr-response-cache-age reports the entry's age in seconds. No header at all means caching is not enabled for the deployment you called.
What is actually held in the cache, and for how long?
A cache entry is a complete, successfully priced response body, keyed to your organization and team along with the model, the exact request body, and the guardrail chain that produced it — so it is only ever replayed for an identical request from the same tenant under the same guardrails. Entries are short-lived and expire on their own after a few minutes. The cache is not queryable, not exported, and separate from your request log, which never carries prompt or completion content.
What do I give up by turning caching off?
Latency and money. An opted-out request always pays the provider round trip and the provider charge, including on repeats an entry would have absorbed for free. Turn it off where you want the guarantee that a response was freshly generated, or where you would rather a body not sit in a serving cache at all, and leave it on everywhere else.
Do my changes take effect the moment I edit them?
The create action is atomic: configuration and deployment candidates either both commit or neither does. The gateway reads the saved Smart Router at request time, with no app or gateway redeploy.