Why We Built nRouter: Routing Was the Easy Half

The position

Choosing a provider and forwarding a request is the easy half of an LLM gateway. The hard half is everything wrapped around it — tenant isolation, credits that cannot go negative, guardrails that run on every call, one bill you can reconcile — and that half is what nRouter exists to own.

Every team building with AI hits the same wall around week two. The prototype works. The model is impressive. Then somebody asks how it ships to production, and a cascade of infrastructure work nobody scoped falls out of that one question.

Which provider? How do keys get issued, scoped, and revoked? What happens when the provider returns 429 for eleven minutes — a status every provider defines its own ceilings for, OpenAI and Anthropic among them? Who can see the cost, and split by which team? What stops a runaway loop from spending a quarter's budget in an afternoon? Can we put a content filter in front of all of it without touching application code?

None of those questions is about routing. Every one of them is about governance, and every team answers them again from scratch. That repetition is the thing we set out to delete. If you are new to the category, what an LLM gateway is is the shorter version of the same argument.

What we were asked for instead

The request we keep declining, at its strongest, is this: ship a thin routing library, not a service. Give us a package we can read in an afternoon. It normalises provider payloads, picks a model, retries on failure, and gets out of the way. We already have an identity system, a billing system, a logging pipeline and an on-call rotation — we will wire your router into ours.

This is a genuinely good ask, and it is not made by naive teams. It is usually made by the best-run platform groups we talk to, for four reasons that are all correct:

We take that seriously enough that we wrote a whole post on the boundary: Managed LLM Gateway vs Self-Hosted: Why We Carry the Pager. Where the library argument breaks is not on any of those four points. It breaks on the fifth thing, the one nobody puts in the ask because it does not feel like part of the router.

What that would cost you

A routing library hands you the horizontal problem — many providers, many models — and leaves you the vertical one: many teams, many keys, many budgets, a compliance review, and a bill somebody has to explain. That vertical column is where the work actually lives, and it is not a weekend of glue code. It is a permanent surface with an on-call rotation attached.

Here is the itemised bill, in the order teams usually discover it:

1. Key issuance and blast radius. One provider key shared by four services is a single revocation away from a company-wide outage, and a single leak away from an unbounded one. You end up building per-team virtual keys, which is a design problem with real trade-offs, not a config flag.
2. Spend attribution. A single provider invoice tells you what the company spent. It cannot tell you which feature, which customer, or which runaway agent spent it, so you build tag-based attribution and a place to store it.
3. Ceilings. A budget that is checked before the call and never reconciled after it is not a budget. Doing it correctly means several distinct ceilings on every request — per key, per team, per organisation, per plan — evaluated in order.
4. Failover that does not double-charge. A retry is a second call and therefore a second bill. Fallback chains that get this wrong quietly bill twice for one user-visible request.
5. Guardrails on both edges. Pre-call filtering is straightforward. Post-call filtering on a streamed response — holding output until it clears — is where homegrown implementations usually stop.
6. Isolation you can prove. Not application-layer checks that a future refactor can bypass, but isolation enforced underneath the application, provable by execution rather than by code review.

Each item is tractable alone. Together they are a product, maintained forever, while the model landscape moves underneath it. Provider prices change, new model families ship with different context windows and different billing units, and every change lands on the team that owns the glue.

What we do instead

We built the vertical column and made routing a feature of it, rather than building a router and calling the column someone else's problem. Concretely, five surfaces ship on every plan:

• Tenant scoping — organisation, team and member, with keys, budgets and guardrails resolved against the authenticated caller rather than anything the caller sends. The shape is documented in org, team, member.
• Credit safety — every call reserves before egress and settles afterwards against the real cost, with every failure path releasing what it held. The mechanism is reserve-and-settle, and it is the reason a balance cannot go negative under concurrency.
• Guardrails — organisation-level content controls applied to every request, not gated behind an enterprise tier and configurable in the guardrails guide.
• Prompt management — Server-Side Prompt Templates: Version, Roll Back, A/B Test, so a prompt change is reviewable the way a code change is.
• Observability — request logs, per-model cost analytics and alerting, described in the analytics guide.

The customer-facing consequence is one seam. Your application keeps talking to one OpenAI-compatible endpoint; adopting a different model is a string, not an integration project:

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.nrouter.ai/v1",
    api_key=os.environ["NROUTER_API_KEY"],
)

resp = client.chat.completions.with_raw_response.create(
    model="claude-sonnet-4-5-20250929",
    messages=[{"role": "user", "content": "Summarise this ticket."}],
)

print(resp.headers.get("x-nr-request-cost"))   # exact cost, absent when unpriced
print(resp.headers.get("x-nr-cost-status"))    # "exact" or "unpriced"

Which models that key reaches is a live question, not a fixed list — the current answer is on the models page, and the quick start is the ten-minute version of the snippet above.

The two bugs that shaped the billing layer

We got things wrong, and two of them changed the architecture rather than just the code.

The first was a race in the original credit check. Under concurrent load, two requests could both read the balance, both pass, and both proceed before either one wrote to the ledger. The window was small — a burst of parallel calls on a single key — but "small window" is not a property you want in the sentence "cannot overspend." The fix was structural: hold the estimated cost before the provider call, settle against the authoritative cost afterwards, release on every failure path, and make the ledger the only writer of record. An unknown cost settles at the amount reserved rather than releasing, because a released reservation is a free request, and free requests are how a gateway leaks money while every dashboard reads green.

The second was ambition about organisations. The early design let a user belong to several organisations and switch between them. That produced a permanent ambiguity: which organisation owns the key you just created, which budget applies to the call in flight, which guardrail set runs? Collapsing to exactly one organisation and one team per user deleted an entire category of bug rather than fixing instances of it. It costs us the multi-org power user, and we took that trade deliberately.

Both fixes have the same shape: prefer the design where the wrong state is unrepresentable over the design where the wrong state is merely handled.

Why the platform fee sits on top and never becomes a feature wall

The commercial decision follows from the same position. If governance is the product, gating governance behind a tier would mean selling the product to some customers and withholding it from the rest. So the fee moves with scale and the feature set does not move at all.

Concretely, one dial moves and the feature list does not: the platform fee on pay as you go is added on top at purchase and is a flat 4% of the credits — the credits you buy land in full — and it is 0% on Pro and Enterprise, with the identical governance surface on all three — the pricing page carries the table.

The arithmetic is public and does not need a call: because the fee is 4% of the credits, it comes to 4% of your monthly provider spend, so Pro pays for itself once that passes $50 — around $1,250/mo of provider spend on the monthly plan and around $1,042/mo on the annual one. Below that, pay as you go is cheaper and we would rather say so than sell you a subscription. The full reasoning is in Every Feature on Every Plan: We Charge a Fee, Not a Gate, and the crossover is worked through in from credits to Pro.

We do not run a free tier, and the honest version of the first mile is that a new account puts a card down for a real $5 minimum purchase with the fee added on top of it. We would rather write that sentence than a friendlier one that is not true.

The trade we are making

Three costs land on you because of how we built this, and they are real.

You do not bring your own provider keys. nRouter holds the provider relationship; you hold an nRouter key and a credit balance. That means you cannot apply a committed-spend discount you negotiated directly with a provider, and you cannot point us at your own account. The full reasoning is in why we do not do BYOK, and the honest summary is that it buys us a coherent billing and revocation story at the cost of your existing contracts.

We are in your request path. A library is not. Our availability becomes part of your availability, which is why the status page is public and why fallback behaviour is documented rather than implied.

One organisation and one team per user. Consultants and contractors working across several customers feel this immediately. It is the price of never having to ask which budget applied to a call.

Who this is wrong for

Some teams should not buy this, and naming them is cheaper for everyone than discovering it in month three.

• Air-gapped or self-host-mandated deployments. If your compliance posture forbids a third party in the request path, no amount of feature parity fixes that. Read Managed LLM Gateway vs Self-Hosted: Why We Carry the Pager and then pick a self-hosted gateway.
• Teams with a large committed-spend contract at one provider. If you have negotiated pricing you cannot bring with you, our fee sits on top of a rate you already beat. Do the arithmetic before you move.
• Single-provider, single-team, single-service shops. If one key serves one service and nobody needs attribution, the vertical column we built is overhead. A direct SDK call is genuinely the right answer.
• Anyone who needs a free tier to evaluate. We do not have one, and we are not going to pretend the $5 minimum is one.

How you can hold us to it

A position is only worth publishing if you can check it without asking us.

1. Read the cost header, not our dashboard. Every response carries x-nr-request-cost when the cost is known and omits it when it is not, paired with x-nr-cost-status. An unknown cost is reported as unpriced, never as $0 — the argument for that is cost honesty.
2. Reconcile the ledger yourself. Every credit movement is a ledger entry, and the entries sum to the balance. Reading a spend ledger shows how, and the billing guide shows where.
3. Check the pricing page against this post. If /pricing and the table above ever disagree, the pricing page is right and this post is a defect. Tell us.
4. Test the feature claim on the cheapest plan. Create a guardrail and a per-team budget on pay as you go. If either one asks you to upgrade, the position in this post is false.
5. Look at what we refuse to fake. We ship no demo mode and no mock data, for the reasons in why we ban mocks and demo mode, and our compliance status is stated plainly on the trust page — SOC 2 Type II is in progress, not certified.

Try it

Create an account, load the $5 minimum, and point an existing OpenAI-compatible client at https://api.nrouter.ai/v1 with NROUTER_API_KEY. The only edit is a base URL and a key. Start at signup, follow the quick start, then create a second virtual key with a per-team budget and watch the ceiling bite — that is the half of the product this whole post is about.

We are also hiring engineers who want to work on this layer. Write to careers@nrouter.ai.

See also

• Managed LLM Gateway vs Self-Hosted: Why We Carry the Pager — the boundary this post skirts, argued against the strongest self-host case.
• Every Feature on Every Plan: We Charge a Fee, Not a Gate — why the fee moves with scale and the feature set never does.
• Why we don't do BYOK — the trade named above, worked through with the revocation and billing consequences.
• Reserve-and-Settle: Never Overspend a Credit Balance — the mechanism behind the first bug in this post.
• Org, team, member: scoping keys, budgets, guardrails — what one organisation per user buys and costs.
• What is an LLM gateway? A 2026 primer — the category definition if this post assumed too much.
• Pricing — the live fee, the $5 minimum, and the Pro crossover, in their authoritative form.

Sources

Verified 2026-08-23. Provider prices move; if any link below has changed, email hello@nrouter.ai and we will correct this post.

• nRouter plans and fees: nrouter.ai/pricing — the $0 subscription with a 4% platform fee, Pro at $50/mo or $500/yr with 0%, and the $5 minimum purchase.
• OpenAI API pricing: openai.com/api/pricing — the moving-price problem described in "what that would cost you".
• Anthropic pricing: anthropic.com/pricing — same, for the Claude model family.
• AWS Bedrock pricing: aws.amazon.com/bedrock/pricing — Bedrock is live on nRouter, and its per-model rates are published by AWS.
• OpenAI API reference: platform.openai.com/docs/api-reference — the request and response shape the https://api.nrouter.ai/v1 endpoint is compatible with.
• RFC 6585 §4, 429 Too Many Requests: datatracker.ietf.org — the status in the opening question, and what it does and does not mean.
• OpenAI rate limits: platform.openai.com/docs/guides/rate-limits — one provider's published ceilings.
• Anthropic rate limits: docs.anthropic.com/en/api/rate-limits — another's, shaped differently, which is half the reason the ceiling problem does not stay solved.

OpenAI, Anthropic and AWS are trademarks of their respective owners. nRouter is not affiliated with or endorsed by them. All claims above are sourced from their public pricing or documentation on the date shown.