LLM Routing for AI Agents: One Model Per Step, One Key

The answer: an agent run is a pipeline of unlike calls, so route it per step rather than per application. Send the model name you want on each call to one OpenAI-compatible endpoint, let the gateway resolve the provider, the fallback chain, the rate limit and the budget from the key, and read one settled cost per call off x-nr-request-cost. The routing decision moves into config; the agent code stops caring which provider is up.

A single-turn chat app has one model decision and you make it once. An agent has one per step, the steps are not alike, and the number of steps is decided at runtime by the model itself. That is the whole difficulty. A planner call is short and needs to reason. A tool-selection call is tiny and needs to emit correct JSON fast. A synthesis call carries every tool result the run accumulated and needs quality across a long context. Those three workloads have nothing in common except that they are all "an LLM call", and treating them as one thing is the most expensive default in agent infrastructure.

This post walks one concrete agent run end to end: where the naive version breaks, what the per-step routing contract actually is from the caller's side, the arithmetic on a real-shaped run, the failure cases we had to pick a side on, and what all of it looks like from outside the gateway.

The user request that becomes eleven calls

Here is the request, as the user sees it: "Summarize this quarter's support tickets and draft a note we can send to affected customers."

Here is the same request as your infrastructure sees it:

user request
  │
  ├─ 1  plan            decide which tools to call, in what order
  │
  ├─ 2..7  tool loop    6 short calls: pick a tool, read its result,
  │                     decide whether to continue
  │
  ├─ 8..9  summarize    2 calls over retrieved ticket batches
  │
  ├─ 10 synthesize      1 long-context call: every result → the draft note
  │
  └─ 11 critique        1 short call: does the draft meet the brief?

Eleven calls. Nothing about that fan-out is exotic — it is the ordinary shape of a ReAct-style loop, the pattern described in Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models and now baked into every mainstream agent framework. What makes it hard is that the eleven are not eleven copies of one thing. They differ by an order of magnitude in context length, by two orders of magnitude in cost, and by which capability actually matters.

And the count is not fixed. Ask a harder question and the tool loop runs twelve times instead of six. That is the point of an agent, and it is also why a per-application model choice cannot be right: you are choosing one model for a workload whose shape is decided after you ship.

Why one hardcoded model breaks the pipeline

The version most teams build first is one client, one model constant, every step. It is defensible on day one — it is the smallest thing that works, and the model is good at everything, so nothing is obviously wrong.

Three things go wrong, in this order.

The bill is dominated by calls that did not need the model. More than half the calls in the run above are short tool-selection turns. They are asking a frontier model to choose between four functions and emit twenty tokens of JSON. You pay flagship input and output rates for a decision a small model makes correctly and considerably faster.

The tail latency is the sum of every step. In a single-turn app, p95 is one model's p95. In an eleven-step pipeline, the user waits for all eleven serially wherever the steps depend on each other, so the run's p95 compounds. Shaving 400 ms off six tool calls is worth more to perceived speed than any prompt tuning on the synthesis call, and you cannot do it while everything shares one model. Reading that number honestly is its own discipline — see LLM Latency: p50, p95, p99, and Time-to-First-Token.

The provider becomes a single point of failure for the whole run. Every provider publishes its own ceilings and its own throttling behaviour — OpenAI's rate limits, Anthropic's, Bedrock's service quotas — and a direct integration inherits whichever one it happens to be pinned to. If step 8 of 11 gets a 429 and your code raises, you have burned the cost of steps 1 through 7 and produced nothing. Single-turn apps retry cheaply. Agent pipelines do not: a failure late in a long chain destroys accumulated work that already cost money.

The obvious next move — a MODEL_BY_STEP dict in the agent — helps with the first problem and not the other two, and it introduces a new one. The moment routing lives in application code, every service that runs an agent carries its own copy, they drift, and changing a model becomes a deploy in n repositories rather than a config change in one place.

The mechanism: model per call, policy resolved from the key

The contract is deliberately small, because a small contract is what lets the routing decision move out of your code without moving your code.

1. One base URL, one key. Every call in the run goes to https://api.nrouter.ai/v1 with NROUTER_API_KEY. There is no per-provider client, no per-provider SDK, no provider credential in your environment — see No BYOK: One nRouter Key for why that boundary is where it is.
2. The model field selects the workload, per call. It is a plain string on the request body, so a per-step choice is a per-step value, not a per-process configuration.
3. Everything else is resolved from the key, not the body. Which provider serves that model, what the fallback chain is, which rate limits apply, which budgets apply, which guardrails apply — all of it is decided from the authenticated key. A caller cannot widen its own limits by sending a header, which is exactly the property you want when the caller is a model deciding its own next step.
4. The response reports what happened. x-nr-model names the model that actually served the request, x-nr-request-cost carries the settled USD cost, and x-nr-request-id identifies the call for support and correlation.

In practice that is one client and a table:

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["NROUTER_API_KEY"],
    base_url="https://api.nrouter.ai/v1",
)

# The routing table. One place, not scattered through the agent.
STEP_MODEL = {
    "plan":       "claude-opus-4-5-20251101",
    "tool_select": "gpt-5.4-mini",
    "summarize":  "gpt-5.4-mini",
    "synthesize": "claude-sonnet-4-5-20250929",
    "critique":   "claude-haiku-4-5-20251001",
}

def step(name: str, messages: list, run_id: str):
    resp = client.chat.completions.with_raw_response.create(
        model=STEP_MODEL[name],
        messages=messages,
        user=f"run:{run_id}",           # attribution, not authorization
    )
    headers = resp.headers
    return resp.parse(), {
        "step": name,
        "routed_model": headers.get("x-nr-model"),
        "cost": headers.get("x-nr-request-cost"),      # absent when unpriced
        "cost_status": headers.get("x-nr-cost-status"),
        "request_id": headers.get("x-nr-request-id"),
    }

Two details in that snippet carry most of the value. with_raw_response is how you get at the headers at all — the parsed object alone does not expose them, and every cost and routing fact you want lives there. And user is attribution only: it groups spend in analytics, it does not grant or restrict anything. The full treatment is in LLM Cost Attribution: Keys, Teams, and the user Field.

Changing the routing table is a config edit. Adding a model that launched this morning is a string. Neither is a code change to the agent, which is the entire reason for putting the seam here.

Worked example: an eleven-call research run

Numbers make the argument better than adjectives. Below is the run from the top of this post with plausible token volumes for each step and three model classes — flagship, mid and small. The per-million rates are illustrative and are there so the arithmetic is checkable; substitute the current numbers from Models before you plan a budget on them.

illustrative rates, per 1M tokens
  flagship   $3.00 in / $15.00 out
  mid        $0.80 in /  $4.00 out
  small      $0.15 in /  $0.60 out

Two facts fall straight out of that table, and neither is visible without per-step attribution.

The six calls that are more than half the run are two percent of its cost. Optimizing them is not where the money is — but they are where the latency is, so they are worth a fast model rather than a cheap one, and those happen to be the same choice.

One call in eleven is half the bill. The synthesis step is 12,000 input tokens at flagship rates. If you want the run cheaper, that is the only line that matters, and the question is whether the draft survives a mid-tier model — which is an A/B test, not an opinion. Hash-Based A/B Tests: Same User, Same Model Variant, Every Call is how to run it without the assignment drifting mid-experiment.

Now the counterfactual. Route every step to the flagship, same token volumes:

all-flagship    60,600 in × $3.00/M   = $0.1818
                 6,700 out × $15.00/M  = $0.1005
                                       ---------
                                         $0.2823  per run

per-step routing                         $0.1129  per run
                                       ---------
difference                               $0.1694  per run   (2.5×)

at 10,000 runs / month
  all-flagship        $2,823
  per-step routing    $1,129
  saved               $1,694 / month

Two and a half times, on the same eleven calls, producing the same artifact, with no change to the agent's logic. That is the size of the prize, and it is entirely a routing-table decision. The general version of the trade-off — which route is worth spending quality on and which is not — is in Cost-vs-Quality LLM Routing: Which Tasks Can Go Cheap.

A key per agent role, and what it buys

The routing table tells you which model each step uses. It does not tell you what the researcher costs versus the writer, and in a multi-agent system that is usually the question being asked. Issue one key per agent role and the answer becomes a grouping instead of an investigation:

Each key carries its own spend, rate limits and budget, and — importantly — its own blast radius: revoking the researcher key stops research and leaves everything else running. That separation, and why the master key never does inference work, is the subject of Virtual Keys vs Master Key: Scoping a Key Per Job. The org/team/key scoping the budgets hang off is in Org, Team, Member: Scoping Keys, Budgets, Guardrails.

Per-role keys and the user field are complementary, not alternatives. The key answers "which role costs what"; user answers "what did this run cost". You want both, and the per-run half is the subject of Multi-Agent Cost Tracking: Attributing Spend Across an Agent Run.

Failure mid-chain: what retries, what falls back, what stops

An agent run fails differently from a single request, because a failure at step 9 has already spent the cost of steps 1 through 8. The taxonomy that matters:

The row people get wrong is the second one. 429 Too Many Requests is defined by RFC 6585 §4 as "the user has sent too many requests in a given amount of time", which is a rate statement — so every naive retry wrapper treats it as "back off and try again". A per-key budget block also arrives as 429, and backing off does not refill a spend cap, so an exhausted agent key produces a polite infinite loop rather than an error. Branch on the code in the body, never on the status alone. The full decision table is in 429 vs 402 on an LLM Gateway: Which to Retry, Which to Stop, and the four ceilings a request passes before it ever reaches a provider are in Credits, Budgets, Rate Limits, Guardrails: Four Pre-Flight Gates.

Provider-side failures are the ones you do not handle. When a route returns a retryable failure, the gateway advances the fallback chain for that model and serves the step from a healthy route, and your agent sees a 200 with a possibly different x-nr-model. Crucially, the credit reservation is taken once per customer request and settled against the route that actually served it, so a failover does not bill you twice for one step. The mechanics are in Provider Fallback Chains: Surviving an OpenAI Outage.

What is left for your code is checkpointing. Fallback protects a step; it does not protect a run. If steps 1 through 8 produced state worth keeping, persist it keyed by run id so a hard failure at step 9 resumes rather than restarts. That is your storage decision, not the gateway's, and it is the single highest-value thing to build once a pipeline goes past three or four steps.

Edge cases we had to decide

These are the calls with no obviously correct answer. Each is the case, the behaviour, and the reason.

1. When a step falls back to a different route, the cost settled is the serving route's cost, not the requested one. The alternative — quoting the price of the model you asked for — would be a fiction, and it would silently mis-price every run during an outage. You are billed for what actually ran, and x-nr-model tells you what that was, so a run whose cost moved has a visible reason.

2. When a rate limit is hit mid-loop, the gateway does not queue the request. It returns 429 immediately with the scope that produced it. Silently holding an agent's call would turn a throttle into unbounded latency inside a pipeline that may already be at a user-facing timeout, and an agent that cannot tell "slow" from "blocked" makes worse decisions than one that gets a fast error. Back-off belongs in your loop, where it can see the whole run — and it should be exponential with jitter rather than a fixed sleep, for the reasons set out in the AWS Builders' Library's Timeouts, retries, and backoff with jitter.

3. When a call cannot be priced, the cost header is absent — never 0. x-nr-cost-status says unpriced and x-nr-request-cost is simply not sent. A zero would be a claim that a step was free, and an accumulator that sums a fabricated zero produces a run total that is wrong in the direction that looks good. Branch on the header's absence. The reasoning is Cost Honesty: Unpriced Is Never $0 on Your LLM Bill.

4. The user field attributes; it never authorizes. Sending user: "run:abc" groups spend for analytics. It does not create a budget, does not raise a limit, and cannot be used to select a route — because it is caller-supplied and a caller-supplied value must never move an enforcement decision. Enforcement hangs off the authenticated key.

5. A budget block stops the step, not just the call. There is no partial state where an agent gets a refusal and the gateway still forwards. Nothing is sent upstream, nothing is charged, and the run is free to decide whether to degrade or fail. What the agent must not do is treat the refusal as a transient error, which is the loop described above.

What you see from the outside

Everything the run did is observable through the customer surface, without instrumenting anything beyond reading headers.

Per call, on the response:

Per run and per role, in the dashboard: spend by key, by model and by user value, which is how "the researcher role is 70% of agent spend this week" becomes a chart rather than an investigation. Analytics covers the views; How to Read Your LLM Credit Ledger covers reconciling them against settled credit movements.

In your own logs: whatever you choose to keep. Request and response content is not stored by the gateway today, so if you want the prompts and completions of a failed run for debugging, that is a decision you make deliberately in your own store — with the retention and redaction consequences described in What an LLM Request Log Should Contain — and What to Leave Out.

Limits

The honest boundaries, because a routing story without them is a brochure.

Per-step routing does not make a bad step good. If synthesis fails on a mid-tier model, routing it there saves money and ships a worse product. The routing table is a place to express a quality decision, not a substitute for making one.

Fallback is availability, not free insurance. Each hop costs a round trip, so a chain that fires often is quietly widening your tail latency. Order chains by reliability first and keep them short.

Cross-step context is your problem. The gateway sees eleven independent requests, not one run. It cannot trim your accumulated history, cannot decide when a tool loop should terminate, and cannot tell you that step 7 asked the same question as step 4. Those are agent-design problems, and no gateway solves them.

A budget is a ceiling, not a plan. Budgets stop a runaway agent; they do not tell you what a run should cost. Get the per-step numbers first, then set the ceiling somewhere sensible above them — How to Set Hard Spend Limits on Your LLM Gateway walks that ordering.

Framework integration is real but not magical. LangChain, CrewAI, AutoGen and the Vercel AI SDK all accept a custom base URL and API key, and that is all the integration needed — see Frameworks. What no framework does for you is decide which step deserves which model.

Try it

Take the agent you already have and change two lines: point the client at https://api.nrouter.ai/v1 and set NROUTER_API_KEY. Nothing else in the pipeline changes, because the contract is OpenAI-compatible.

curl -i https://api.nrouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $NROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.4-mini",
    "messages": [{"role":"user","content":"Pick a tool: search, calculator, or none."}],
    "user": "run:smoke-test"
  }'

Read x-nr-model, x-nr-request-cost and x-nr-request-id off the response. Then run one real pipeline end to end, log those three per step, and sort by cost — the step that dominates will surprise you, and it is the only one worth optimizing first. Put a ceiling under the whole thing with budget controls before you let an autonomous loop run unattended, and configure the fallback chain in Router Settings.

No account yet? Sign up — a card is required and a $5 minimum charge is taken, with the platform fee on top. Questions belong in the community.

See also

• Multi-Agent Cost Tracking: Attributing Spend Across an Agent Run — the per-run half of this post, including what to do when one user action fans out into fifty calls.
• Provider Fallback Chains: Surviving an OpenAI Outage — what happens to step 8 of 11 when its provider goes down, and why the failover is billed once.
• 429 vs 402 on an LLM Gateway: Which to Retry, Which to Stop — the decision table that stops an agent retrying a budget block forever.
• Cost-vs-Quality LLM Routing: Which Tasks Can Go Cheap — how to decide which steps deserve the flagship, generalized past agents.
• RPM and TPM Rate Limiting Per Key, Team, and Org — the ceilings a tool loop hits first, and why they are not negotiable from the client.
• nRouter for Agentic Workflows: One Gateway, Any Model — the product view of the same pattern, for a reader who is evaluating rather than building.
• Models — the current catalog and rates, so your routing table names real options and your arithmetic uses real numbers.

Sources

Verified 2026-08-23. The token volumes, per-million rates and run costs in the worked example are illustrative arithmetic for one pipeline shape — they are shown so the reasoning is checkable, not quoted as a rate card. Current rates are on Models and current fees on Pricing. Corrections to hello@nrouter.ai and we will update.

• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022): arxiv.org/abs/2210.03629
• OpenAI chat completions API reference, including the user field: platform.openai.com/docs/api-reference/chat
• Anthropic Messages API reference: docs.anthropic.com/en/api/messages
• LangChain custom OpenAI-compatible endpoints: python.langchain.com
• nRouter framework integrations: nrouter.ai/docs/frameworks
• nRouter router settings: nrouter.ai/docs/guides/router-settings
• RFC 6585 §4, the 429 Too Many Requests status code: datatracker.ietf.org — the definition the retry taxonomy above is written against.
• OpenAI rate limits: platform.openai.com/docs/guides/rate-limits
• Anthropic rate limits: docs.anthropic.com/en/api/rate-limits
• Amazon Bedrock quotas: docs.aws.amazon.com
• AWS Builders' Library, Timeouts, retries, and backoff with jitter: aws.amazon.com/builders-library — how the back-off in your loop should be shaped.

OpenAI, Anthropic, AWS and LangChain are trademarks of their respective owners. nRouter is not affiliated with or endorsed by them.