Auto-failover for long runs
Agents fan out dozens of calls per task. A provider hiccup at step 11 triggers the next fallback link transparently. Step 12 never knows, and the failover is logged.
Use Case · AI Agents
Agents that stay up and stay in budget.
An autonomous agent fans out dozens of LLM calls per task. NemoRouter is the agent control plane. It keeps a long run alive when a provider degrades, caps each agent with a per-key budget, and logs every tool call for audit.
agent-run · key sk-nemo-7f2a
One agent, one key, one budget
Steps this run24 calls
Fallback usedstep 11
Run statuscompleted
Budget$2.40 / $5.00
Tool calls logged24 / 24
Runaway protection402 ceiling
per-agent keyfailover-safeaudited
- Long runs
- Stay up
- Per-agent spend
- Budgeted
- Tool calls
- 100% logged
- Catalog
- 240+
Fallback chains survive a provider blip
A 402 stops a runaway loop
Every step in the request log
models on Alibaba US, OpenAI, AWS Bedrock, Azure Foundry, Google Vertex AI & Anthropic
Why Nemo for agents
Reliability, spend control, and a trail
Agents are unpredictable by design. The gateway makes them dependable: a run that survives a provider blip, a budget it cannot exceed, and a log of everything it did.
Per-agent budgets
One virtual key per agent with its own budget. A runaway reasoning loop hits a hard 402 ceiling instead of burning credits; per-key RPM/TPM throttles a misbehaving agent.
Every tool call audited
Planning, tool selection, reflection: every LLM call lands in the request log with model, latency, cost, and result. Filter by the agent’s key to replay a whole run.
Model choice per step
Strong reasoning model for planning, fast model for tool-argument extraction. Pick per call from one catalog, no provider account, no SDK swap.
How it works
An agent run, end to end
Each agent carries its own virtual key. Every planning and tool-selection step is a separate LLM call — routed, failover-protected, budget-checked, and logged.
Agent run flow
Agent key
sk-nemo-... · per agent
One virtual key per agent — its own budget and rate limit.
Plan + tool calls
many /chat/completions
Each reasoning + tool-selection step is a separate call.
Routing + failover
fallback chain
A degraded provider triggers the next link — the run survives.
Budget check
reserve + settle
Out of budget → 402, before the call. No runaway spend.
Logged per step
request log
Model, latency, cost, result — replay the whole run.
The budget check is a reserve-then-settle: credits are reserved before the call and settled at the real cost after. If an agent is out of budget, it gets a 402 — never a surprise overspend.
The code
Point your agent framework at one endpoint
NemoRouter speaks the OpenAI API, so any agent framework that targets the OpenAI SDK works with a base-URL and key change. These snippets are generated from the same SDK examples the playground uses. Give each agent its own key and the per-key budget does the rest.
Install
pip install openai| 1 | # Cache: enabled (org default). Pass nemo_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 | |
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NEMOROUTER_API_KEY"], |
| 7 | base_url="https://api.nemorouter.ai/v1", |
| 8 | ) |
| 9 | |
| 10 | response = client.chat.completions.create( |
| 11 | model="gemini-2.5-flash-lite", |
| 12 | temperature=1, |
| 13 | max_tokens=1024, |
| 14 | messages=[ |
| 15 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 16 | ], |
| 17 | extra_body={ |
| 18 | # "nemo_cache": False, # Uncomment to skip cache |
| 19 | }, |
| 20 | ) |
| 21 | |
| 22 | print(response.choices[0].message.content) |
Issue one virtual key per agent — spend, rate limits, and the request log all scope to that key.
FAQ
Common agent questions
Why do AI agents need a gateway?
An agent fans out many LLM calls per task: planning, tool selection, reflection, synthesis. A single provider hiccup can break a long run. A gateway adds auto-failover, spend caps, and a unified request log so an agent run is reliable, bounded, and auditable.
Can I cap how much an agent can spend?
Yes. Issue each agent its own virtual key and set a per-key budget. When the agent exhausts its budget, the gateway returns a 402 instead of silently burning credits. That is a hard ceiling on a runaway loop.
How do I audit what an agent did?
Every LLM call an agent makes lands in the request log with the model, prompt metadata, latency, cost, and result. Filter by the agent’s virtual key to replay an entire run, or export logs to Langfuse, Datadog, or S3 via a logging callback.
Does the OpenAI Agents SDK or LangChain work with NemoRouter?
Yes. NemoRouter exposes an OpenAI-compatible API, so any agent framework that targets the OpenAI SDK works by changing the base URL and key. Tool use, streaming, and JSON mode are all proxied transparently.
AI is becoming autonomous; control must become automatic.
Run agents on a gateway built for long runs
Auto-failover, per-agent budgets, and a full tool-call log — unlocked on every plan, behind one NemoRouter key.