Pick the model per task
Fast model for inline autocomplete, strong reasoning model for refactors and test generation. Set the model per request from one catalog, no SDK swap.
Use Case · Code Generation
Coding assistants with model choice and per-seat budgets.
A coding tool mixes fast autocomplete with deep reasoning. NemoRouter lets you pick the model per task from one catalog, route latency-sensitive calls to the quickest endpoint, and cap spend per developer seat.
code-assistant · seat key
One catalog, the right model per task
Inline completiongemini-2.5-flash
Refactor / testsgemini-2.5-pro
Route strategylatency
Streamingtoken-by-token
Seat budget$31 / $80
Provider confignone
model choicelow-latencyper-seat budget
- Model catalog
- 240+
- Gateway overhead
- 95 ms
- Per-seat spend
- Budgeted
- Every call
- Logged
models on Alibaba US, OpenAI, AWS Bedrock, Azure Foundry, Google Vertex AI & Anthropic
p50 — LLM inference dominates
One key per developer, capped
Model, latency, and cost per request
Why Nemo for code
The right model, fast, and within budget
A coding assistant has two competing needs (speed for completion, depth for reasoning) and cost that scales with every seat. NemoRouter handles all three behind one key.
Low-latency completion
A developer is waiting on every keystroke. Latency-based routing steers each request to the model with the lowest recent p95 for your org; streaming is proxied token-by-token.
Per-seat budgets
One virtual key per developer with a hard 402 ceiling. Cost-per-seat is a number, not a guess, and one seat can never hog capacity (per-key RPM/TPM).
Every generation logged
When a suggestion is wrong or slow, filter the request log by the developer’s key: model, latency, tokens, and real cost per call. A/B test two models on real traffic.
How it works
An editor request, end to end
Each developer carries a seat key. Completion and reasoning requests route to the model the task needs, stream back token-by-token, and land in the log attributed to that seat.
Code-assistant request flow
Editor request
completion or chat
A keystroke completion or a refactor prompt from the IDE.
Seat key
sk-nemo-... · per developer
Budget and rate limit scope to the developer’s key.
Model + latency route
catalog · latency
Fast model for completion, strong model for reasoning.
Stream back
token-by-token
Streaming proxied transparently to the editor.
Logged per seat
request log
Model, latency, tokens, cost — attributed to the seat.
The gateway adds about 95 ms at p50 — LLM inference is the dominant latency factor. Streaming is proxied with no hot-path buffering.
The code
Set the model per request
A coding assistant just sets the model field per call — fast for completion, strong for reasoning. These snippets come from the same SDK examples the playground uses; change the model string and the catalog does the rest.
Install
pip install openai| 1 | # Cache: enabled (org default). Pass nemo_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 | |
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NEMOROUTER_API_KEY"], |
| 7 | base_url="https://api.nemorouter.ai/v1", |
| 8 | ) |
| 9 | |
| 10 | response = client.chat.completions.create( |
| 11 | model="gemini-2.5-flash-lite", |
| 12 | temperature=1, |
| 13 | max_tokens=1024, |
| 14 | messages=[ |
| 15 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 16 | ], |
| 17 | extra_body={ |
| 18 | # "nemo_cache": False, # Uncomment to skip cache |
| 19 | }, |
| 20 | ) |
| 21 | |
| 22 | print(response.choices[0].message.content) |
One key reaches every model in the catalog — no per-model provider account to manage.
FAQ
Common code-assistant questions
Can I use different models for autocomplete versus deep code reasoning?
Yes. The model catalog exposes every model behind one key. Set the model per request (a fast model for inline completion, a stronger reasoning model for refactors or test generation) with no separate provider account or SDK swap.
How does NemoRouter keep code completion latency low?
Routing decisions happen in-memory and add under a millisecond. LLM inference time dominates. Latency-based routing steers each request to the model with the lowest recent p95 latency for your org.
Can I budget LLM spend per developer seat?
Yes. Issue one virtual key per developer or per team and set a per-key budget. Spend, rate limits, and the request log all scope to that key, so you can see and cap cost per seat.
Does streaming work for token-by-token completion?
Yes. NemoRouter is OpenAI-compatible and proxies streaming responses transparently, so an editor extension gets tokens as they are generated — the gateway adds no buffering on the hot path.
Model choice, low latency, per-seat budgets
Build a coding assistant your finance team can read
Pick the model per task, route for latency, and cap spend per developer — all unlocked on every plan.
OpenAI-compatible — works with any IDE extension that targets the OpenAI SDK.