Embeddings and chat, one key
The same endpoint serves the embeddings call that builds your index and the chat call that synthesizes the answer. Zero provider config.
Use Case · RAG
Retrieval-augmented generation, one key for embed and chat.
A RAG pipeline calls two model families: embeddings to index and query, a chat model to synthesize the answer. Route both through one NemoRouter endpoint, cache the repeats, and track cost per stage.
rag-pipeline · request trace
One pipeline, two model families
Index embedgemini-embedding
Query embedgemini-embedding
Synthesisgemini-2.5-flash
Cachehit · 0 ms
Stages taggedindex · query
Provider confignone
embed + chatcachedcost-tracked
- Model families
- Embed + chat
- Repeat queries
- Cached
- Cost attribution
- Per stage
- Catalog
- 240+
Both on one OpenAI-compatible endpoint
Exact-match hits skip the provider call
Index vs. query, split by metadata tag
models on Alibaba US, OpenAI, AWS Bedrock, Azure Foundry, Google Vertex AI & Anthropic
Why Nemo for RAG
The four things a RAG pipeline needs
Two model families, repetitive traffic, real cost pressure, and providers that occasionally fail. NemoRouter handles all four behind one key.
Caching for repetitive traffic
FAQs re-asked, identical context windows. Exact-match repeats are served from cache and skip the provider call entirely. Override with nemo_cache: false.
Cost per pipeline stage
Tag index vs. query traffic in request metadata and the dashboard attributes real per-call cost to each stage of the pipeline.
Failover keeps retrieval answering
A degraded embedding or chat provider triggers the next fallback link transparently. Your index build finishes and the query path keeps answering.
How it works
A RAG request, end to end
Index once, then query: embed the question, retrieve context from your own vector store, and synthesize with a chat model. Nemo sits on the two LLM hops (embeddings and synthesis) and logs the cost of each.
RAG pipeline flow
Index documents
POST /v1/embeddings
Chunk + embed your corpus once; store vectors in your DB.
Query embedding
POST /v1/embeddings
Embed the user question with the same model.
Retrieve context
your vector store
Nearest-neighbour search runs in your own database.
Synthesize answer
POST /v1/chat/completions
Chat model answers from retrieved context — cached if repeated.
Settled + logged
cost per stage
Embed cost, chat cost, cache hit — all in the request log.
Nearest-neighbour search stays in your database. NemoRouter handles the two LLM hops (embeddings and synthesis) with caching, failover, and per-stage cost & usage tracking.
The code
Same client for embeddings and chat
A RAG pipeline is just two endpoint calls against one key. These snippets come straight from the SDK examples the playground and dashboard use. Set NEMOROUTER_API_KEY and the chat call runs as-is; the embeddings call uses the same client and base URL.
Install
pip install openai| 1 | # Cache: enabled (org default). Pass nemo_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 | |
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NEMOROUTER_API_KEY"], |
| 7 | base_url="https://api.nemorouter.ai/v1", |
| 8 | ) |
| 9 | |
| 10 | response = client.chat.completions.create( |
| 11 | model="gemini-2.5-flash-lite", |
| 12 | temperature=1, |
| 13 | max_tokens=1024, |
| 14 | messages=[ |
| 15 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 16 | ], |
| 17 | extra_body={ |
| 18 | # "nemo_cache": False, # Uncomment to skip cache |
| 19 | }, |
| 20 | ) |
| 21 | |
| 22 | print(response.choices[0].message.content) |
The same client object also calls client.embeddings.create() — one key covers the whole pipeline.
FAQ
Common RAG questions
Does NemoRouter support embedding models for RAG?
Yes. The same OpenAI-compatible endpoint serves embeddings and chat completions. Your indexing job and your query path both call the gateway with one NemoRouter key. No separate embedding provider to configure.
How does caching reduce RAG costs?
RAG traffic is repetitive: popular questions, re-asked queries, identical context windows. Nemo caches responses by default; an exact-match repeat is served from cache and skips the provider call entirely. Pass nemo_cache: false per request to force a fresh generation.
Can I track cost for each stage of a RAG pipeline?
Yes. NemoRouter reports the real cost of every call, and the request log records the model and metadata for each. Tag your embedding calls and chat calls separately to attribute spend per pipeline stage.
What happens if my embedding provider goes down mid-index?
The fallback chain retries the next provider in the configured order. A 5xx or timeout on the primary triggers the next link transparently. Your index build keeps going and the failover is logged.
One key for the whole pipeline
Ship a RAG pipeline without juggling providers
Embeddings, chat, caching, and per-stage cost & usage tracking. All behind one NemoRouter key. Every feature is unlocked on every plan.
Building autonomous workflows on top of retrieval? See the AI agents use case.