RAG on One Key: Costing Embeddings and Chat as One Answer

What it does

A RAG app makes two different model calls — embeddings and chat — and most teams wire them to two providers, two SDKs, two keys and two invoices. Behind one gateway key they become one surface: one combined cost per answer, one budget that covers index-time and query-time alike, and fallback on both halves rather than one.

The two-call shape is intrinsic to the pattern, not to any one implementation of it: retrieval-augmented generation was defined that way in the original RAG paper — retrieve over a dense index, then condition a generator on what came back — and every framework since has kept the split. The unification sounds like a convenience and is actually a measurement fix. Teams that price only the chat call undercount what an answer costs, sometimes badly, because embeddings run constantly — once per document at index time, once per query at query time, and a whole corpus at a time whenever anyone re-indexes. This post is about what changes when both halves pass through one place: what you configure, what a real month costs written out, and where the approach stops helping.

The job it does for you

The workflow before. The retrieval layer holds one provider's key and the generation layer holds another's. Two SDKs, two error taxonomies, two rate-limit regimes. The embedding bill arrives separately from the chat bill, on a different date, so "what does it cost to answer one question" is a question nobody in the room can answer. A budget, if one exists at all, covers generation only — which means the single largest cost event in a RAG stack, a full re-index, is outside every ceiling you have. When the embedding provider degrades, retrieval returns nothing and the app confidently answers from an empty context.

The workflow after. One key reaches both call types. Tag the embedding call and the chat call for a query with the same identifier and the cost of one answer is a sum. A budget covers both halves, so the re-index burst hits the same ceiling as everything else. A fallback chain covers both, because an embeddings outage is exactly as fatal to RAG as a chat outage. And you can tune each half separately — a cheaper embedding model with an unchanged chat model, or the reverse — because they are two knobs on one control plane rather than two vendor relationships.

The two halves of a RAG request

The shape is worth drawing, because the cost asymmetry between the two halves is what makes the combined number surprising.

INDEX TIME                          QUERY TIME
docs → chunk → embeddings → store   query → embeddings → store → top-k
                                                                  │
                                              retrieved context + query
                                                                  │
                                                        chat completion → answer

Two calls, two cost profiles, one user-visible outcome. Treating them as unrelated integrations is what hides the real number.

How it works

What you configure. One virtual key, scoped to the team that owns the RAG service, carrying a budget ceiling, RPM and TPM limits, and a model allowlist covering both the embedding model and the chat model. The scoping model is org, team, member. If index-time and query-time traffic have genuinely different risk profiles — and on a large corpus they do — give them two keys with two budgets rather than one.

What happens to a call. Both call types take the same path: the key resolves to an organisation and a team, guardrails run pre-call, every ceiling is checked before the provider is called, credit is held, the provider is called, post-call guardrails run, and the hold settles against the real cost. Holding credit before the call rather than reconciling afterwards is what keeps a parallel embedding job from blowing through a ceiling it technically had not reached when it started — the mechanism is reserve-and-settle.

What comes back. The provider's response plus the x-nr-* headers, including x-nr-request-cost on both call types. When a cost cannot be determined, the header is absent and x-nr-cost-status reads unpriced — never a zero, for the reason in cost honesty. This matters more in RAG than elsewhere: a zero silently folded into a per-answer total is invisible precisely because embedding costs are small enough that nobody double-checks them.

When a ceiling stops a call, the code says which one:

Both are ordinary HTTP — 429 is RFC 6585 §4 and 402 is RFC 9110 §15.5.3 — so an index job's existing backoff already covers one of them and almost certainly has no rule for the other. An index job should treat these as pause-and-resume rather than as a reason to abandon a half-embedded corpus. The client-side half is handling 429 and 402.

Set it up

1. Create a key for the RAG service in the dashboard, scoped to its owning team. The full sk-nrouter-… is shown exactly once; store it in your secret manager. See API key management.
2. Set a budget that covers index time, not just query time. Size it against a full re-index, because that is the largest thing it will ever have to absorb. See budget controls.
3. Allow both models on the key — the embedding model and the chat model you picked off /models — and nothing else.
4. Point both call types at the same client. One base URL, one key.
5. Tag both calls with the same query identifier so the combined cost is queryable.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.nrouter.ai/v1",
    api_key=os.environ["NROUTER_API_KEY"],
)

# Both slugs come from /models and live in config, not in source. Changing the
# embedding one means re-embedding the corpus, so it is a deployment decision.
EMBEDDING_MODEL = os.environ["RAG_EMBEDDING_MODEL"]
CHAT_MODEL = os.environ["RAG_CHAT_MODEL"]

# index time and query time: embeddings
emb = client.embeddings.create(
    model=EMBEDDING_MODEL,
    input=chunks,
    extra_body={"metadata": {"tags": ["feature:rag", "phase:embed", f"query:{query_id}"]}},
)

# answer: chat completions, same client, same key
ans = client.chat.completions.create(
    model=CHAT_MODEL,
    messages=[{"role": "user", "content": prompt_with_context}],
    extra_body={"metadata": {"tags": ["feature:rag", "phase:generate", f"query:{query_id}"]}},
)

The embedding contract is in the embeddings reference and the generation contract in chat completions; both are the OpenAI-compatible shapes documented in the OpenAI embeddings guide and the OpenAI API reference, called through the official openai client. Which models your key can reach is a live question and this post deliberately does not answer it — read the current catalogue off the models page before you pin either half, and treat any model slug you see in a blog post, ours included, as illustrative. If your retrieval layer is a framework rather than raw SDK calls, the drop-in pages for LlamaIndex and LangChain cover the same base-URL swap.

Worked example with numbers

Take a documentation assistant over a 200,000-chunk corpus, serving 50,000 queries a month, re-indexed once a month.

Which looks like it proves embeddings do not matter — until you change one assumption. Re-index daily instead of monthly, which is what happens the moment the docs are updated continuously, and the index-time line becomes $120.00/month: embeddings go from 0.9% of the bill to 18% of it, and they did it without a single line of code changing. That is the class of surprise a combined budget exists to catch.

The per-answer number is the other output: $555.00 ÷ 50,000 = $0.0111 per answer, embeddings included. A team pricing only the chat call would have quoted $0.0110 and been roughly right this month and roughly wrong the month the re-index cadence changed. The rates above are illustrative placeholders labelled as such — substitute the live per-token rates for your models from the models page and the vendor pricing pages in ## Sources, then replace the whole table with measured numbers using cost vs usage once a week of real traffic has run. Tagging both phases with the same query identifier, per attribution tags, is what makes that replacement a query rather than a project.

Re-indexing is a budget event

The classic RAG cost surprise is not a slow leak, it is a spike: somebody re-embeds a large corpus and the burst dwarfs a day of query traffic. Three things make that survivable, and all three are the same controls used everywhere else.

1. A budget that covers index time. Because embeddings are metered through the same key as chat, the burst hits a ceiling instead of an invoice. Hard spend limits is the how.
2. A rate limit on the index key. A budget bounds the money; an RPM limit bounds how fast it can be spent, which is what buys a human time to notice. Choosing between the two is budgets vs rate limits.
3. An alert before the ceiling, not at it. A soft threshold that pages someone at 70% is worth more than a hard stop at 100%, because a stopped re-index that nobody knew about is a corpus in a half-embedded state.

What it costs

From the pricing page:

• Pay as you go — $0 subscription and a platform fee added on top at purchase, set at 4% of your credits. Buying the $555/month of credits above is a $577.20 charge, so the fee is $22.20/month.
• Pro — $50/mo or $500/yr, 0% platform fee.
• Enterprise — custom, 0% fee, contact sales only.

At $555/month, pay as you go is cheaper and we would rather you stayed on it. The crossover is where the fee on a month of spend, 4% of it, passes the subscription — around $1,250/month on the monthly plan and around $1,042/month on the annual one — and a RAG app crosses it as soon as query volume grows or the re-index cadence tightens. The walkthrough is from credits to Pro.

Every control described here — budgets, rate limits, fallback, tagging, A/B tests, guardrails — is on every plan; plans vary the fee and the rate limits and nothing else, for the reasons in we charge a fee, not a gate. Signup is not free: a card is required and a real $5 minimum charge is taken, with the fee on top.

Where it fits with the rest of the platform

• Fallback on both halves. An embeddings outage strands retrieval as completely as a chat outage strands generation; one chain covers both. Provider fallback chains.
• Independent cost tuning. Embeddings and chat sit on different cost/quality frontiers, so route each on its own terms. Route by cost vs quality.
• Testing an embedding-model swap. Change the retrieval model the same way you would change a chat model: on live traffic, with a stable assignment. Deterministic A/B testing.
• Guardrails on retrieved context. Retrieved documents are untrusted input to the generation call — indirect prompt injection through a retrieved chunk is the first entry in the OWASP Top 10 for LLM Applications, and the whole point of RAG is that a stranger's text reaches the model. Rules run pre-call and post-call on every request. Guardrails on every request.
• Redacting what you log. Retrieved chunks are the most PII-dense thing in a RAG pipeline, and the GDPR's minimisation principle applies to a log line as much as to the corpus it came from. Redacting PII from LLM logs.
• Per-customer billing. If each customer has their own corpus, the per-answer number above is the billing input. Per-customer LLM billing.

Limits and what it will not do

• The gateway does not do retrieval. Your vector store, chunking strategy and ranking are yours — whether that is pgvector, a managed index, or something you wrote. It meters, caps and routes the two model calls at either end; it has no opinion about what sits between them.
• A cheaper embedding model is not a free lunch. Changing embedding models means re-embedding the corpus, and the new vectors are not comparable to the old ones. Budget the migration, not just the rate difference.
• Fallback across embedding providers is not transparent. Two providers produce vectors in different spaces. A fallback that keeps query-time embedding available is only useful if the fallback model is the one the index was built with — otherwise availability is preserved and retrieval quality is not. Plan the chain accordingly.
• A retry is a second call and a second provider charge. The credit hold is taken once per customer request rather than once per attempt, but the provider still bills for attempts it served.
• An unpriceable call is reported, not guessed. x-nr-request-cost is absent when the cost is unknown; treat a missing header as unknown rather than as zero.
• No BYOK. You cannot bring a provider key. Why we do not do BYOK.
• Bodies are not logged or retained for you. Retrieved context is often the most sensitive text in your system, and it does not enter the audit trail, which is metadata only. One qualification: a response body may be held for a few minutes in a short-lived serving cache, keyed to your organisation and team, so a byte-identical repeat query skips the provider call — a cache you cannot read back, not a copy of your corpus. If you want that text retained, forward it to a sink you control and read what to log and not log first.
• SOC 2 Type II is in progress, not certified. The AICPA defines what that examination covers, and "in progress" is not one of its outcomes. Posture is on the trust page.

Try it

Create an account at signup and load the $5 minimum, with the platform fee on top. Then run the smallest honest version of this post: point both your embedding calls and your chat calls at https://api.nrouter.ai/v1 with one key, tag both phases with the same query identifier, and let a day of real query traffic through. Sum the tagged rows and you will have a true cost per answer — embeddings included — which is almost certainly not the number you have been quoting. Then set a budget sized against a full re-index and fire a deliberate small one to watch the ceiling work.

See also

• An LLM gateway for coding agents — the same ceilings under the other workload that fans one action into hundreds of calls.
• Attribute LLM spend by team, customer, and feature — the tagging that turns the two RAG phases into one queryable number.
• How to set hard spend limits on your LLM gateway — sizing the budget that has to absorb a full re-index.
• Provider fallback chains: surviving an OpenAI outage — why the embedding half needs a chain as much as the chat half does.
• Cost vs usage: finding the quietly expensive model — replacing the worked example's illustrative rates with measured ones.
• Write-Time PII Redaction in LLM Logs, Without Losing Debug Detail — what to do about retrieved chunks before they reach a log sink.
• Models — the live catalogue; check it for both halves before you pin a model on either.

Sources

Verified 2026-08-23. If a linked page has changed since, email hello@nrouter.ai and we will correct this post.

• nRouter plans and fees: nrouter.ai/pricing — $0 subscription with a fee on top that is 4% of your credits, Pro at $50/mo or $500/yr at 0%, $5 minimum purchase.
• nRouter live model catalog: nrouter.ai/models — the authoritative list of what a key can reach today. This post names no model slug on purpose; read it here.
• OpenAI embeddings guide: platform.openai.com/docs/guides/embeddings — the embeddings request shape used above.
• OpenAI API pricing: openai.com/api/pricing — per-model embedding and chat rates to substitute into the worked example.
• Anthropic pricing: anthropic.com/pricing — per-model rates for the Claude family used in the generation half.
• AWS Bedrock pricing: aws.amazon.com/bedrock/pricing — Bedrock is live on nRouter and AWS publishes its own rates.
• OpenAI API reference: platform.openai.com/docs/api-reference — the chat-completions half of the sample.
• Official OpenAI client library: github.com/openai/openai-python — the SDK both calls are made through.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.): arxiv.org/abs/2005.11401 — where the retrieve-then-generate split this post costs comes from.
• pgvector: github.com/pgvector/pgvector — one example of the retrieval half the gateway deliberately has no opinion about.
• RFC 6585 §4 (429) and RFC 9110 §15.5.3 (402) — the standard meanings of the two ceiling responses: datatracker.ietf.org/rfc6585, datatracker.ietf.org/rfc9110.
• OWASP Top 10 for LLM Applications: owasp.org — indirect prompt injection through retrieved context.
• GDPR Article 5: gdpr-info.eu/art-5-gdpr — minimisation, applied to retrieved chunks and to what you log of them.
• AICPA on SOC 2: aicpa-cima.com — what a Type II report is, and why "in progress" is not it.

OpenAI, Anthropic and AWS are trademarks of their respective owners. nRouter is not affiliated with or endorsed by them. The per-call rates in the worked example are illustrative placeholders and are labelled as such; substitute the live rates from the pages above.