A/B Testing

An A/B test lets you compare model variants on real production traffic without changing a line of application code. You declare two or more variants, assign weights, start the test, and nRouter rewrites the model your application asked for to one of the variants on the way through. Per-variant request counts, latency, cost, and success rate are recorded against the test so you can decide which variant wins.

A/B tests are administered on the dashboard API at https://nrouter.ai/api/ab-tests and applied by the gateway on the inference path at https://api.nrouter.ai/v1/*. Those are two different services with two different credentials — see below.

Authentication

/api/ab-tests is a dashboard API, not an inference endpoint. It authenticates with your logged-in dashboard session cookie, and every call must name the organization:

https://nrouter.ai/api/ab-tests?orgId=<your-organization-uuid>

There is no bearer-token path on this route. Your nRouter virtual key (sk-nrouter-…) authenticates inference at api.nrouter.ai/v1/* and nothing else — never send it to nrouter.ai/api/*, and never send a management credential to an inference endpoint.

To drive the API from a shell, export the Supabase session cookie from a browser session you are already signed in to and pass it with -b:

ORG_ID="<your-organization-uuid>"
API="https://nrouter.ai/api/ab-tests"
# COOKIE holds your dashboard session cookie header, copied from a signed-in browser.
# Treat it as a credential: it is equivalent to being logged in.

Mutating calls (POST, DELETE) additionally require:

• Content-Type: application/json — otherwise 415 Content-Type must be application/json.
• An Origin (or Referer) header matching https://nrouter.ai — otherwise 403 Forbidden: invalid request origin.

Missing or malformed orgId returns 400 Missing required parameter: orgId. A signed-out caller gets 401 Unauthorized; a caller who is not a member of that organization gets 403.

Where A/B selection happens on the request path

A/B selection runs inside the gateway's prompt-runtime step, on buffered requests to four endpoints:

• POST /v1/chat/completions
• POST /v1/messages
• POST /v1/responses
• POST /v1/completions

Five conditions must all hold for a request to enter a test:

1. The organization's Prompts feature is on. A/B selection lives in the same runtime step as managed prompts and is gated by organization_billing.prompts_enabled. With Prompts off, no test ever matches. The toggle is on the dashboard Prompts page.
2. A test is running in your organization.
3. The test's model field matches the requested model — or the test was created without a model, in which case it matches every model. This is the test-level model field, not the models listed in its variants.
4. The request is not streaming. A streaming request that matches a running test is rejected with 400 streaming requests cannot enter an A/B test until completion telemetry is available. Terminal telemetry for a stream is not available yet, so the gateway refuses rather than record a result it cannot measure.
5. The request falls inside traffic_percentage. Requests above that fraction bypass the test and are served exactly as sent.

When a variant is chosen, the gateway overwrites body.model with the variant's model and applies the variant's config_override. Everything after that — guardrails, prompt injection, settlement — sees the selected variant. Settlement uses the served model's cost, so billing reflects what actually ran.

config_override accepts only these five keys; anything else fails the request with 400 A/B config override <key> is not allowed:

temperature · top_p · max_tokens · max_completion_tokens · seed

Bucketing is sticky per key, not random per request

This is the single most important thing to understand before you read a rollup.

The gateway hashes SHA256("<test_id>:<identity>"), where identity is your virtual key id, falling back to the user id, then the organization id. It does not hash the request id. The first half of the digest decides whether the request enters the test at all (against traffic_percentage); the second half maps into the variants' cumulative-weight range.

The consequence: every request from one virtual key lands in the same variant, forever. A 70/30 test does not send 70% of one key's calls to the control — it sends all of that key's calls to whichever variant that key hashes into. The weights describe how your keys are distributed across variants, not how one key's requests are distributed.

So: run the experiment across many virtual keys, and read the split as a per-key assignment. A single-key smoke test will show 100% on one variant, and that is correct behaviour, not a bug.

The bucketing is also integrity-protected: the identity comes from the authenticated caller, never from a header or body field, so a client cannot pin itself to a variant.

Results are written per request, not batched

After a buffered attempt completes, the gateway writes one immutable row per request carrying the test id, variant name, served model, latency, cost, token counts, success flag, and the authenticated org/team/user/key chain. There is no flush interval to wait out — a rollup read immediately after a completed call already includes it.

Refusals before the provider call (credit, credential, routing, guardrail) deliberately record no variant outcome, so request_count counts attempts that actually reached a provider.

Lifecycle

   draft ── start ──► running ── pause ──► paused ── start ──► running
                         │                    │
                         └─ complete/stop ─┬──┘
                                           ▼
                                    completed (terminal)

A transition from a status the action does not allow returns 409 naming the current status. completed is terminal — there is no reopen action; clone the configuration into a new test instead.

Status changes are read from the database on every request. There is no cache to wait out: a pause stops routing on the next call.

A worked example

The scenario: you use gpt-4.1-mini for summarising support tickets and want to know whether gpt-4.1-nano holds up. Split 70/30 across your keys.

1. Create the test

POST /api/ab-tests with no action creates. Body fields (strict — an unknown field is rejected):

Each variant is also strict: name (1–255), model (1–255), weight (greater than 0, at most 1), optional config_override, and the optional pair prompt_template_id + prompt_version_id — which must be supplied together or not at all.

curl -X POST "$API?orgId=$ORG_ID" \
  -b "$COOKIE" \
  -H "Origin: https://nrouter.ai" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Mini vs Nano for support summaries",
    "description": "Does 4.1-nano hold up for one-line ticket summaries?",
    "model": "gpt-4.1-mini",
    "traffic_percentage": 1,
    "variants": [
      { "name": "control",    "model": "gpt-4.1-mini", "weight": 0.7 },
      { "name": "challenger", "model": "gpt-4.1-nano", "weight": 0.3 }
    ]
  }'

201 Created, and the response is the row itself — no envelope:

{
  "id": "7f3c1a8e-2b4d-4a91-bf7c-1d2e3a4b5c6d",
  "organization_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "name": "Mini vs Nano for support summaries",
  "description": "Does 4.1-nano hold up for one-line ticket summaries?",
  "model": "gpt-4.1-mini",
  "status": "draft",
  "traffic_percentage": 1,
  "variants": [
    { "name": "control",    "model": "gpt-4.1-mini", "weight": 0.7 },
    { "name": "challenger", "model": "gpt-4.1-nano", "weight": 0.3 }
  ]
}

Save the id. Every later call passes it as the testId query parameter, and it must be a UUID — anything else returns 400 Invalid testId format — must be a UUID.

Every variant model is checked against the models the gateway actually serves before the row is written:

• An unknown model returns 400 with "error": "invalid_model_group" and a missing array naming them.
• If the catalogue itself cannot be read, the create is refused with 503 and "code": "GATEWAY_MODEL_LIST_UNAVAILABLE" rather than accepting a model that may not exist.
• A duplicate test name returns 409 An A/B test with this name already exists.

2. Start it

TEST_ID="7f3c1a8e-2b4d-4a91-bf7c-1d2e3a4b5c6d"

curl -X POST "$API?orgId=$ORG_ID&action=start&testId=$TEST_ID" \
  -b "$COOKIE" \
  -H "Origin: https://nrouter.ai" \
  -H "Content-Type: application/json" \
  -d '{}'

The response is the updated row with "status": "running". From this point, non-streaming calls to gpt-4.1-mini from keys that bucket into the test are rewritten to their assigned variant.

3. Your application keeps calling the same model

No code change. It still sends model: gpt-4.1-mini to https://api.nrouter.ai/v1/chat/completions with its own virtual key. Cost and latency settle against whichever model actually served the response.

4. Read the rollup

GET /api/ab-tests?orgId=…&testId=… returns the test row plus aggregated metrics:

curl "$API?orgId=$ORG_ID&testId=$TEST_ID" -b "$COOKIE"

{
  "id": "7f3c1a8e-2b4d-4a91-bf7c-1d2e3a4b5c6d",
  "status": "running",
  "model": "gpt-4.1-mini",
  "variants": [
    { "name": "control",    "model": "gpt-4.1-mini", "weight": 0.7 },
    { "name": "challenger", "model": "gpt-4.1-nano", "weight": 0.3 }
  ],
  "metrics": [
    {
      "variant_name": "control",
      "request_count": 706,
      "requests": 706,
      "avg_latency_ms": 412.4,
      "latency_samples": 706,
      "total_cost": 0.129198,
      "avg_cost": 0.000183,
      "cost_samples": 706,
      "total_tokens": 84720,
      "error_count": 2,
      "success_rate": 0.9972
    },
    {
      "variant_name": "challenger",
      "request_count": 294,
      "requests": 294,
      "avg_latency_ms": 287.1,
      "latency_samples": 294,
      "total_cost": 0.027636,
      "avg_cost": 0.000094,
      "cost_samples": 291,
      "total_tokens": 33810,
      "error_count": 2,
      "success_rate": 0.9932
    }
  ],
  "results_recorded": true
}

Read these three field pairs carefully:

• avg_latency_ms and avg_cost are null, never 0, when nothing was measured. A null means no row in that variant carried a latency (or a cost) at all. "0 ms" and "$0.00" are assertions; null is the fact.
• latency_samples and cost_samples are the denominators behind those averages. When cost_samples is lower than request_count, the difference is unpriced requests — the gateway records no cost rather than a zero. This mirrors the inference contract, where x-nr-request-cost is absent on an unpriced request and never 0.
• request_count and requests are the same number, kept for both spellings.

metrics has one entry per variant that has recorded at least one result. A variant with no traffic yet does not appear.

5. Read the raw per-request rows

curl "$API?orgId=$ORG_ID&action=results&testId=$TEST_ID&limit=100&offset=0" -b "$COOKIE"

Returns { "data": [ … ], "total": <exact count> }, newest first. limit must be an integer between 1 and 1000 (default 100); offset a non-negative integer (default 0). Out-of-range values return 400 naming the constraint.

6. Decide

Once a test leaves draft it is immutable: update is accepted only on a draft test, so a single test can never silently mix two configurations. On a running or paused test it returns 409 Only draft A/B tests can be edited.

That gives you three moves:

Keep gathering signal — do nothing.

Roll the winner out — change the model your application sends to the winner, then freeze the test for the audit trail:

curl -X POST "$API?orgId=$ORG_ID&action=complete&testId=$TEST_ID" \
  -b "$COOKIE" \
  -H "Origin: https://nrouter.ai" \
  -H "Content-Type: application/json" \
  -d '{}'

Try a different split — complete the current test and create a fresh one at the new weights. A new test starts in draft with a new id.

Editing a draft test uses action=update, and the body carries only the fields you are changing:

curl -X POST "$API?orgId=$ORG_ID&action=update&testId=$DRAFT_TEST_ID" \
  -b "$COOKIE" \
  -H "Origin: https://nrouter.ai" \
  -H "Content-Type: application/json" \
  -d '{
    "variants": [
      { "name": "control",    "model": "gpt-4.1-mini", "weight": 0.3 },
      { "name": "challenger", "model": "gpt-4.1-nano", "weight": 0.7 }
    ]
  }'

update re-validates weights and re-checks every variant model against the served catalogue, exactly as create does. Note that update cannot change the test's model field — that is fixed at creation.

Checks you can run yourself

Prerequisites: the worked example is created and running, TEST_ID, ORG_ID, API and COOKIE are set, the org's Prompts toggle is on, and the virtual key you send inference with has credits.

Check 1 — one key is stable

What it proves: bucketing is per key, not per request. This is the invariant to internalise before reading any rollup.

# Send 20 non-streaming calls from ONE virtual key.
seq 1 20 | xargs -P 5 -I {} curl -s -X POST https://api.nrouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $NROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"Reply with the single word: ok"}],"max_completion_tokens":4}' \
  -o /dev/null

curl -s "$API?orgId=$ORG_ID&testId=$TEST_ID" -b "$COOKIE" \
  | jq '[.metrics[] | {variant: .variant_name, count: .request_count}]'

Expected: all 20 land on exactly one variant. To see both variants move, repeat with a second virtual key.

Check 2 — a model the test does not name passes through

What it proves: the test intercepts only its own model.

BEFORE=$(curl -s "$API?orgId=$ORG_ID&testId=$TEST_ID" -b "$COOKIE" \
  | jq '([.metrics[].request_count] | add) // 0')

seq 1 10 | xargs -P 5 -I {} curl -s -X POST https://api.nrouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $NROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-5.4-mini","messages":[{"role":"user","content":"hi"}],"max_completion_tokens":4}' \
  -o /dev/null

AFTER=$(curl -s "$API?orgId=$ORG_ID&testId=$TEST_ID" -b "$COOKIE" \
  | jq '([.metrics[].request_count] | add) // 0')

echo "delta: $((AFTER - BEFORE))"

Expected: delta: 0.

Check 3 — a streaming request is refused, not silently excluded

What it proves: the gateway will not quietly drop a matching request out of the experiment.

curl -s -o /dev/null -w '%{http_code}\n' -X POST https://api.nrouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $NROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4.1-mini","stream":true,"messages":[{"role":"user","content":"hi"}],"max_completion_tokens":4}'

Expected: 400, with a body naming streaming as the reason. Keep streaming traffic on a model no running test names.

Check 4 — a started test is immutable

curl -s -o /tmp/ab-update-resp.json -w 'HTTP %{http_code}\n' \
  -X POST "$API?orgId=$ORG_ID&action=update&testId=$TEST_ID" \
  -b "$COOKIE" \
  -H "Origin: https://nrouter.ai" \
  -H "Content-Type: application/json" \
  -d '{
    "variants": [
      { "name": "control",    "model": "gpt-4.1-mini", "weight": 0.2 },
      { "name": "challenger", "model": "gpt-4.1-nano", "weight": 0.8 }
    ]
  }'
cat /tmp/ab-update-resp.json

Expected: HTTP 409 and {"error":"Only draft A/B tests can be edited."}.

Cleanup

curl -X POST "$API?orgId=$ORG_ID&action=complete&testId=$TEST_ID" \
  -b "$COOKIE" \
  -H "Origin: https://nrouter.ai" \
  -H "Content-Type: application/json" \
  -d '{}'

Things to watch for

• Weights describe key assignment, not request assignment. Covered above, and it is the mistake that makes a rollup look broken.
• Prompts must be on. With prompts_enabled off, tests sit in running and intercept nothing. Nothing errors; the metrics simply stay empty.
• Streaming is refused, not bypassed. A matching streaming request gets a 400.
• Only one running test intercepts a given model. The gateway takes the first matching running test by id and stops. A second running test that also matches never sees traffic.
• A test with no model matches everything. Create one deliberately or not at all.
• Keep variants in the same pricing class. The credit hold is taken before selection, sized for the model the caller requested; a variant that is far more expensive can under-reserve on bursty traffic.
• Rate limits are per virtual key, not per variant. A variant pointing at a model with stricter provider limits shows more 429s. That is real signal.
• Mutations are rate-limited to 20 per minute per organization. Exceeding it returns 429.
• Tests are org-scoped, not team-scoped. Every team's matching traffic is eligible; per-team experiment scoping does not exist.

Endpoint reference

All paths are on https://nrouter.ai, all require ?orgId=<uuid> and a dashboard session, and every testId must be a UUID.

The list form returns a bare JSON array. Single-test and mutation forms return the row object. action=results returns an object with data and total.

Roles. Reads (GET, any form) accept any member of the organization. Every write — create, update, start, pause, complete, delete — requires organization Owner or org_admin; anyone else gets 403 Insufficient permissions. Owner is derived from the organization's billing owner, not from a membership role string.

You can do all of this from the dashboard instead, at Prompts → A/B Tests.

FAQ

Do I need to change my application code to run an A/B test?

No. You keep calling the same model your app already uses, with the same virtual key. The rewrite happens inside the gateway.

How is traffic split between variants?

By a deterministic hash of the test id and the caller's virtual key id. One key always resolves to the same variant. Weights distribute keys across variants, so run the experiment across many keys.

Can a client pin itself to a specific variant?

No. The identity that decides bucketing comes from the authenticated caller — never from a header, body field, or query parameter — so there is no override to send.

What happens to a model the test does not name?

It passes straight through. A test intercepts only the model in its model field; a test created without one intercepts everything.

Does A/B testing work with streaming?

No. A streaming request that matches a running test is rejected with 400, because terminal telemetry for a stream is not available to record a result from. Point streaming traffic at a model no running test names.

Can I edit the variants or weights after I start a test?

No. update is accepted only on a draft test; on running or paused it returns 409. To change the split, complete the test and create a new one.

How quickly do start, pause, and complete take effect?

Immediately. The status is read from the database on each request — there is no propagation window to wait out.

Why is avg_cost null when there were requests?

Because no request in that variant produced a priced result. nRouter records no cost rather than a zero for an unpriced request, exactly as the inference response omits x-nr-request-cost instead of reporting 0. Compare cost_samples against request_count to see how many were priced.

Which variant's cost am I billed for?

The model that actually served the request. One caution: the credit hold is taken before selection, sized for the model you requested, so keep variants in a similar pricing class.

Can I run two tests on the same model at the same time?

Effectively no. The gateway takes the first matching running test by id; a second one never receives traffic. Complete or pause the first.

Who can create and manage tests, and does it cover my whole org?

Any org member can read. Only Owner and org_admin can write. Tests are organization-scoped — every team's matching traffic is eligible, and per-team scoping does not exist.

Next Steps

• Playground — Send a few calls through your model to confirm the test is intercepting before you scale traffic up.
• Budget Controls — Cap experiment spend with a per-key budget.
• Authentication — Virtual keys for inference, dashboard sessions for management.
• Chat Completions — The endpoint A/B selection sits in front of.