Hash-Based A/B Tests: Same User, Same Model Variant, Every Call

The rule: a variant is a function of who is calling, not a coin flip on each call. Hash a stable key together with the experiment id, and the same user lands in the same variant on every request for the life of the test — uniformly distributed across users, perfectly stable for each one.

You want to know whether a cheaper model is good enough, or whether the rewritten prompt actually helps. So you split traffic 50/50 and compare. If that split is random per request, the experiment is broken before the first data point: the same user bounces between variants mid-conversation, every metric is contaminated, and re-running the test gives a different answer.

This post is the design that fixes it, described as a contract you can verify from the outside rather than as a tour of anything internal.

The problem in one request

Here is the request. Nothing about it is unusual, which is the point.

POST https://api.nrouter.ai/v1/chat/completions
Authorization: Bearer $NROUTER_API_KEY

model: "chat-experiment"   ← logical model splitting between gpt-5.5 and claude-sonnet-4-5
user:  u_8813             ← the same person who called 40 seconds ago

Under random per-request splitting, the correct answer to "which variant did u_8813 get?" is this time? — and that question has no stable answer, so neither does anything you build on top of it. The user's third message in one conversation may be answered by a different model than their first, in a different voice, at a different quality level, and your satisfaction metric has no idea which variant to credit.

Why the naive approach breaks

Random-per-request looks like the textbook definition of a fair split, and it even is one — it is just fair about the wrong unit. Choosing that unit is the first decision in any controlled experiment, and the standard practitioner guidance is unambiguous that a user-level outcome needs a user-level randomization unit: see Kohavi, Longbotham, Sommerfield and Henne, Controlled experiments on the web: survey and practical guide (Data Mining and Knowledge Discovery, 2009). Three consequences follow, and they compound.

• The user experience flips mid-session. Two models rarely match in tone, verbosity or refusal behaviour. A user who gets A, then B, then A inside one conversation experiences a product that is inconsistent, and that inconsistency is itself a quality signal you did not intend to test.
• Metrics are confounded. A thumbs-down arrives. Which variant earned it? The user saw both. Every per-user outcome — task success, retention, conversion, a support ticket — becomes unattributable, and those are exactly the outcomes worth measuring.
• The result is unrepeatable. Re-run the same experiment and a different random draw produces a different number. There is nothing to reproduce, so there is nothing to defend when someone asks whether the result is real.

The exact input that defeats it is the ordinary one: a user who sends more than one request. Any multi-turn product hits this on day one.

There is a second naive fix that fails differently — assigning by modulo on a user id (user_id % 2). That is stable, which is progress, but it is not uniform: ids are rarely random, and sequential or prefixed ids cluster in a way that puts systematically similar users into the same bucket.

Deterministic assignment via hashing

The fix is to derive the variant from a hash of a stable key plus the experiment identifier, instead of from a random number:

bucket  = hash(experiment_id + assignment_key) % 100
variant = bucket < 50 ? "A" : "B"

assignment_key is whatever unit your experiment is really about — an end-user id for a consumer product, an account id for a B2B one, a session id where a session is the meaningful unit.

Two properties fall out, and you need both:

• Deterministic. The same key always produces the same bucket, so the same user always gets the same variant for the life of the experiment. No state to store, no lookup to keep consistent, nothing to go stale.
• Uniform. A good hash spreads keys evenly across 0–99, so a 50/50 split is genuinely 50/50 across the population even though it is fixed for each individual. This is the same property bucketing schemes in distributed systems rely on — Lamping and Veach's jump consistent hash is the compact statement of it: a key maps to a bucket by arithmetic alone, evenly, with no stored assignment table. Random assignment, stable thereafter — which is what "random assignment" was supposed to mean all along.

The mechanism: what the gateway guarantees

Stated as a contract, there are five promises.

1. Assignment happens at the gateway, not in your application. Your code asks for the logical model the experiment is defined on and receives the variant that user is assigned to. No branching in your call sites, no experiment config shipped to clients, no drift between services that all call the same model.
2. Assignment is stable for the life of the experiment. Same key, same experiment, same variant — across requests, across processes, across restarts.
3. Assignment is not a request parameter. A caller cannot ask to skip the experiment or pin a variant. If they could, your sample would be self-selected and the result meaningless — and a client could simply dodge the cheaper variant, defeating the thing you were testing. This is the same principle as rate limits being boundaries rather than knobs: some controls stop being controls the moment they are overridable.
4. Every request records which variant served it. Cost, latency and outcome are attributable per variant without you instrumenting anything, which is what makes the comparison at the end cheap.
5. Changing the split changes future assignment, not history. Move a 50/50 to 80/20 and users re-bucket from that point. Requests already recorded keep the variant that actually served them.

Promise five is the one people trip over, so it is worth being blunt: if you change the split mid-test, you have started a new test. Do it deliberately or not at all.

What you can put in an A/B test

The same deterministic seam handles several kinds of comparison, which is why it is worth having one rather than four.

The prompt-change case pairs naturally with server-side prompt templates, since a versioned template is exactly the kind of thing you want to A/B rather than swap and hope.

Worked example: a 50/50 split on the expensive route

Continuing the whale from cost vs usage: a summarization route, 10,000 calls a month, $0.0665 per call, $665.00 a month, 80% of the bill. The hypothesis is that a smaller model is good enough.

Set up the split. Experiment sum-2026-06, assignment key = end-user id, 50/50 between flagship variant A (such as openai/gpt-5.5 or claude-sonnet-4-5) and cost-optimized challenger variant B (such as openai/gpt-5-mini or claude-haiku-4-5).

Check the split is actually even. After a fortnight:

1,204 against 1,196 is the uniformity property doing its job. A meaningfully lopsided count here would be a sample ratio mismatch — the single most useful early sign that an experiment is broken rather than merely inconclusive, and the subject of Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD 2019). Requests split 4,970/5,030 rather than exactly evenly because users were split evenly and users differ in how much they call — which is correct, and is a thing you can only see because assignment was per user.

Do the arithmetic.

projected monthly, all A      10,000 × $0.0665            = $665.00
projected monthly, 50/50       5,000 × $0.0665
                             + 5,000 × $0.0222            = $443.50
projected monthly, all B      10,000 × $0.0222            = $222.00

saving at 50/50   $221.50 / month
saving at 100% B  $443.00 / month

Then read the half that decides it. Cost is the easy column. Compare the quality signal you actually capture — thumbs, task completion, follow-up rate, escalation to a human — between cohorts that each saw exactly one variant. That comparison is only meaningful because of stable assignment; under random-per-request it would have been arithmetic on noise.

If B holds up, you complete the experiment onto B and the next month's chart should show roughly $222 on that route. If it does not, the experiment was still worth running, and you know something you did not know a fortnight ago.

Lifecycle: draft → running → paused → completed

"Is this test live?" has to be unambiguous, so an experiment is a small state machine rather than a boolean.

• Draft — configured, reviewable, splitting nothing. Traffic behaves as if the experiment does not exist.
• Running — assigning variants and recording results.
• Paused — assignment frozen. Users who already have a variant keep it; no new keys are assigned. This is the state you want during an incident, because it stops the experiment growing without discarding what it has learned.
• Completed — a winner is chosen and all traffic consolidates onto it. Nobody is stranded on the losing variant.

Because assignment was deterministic throughout, the numbers you are deciding on are numbers you can reproduce and, more importantly, defend.

Edge cases we had to decide

1. When a request arrives with no stable assignment key, we fall back to a per-request assignment and mark it as such, because silently inventing a sticky identity would be worse than a visible gap. Unkeyed traffic is uniform in aggregate but useless for per-user outcomes, so it is separable in the results rather than blended in.

2. When the split ratio changes mid-experiment, users re-bucket from that moment and prior requests keep the variant that served them, because retroactively relabelling history would make the record disagree with what users actually experienced. The honest read is that you now have two periods, and they should be analysed as two.

3. When a variant's model is unavailable, the request follows the ordinary fallback chain and the result is recorded against what actually served it, because an experiment must never become an availability risk. An outage should degrade your test, not your product.

4. When an experiment is paused, existing assignments are preserved rather than cleared, because clearing them would re-randomize everyone on resume and destroy the comparison. Pause is a freeze, not a reset — and the difference matters exactly when you are under pressure and reaching for the button.

5. A caller cannot pin a variant, and we do not offer an escape hatch for "just this one service", because one exempted service is one self-selected cohort, and a self-selected cohort invalidates the whole test. If a service genuinely must not participate, define the experiment so it is out of scope rather than letting it opt itself out at call time.

What you see from the outside

• Experiment configuration and results in the dashboard — variants, split, state, and per-variant cost, latency and volume. Field by field in A/B testing.
• Per-request headers in the x-nr-* namespace — x-nr-request-id to correlate an individual call with your own logs, x-nr-request-cost for the settled cost of that call, and x-nr-cost-status telling you whether it was exact or unpriced. When a call cannot be priced the cost header is absent, not zero, so an unpriced variant never looks artificially cheap.
• Analytics filtered by variant, so cost-per-variant is a view rather than a spreadsheet exercise. See Analytics.
• Unchanged application code. The same model value in the same request; the split moves under it.

All of it is on every plan. A/B tests, guardrails, prompt management, evals and per-team budgets are not gated behind a tier — plans vary the platform fee and the rate limits, not the feature set.

Limits

• Determinism is not significance. A stable split makes the comparison valid; it does not make it powerful. If the effect is small and the traffic is thin, you will not detect it, and running the test longer is the only honest answer. Watching a p-value until it crosses a threshold is not that answer — Johari, Pekelis and Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing, set out both why continuous peeking inflates false positives and what you can legitimately do instead.
• A cost win is not a quality verdict. The cost column will always be cleaner than the quality column. Decide on both, and be suspicious of any experiment where the only reported improvement is the bill.
• Per-user assignment measures per-user outcomes. If what you care about is a per-document or per-request property, key the experiment on that unit instead — and accept that per-user metrics then no longer apply.
• Changing the assignment key restarts the experiment. Different key, different buckets, different cohorts. There is no migration.
• It is not an offline eval. A/B tests measure live traffic. Comparing models on a fixed dataset before you expose anyone is a different tool, and doing it first is usually cheaper.

Try it

Pick your most expensive route — sort by cost in analytics and take the top row. Pick the cheaper candidate from its own vendor's model list — OpenAI's or Anthropic's — so you are comparing two models whose documented context and capability limits you have actually read — for instance, testing gpt-5.4 against gpt-5-mini, or claude-sonnet-4-5 against claude-haiku-4-5. Define a 50/50 experiment against it, key it on your end-user id, and leave it running for two comparable weeks.

Then check the three things that tell you the test is sound before you look at the result: distinct users are close to even between variants, each user appears in exactly one variant, and the per-variant cost matches the arithmetic you expected. If all three hold, the quality comparison is worth having.

Put a ceiling under it with budget controls while it runs, and read the settled numbers from your ledger rather than from a projection. No account yet? Sign up — a card is required and a $5 minimum charge is taken, with the platform fee on top.

See also

• 5% of Requests, 60% of the Bill: Reading Cost Against Usage — how to find the route that is worth testing before you design the test.
• Server-Side Prompt Templates: Version, Roll Back, A/B Test — the versioned artifact that makes a prompt A/B test reproducible.
• Provider Fallback Chains: Surviving an OpenAI Outage — what happens to an experiment when a variant's provider goes down.
• RPM and TPM Rate Limiting Per Key, Team, and Org — the other control a caller cannot override, and why that is the point.
• Cost-vs-Quality LLM Routing: Which Tasks Can Go Cheap — turning a completed experiment into a standing routing rule.
• One Authoritative Cost Per LLM Request, Across Providers — where the per-variant cost figures come from.
• Models — the current catalog, so your cheaper candidate is a real option rather than a guess.

Sources

Verified 2026-08-23. The traffic and cost figures in the worked example are illustrative arithmetic for one organization's own settled data, not a quoted rate card. Current fees and minimums are on pricing. Corrections to hello@nrouter.ai and we will update.

• nRouter A/B testing documentation: nrouter.ai/docs/guides/ab-testing
• nRouter router settings: nrouter.ai/docs/guides/router-settings
• nRouter analytics documentation: nrouter.ai/docs/guides/analytics
• nRouter pricing: nrouter.ai/pricing

External references, all checked 2026-08-23:

• Kohavi, Longbotham, Sommerfield & Henne, Controlled experiments on the web: survey and practical guide (2009): link.springer.com — why the randomization unit has to match the unit the outcome is measured on.
• Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD 2019): exp-platform.com — the uneven-cohort check in the worked example, and what an uneven count usually means.
• Johari, Pekelis & Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing: arxiv.org/abs/1512.04922 — the peeking problem behind "determinism is not significance".
• Lamping & Veach, A Fast, Minimal Memory, Consistent Hash Algorithm: arxiv.org/abs/1406.2294 — even, stateless bucket assignment from a hash.
• OpenAI model list: platform.openai.com/docs/models
• Anthropic model overview: docs.claude.com

OpenAI and Anthropic are trademarks of their respective owners. nRouter is not affiliated with or endorsed by them.