Tag

ab-testing

5 posts tagged "ab-testing".

Posts

Latest first

Server-Side Prompt Templates: Version, Roll Back, A/B Test
Engineering

Server-Side Prompt Templates: Version, Roll Back, A/B Test

A one-word prompt fix should not need a deploy, and a bad prompt should not need an incident bridge. Here is how referencing a versioned server-side template by id makes prompt changes reversible, auditable, and testable on real traffic.

nRouter team
11 minRead →
Cost-vs-Quality LLM Routing: Which Tasks Can Go Cheap
Guides

Cost-vs-Quality LLM Routing: Which Tasks Can Go Cheap

Quality is a property of a task, not of a model. Here is how to inventory your traffic by task, point a router alias at a candidate set, prove each downgrade with an A/B test, and read the saving off the Cost vs Usage report.

nRouter team
11 minRead →
Hash-Based A/B Tests: Same User, Same Model Variant, Every Call
Engineering

Hash-Based A/B Tests: Same User, Same Model Variant, Every Call

A coin flip on every request is not an experiment — it is noise with a dashboard. Here is how deterministic hash-based assignment gives each user a stable variant for the life of a test, why the experiment id belongs in the hash, and what the gateway refuses to let a caller override.

nRouter team
11 minRead →
NotDiamond alternative: a router picks a model, a gateway runs the call
Comparison

NotDiamond alternative: a router picks a model, a gateway runs the call

Not Diamond returns a model recommendation and charges $0.05 per million tokens routed — you still hold every provider key and make the call yourself. What that leaves you to build, and how deterministic A/B tests compare to a trained router when you have to reproduce a decision.

nRouter team
12 minRead →
Unify AI alternative: measure quality-per-dollar on your own traffic
Comparison

Unify AI alternative: measure quality-per-dollar on your own traffic

nRouter vs Unify AI for teams weighing benchmark-driven model arbitration against operator-pinned routing. Why a vendor leaderboard is not your eval, and how to compute cost-per-passing-answer from your own requests.

nRouter team
12 minRead →