A managed LLM gateway adds a hop between your code and the provider. The honest question is: how much does that hop cost in latency and reliability? This page is our methodology for measuring it, and a commitment to publish real numbers, not a fabricated leaderboard.
What this page measures
Read this first. The latency figures below are SLA targets we hold ourselves to — not the results of a published study. A full reproducible gateway-latency report is on the roadmap. We would rather state that plainly than print numbers we cannot defend. For live, real-time health see the status page.
Gateway time on top of the provider call
The slow tail we hold ourselves to
Contractual: see the legal SLA
Switch models with a string, no re-integration
Targets, not measurements. p50 mirrors SLA_LATENCY_MS from the live stats endpoint; p95 is the tail bound we commit to.
Routing a request does not change the model's answer. So NemoRouter benchmarks the things routing actually affects: added latency, failover, and throughput. Model quality stays where it belongs: with the model provider and independent evaluators.
The single number that is genuinely ours: how many milliseconds NemoRouter adds on top of the provider round-trip. Reported at p50/p95/p99 and broken down by stage (auth, credit reserve, guardrails, routing, settle), so the number is auditable, not a black box. A pass-through request (no guardrails) is the floor case.
When a provider 5xxs, times out, or rate-limits, the routing engine retries the next model in the fallback chain, bounded and budgeted per provider. We measure how often that fires and how much latency the retry adds, reported separately from the happy-path p50. A failed attempt costs zero credits (reserve+settle).
Sustained requests-per-minute and tokens-per-minute the gateway holds before queueing — the number behind the per-tier RPM/TPM guarantees on the pricing page, including the latency degradation curve and 429 behavior at the ceiling.
Model quality (MMLU, HumanEval, GPQA, reasoning scores) is the model provider’s to measure and publish. Routing a request does not change the model’s answer, so we run no model leaderboard and invent no quality scores. Honest gap: a published latency study is on the roadmap, not done.
NemoRouter adds no separate network service and no second hop on the path to a provider. Browse the live catalog on the models page.
A benchmark is only worth printing if someone else can reproduce it. Every study we publish follows the same four rules, and ships the config so you can run it yourself.
Run against the zero-cost mock provider so provider latency is constant. Whatever moves is gateway overhead. Nothing else.
Discard cold-start requests, then collect a large fixed sample at steady state. Cold starts are reported separately, never blended into p50.
Publish p50, p95, and p99. Never just the average. A gateway that looks fast at p50 and stalls at p99 is a slow gateway.
Every run ships its config: region, instance size, guardrails enabled, concurrency, sample size, and date. A benchmark you cannot reproduce is marketing.
A dated report, not a moving number. Each study is timestamped and versioned. Older reports stay live so you can see the trend, not just today’s headline.
Full run configuration. Region, instance size, concurrency, guardrails enabled, sample size, and provider mix — everything needed to reproduce the run.
p50, p95, and p99 — always the tail. Every latency claim ships all three percentiles. A single average is not a benchmark; it is a hope.
Failover and error rates alongside latency. Reliability and speed are reported together. A gateway that is fast only when nothing fails is not actually fast.
Want the methodology in detail, or to run a benchmark against your own workload before committing? Email sales@nemorouter.ai.
The 99.9% uptime SLA is a contractual commitment in the legal SLA, not a figure we picked because it looked good on a slide. The gateway runs on managed autoscaling container infrastructure and managed Supabase Postgres.
99.9%
Committed, monitored, and public — component health and incident history are never behind a login.
Measure it yourself
Point a real workload at NemoRouter and watch the overhead in your own dashboard. No fabricated leaderboard to trust. Just the gateway, your requests, and the numbers.