Notes on shipping reliable LLM infrastructure.

Architecture & internals·13

How-tos & best practice·16

How we compare·13

Guides

All posts

Browse the catalog

Filter by category, or browse the full archive of deep-dives.

63 posts found

Guides

Handling 429 and 402 Errors From an LLM Gateway

A 429 and a 402 mean different things and need different client logic. Here is how to handle rate limits, budget caps, and out-of-credits responses gracefully — with backoff, not blind retries.

2026-06-108 min

RPM and TPM Rate Limiting Per Key, Team, and Org

Rate limits cap velocity, not total spend — and they're a security boundary, not a knob. Here is how RPM/TPM limits work per key, team, and org, and why the caller can never override them.

2026-06-108 min

Guides

Routing LLM Traffic by Cost vs Quality

Not every request needs your most expensive model. Here is a decision framework for routing LLM traffic by cost and quality — which tasks to send cheap, which to send premium, and how to prove the split works.

2026-06-108 min

Deterministic A/B Testing Across Model Variants

Randomly splitting LLM traffic gives you flaky, unrepeatable experiments. Here is how hash-based deterministic A/B testing splits traffic consistently per user, so your model comparison is actually measurable.

2026-06-109 min

Provider Fallback Chains: Surviving an OpenAI Outage

When a provider 5xxs or rate-limits you, your app shouldn't go down with it. Here is how fallback chains on an LLM gateway reroute to a healthy provider mid-request — without changing your code.

2026-06-109 min

Langfuse alternative: observability AND routing AND governance included on every plan

Head-to-head: NemoRouter vs Langfuse. A self-hostable observability + prompt + evals specialist vs a hosted LLM gateway that bundles observability, routing, and governance — every feature included on every plan. 4% pay-as-you-go, 0% on Pro. 240+ models behind one API key.

2026-06-109 min

Buyer's Guide

LLM routing strategies 2026: benchmark-anchored vs ML-classifier vs operator-controlled — how to pick the routing intelligence that fits your team

Vendor-neutral buyer's-guide decision tree across the three durable LLM routing-intelligence shapes — benchmark-anchored (Unify-style), ML-classifier (NotDiamond-style), and operator-controlled (NemoRouter-style). Three questions, one shape, one product. Pick the failure mode your team is best equipped to own.

2026-06-0913 min

TrueFoundry AI Gateway alternative: an AI-native LLM gateway with every governance feature free, instead of an AI gateway bundled inside a broader MLOps platform

Head-to-head: NemoRouter vs TrueFoundry AI Gateway. Guardrails, A/B tests, prompt management, evals, and per-team budgets — included on every plan, on an AI-native LLM gateway with 240+ models behind one API key. 4% pay-as-you-go, 0% on Pro. Focused LLM governance depth, not an AI Gateway bundled inside a broader MLOps platform.

2026-06-089 min

NotDiamond alternative: an AI-native LLM gateway with every governance feature free, instead of an ML-trained per-query routing-decision layer that delegates governance to the operator

Head-to-head: NemoRouter vs NotDiamond. Guardrails, A/B tests, prompt management, evals, and per-team budgets — included on every plan, on an AI-native LLM gateway with operator-controlled routing across 240+ models behind one API key. 4% pay-as-you-go, 0% on Pro. Governance-first depth, not an opaque ML-classifier routing layer.