Notes on shipping reliable LLM infrastructure.

Architecture & internals·15

Features & releases·12

Engineering

Comparison

How we compare·12

How-tos & best practice·16

Featured · Latest deep-dive

Featured· Engineering

LLM Streaming Failures: Why Mid-Stream Retry Duplicates Output

Streaming makes responses feel fast, but it creates a hard recovery boundary. Learn when an LLM request can fail over safely, why emitted tokens cannot be replayed silently, and how clients should handle partial output.

2026-08-2611 min

All posts

Browse the catalog

Filter by category, or browse the full archive of deep-dives.

64 posts found

Company

Four Outcomes That Get an LLM Gateway Funded

A feature matrix does not survive a finance review. The internal case for an LLM gateway is four measurable outcomes — total spend, unit cost, performance under load, and shipping speed — each with a number attached.

2026-06-2410 min

Budget Ceilings That Turn AI Spend Into a Forecast

A forecast is only a forecast if the worst case is bounded. Set hard dollar ceilings at the org, team, user and key scope, decompose next month's number into them, and know exactly which code your client gets when one bites.

Ship Your First AI Feature: Signup to Production in an Afternoon

A step-by-step path from a funded nRouter account to a production LLM call with a spend ceiling on it — create a scoped key, point your existing OpenAI client at one base URL, cap the key, and verify the cost header came back.

Latency Routing and Automatic Failover Without Retry Code

Point a router alias at a candidate set with the Latency strategy, keep the automatic cross-provider failover, tune the two settings that decide how fast a bad provider is abandoned, and prove the effect on your own p95.

Cut LLM Costs by Routing, Not by Rewriting Your App

Routing is the one cost lever you can pull from a dashboard. Point an alias at a set of models, choose cost or latency or weighted, and change what a request costs without touching a line of application code.

One API Key Across Providers: The Integration You Stop Writing

Provider breadth behind a single OpenAI-compatible key. What the swap changes in your codebase, what the model string buys you, which providers are live today, and the integration work that stops being yours to maintain.

Engineering

5% of Requests, 60% of the Bill: Reading Cost Against Usage

Request count and dollar cost tell different stories, and the gap between them is where the savings are. Here are the four shapes an overlay of cost and usage produces, which one to chase first, and what makes the numbers trustworthy enough to act on.

2026-06-1511 min

Set Up LLM Log Callbacks: Datadog, Langfuse, S3, Slack

Configure log destinations once at the gateway instead of instrumenting every call site. Here is how to add and verify a callback today, what the Beta does and does not deliver yet, and the live paths that get data out in the meantime.

2026-06-1510 min

Engineering

Write-Time PII Redaction in LLM Logs, Without Losing Debug Detail

"Redact the logs and you cannot debug" is a false trade. Here is how write-time redaction, typed structure-preserving placeholders and metadata scrubbing keep LLM request logs reproducible while the sensitive values never reach durable storage.