How the nRouter Enterprise LLM Gateway Is Implemented. 5 phases in Rust, Cortex mTLS 1.3 & atomic credit ledger.
Every request to api.nrouter.ai/v1/* executes a deterministic 5-phase lifecycle in our Rust data plane (:4000): Gate 0 safety holds, SHA-256 virtual key ACLs, air-gapped Cortex mTLS 1.3 inspection (:7443), atomic dual-pot credit reservation, and exact list-price settlement across 286+ models and 8 provider clouds.
5-Phase Preflight & Settlement Lifecycle
The 5-Phase Inference Lifecycle in Sub-Millisecond Rust
A single shared preflight pipeline enforces security, governance, and funding invariants before any upstream provider socket is opened.
5-Phase Request Lifecycle: Gate 0 Safety Holds -> Cortex mTLS 1.3 -> Atomic Credit Hold -> Provider Settle
Every LLM, audio, image, and video route runs through this exact sequence. Refusals in Gate 0, Phase 1, Phase 2, or Phase 3 exit before Phase 4 — holding $0.00 and spending $0.00.
3840 × 2160 4K Master ↗Gate 0 & Phase 1–2
Edge WAF, SHA-256 Key & ACLs
Validates payload bounds, checks 5-tier kill switches, resolves SHA-256 virtual key in <1ms, and enforces CIDR/route/model ACLs.
Phase 3 · Cortex :7443
Air-Gapped mTLS 1.3 Inspection
Scores moderation, prompt injection, and 21 PII families over mutual TLS 1.3 with zero DB credentials and zero internet egress.
Phase 4 · Atomic Ledger
Dual-Pot Credit Reservation
Checks Org > Team > User > Key ceilings and locks a worst-case hold in Allowance or Credits (Single Pot Invariant M3).
Phase 5 · Egress & Settle
Smart Routing & Exact List Price
Dispatches via Smart Router or Jev System One, fails over on 429/529/503, and settles at exact provider list price (0% fee).
curl https://api.nrouter.ai/v1/chat/completions \
-H "Authorization: Bearer sk-nrouter-prod-9f8a..." \
-H "Content-Type: application/json" \
-d '{
"model": "nrouter/auto",
"messages": [
{"role": "user", "content": "Summarize Q3 cloud margin variance and top 3 cost drivers."}
],
"nrouter_guardrails": ["pii-strict"]
}'HTTP/1.1 200 OK
x-nr-request-id: req_01j9z8k4m2p7x6q5r3v1w0y2z4
x-nr-model: anthropic/claude-sonnet-4-5
x-nr-routing: direct
x-nr-attempts: 1
x-nr-guardrails: pass
x-nr-funding-source: credits
x-nr-request-cost: 0.001420
x-nr-cost-status: exact| Phase | Executing Subsystem | How It Is Implemented in Production | Wire Header / Error Contract | Financial Impact on Refusal |
|---|---|---|---|---|
| Gate 0 & WAF | Pre-Auth Edge WAF & Safety Holds (:4000) | Validates HTTP method, URI bounds, JSON nesting depth, and payload size before touching the database. Evaluates 5-tier emergency kill-switches (Platform freeze -> 503 maintenance; Org/Team hold -> 403). | HTTP 400 / 413 / 415 / 503 · x-nr-auth-reason: maintenance | organization_hold | team_hold | $0.00 held · $0.00 spent |
| Phase 1 & 2 | SHA-256 Virtual Key Auth, ACLs & Rate Slots | Hashes Bearer sk-nrouter-* in <1ms against DashMap cache, verifies IP CIDR allowlists (allowed_ips), canonical route ACLs, model ACLs (auto_model_policy a108), and sliding-window RPM/TPM slots. | HTTP 401 / 403 / 429 + Retry-After · x-nr-auth-reason: key_revoked | ip_not_allowed | model_not_allowed | $0.00 held · $0.00 spent |
| Phase 3 | Cortex Air-Gapped mTLS 1.3 Sidecar (:7443) | Resolves guardrail scope (Key > Team > Org > Floor + add-only nrouter_guardrails) and scores moderation (OP_MODERATE), prompt injection (OP_INJECTION), and 21 PII families over mTLS 1.3 before any credit hold. | x-nr-guardrails: none | monitor | pass | redacted | partial | blocked | unavailable | $0.00 held · $0.00 spent |
| Phase 4 | Two-Tier Atomic Credit Reservation | Evaluates hierarchical spend ceilings (Org > Team > User > Key) and atomically reserves worst-case candidate cost from Tier 1 Allowance or Tier 2 Credits (Cash Pool 1 vs Promo Pool 2 under Single Pot Invariant M3). | HTTP 402 budget_exceeded | HTTP 429 key_budget_exceeded · x-nr-funding-source: allowance | credits | $0.00 spent (refused pre-dispatch) |
| Phase 5 | Smart Routing, Cross-Cloud Failover & Exact Settlement | Selects candidate via Direct, Per-Request Fallbacks, Smart Router, or TypeSafe Jev System One (nrouter/auto). Dispatches to upstream provider, settles at exact served model list price (0% platform fee), and releases unused hold. | x-nr-model · x-nr-routing: direct | fallback:<n> · x-nr-attempts · x-nr-request-cost · x-nr-cost-status: exact | 100% hold released on total upstream failure |
SHA-256 Virtual Keys, CIDR/Route/Model ACLs & 5-Tier Kill Switches
Plaintext sk-nrouter-* secrets are displayed once at creation and never stored. The gateway authenticates SHA-256 digests in <1ms and enforces four independent ACL gates.
Zero-Plaintext Virtual Keys, 4-Dimension ACLs & Sub-10ms Revocation
Every virtual key binds an organization, optional team, IP CIDR allowlist, canonical route ACL, model allowlist, and dedicated RPM/TPM + spend ceiling.
3840 × 2160 4K Master ↗Cortex (:7443) mTLS 1.3 Inspection & 5 Built-In Guardrails
Content moderation, prompt-injection classification, Presidio 21-entity PII redaction, and secret scanning execute inside an air-gapped sidecar before Phase 4 credit holds.
Zero-Egress ML Inspection & Specificity Scope Resolution (Key > Team > Org > Floor)
Under Rule #33, Cortex holds zero database credentials and has zero internet egress. Blocked prompts return HTTP 400/403 before Phase 4 with $0.00 held and $0.00 spent.
3840 × 2160 4K Master ↗Four Model Selection Modes, TypeSafe Jev System One & Cross-Cloud Failover
Keep one OpenAI-compatible client and switch seamlessly between direct model calls, per-request fallback chains, Smart Routers, and nrouter/auto.
Four Ways to Choose a Model & 5 Smart Router Strategies
Configure priority, cost, latency, weight, or intent routing in the dashboard — or send nrouter/auto for sub-400ms per-request selection.
3840 × 2160 4K Master ↗Cross-Cloud Provider Failover & Circuit-Breaker State Machine on a Single Hold
Automatically steps across Anthropic Direct, AWS Bedrock, Google Vertex AI, OpenAI, and Azure Foundry on 429/529/503 or pre-first-byte stall without double-billing.
3840 × 2160 4K Master ↗Two-Tier Funding, Dual Credit Pools & Sub-5ms SHA-256 Response Caching
Zero platform markup (0% permanent standard), absorbed Stripe deposit fees, hierarchical spend caps, and deterministic cache-read settlement.
Two-Tier Funding (Allowance vs Credits), Single Pot Invariant (M3) & Reserve-Forward-Settle
Hard spend caps across Org > Team > User > Key refuse over-budget traffic pre-dispatch with HTTP 402/429. Every call settles at the exact served model list price.
3840 × 2160 4K Master ↗Composite SHA-256 FieldHasher Response Cache & Discounted CacheRead Settlement
Tenant-isolated cache keys ({org_id} hash tag + guardrail policy hash) serve buffered hits in <5ms with $0.00 output token fee.
3840 × 2160 4K Master ↗Postgres NOBYPASSRLS Isolation, Zero-Leak /mcp Proxy & 6-Stage Trace Waterfalls
Every product capability shares one database isolation boundary, one virtual key identity, and one privacy-safe telemetry pipeline.
Dual-Axis 5-Role RBAC & Postgres NOBYPASSRLS Row-Level Security
The Rust gateway connects as nrouter_runtime (NOBYPASSRLS) and pins every transaction to the authenticated organization.
3840 × 2160 4K Master ↗6-Stage Request Trace Waterfall, Metadata-Only Logging & OTLP Export
Correlate client logs to dashboard Gantt waterfalls via x-nr-request-id while storing 0 bytes of prompt or completion text.
3840 × 2160 4K Master ↗Complete Enterprise Product Architecture Directory (12 Modules)
Every module below includes its own 4K architectural blueprint and real wire implementation walkthrough.
Smart Routing & Fallbacks
Four model selection modes, 5 Smart Router strategies (priority, cost, latency, weight, intent), and safe failover status codes.
Automatic Routing (nrouter/auto)
Probabilistic intent classification (<400ms p50, 2s deadline) + 4 structural complexity signals across your org candidate pool.
Cortex mTLS Guardrails
Air-gapped mTLS 1.3 sidecar (:7443), 5 built-in guardrails, Presidio 21-entity PII masking, and 7 x-nr-guardrails posture tokens.
Budgets & Atomic Ledger
Org/Team/User/Key spend ceilings, Pool 1 Cash vs Pool 2 Promo, Single Pot Invariant (M3), and 0% platform fee settlement.
SHA-256 Response Cache
Tenant-isolated FieldHasher SHA-256 keying, single-flight coalescing, and discounted CacheRead billing with $0 output fee.
Virtual Keys & ACLs
SHA-256 hashed sk-nrouter-* keys, CIDR allowlists, route/model ACLs, RPM/TPM ceilings, and <10ms emergency revocation.
6-Stage Observability
Unified x-nr-request-id trace waterfall, metadata-only SpendLogs (zero prompt storage), and OTLP/HTTP export.
FinOps FOCUS 1.4 Analytics
Disjoint 7-bucket token accounting, 3-way reconciliation (Stripe <-> SpendLogs <-> Cloud Invoices), and tag attribution.
Zero-Leak MCP Gateway
Reverse proxy for Model Context Protocol tool servers with server-side vault header injection and sensitive header stripping.
Multi-Tenant Teams & RLS
Dual-axis 5-role RBAC (Owner, Admin, Billing, Developer, Viewer) and database-enforced tenant isolation.
Managed Prompts & A/B
Immutable version history, server-side Jinja2 variable rendering, and deterministic sticky A/B traffic splits.
Streaming Playground
Side-by-side multi-model comparison with live SSE timing badges and instant production SDK snippet export.
Frequently Asked Questions
Direct engineering answers on how the nRouter Rust gateway, Cortex mTLS 1.3 sidecar, and atomic credit ledger operate.
How does the nRouter Rust gateway execute all 5 preflight and settlement phases with sub-millisecond overhead?
The data plane (nrouter-rust-gateway on :4000) keeps hot path lookups in memory: virtual keys are SHA-256 hashed and resolved against an in-process DashMap cache, the servable model catalog and organization ACLs live in lock-free ArcSwap snapshots, and rate-limit slots evaluate without blocking. Only when a request passes Gate 0 through Phase 3 does Phase 4 place a single atomic credit reservation before upstream provider dispatch.
Why does Phase 3 (Cortex mTLS 1.3 guardrails) run before Phase 4 (credit reservation)?
Running Phase 3 before Phase 4 guarantees that any request blocked by pre-call content moderation, prompt-injection detection, or PII/secret policy halts immediately with HTTP 400 or 403, $0.00 held, and $0.00 spent. Your organization never pays provider token fees or locks up credit capacity on rejected attacks.
How is the Cortex ML sidecar (:7443) isolated from customer databases and the public internet?
Under Permanent Rule #33, nrouter-cortex is a stateless gRPC sidecar reachable exclusively from the Rust gateway over mutual TLS 1.3 (:7443). It runs in an air-gapped container with zero database credentials, zero internet egress, and zero disk persistence — inspecting text in memory and returning numeric scores to the gateway.
How does atomic credit reservation prevent both negative balances and double-billing during provider failover?
In Phase 4, the gateway calculates a single worst-case reservation across the candidate chain (for text: estimated prompt tokens + max_tokens at the highest candidate list price) and locks that hold atomically in Postgres. If Rank 0 returns 429, 529, or 503 and the gateway fails over to Rank 1, both attempts share that single reservation. In Phase 5, the gateway settles at the exact list price of the model that actually answered and releases the unused hold.
What is the Single Pot Invariant (M3) across Allowance, Cash Credits (Pool 1), and Promotional Credits (Pool 2)?
Every inference request is funded from exactly one source: Tier 1 plan allowance if active and sufficient, otherwise Tier 2 prepaid credits. Within Tier 2 credits, Pool 2 promotional grants are checked before Pool 1 cash deposits, and Single Pot Invariant (M3) ensures a single request never splits across Pool 2 and Pool 1.
Does nRouter charge a platform fee, routing fee, or Stripe deposit processing fee?
No. nRouter operates with a 0% platform fee and 0% token markup as our permanent standard — every model call settles at the exact published provider list price reported in x-nr-request-cost. On self-serve Stripe top-ups, nRouter absorbs the 2.9% + $0.30 processing fee so a $100.00 payment credits exactly $100.00 to Pool 1 Cash.
Does the gateway store customer prompts or model completions in request logs?
No. Request logs and SpendLogs persist operational metadata only: x-nr-request-id, timestamps, serving model, token breakdown, exact USD cost, latency, routing rank, and guardrail posture token. Raw prompt bodies and completion text are never written to the database.
Deploy on the Enterprise LLM Gateway
One OpenAI-Compatible Endpoint. Every Provider. Zero Platform Markup.
Create a virtual key in 30 seconds, inspect every phase via x-nr-* headers, and enforce hard spend ceilings from day one.