Semantic Cache. Zero duplicate spend.
Eliminate redundant LLM inference calls with sub-millisecond exact and semantic caching built directly into our pure Rust gateway core. Slash response latency from seconds to under 5ms, eliminate 100% of token spend on hits with zero broker markup, and preserve strict multi-tenant isolation. Read our pure Rust engine benchmark.
Edge Cache Acceleration
- Token spend on cache hits
- $0.00
- P50 hit latency
- < 5 ms
- Supported modalities
- Text & Embeddings
- Broker markup on hits
- $0.00
100% token savings on cached traffic
Pure Rust in-memory exact & semantic retrieval
All OpenAI-compatible model routes
Zero broker markup; exact $0 cost on hits
Hierarchical exact and semantic evaluation
Incoming prompts pass through L1 in-memory exact hash checking and L2 semantic vector similarity before any upstream model egress is attempted.
Intelligent Response Caching Architecture
Inbound Request
Prompt + Model Parameters
Gateway intercepts request and derives cache lookup keys (tenant ID, model, prompt hash).
Exact Hash Check
<1ms Local In-Memory
Direct SHA-256 match against active memory / Redis cache. Returns instant hit if found.
Semantic Evaluation
Vector Similarity Threshold
Evaluates cosine distance between prompt embeddings. Matches intent within configured epsilon.
Zero-Cost Return
x-nr-cache: HIT · $0.00
Returns cached output stream in <5ms. Provider credit hold is released with $0 charged.
Dramatic cost reduction for repetitive and conversational workloads
Performance & Savings
Turn high-volume customer queries into zero-cost instant responses
Documentation assistants, customer support chatbots, and programmatic evaluation runs frequently answer identical or near-identical questions. With nRouter semantic caching, pay once and serve thousands of repeat requests with zero extra provider egress.
- Configurable semantic threshold (0.90 to 0.99 cosine similarity)
- Automatic normalization of whitespace, casing, and JSON formatting
- Full support for streaming completions with instant buffer replay
- Granular TTL controls and instant programmatic cache purging
Live Production Cache Telemetry
Full response cache technical reference — algorithms, storage & headers
- Dual-mode caching: Exact + Semantic
- Exact caching matches normalized prompt strings and inference hyperparameters (temperature, top_p, model). Semantic caching computes embedding cosine similarity, serving identical answers to rephrased queries within a configurable similarity score (e.g. >= 0.96).
- 100% token cost elimination ($0 spent)
- Cache hits bypass external model provider endpoints entirely. No upstream egress occurs, no provider API tokens are consumed, and zero broker markup applies. You pay $0.00 for every served cache hit.
- Sub-5ms global response latency
- Instead of waiting seconds for frontier LLM reasoning tokens, cached completions are streamed or delivered in single-digit milliseconds directly from edge and in-memory caches.
- Strict multi-tenant data isolation
- Cache keys are cryptographically partitioned by organization ID and optional team/key boundaries. Organization A can never inspect or retrieve cached completions belonging to Organization B.
- Configurable TTLs & programmatic invalidation
- Define custom time-to-live (TTL) policies per model, route, or key (from 60 seconds to 30 days). Invalidate cached entries on demand via REST API or dashboard whenever reference data changes.
- Observability & x-nr-cache headers
- Every gateway response reports cache status in standard HTTP headers (x-nr-cache: HIT, MISS, or BYPASS) along with precise latency savings and original generation timestamps.
- Enterprise compliance & SOC 2
- Cache storage keyspaces are cryptographically isolated per tenant organization ID. Combined with our SOC 2 Type II audit in progress, cached completions adhere to strict enterprise data isolation standards.
Frequently asked questions about response caching
How does semantic caching differ from exact caching?
Exact caching requires character-by-character identity of the input prompt, model, and generation parameters. Semantic caching computes high-dimensional vector embeddings of the user prompt and returns cached answers when cosine similarity exceeds your configured threshold (e.g. 0.95), handling spelling differences and phrasing variations.
Are cache hits charged against my credit balance?
No. Cache hits bypass the LLM provider completely and cost $0.00. Preflight credit reservations are instantly released without deducting from your prepaid balance or plan allowance.
How do you prevent data leaks between different tenants in the cache?
The cache partition key embeds the cryptographic tenant organization ID and customer-configured salt. Cache lookup is physically scoped to your tenant. It is mathematically impossible for another tenant to query or retrieve your cached prompts or responses.
Can I bypass the cache for specific requests?
Yes. Clients can bypass the cache at any time by including standard cache control headers (e.g., Cache-Control: no-cache) or passing "no_cache": true in the extra_body JSON payload.
What models and modalities support response caching?
All text, chat completion, and embedding models across all providers (OpenAI, Anthropic, Google Vertex, AWS Bedrock, Azure AI) support exact and semantic response caching.
How do I invalidate outdated cache entries?
You can invalidate cache keys by prompt prefix, model ID, or tag via the nRouter Dashboard or the /v1/cache/purge REST endpoint, with immediate propagation across edge nodes.
Pure Rust Performance · Zero Broker Markup
Accelerate your AI applications with semantic caching
Start serving repeat requests in under 5ms while eliminating duplicate token spend. Zero broker markup, SOC 2 Type II audit in progress, and simple drop-in configuration with zero SDK changes.