Instant completions. Zero duplicate spend.
Eliminate redundant LLM inference calls with sub-millisecond exact and semantic caching. Slash response latency from seconds to under 5ms, preserve 100% of your token budget, and maintain strict multi-tenant data isolation.
Edge Cache Acceleration
- Token spend on cache hits
- $0.00
- P50 hit latency
- < 5 ms
- Supported modalities
- Text & Embeddings
- Tenant isolation
- 100%
100% token savings on cached traffic
In-memory exact & semantic retrieval
All OpenAI-compatible model routes
Cryptographically partitioned keyspaces
Hierarchical exact and semantic evaluation
Incoming prompts pass through L1 in-memory exact hash checking and L2 semantic vector similarity before any upstream model egress is attempted.
Intelligent Response Caching Architecture
Inbound Request
Prompt + Model Parameters
Gateway intercepts request and derives cache lookup keys (tenant ID, model, prompt hash).
Exact Hash Check
<1ms Local In-Memory
Direct SHA-256 match against active memory / Redis cache. Returns instant hit if found.
Semantic Evaluation
Vector Similarity Threshold
Evaluates cosine distance between prompt embeddings. Matches intent within configured epsilon.
Zero-Cost Return
x-nr-cache: HIT · $0.00
Returns cached output stream in <5ms. Provider credit hold is released with $0 charged.
Dramatic cost reduction for repetitive and conversational workloads
Performance & Savings
Turn high-volume customer queries into zero-cost instant responses
Documentation assistants, customer support chatbots, and programmatic evaluation runs frequently answer identical or near-identical questions. With nRouter semantic caching, pay once and serve thousands of repeat requests for free.
- Configurable semantic threshold (0.90 to 0.99 cosine similarity)
- Automatic normalization of whitespace, casing, and JSON formatting
- Full support for streaming completions with instant buffer replay
- Granular TTL controls and instant programmatic cache purging
Live Production Cache Telemetry
Full response cache technical reference — algorithms, storage & headers
- Dual-mode caching: Exact + Semantic
- Exact caching matches normalized prompt strings and inference hyperparameters (temperature, top_p, model). Semantic caching computes embedding cosine similarity, serving identical answers to rephrased queries within a configurable similarity score (e.g. >= 0.96).
- 100% token cost elimination ($0 spent)
- Cache hits bypass external model provider endpoints entirely. No upstream egress occurs, no provider API tokens are consumed, and no platform markup applies. You pay $0.00 for every served cache hit.
- Sub-5ms global response latency
- Instead of waiting seconds for frontier LLM reasoning tokens, cached completions are streamed or delivered in single-digit milliseconds directly from edge and in-memory caches.
- Strict multi-tenant data isolation
- Cache keys are cryptographically partitioned by organization ID and optional team/key boundaries. Organization A can never inspect or retrieve cached completions belonging to Organization B.
- Configurable TTLs & programmatic invalidation
- Define custom time-to-live (TTL) policies per model, route, or key (from 60 seconds to 30 days). Invalidate cached entries on demand via REST API or dashboard whenever reference data changes.
- Observability & x-nr-cache headers
- Every gateway response reports cache status in standard HTTP headers (x-nr-cache: HIT, MISS, or BYPASS) along with precise latency savings and original generation timestamps.
Frequently asked questions about response caching
How does semantic caching differ from exact caching?
Exact caching requires character-by-character identity of the input prompt, model, and generation parameters. Semantic caching computes high-dimensional vector embeddings of the user prompt and returns cached answers when cosine similarity exceeds your configured threshold (e.g. 0.95), handling spelling differences and phrasing variations.
Are cache hits charged against my credit balance?
No. Cache hits bypass the LLM provider completely and cost $0.00. Preflight credit reservations are instantly released without deducting from your prepaid balance or plan allowance.
How do you prevent data leaks between different tenants in the cache?
The cache partition key embeds the cryptographic tenant organization ID and customer-configured salt. Cache lookup is physically scoped to your tenant. It is mathematically impossible for another tenant to query or retrieve your cached prompts or responses.
Can I bypass the cache for specific requests?
Yes. Clients can bypass the cache at any time by including standard cache control headers (e.g., Cache-Control: no-cache) or passing "no_cache": true in the extra_body JSON payload.
What models and modalities support response caching?
All text, chat completion, and embedding models across all providers (OpenAI, Anthropic, Google Vertex, AWS Bedrock, Azure AI) support exact and semantic response caching.
How do I invalidate outdated cache entries?
You can invalidate cache keys by prompt prefix, model ID, or tag via the nRouter Dashboard or the /v1/cache/purge REST endpoint, with immediate propagation across edge nodes.
Immediate Speed & Savings
Accelerate your AI applications with semantic caching
Start serving repeat requests in under 5ms while eliminating duplicate token spend. Simple drop-in configuration with zero SDK changes.