Secure LLM Gateway: API Keys, Tool Calls, and JSON Validation

A secure LLM gateway protects the boundary between your application, model providers, and connected tools. It centralizes API-key management, applies request policies, and gives applications a safer way to handle tool calls and structured JSON responses.

Moving an LLM application into production introduces more risk than simply sending prompts to a model. Production applications may retrieve customer records, query internal systems, update tickets, issue refunds, or send messages. Once an AI application can perform these actions, every model response becomes part of a security-sensitive workflow.

A secure LLM gateway helps centralize the model-access layer. It can provide one controlled API endpoint, authentication, routing, usage controls, guardrails, and observability. The gateway is not a replacement for application authorization: the model can suggest an action, but application code must decide whether that action is allowed.

The problem in one request

Consider a support assistant that can look up an order and recommend a refund. The user asks:

Check order 1842 and refund it if the customer was charged twice.

The model may return a tool call:

{
  "name": "lookup_order",
  "arguments": { "order_id": "1842" }
}

After receiving the order details, the model may suggest another tool call:

{
  "name": "issue_refund",
  "arguments": {
    "order_id": "1842",
    "reason": "duplicate_charge"
  }
}

That workflow contains separate security decisions:

A model response is not authorization. A valid tool call is not proof that the action should execute.

Why direct provider access becomes difficult

A prototype may give every service its own provider credentials:

Checkout service  → Provider key
Support service   → Provider key
Agent service     → Provider key
Background worker → Provider key

This creates operational problems as the system grows:

A gateway reduces the number of provider-facing integration points. It does not remove the need for secure application design. The safer model is to authenticate at the gateway, apply gateway controls, treat model output as untrusted data, validate tool calls in application code, and validate the final JSON response.

How a secure LLM gateway works

The request path should make the trust boundaries visible:

The gateway manages the provider-facing part of the request. The application remains responsible for business authorization and tool execution.

nRouter provides an OpenAI-compatible API in front of major model providers. Teams can use one integration while managing routing, usage controls, and model access through a centralized gateway. For enterprise deployment and governance requirements, see nRouter Enterprise.

The important boundary is the difference between control-plane policy and application behavior. Gateway policy can determine whether a request has a valid key, whether it is within a rate or budget limit, and whether configured guardrails allow it to continue. Application policy determines whether a particular user can read an order, whether a refund amount is acceptable, and whether a tool may change state. Keeping those decisions separate makes reviews easier because each control has one owner and one observable failure mode.

This layered approach also avoids a common mistake: treating a model provider as the place where security policy lives. Providers generate responses. Your gateway and application decide whether the request may leave, which route it may use, and what the application will do with the result.

The production design is therefore a sequence of gates rather than a single claim that the model is secure:

key → limits → guardrails → provider → schema → authorization → action

API-key protection through one gateway

One practical benefit of a secure LLM gateway is reducing provider-key sprawl. The application uses an nRouter key instead of placing separate provider keys in every service:

export NROUTER_API_KEY="sk-nrouter-..."

curl https://api.nrouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $NROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nrouter/auto",
    "messages": [{
      "role": "user",
      "content": "Classify this support request."
    }]
  }'

Use the nRouter API key management guide and authentication guide for the supported workflow.

A production key checklist includes:

1. Store gateway keys in a secret manager.
2. Never commit keys to source control.
3. Use separate keys for separate workloads.
4. Rotate keys when ownership changes.
5. Set budgets and rate limits at the narrowest useful scope.
6. Never print authorization headers in logs.
7. Revoke unused keys.

Tool calls must be authorized by application code

Tool calling lets a model describe an action in a structured format. It does not mean that the model should execute that action directly. nRouter supports tool-calling workflows through its OpenAI-compatible API; the tool-calling guide explains how tools are defined with JSON Schema and executed by the client application.

type ToolCall = {
  name: string;
  arguments: unknown;
};

async function handleToolCall(call: ToolCall, user: User) {
  if (!allowedToolsForUser(user).includes(call.name)) {
    throw new Error("Tool is not allowed for this user");
  }

  const args = validateToolArguments(call.name, call.arguments);

  if (requiresHumanApproval(call.name, args)) {
    return requestApproval(call, user);
  }

  return executeTool(call.name, args, user);
}

Keep an explicit tool registry:

Do not allow the model to invent tool names or expose unrestricted database access through a generic tool. Tools should be explicitly registered, narrowly scoped, schema-defined, permission-checked, auditable, and idempotent where side effects are possible.

JSON responses still need validation

Structured output features make model responses easier to parse, but valid JSON is not automatically safe or correct. A response such as this may be syntactically valid:

{
  "category": "billing",
  "priority": "high",
  "requires_human": true,
  "confidence": 0.91
}

The application should validate required fields, allowed enum values, number ranges, string lengths, nested object structure, unknown fields, tenant identifiers, and business rules.

import { z } from "zod";

const TicketDecision = z.object({
  category: z.enum(["billing", "technical", "account", "other"]),
  priority: z.enum(["low", "medium", "high"]),
  requires_human: z.boolean(),
  confidence: z.number().min(0).max(1),
}).strict();

const decision = TicketDecision.parse(JSON.parse(modelText));

if (decision.requires_human) {
  await sendToHumanReview(decision);
}

The nRouter structured output guide covers the gateway-compatible request format. nRouter provides the consistent API and gateway policy layer; domain-specific validation belongs in the application because only the application knows whether an order, refund, account, or workflow transition is valid.

Validation should be treated as a normal application boundary, not as a recovery trick added after an incident. Keep the schema close to the code that consumes it, version the schema when downstream behavior changes, and test both accepted and rejected examples. A useful test set includes a complete response, a missing required field, an unexpected enum value, a number outside its permitted range, an extra property, and a response that is valid JSON but refers to a different tenant.

That test set also makes model changes safer. When a team changes a prompt, provider, model, or routing rule, it can replay the same fixtures and check whether the contract still holds. The gateway supplies consistent access and telemetry; the application’s validation tests provide the final compatibility check before a response reaches a business system.

Worked example: refund approval

Suppose a support application asks the model to classify a refund request. The expected JSON structure is:

{
  "decision": "approve | reject | review",
  "order_id": "string",
  "reason": "string",
  "amount": "number"
}

A safe process is:

The model may recommend an action, but the order system remains authoritative for the order, amount, and refund status. This distinction prevents a syntactically correct model response from becoming an unauthorized business action.

Edge cases to handle

Unknown tool name

Reject the request. Do not dynamically execute a function based only on a model-generated string.

Valid JSON with invalid business data

A valid order_id does not prove that the current user owns the order. Apply authorization after schema validation.

Repeated side effects

Retries can create duplicate refunds, emails, or updates. Use idempotency keys for actions that change state.

Failed output validation

Do not automatically store raw prompts and responses in logs. Record safe metadata such as request ID, model, route, status, and validation category.

High-impact actions

Require human approval for account deletion, permission changes, high-value refunds, or sensitive communications.

What teams can observe

A secure LLM gateway should expose useful operational metadata without exposing secrets or unnecessary prompt content. nRouter’s observability documentation and request-log guide cover related implementation guidance.

For administrative events such as key changes and permission updates, see Who Rotated That Key?.

Limits of a secure LLM gateway

A secure LLM gateway does not make an application automatically secure. It cannot replace application-level authorization, tenant ownership checks, tool-specific validation, idempotency controls, human approval workflows, secure secret storage, database access policies, dependency security, red-team testing, or incident response.

nRouter provides the controlled model-access layer. The application still decides what each user is allowed to do.

Teams should also distinguish between managed multi-tenant access, dedicated tenancy, and VPC or BYO-cloud requirements. These are different deployment models and should be reviewed against the organization’s security requirements. See nRouter Enterprise for enterprise deployment and governance discussions.

The same boundary applies to compliance language. A gateway can provide useful controls and evidence, but it does not turn a workload into a certification by itself. Procurement teams should ask which controls are active today, what data is retained, who can administer keys, and which deployment model is in scope. nRouter’s security page and trust page should be treated as the current product and compliance references.

For engineering teams, the practical test is whether every important decision has an owner and a failure path. A rejected key should be distinguishable from a blocked prompt. A malformed tool call should be distinguishable from an unauthorized action. A provider error should not be silently converted into a successful business operation. That clarity is what makes the system reviewable after an incident.

Frequently asked questions

What is a secure LLM gateway?

A secure LLM gateway is a controlled service between an application and model providers. It centralizes authentication, routing, usage controls, guardrails, and observability while allowing the application to validate model outputs and authorize tool execution.

Does a secure LLM gateway replace application authorization?

No. The gateway can authenticate the application and apply gateway policies, but the application must verify user permissions, tenant ownership, tool access, and business rules.

Can an LLM execute tools directly?

The model should propose a tool and its arguments. Application code should validate, authorize, and execute the action.

Why is JSON validation necessary?

Valid JSON can still contain incorrect, unsafe, or unauthorized values. Applications must validate both the response structure and business rules.

Does nRouter require separate provider API keys?

nRouter uses an nRouter API key for the application integration. Follow the current authentication documentation and avoid placing provider credentials throughout application services.

Try it

Start with a low-risk, read-only workflow such as ticket classification or order lookup.

1. Create an nRouter key using the API key management guide.
2. Store it securely as NROUTER_API_KEY.
3. Send requests through https://api.nrouter.ai/v1.
4. Define a small, explicit tool registry.
5. Validate tool arguments before execution.
6. Add approval for high-impact actions.
7. Validate the final JSON response.
8. Review request metadata in nRouter observability.

Start with read-only tools, verify the entire request path, and expand gradually to approved write actions.

See also

• Inline LLM Guardrails: Redact, Block, or Flag Every Request — request policies for unsafe input.
• LLM Routing for AI Agents: One Model Per Step, One Key — model routing across agent steps.
• Migrate Off OpenRouter: The Base-URL Swap and What Does Not Map — OpenAI-compatible requests and tool-call migration.
• What an LLM Request Log Should Contain — and What to Leave Out — safe request metadata.
• Who Rotated That Key? An Audit Trail That Answers in One Query — administrative key and permission events.
• nRouter Enterprise — enterprise deployment and governance information.
• nRouter pricing — current usage and rate-limit information.

Sources

Verified 2026-11-10. Product behavior and current limits should be rechecked before publication.

• OWASP GenAI Security Project: LLM01:2025 Prompt Injection
• OpenAI: Function calling guide
• OpenAI: Structured outputs guide
• NIST: AI Risk Management Framework: Generative AI Profile
• Google SRE: Monitoring Distributed Systems
• nRouter: Authentication
• nRouter: API key management
• nRouter: Tool calling
• nRouter: Structured output
• nRouter: Observability
• nRouter: Security
• nRouter: Trust and compliance