Stream Responses

Streaming returns a model's answer as it is produced instead of all at once — the reader sees the first words in well under the full generation time, which is what makes chat interfaces and agent loops feel responsive. On nRouter it is one field on the request you already send.

Turn it on

Set "stream": true on a Chat Completions request (the Responses endpoint takes the same boolean). The gateway opens an HTTP chunked connection immediately and sends the answer as server-sent events (SSE). With the Python SDK:

from nroutersdk import nRouter

client = nRouter()

stream = client.chat.completions.create(
    model="gpt-5.4-mini",
    messages=[{"role": "user", "content": "Write a 3-paragraph summary of quantum computing."}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

The TypeScript client iterates the same stream with for await. The Python SDK page also links a complete streaming demo.

What arrives

Each event is a data: line carrying one delta — a small piece of the answer — until a final frame closes the completion and a [DONE] line ends the stream. The sequence documented in the reference:

data: {"id":"chatcmpl-98a","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-98a","choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]}

data: {"id":"chatcmpl-98a","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Append each delta.content in order and you hold the full answer — the same text a non-streamed request would have returned whole.

Telemetry and billing on a stream

Two behaviours are specific to streams, both documented in the reference:

• Latency means time to first byte. On a stream, the x-nr-latency-ms header measures the time from the request reaching the edge until the initial response headers are flushed — not the total generation time of the stream.
• Billing settles at the end. Spend is computed and committed atomically at end-of-stream, from the tokens actually emitted — a streamed request is billed on real usage, never on an estimate taken before generation.

Everything else behaves as it does on a whole-response request: the served model is named in x-nr-model (and in every streamed frame when nrouter/auto chose it), and the request lands in your logs with its exact cost.

Related guides

• Use Tool Calling — Streams inside agent loops: calls and answers as they form
• Get Structured Output — Streamed answers in a strict shape, through Instructor
• Use Reasoning Models — Long, worked-through answers worth streaming

Frequently asked questions