LLM Streaming Failures: Why Mid-Stream Retry Duplicates Output

The answer: an LLM request can be retried safely only before response data reaches the caller. Once a streamed chunk has been delivered, silently starting again can duplicate text, repeat tool-call arguments, and create a second billable generation. At that boundary, nRouter surfaces the interrupted stream and its request ID so the application can decide whether to discard, resume at the workflow level, or ask the user to retry.

Streaming improves perceived latency because the reader sees useful output before the full completion exists. That same property removes a convenience buffered responses have: the gateway can no longer replace a failed attempt invisibly after bytes have crossed the customer boundary. The reliable design is therefore not “retry every network error.” It is “classify the failure by whether output escaped.”

This deep dive explains that contract for OpenAI-compatible server-sent events, how it relates to LLM Latency: p50, p95, p99, and Time-to-First-Token, and why provider fallback chains can protect the beginning of a stream but not rewrite its middle.

The problem in one request

Consider a support assistant asked to produce a short incident summary. The application renders each delta as soon as it arrives:

request accepted
  → chunk: "The outage began"
  → chunk: " after the primary"
  → chunk: " provider stopped"
  → connection closes

Nothing tells the browser whether the provider generated more text that failed in transit, stopped generation entirely, or completed while the downstream connection disappeared. The client knows only that it received a prefix and never received the terminal marker.

A generic retry wrapper starts the same request again. The second generation begins:

The outage began when the primary provider...

If the application appends that retry to the existing buffer, the user sees a duplicated opening. If it clears the buffer, the user watches already-visible content disappear. If the completion contains tool calls, replaying can be worse than duplicated prose: the application can parse a partial argument object, then receive a different complete object from the retry.

The failure is not that streaming uses server-sent events. The failure is treating “connection closed” as enough information to prove that replay is safe.

Why the naive approach breaks

The common implementation has one catch block around the whole iterator:

async def generate_with_retry(client, request, attempts=2):
    for attempt in range(attempts):
        try:
            stream = await client.chat.completions.create(**request, stream=True)
            async for chunk in stream:
                yield chunk.choices[0].delta.content or ""
            return
        except Exception:
            if attempt + 1 == attempts:
                raise

That loop knows how many attempts ran, but not whether any attempt produced externally visible work. The yield is the important line: after it executes, the function has transferred ownership of the prefix to its caller. A later exception cannot retract it.

HTTP also gives no general permission to replay this POST. RFC 9110’s idempotency rules explain why automatic retry is straightforward for idempotent methods and unsafe for non-idempotent requests unless the client has additional knowledge. An LLM generation is not merely a database mutation, but the same uncertainty applies: the first attempt may have produced output and cost even when the client missed its end.

Three naive assumptions fail:

1. “The socket closed, so the provider did no work.” A connection failure reports transport state, not provider billing state.
2. “The same prompt produces the same continuation.” Generated text can differ between attempts, so byte-offset concatenation is not semantic resumption.
3. “SSE reconnect means generation resume.” The HTML event-stream format supports event IDs and reconnection, but a chat-completion stream does not thereby promise resumable model generation. The WHATWG event-stream specification defines transport behavior, not application-level replay semantics.

The fix is to track whether output crossed the boundary and make that state part of the retry decision.

The mechanism

nRouter separates failures into two customer-visible phases: before delivery and after delivery. The distinction is intentionally observable rather than tied to private implementation details.

authenticate → apply controls → select eligible route → connect upstream
                                                     │
                          safe failover window ──────┤
                                                     ▼
                                      response starts / chunk exits
                                                     │
                          surface interruption ──────┘

The contract is:

1. The request passes authentication, spend controls, rate limits, and guardrails before provider work begins. The ordering of those controls is covered in Credits, Budgets, Rate Limits, Guardrails.
2. A configured router alias may advance to another eligible route when the failure is known to occur before output delivery and the saved policy permits it. A concrete model request has no hidden fallback chain; see Router Settings.
3. Once a response chunk is delivered, nRouter does not splice a new provider generation behind the prefix. The stream either completes or surfaces as interrupted.
4. The application retains the shared x-nr-request-id for correlation. It should record whether it displayed, persisted, or executed anything from the partial stream.
5. Recovery happens at the application boundary, where “discard and regenerate,” “keep as a draft,” and “require confirmation” have product meaning.

This is deliberately narrower than promising exactly-once generation. The gateway can prevent a hidden mid-stream reroute; it cannot make a caller’s side effects idempotent. If an application executes a tool call before the stream is complete, it owns the deduplication key and commit boundary for that tool.

The OpenAI streaming reference notes that usage totals can arrive in a final chunk and may be missing when a stream is interrupted. That is why absence must remain absence: an unavailable final usage event is not evidence of zero usage or zero cost. nRouter follows the same truth principle described in one authoritative cost per request: an unknown value is not rewritten as $0.

Worked example

Use a deliberately small hypothetical request to see the arithmetic. This is an example of client behavior, not a production performance claim.

Suppose the intended answer contains 120 generated tokens. The first stream delivers 45 tokens before the connection fails. A blind retry produces a fresh 120-token answer.

The unsafe arithmetic is 45 + 120 = 165 handled token fragments for one intended 120-token answer. That does not mean the bill is exactly 165 tokens; the interrupted attempt’s final usage may be unavailable. It means the application has processed more output than the user asked for and cannot infer a clean continuation point.

A safer client buffers by default and promotes chunks only according to product policy:

async def collect_stream(client, request):
    parts: list[str] = []
    exposed = False

    try:
        stream = await client.chat.completions.create(**request, stream=True)
        request_id = stream.response.headers.get("x-nr-request-id")

        async for chunk in stream:
            text = chunk.choices[0].delta.content or ""
            parts.append(text)

            # Set this to True only if your UI or workflow consumes the prefix.
            exposed = exposed or publish_delta(text)

        return {"status": "complete", "text": "".join(parts), "request_id": request_id}

    except Exception as exc:
        return {
            "status": "partial" if exposed else "failed_before_display",
            "text": "".join(parts),
            "error": str(exc),
        }

The code does not auto-retry. It returns a state the product can interpret. A batch job may discard partial output and enqueue a new generation. A chat UI may show “generation interrupted” beside the retained prefix. An agent that has already acted on a tool call should stop and reconcile the side effect before doing anything else.

Edge cases we had to decide

Each boundary below names the failure, the behavior, and the reason.

1. When the upstream connection fails before any response chunk exits, nRouter may advance a configured route, because no partial answer has crossed the customer boundary. The saved route policy still decides whether another candidate is eligible; the mere existence of another model does not create implicit failover.

2. When a provider fails after a content chunk exits, nRouter surfaces the interrupted stream instead of silently switching providers, because a new generation cannot be guaranteed to continue the same text. This prevents hidden duplication and mixed-provider answers.

3. When the client disconnects after receiving content, nRouter cannot promise that upstream work stopped at the same instant, because cancellation and network failure propagate asynchronously. The caller must not translate “I stopped reading” into “nothing was generated or billed.”

4. When the final usage chunk is missing, nRouter leaves cost unavailable rather than reporting zero, because missing settlement evidence is not proof of a free request. The x-nr-request-cost header can therefore be absent; branch on presence rather than coercing it. The broader rule is explained in Cost Honesty: Unpriced Is Never $0.

5. When partial output contains a tool call, the application must not execute and then blindly replay it, because a retry can produce the same side effect twice or produce different arguments. Commit tool execution only after complete validation, and give side-effecting tools their own idempotency controls. AWS retry guidance makes the same general point: retry safety depends on idempotency, not only backoff.

These choices optimize for a debuggable failure over a seamless-looking corruption. A visible interruption can be retried deliberately. A duplicated action can be much harder to unwind.

What you see from the outside

The application should retain four pieces of evidence for every streamed request:

The protocol itself is ordinary SSE: UTF-8 events separated by blank lines, as documented by MDN’s SSE guide. nRouter’s Chat Completions reference shows the chunk shape and terminal marker.

Operationally, the useful log entry is not “stream failed.” Record a compact state machine:

{
  "request_id": "from-x-nr-request-id",
  "stream_state": "partial",
  "chunks_received": true,
  "done_received": false,
  "output_committed": false,
  "retry_decision": "manual-or-workflow-level"
}

That record pairs naturally with an LLM request log that excludes sensitive content. You can diagnose the failure without copying the prompt or generated text into every log destination.

Limits

This contract does not make streaming resumable, deterministic, or exactly once.

Streaming is also not always the right mode. If the result must be validated as a complete JSON object before any consumer sees it, buffering may be the simpler reliability choice. If perceived responsiveness matters more, stream—but design the UI and workflow around a possible partial state.

The trade-off is visible in LLM Latency: p50, p95, p99, and Time-to-First-Token: streaming improves the first visible moment without shortening every part of the request. Reliability requires measuring both the start and the completion boundary.

Try it

Use the Playground or an OpenAI-compatible client configured with https://api.nrouter.ai/v1. Send a streaming request, log x-nr-request-id, and track two booleans: whether any content was displayed and whether [DONE] arrived.

Then test your application-level state table:

1. No chunks and no completion: classify as failed before display.
2. Chunks without completion: classify as partial and do not silently append a retry.
3. Chunks with completion: commit the assembled result.
4. A partial tool call: stop before execution and require a validated complete call.

Start with the streaming request shape in the API reference, then use Router Settings if you want an explicit alias and pre-delivery failover policy.

See also

• Provider Fallback Chains: Surviving an OpenAI Outage — learn which pre-delivery failures can advance a configured route and why mid-stream failure cannot.
• LLM Latency: p50, p95, p99, and Time-to-First-Token — separate first-token responsiveness from full-response completion and tail latency.
• One Authoritative Cost Per LLM Request, Across Providers — understand why missing usage or cost remains unknown instead of becoming zero.
• What an LLM Request Log Should Contain — and What to Leave Out — capture request IDs and stream states without leaking prompt content.
• 429 vs 402 on an LLM Gateway: Which to Retry, Which to Stop — build the complementary status-code retry branch for requests rejected before streaming starts.
• Models — choose a currently served model before testing the streaming failure path.

Sources

Verified 2026-08-26. If a source or behavior has changed, email hello@nrouter.ai and we will update this article.

• HTTP retry and idempotency semantics: RFC 9110 §9.2.2.
• Server-sent event format and reconnection model: WHATWG HTML Standard.
• Practical SSE parsing and error handling: MDN Web Docs.
• Streaming chunks and interrupted final usage: OpenAI Chat Completions API reference.
• Typed streaming lifecycle events: OpenAI Responses streaming reference.
• Retry safety depends on idempotency: AWS Prescriptive Guidance.