Streaming Connection Dropped — Fixing SSE Stream Errors

Tokens flow for a while, then the connection dies mid-sentence: stream disconnected, peer closed connection, or an EventSource error. Here is why SSE streams from the OpenAI and Claude (Anthropic) APIs drop, and how to fix each cause.

Quick answer

A dropped SSE stream is almost always an intermediary problem — a proxy, load balancer, or firewall closing what it thinks is an idle connection — not the model failing. First fix: retry with exponential backoff, keeping the partial text so a late drop only needs a "continue" follow-up. If you run your own reverse proxy, disable response buffering and raise read timeouts; for very long unattended generations, go non-streaming or split the task.

Symptoms and What They Mean

You are in this failure mode if you see:

• httpx.RemoteProtocolError: peer closed connection without sending complete message body
• openai.APIConnectionError / Stream ended prematurely (Python SDK)
• Anthropic SDK: an incomplete read mid-stream, or an overloaded_error event
• Browser: net::ERR_INCOMPLETE_CHUNKED_ENCODING or EventSource firing onerror
• The text just stops mid-sentence and no [DONE] / message_stop event ever arrives

The common thread: the HTTP connection closed before the stream signaled completion. The generation itself may have finished fine — the bytes simply never reached you.

Causes

1. Idle timeouts on an intermediary. SSE streams sit silent between tokens (extended thinking, tool use, a slow first token), and load balancers, corporate firewalls, and NAT routers often kill connections after 30–120 quiet seconds.
2. Reverse proxy buffering. nginx and similar proxies buffer responses by default; buffered SSE arrives as one late burst or times out before the buffer flushes.
3. CDN limits. Cloudflare and other CDNs enforce their own idle timers (roughly 100 seconds without bytes on Cloudflare, surfacing as a 524).
4. Client read timeouts. Default read timeouts are often shorter than the gap before a reasoning model emits its first token.
5. Long outputs. The longer a stream lives, the higher the odds that some hop resets it — multi-minute generations fail disproportionately often.

Fixes

1. Retry with exponential backoff — and keep the partial text

Streams cannot be resumed server-side, so retry the request — but if the drop happened late, keep the partial text and ask the model to continue:

import time
from openai import OpenAI, APIConnectionError, APITimeoutError

client = OpenAI(api_key="sk-YOUR_TOKEN", base_url="https://byesu.com/v1")

def stream_with_retry(messages, retries=3):
    text = ""
    for attempt in range(retries + 1):
        try:
            stream = client.chat.completions.create(
                model="grok-4.5", messages=messages, stream=True,
            )
            for chunk in stream:
                text += chunk.choices[0].delta.content or ""
            return text
        except (APIConnectionError, APITimeoutError):
            if attempt == retries:
                raise
            time.sleep(2 ** attempt)  # 1s, 2s, 4s
            if text:  # resume instead of regenerating from zero
                messages = messages + [
                    {"role": "assistant", "content": text},
                    {"role": "user", "content": "Continue exactly where you stopped. Do not repeat anything."},
                ]
    return text

The same pattern works on the Anthropic-native endpoint — accumulate content_block_delta events and re-prompt on failure.

2. Raise client timeouts and use connection keep-alive

The read timeout must cover the longest gap between bytes, not the total response time:

import httpx
from openai import OpenAI

client = OpenAI(
    api_key="sk-YOUR_TOKEN",
    base_url="https://byesu.com/v1",
    timeout=httpx.Timeout(connect=10.0, read=300.0, write=30.0, pool=10.0),
    max_retries=2,  # SDK-level retry for connection errors
)

Reuse one client instance so pooled connections stay warm — repeated TLS handshakes add latency and multiply the chances of hitting a flaky path. Where possible, enable TCP keep-alive at the socket level so NAT devices see periodic traffic during quiet stretches.

3. nginx: turn off buffering, raise read timeouts

If your own nginx sits anywhere on the path, the defaults will break streaming:

location /v1/ {
    proxy_pass          http://upstream_api;
    proxy_http_version  1.1;
    proxy_set_header    Connection "";
    proxy_buffering     off;      # forward SSE chunks immediately
    proxy_cache         off;
    proxy_read_timeout  3600s;    # long gaps between tokens are normal
    proxy_send_timeout  3600s;
    chunked_transfer_encoding on;
}

Alternatively, have your app send the X-Accel-Buffering: no response header on streaming routes — it disables buffering per-response without touching global config.

4. Cloudflare and other CDNs

Cloudflare proxies SSE fine as long as bytes keep flowing, but ~100 seconds of silence triggers a 524. If you control the origin:

• Emit SSE comment heartbeats (: ping\n\n) every 15–30 seconds during silent phases — clients ignore comments, but they reset every idle timer on the path.
• Serve the API hostname in DNS-only (grey-cloud) mode so streams skip the CDN entirely.

If you are only a consumer of an API behind a CDN, fall back to fix #1.

5. Very long outputs: segment the task or go non-streaming

For unattended batch jobs, "stream": false removes every mid-stream failure mode: one response, one status code, and standard HTTP retries just work. Two caveats:

• Some providers require streaming above a certain requested output length, precisely because long non-streaming requests tie up connections for minutes.
• Either way, a single multi-minute generation is fragile. The robust pattern is segmentation: ask for an outline, then generate section by section with a modest max_tokens per call. Each segment completes in seconds and retries independently.

Streaming Through byesu

byesu is an AI API gateway with both OpenAI-compatible (/v1/chat/completions) and Anthropic-native (/v1/messages) endpoints, and "stream": true works on both. SSE chunks are forwarded as they arrive from the upstream provider, so the gateway adds no buffering stage of its own. Two properties help with dropped streams specifically: a stream that fails before the first token usually succeeds on retry, because the retry can land on a different upstream channel; and since one key covers Claude, GPT-5.6, Grok and Gemini, a model whose upstream is having a bad hour can be swapped by changing a single model string. Billing is pay-as-you-go per token, so an interrupted stream only bills the tokens actually generated — see your live billed price in the console.

FAQ

Why does my OpenAI API stream keep disconnecting?

Most drops are caused by an intermediary — a reverse proxy, load balancer, CDN, or corporate network appliance — closing an HTTP connection it considers idle, or by response buffering that stalls the SSE stream until it times out. The model itself rarely fails mid-generation. Raise client read timeouts, disable proxy buffering on any hop you control, and add automatic retry with exponential backoff.

How do I stop nginx from breaking SSE streaming?

Set proxy_buffering off and proxy_cache off, raise proxy_read_timeout well above the longest expected gap between tokens, and use HTTP/1.1 with an empty Connection header. If nginx fronts your own app, also send the X-Accel-Buffering: no response header. Buffered SSE either arrives in one late burst or times out entirely.

Should I retry a dropped stream from scratch or resume it?

Chat completion streams cannot be resumed server-side, so you retry the whole request — but keep the partial text you already received. If the drop happened late in a long answer, append the partial output as an assistant message and ask the model to continue exactly where it stopped, instead of regenerating everything.

Does Cloudflare drop SSE connections?

Cloudflare proxies SSE, but it enforces an idle timeout of roughly 100 seconds without bytes, returning a 524 when it triggers. Long silent gaps — extended thinking, a slow first token — are the usual cause. If you control the origin, keep bytes flowing with SSE comment heartbeats every 15 to 30 seconds, or serve the API hostname in DNS-only mode.

When should I use non-streaming instead of streaming?

For unattended batch jobs where nobody is watching tokens arrive, non-streaming is simpler: one response, one status code, no mid-stream failure modes. Note that some providers require streaming for very long generations, so for very long outputs the most reliable pattern is splitting the task into segments rather than one giant request.

Does byesu support streaming for all models?

Yes. Both the OpenAI-compatible /v1/chat/completions endpoint and the Anthropic-native /v1/messages endpoint accept stream: true, and SSE chunks are forwarded as they arrive from the upstream provider. If a stream fails before the first token, retrying usually succeeds because the gateway can route the new attempt to a different upstream channel.