Streaming LLM Responses in Web Apps Without Making It Fragile

Streaming LLM responses is one of those things that looks trivial in a demo and turns out to be surprisingly annoying in production. The token-by-token effect is the reason people tolerate AI features feeling slow, so it's worth getting right.

This is a working guide to streaming LLM responses in web apps — how the transport actually works, where teams get it wrong, and the small details that decide whether a streamed AI UI feels responsive or feels haunted.

Why streaming matters more than raw speed

An LLM response that takes eight seconds to arrive as a single blob feels broken. The same response, streamed, feels acceptable at fifteen seconds. Perceived latency is a different problem than actual latency, and streaming is how you solve it.

The user's brain reads at 200 words per minute. If the model is producing 40 tokens per second, streaming gives you a UI that's always ahead of the reader. That's the whole trick.

The transport, briefly

Almost all LLM providers support streaming over Server-Sent Events. SSE is a one-way stream from server to client over plain HTTP. It's not WebSockets and it doesn't need a special server. Anything that can hold a response open works.

The typical flow: your client calls your backend, your backend calls the LLM with stream: true, and your backend forwards chunks to the client as they arrive. Don't call the LLM from the browser directly. You'll leak the API key, lose the chance to add rate limits, and lose the chance to log anything.

What the chunks look like

Each SSE event is a line prefixed with data: and terminated with a blank line. The body is usually JSON with a delta field. A well-behaved client parses the JSON, extracts the delta, and appends it to the visible response.

Watch out for two things. First, chunks can split mid-token or mid-JSON — buffer until you see a full event boundary. Second, the last event is usually a sentinel like [DONE], not JSON. Handle it explicitly instead of letting JSON.parse throw.

Where teams get streaming wrong

Rendering on every token

A naive React implementation calls setState on every token. For a long response that's thousands of renders. On slower devices you'll see the UI stutter or the tab go warm.

The fix is small. Accumulate tokens in a ref and flush to state on a requestAnimationFrame tick. You keep the streaming feel and stop doing pointless work.

Not handling disconnects

Users close tabs. Networks flap. If your backend keeps generating tokens after the client is gone, you're paying for output nobody will see. Propagate the client's abort to the LLM SDK. Every mainstream SDK supports an AbortSignal. Wire it up.

Losing the response on refresh

A common pattern we see: the user refreshes mid-stream and the entire response is gone. If the response matters (a summary, a draft, an answer), persist it as tokens arrive. Write to your database on the server, not on the client. That way the user's refresh reads the partial response instead of starting from nothing.

Middleware that buffers the whole response

Some proxies and edge platforms buffer responses by default and only forward once the upstream closes. That defeats streaming completely. Test in production, not just locally. If you see all tokens arrive at once at the very end, the culprit is almost always a buffering layer between your server and the browser.

A minimal working shape

On the server, open the LLM stream, iterate the async iterator, and write each delta to the response as an SSE event. Flush after every write. Close the response when the stream ends or the client disconnects.

On the client, use fetch with a ReadableStream reader. EventSource is convenient but only supports GET and doesn't let you send the prompt as a body — for anything non-trivial, prefer the fetch approach and parse SSE frames manually. It's about 30 lines.

Cancellation, correctly

Streaming without cancellation is a bug waiting to bill you. Three cancellation paths matter:

The user clicks stop. Wire a button to abortController.abort(). The client's fetch tears down. The backend sees the socket close and aborts the LLM call.

The user navigates away. Same abort, triggered from a cleanup effect. Test this by clicking a link mid-stream. The token counter on your provider dashboard should stop climbing within a second.

The LLM stalls. Set a max duration and a max token count on the backend. If neither has produced output in ten seconds, tear it down and return what you have. Better a truncated answer than a hung page.

Streaming with structure

Streaming plain text is easy. Streaming structured output — JSON that the client needs to render as a UI — is where teams fall over.

The naive approach is to wait for the full response, parse the JSON, then render. That kills the streaming feel. The alternative is to parse the JSON incrementally as it arrives. Small utilities like partial-json can turn a half-finished JSON string into a best-effort object you can render against.

One common pattern we see: the model streams a list of items, and the UI adds each item as it appears. Users love it. It requires a schema you can parse forgivingly, which usually means top-level arrays and shallow objects.

Errors mid-stream

The stream can fail after the first token. Rate limit, content filter, provider hiccup. The right behavior is almost never to throw the partial response away.

Show the partial content. Add a small inline notice ("connection dropped, tap to retry"). If the response was important, offer a retry that continues from where it stopped, or that regenerates cleanly. Which one you pick depends on whether determinism matters more than latency.

Rate limits and quotas

Streaming responses hold a connection open for seconds. That has knock-on effects.

Concurrent connection limits on your host are a real ceiling. Serverless platforms often cap this hard. If you're on a platform with a 100-concurrent-request limit and each user holds a connection for 20 seconds, you cap at about five concurrent users per second of arrival. Model your load accordingly.

Provider rate limits are usually measured in tokens per minute across your whole account. Streaming doesn't change that math but does make bursty traffic more visible. A queue in front of the LLM call is worth the small extra latency.

Testing something you can't easily replay

Streaming responses are annoying to test because the timing matters. A response that arrives all at once is a different UX than one that arrives token by token. Record a real stream once, replay it with a fake server that emits chunks on a timer, and you can integration-test the UI properly.

Common misconceptions

"Streaming is faster." It isn't. The total time is the same. Perceived time is shorter because the user starts reading sooner.

"We need WebSockets for streaming." You don't. SSE is fine for one-way server-to-client streams and needs no special infrastructure.

"Streaming makes evals harder." Not really. Evals run on the final response. Log the assembled string on the server after the stream closes.

Frequently asked questions

Should I use SSE or WebSockets for LLM streaming?

SSE for almost everything. WebSockets only if you also need to send tokens from client to server after the stream starts — voice, collaborative editing, live commands. For a chat UI, SSE is simpler and works through more networks.

Can I stream from a serverless function?

On most modern serverless platforms, yes. Check that your platform supports streaming responses and that no CDN or proxy in front buffers them. If you see all tokens arrive at once, that's the culprit.

How do I stream with structured output?

Use JSON mode plus an incremental JSON parser. Design the schema so partial parses render something usable — top-level arrays and shallow objects work best. Anything nested and deeply required breaks the streaming feel.

What if the connection drops mid-stream?

Show what you have, offer a retry, and never throw the partial response away. If the response was important, persist tokens on the server as they arrive so a refresh recovers.

Does streaming increase cost?

No. Cost is per token in and out. Streaming changes when the tokens arrive, not how many there are.

Conclusion

Streaming is the difference between an AI feature that feels usable and one that feels stuck. The transport is boring — SSE and a fetch reader. The details are where the polish lives: throttled rendering, wired-up cancellation, partial-response persistence, and a UI that survives a mid-stream error.

If you're shipping AI features and streaming still feels fragile, that's usually a sign the plumbing between the model and the user has too many hops. Fewer layers, cleaner cancellation, and honest error handling fix most of it.

If you'd like a second pair of eyes on your AI feature — streaming, evals, cost, or the whole pipeline — the AI Audit is the fastest way to get one.