Eight seconds staring at a spinner kill conversion; the same eight seconds arriving in chunks feel instant. Streaming AI responses cuts perceived latency from 8s to under 1s because the first token arrives fast and the rest flows in, and abandonment rates drop along with it. For most cases, Server-Sent Events deliver that with far less complexity than WebSockets.

How SSE works

SSE is plain HTTP: a response carrying the text/event-stream content type, events separated by blank lines, auto-reconnect built into the browser's EventSource. Each event fits in a few bytes, so proxies and load balancers handle the flow without strain. Idle connections die inside intermediate proxies; a keepalive comment (: ping every 15s) keeps the path open. With LLMs the flow runs one way, server to client, so SSE gives you everything WebSockets offer through an ordinary HTTP request.

When do WebSockets earn their keep?

  • Genuine bidirectional flow: voice agents, collaborative sessions, users interrupting mid-sentence
  • Message latency below 100ms changes the perceived experience
  • Multiplexing several logical streams over one connection

Token streaming on the backend

The server forwards provider deltas as they arrive and flushes buffers without waiting for accumulation. Nginx holds chunks by default and wrecks the experience: proxy_buffering off; on the location block or an X-Accel-Buffering: no response header fixes it. Gzip compression on partial responses also holds bytes back; switch it off on that endpoint.

Cancellation: stop paying for invisible tokens

The user hit stop? The client aborts the fetch, the server detects the closed connection (request.on('close') in Express, generator cancellation in FastAPI), and cancels the upstream provider call. Providers bill tokens generated up to the cut, so the cut has to happen on both sides. Log cancellations with the session ID; support will thank you when a user reports a cut-off answer. Without that chain, every abandoned session pays for tokens nobody read.

Errors mid-stream

The provider drops the connection halfway through a completion. Ship what arrived plus one final structured error event; clients render the partial with a clear failure marker and a regenerate button. When the provider supports continuation, resume from the last received chunk. Silence until timeout is the bad alternative.

Framework notes

  • Next.js route handlers stream a ReadableStream without extra libraries
  • Express needs Content-Type: text/event-stream, Cache-Control: no-cache, and Connection: keep-alive headers, plus res.flush() after each chunk
  • FastAPI ships StreamingResponse out of the box, fed by async generators