SSE for LLM Streaming Responses

Server side

response.setHeader("Content-Type", "text/event-stream");
for (var delta : llm.generateStream(prompt)) {
  writer.write("data: " + escape(delta.text()) + "\n\n");
  writer.flush();
}
writer.write("event: done\ndata: {}\n\n");

Advertisement

Client side

const es = new EventSource('/chat/stream');
es.onmessage = (e) => appendToken(e.data);
es.addEventListener('done', () => es.close());

Advertisement

Escape newlines in data

SSE splits on \n\n. Payload newlines break framing. Escape or use JSON.

Flush every token

Without flush, tokens buffer and arrive together. Defeats the point of streaming.

End marker

Custom done event tells client to close. Prevents automatic reconnect.