Most teams switch on streaming because an answer that appears all at once, after a long blank pause, feels broken. Streaming fixes that first impression, but it creates smaller moments nobody designed: the gap before the first word, the silence while an SI agent calls a tool, the answer that stops mid-sentence, and the screen reader that tries to read every fragment.
Below, each state in the order a user meets it, with the checks we would run before launch.
Why does streaming change the design problem?
Nielsen Norman Group's response-time limits still frame the problem. Around 0.1 second feels instant. Around 1 second keeps the user's train of thought intact. Around 10 seconds is the limit for holding attention, and beyond it users want to know how long the rest will take. A full LLM answer often runs past 10 seconds, so without streaming you ask users to wait beyond the point where attention drifts.
Streaming moves the critical number from total response time to time to first token: the gap between sending a message and seeing the first words. Measure it on the user's device, not only on your server, because retrieval, tool calls, network hops and proxies all sit inside that gap.
How should I design the UX for streaming LLM responses?
Treat streaming as a set of states, not one animation. Acknowledge the message instantly, then show real status text (such as which documents are being searched) until the first token arrives. Label tool-call pauses, render text at a steady pace without layout jumps, and let users stop at any time while keeping the partial answer. If a stream fails, mark the answer as incomplete and offer retry. For screen readers, keep the streaming text out of assertive live regions and announce status changes politely instead.
What should users see before the first token?
Respond to the click straight away: the user's message appears in the thread, the input clears, and something shows the request was accepted. None of that needs a model, so it should never lag.
For the wait that follows, use status text rather than a bare spinner. NN/g's guidance on progress indicators recommends explaining why the user is waiting and warns that static text like "Loading..." gives no sense of progress. In an SI feature you usually know what is happening, so say it: "Searching your policy documents", "Reading 3 matching files", "Drafting the answer".
- Drive status labels from real pipeline events, not a timer. A label that says "Checking sources" when nothing is being checked will be caught out the first time it hangs.
- Keep labels short and specific to the user's data. "Searching 2023 contracts" builds more confidence than "Thinking".
- If the gap before the first token is regularly long, fix the pipeline as well as the copy. Status text makes a wait tolerable; it does not make it short.
How do you handle tool-call pauses mid-answer?
Agents stream unevenly. Text flows, then stops while the model decides on a tool call, then stops again while your code runs that tool. Anthropic's streaming documentation notes that when tools are in use there can be delays between streaming events while the model is working. To a user, that pause looks exactly like a frozen app.
Give each tool call a visible step: a short line such as "Looking up order 4471" that appears when the call starts and changes to a done state when it returns. Once the answer is complete, collapse those steps into a small trail the user can expand, so the final answer stays readable but the work behind it is still checkable. For tools that change something (sending an email, issuing a refund), the pause is not a loading state at all; it is a decision point, and the patterns in our guide to human-in-the-loop approvals for SI agents apply.
How should the streaming text itself behave?
Tokens rarely arrive at an even rate, and rendering each the moment it lands reads as stutter. A small client-side buffer that releases text at a steady pace looks calmer, as long as it never holds text back for long.
- Render formatting incrementally without layout jumps. Half-written tables, lists and code blocks are the usual culprits; hold back a block until it can render sensibly.
- Do not fight the user's scroll. Follow new text only while the user is at the bottom of the thread. If they scroll up to reread, stop following and offer a "Jump to latest" control.
- Respect the prefers-reduced-motion setting for any typing cursor or fade effects.
These details sit inside the wider question of how an SI surface earns trust, covered in designing for SI surfaces.
What should stop and retry controls do?
Every streamed answer needs a stop control, placed where the send button was, so users can find it without searching. NN/g's progress-indicator guidance also recommends giving users a way to stop long processes. In a browser, an AbortController can cancel both the request and the reading of the response stream. Whether closing the connection also stops generation and billing on the provider side depends on your architecture, so confirm it for your own stack rather than assuming.
- On stop, keep the partial answer on screen and label it "Stopped". Deleting text the user may already have read is disorienting.
- Offer retry next to any stopped or failed answer. Decide whether a retry replaces the old version or keeps both, and make that visible.
- Let users edit their last message and resend. Often the reason they stopped was a badly worded question.
What happens when a stream fails halfway?
A streamed answer can fail after it has started. Anthropic's docs note that errors can arrive inside the event stream, for example an overloaded error during heavy usage. Mobile connections drop, laptops sleep, and proxies time out. The worst outcome is a half answer that looks finished.
- Keep the partial text, but mark it clearly as incomplete, with a short reason in plain language and a retry button.
- Persist the answer on the server as it is generated, so a page reload shows what was produced instead of an empty thread.
- If you use the browser's EventSource, know its limits. The WHATWG specification defines automatic reconnection with a Last-Event-ID header, but resuming only works if your server can replay from that ID. EventSource also takes only a URL, so teams that need to send a request body usually read the stream through fetch instead.
- Plan for retraction. OpenAI's streaming guide notes that streaming makes completions harder to moderate, because partial output is shown before any full-output check can run. If a later check fails, replace the text with a clear notice rather than leaving it up.
How you word these states matters as much as the mechanics; see what an SI assistant should say when it does not know.
How do users know the answer is complete?
When the text stops moving, users need to know whether it finished or broke. Providers report why generation ended. Anthropic's API, for example, returns a stop reason that distinguishes a natural finish from hitting the token limit, a refusal, or filling the context window. Map those reasons to visible states: a finished answer shows its actions and sources; a truncated one says so and offers to continue; a refusal shows your own refusal message rather than a fragment.
How do you make streaming accessible to screen readers?
Streaming and screen readers pull in opposite directions. If the message bubble is a live region, each appended fragment can be announced as a separate update, and an assertive region interrupts whatever the user is listening to. MDN's live region guidance says polite announcements should be the normal choice and assertive ones used sparingly, for urgent updates only.
- Keep a separate, visually quiet status region (role="status", which is polite by default) that announces state changes: "Generating answer", "Answer ready", "Answer stopped". WCAG 2.2 success criterion 4.1.3, Status Messages, at level AA, expects status messages like these to reach assistive technology without moving focus.
- Make sure live regions exist in the page before their content changes. MDN notes that regions added at the same moment as their content are often not announced.
- Consider aria-busy="true" on the answer container while it streams, then set it to false when done. MDN describes it as telling assistive technology it may wait until changes are complete; support varies, so test with real screen readers rather than relying on it.
- Do not move focus into the answer automatically. Let users navigate to it, and give each answer a heading or label so it is easy to find.
What about slow and unreliable networks?
Streaming that works on office Wi-Fi can fail quietly elsewhere. Some reverse proxies buffer responses by default (nginx's proxy_buffering setting is on unless you turn it off), which turns a stream into one late block. The WHATWG specification also warns that browsers with per-server connection limits can struggle when several tabs each hold an event stream open to the same site. Test on a throttled connection, on a phone, and through the same proxies and gateways production uses.
A pre-launch checklist
- Time to first token is measured on real devices.
- Status text reflects real pipeline steps, including tool calls.
- Stop keeps the partial answer; retry and edit-and-resend work.
- A broken stream is marked incomplete and survives a reload.
- Finished, truncated and refused answers look different.
- Screen readers hear state changes, not every fragment.
- The stream works through production proxies on a slow connection.
Users forgive a wait they understand. They abandon one that looks like a freeze.
Where Octalcode fits
Octalcode designs and builds LLM applications, SI agents and RAG systems from a studio in Lahore, with a US presence in Austin. Streaming states are where design and engineering have to be worked out together, because every label and control depends on events the backend actually emits.
If your SI feature works but feels uncertain to use, our SI-augmented design work starts by walking through each state above on your real product and network, and ends with a written list of what to change first.