Streaming LLM Responses in a Next.js App

Waiting several seconds for a complete LLM response kills the experience of an AI-powered feature. Streaming solves this by sending tokens to the browser as they are generated, so users see text appearing word-by-word rather than staring at a spinner.

This post covers the mechanics of LLM streaming in a Next.js App Router application using the Anthropic SDK, from the server-side Route Handler all the way to the React component that renders the output.

Why Streaming Matters for LLM UIs

LLMs generate text autoregressively — one token at a time. A typical response might take two to five seconds to complete on the server. Without streaming, the user waits for the entire generation before the browser renders anything. With streaming, the first tokens arrive in under a second and the UI updates continuously, which dramatically improves perceived performance.

This pairs well with everything covered in React Suspense and Streaming, Explained — Next.js streaming and LLM streaming are complementary ideas.

Setting Up the Route Handler

The Anthropic SDK's messages.stream() method returns an async iterator you can pipe directly into a ReadableStream. Next.js Route Handlers accept Response objects that wrap a ReadableStream, so the wiring is straightforward.

ts
// app/api/chat/route.ts
import Anthropic from "@anthropic-ai/sdk";
import { NextRequest } from "next/server";

const client = new Anthropic();

export async function POST(req: NextRequest) {
  const { messages } = await req.json();

  const stream = await client.messages.stream({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    messages,
  });

  const readable = new ReadableStream({
    async start(controller) {
      for await (const event of stream) {
        if (
          event.type === "content_block_delta" &&
          event.delta.type === "text_delta"
        ) {
          controller.enqueue(
            new TextEncoder().encode(event.delta.text)
          );
        }
      }
      controller.close();
    },
  });

  return new Response(readable, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

A few things worth noting:

• client.messages.stream() handles reconnection and error propagation internally.
• We only enqueue text_delta events. Anthropic's streaming API emits several event types (message start, content block start, ping, stop) — filtering keeps the byte stream clean.
• The Content-Type header tells the browser to treat the response as plain text rather than JSON, which is what the client-side fetch reader expects.

Reading the Stream in React

On the client, use the fetch API with response.body.getReader(). Keep the accumulated text in state and update it on each chunk.

ts
// components/ChatBox.tsx
"use client";
import { useState } from "react";

export function ChatBox() {
  const [prompt, setPrompt] = useState("");
  const [output, setOutput] = useState("");
  const [loading, setLoading] = useState(false);

  async function handleSubmit(e: React.FormEvent) {
    e.preventDefault();
    setOutput("");
    setLoading(true);

    const res = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        messages: [{ role: "user", content: prompt }],
      }),
    });

    const reader = res.body!.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      setOutput((prev) => prev + decoder.decode(value));
    }

    setLoading(false);
  }

  return (
    <form onSubmit={handleSubmit}>
      <textarea value={prompt} onChange={(e) => setPrompt(e.target.value)} />
      <button type="submit" disabled={loading}>Send</button>
      <pre>{output}</pre>
    </form>
  );
}

This pattern accumulates text correctly even when the browser coalesces multiple chunks into a single read() call.

Aborting In-Flight Requests

Users often want to stop a long generation. Pass an AbortController signal to fetch and call abort() on the controller when needed. On the server side, wrap the streaming loop in a try/catch — when the client disconnects, the ReadableStream controller will throw, letting you clean up.

ts
const controller = new AbortController();
const res = await fetch("/api/chat", {
  method: "POST",
  signal: controller.signal,
  body: JSON.stringify({ messages }),
});

// Later:
controller.abort();

Rate Limiting and Protecting the Endpoint

A streaming endpoint is still an API endpoint and needs the same protection as any other. Without rate limiting, a single user can repeatedly fire requests and rack up token costs. See Rate Limiting Your Next.js APIs for token-bucket and sliding-window patterns that work with Route Handlers.

Also store your ANTHROPIC_API_KEY in a server-only environment variable — never expose it to the client bundle. The Environment Variables and Secrets in Next.js guide covers how to do this safely.

Handling Errors Gracefully

Network errors can interrupt a stream mid-generation. Two failure modes to handle:

1. HTTP-level errors (4xx, 5xx): Check res.ok before opening the reader and show a user-facing message.
2. Mid-stream errors: The Anthropic API emits an error event type in the stream. Detect it in the server loop, enqueue a special error marker, and handle it on the client.

For general error handling patterns, Error Handling Patterns in Node.js APIs is a good reference.

Going Further

Once basic streaming is working you can layer on:

• Conversation history: Append each exchange to the messages array and send the full history on each request. See Building an AI Chatbot with Memory.
• Tool use: Pause the stream when the model emits a tool_use block, run the tool, and resume. Covered in depth in Tool Use and Function Calling with LLMs.
• Cost optimisation: Streaming does not change token cost, but smart model selection and caching do. See Cost and Latency Optimization for LLM Apps.

If you are building a complete AI product and want to see these patterns in context, browse the projects page for working examples.

Conclusion

Streaming LLM responses is a small amount of code for a large improvement in perceived speed. The core pattern — client.messages.stream() on the server, ReadableStream in the Response, and a getReader() loop on the client — is stable and framework-agnostic. Get this right early and the rest of your AI UI will be much easier to build on top of it.