Building an AI Chatbot with Memory

A stateless LLM forgets everything after every request. Building a chatbot that feels natural requires you to manage memory explicitly — deciding what to include in the context window, when to summarise old turns, and how to persist conversations across page reloads.

This post is a practical walkthrough of the three layers of chatbot memory: in-context history, summarisation, and persistent storage.

The Baseline: Conversation History in Context

The simplest form of memory is including all prior messages in every API request. The Anthropic API accepts a messages array where each element has a role ("user" or "assistant") and a content field.

ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

type Turn = { role: "user" | "assistant"; content: string };

async function chat(history: Turn[], newMessage: string): Promise<string> {
  const messages: Turn[] = [
    ...history,
    { role: "user", content: newMessage },
  ];

  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    messages,
  });

  const assistantText =
    response.content[0].type === "text" ? response.content[0].text : "";

  return assistantText;
}

The caller is responsible for appending both the user message and the assistant reply to history after each turn. This is straightforward, but it has a hard limit: context windows are finite. As the conversation grows, so does the token count — and the cost.

Understanding the Context Window Constraint

Every LLM has a maximum context length. Once the cumulative token count of system + messages + new_tokens exceeds it, the API returns an error. In practice, you hit cost and latency problems before hitting the hard cap because longer contexts mean more tokens processed per request.

The strategies below address this progressively.

Strategy 1: Sliding Window

Keep only the last N turns. This is the simplest mitigation and works well for conversational tasks where very old context is rarely relevant:

ts
const MAX_TURNS = 20;

function trimHistory(history: Turn[]): Turn[] {
  // Always keep an even number so the array starts with a user turn
  const capped = history.slice(-MAX_TURNS);
  return capped.length % 2 === 0 ? capped : capped.slice(1);
}

The tradeoff: the bot loses access to things said early in the conversation. For support bots or document assistants this is often unacceptable, which motivates summarisation.

Strategy 2: Summarisation

When the conversation history exceeds a threshold, ask the model to condense the older turns into a compact summary, then replace those turns with a single synthetic message.

ts
async function summariseHistory(history: Turn[]): Promise<string> {
  const transcript = history
    .map((t) => `${t.role}: ${t.content}`)
    .join("\n");

  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 512,
    messages: [
      {
        role: "user",
        content: `Summarise the following conversation into 3-5 sentences, preserving key facts and decisions:\n\n${transcript}`,
      },
    ],
  });

  return response.content[0].type === "text" ? response.content[0].text : "";
}

async function maybeSummarise(history: Turn[]): Promise<Turn[]> {
  if (history.length < 30) return history;

  const toSummarise = history.slice(0, -10);
  const recentTurns = history.slice(-10);
  const summary = await summariseHistory(toSummarise);

  return [
    {
      role: "user",
      content: `[Conversation summary so far]: ${summary}`,
    },
    { role: "assistant", content: "Understood, I'll keep that context in mind." },
    ...recentTurns,
  ];
}

The synthetic summary turn acts as a memory anchor. The model treats it as prior context even though it was generated, not literally exchanged.

Strategy 3: Persistent Storage

In-memory history disappears when the user closes the tab. To support multi-session or multi-device continuity, persist history to a database and associate it with a user ID or session ID.

A minimal schema in MongoDB:

ts
interface Conversation {
  _id: string;
  userId: string;
  messages: Turn[];
  summary?: string;
  updatedAt: Date;
}

On each request:

1. Load the conversation document for the current session.
2. Pass messages (or the summarised form) to the API.
3. Append the new user and assistant turns.
4. Save back to the database.

For multi-tenant apps, add access control so users can only read their own conversations. The Authentication with Refresh Tokens post covers session management patterns that integrate cleanly here.

Adding Long-Term Semantic Memory

Summarisation helps with recency but is lossy. A complementary approach is semantic memory: store key facts as embeddings and retrieve the most relevant ones per turn using vector similarity search.

This is the same retrieval pattern used in RAG systems — instead of retrieving from a document corpus, you retrieve from a personal fact store. See Retrieval-Augmented Generation (RAG), Explained and Vector Databases and How Semantic Search Works for the underlying mechanics.

Streaming the Response

For a responsive UI, stream the assistant's reply as tokens arrive rather than waiting for the full response. The conversation history management described here is orthogonal to streaming — collect the full streamed text, then append it to history. See Streaming LLM Responses in a Next.js App for the implementation.

Putting It Together in Next.js

A production chatbot endpoint in Next.js:

1. POST /api/chat receives { sessionId, message }.
2. Load history from the database.
3. Run maybeSummarise if the history is long.
4. Call the API with the trimmed history.
5. Append turns and persist.
6. Return the response (or stream it).

For structuring the API layer, see How to Design APIs for Next.js Applications. You can also see working chatbot implementations on the /projects page.

Conclusion

Chatbot memory is a spectrum. Start with a simple sliding-window history, add summarisation once you hit context limits, and layer in persistent storage when sessions need to survive across page loads. Each layer adds complexity but solves a real problem. Understand the trade-off at each step and add only as much complexity as your use case demands.