Retrieval-Augmented Generation (RAG), Explained

Large language models are trained on broad web data, not your documentation, your codebase, or your company's knowledge base. When a user asks a question that requires that private knowledge, the model either hallucinates an answer or admits it doesn't know. Retrieval-Augmented Generation (RAG) solves this by retrieving relevant context at query time and including it in the prompt.

RAG is not a complex system. It is a pipeline with four steps: chunk, embed, retrieve, generate.

Why RAG Instead of Fine-tuning?

Fine-tuning adjusts a model's weights to embed knowledge permanently. It is expensive, requires retraining whenever data changes, and does not give the model access to new information after training. RAG stores knowledge externally, retrieves it on demand, and updates instantly when the underlying data changes. For most production use cases—documentation assistants, internal search, customer support bots—RAG is the right choice.

Step 1: Chunking

Documents are too long to fit in a single prompt, and even if they did fit, irrelevant sections dilute the model's attention. Chunking splits documents into smaller, self-contained passages that can be retrieved selectively.

Chunking strategies:

• Fixed-size with overlap: Split every N tokens, with a K-token overlap between adjacent chunks so sentences are never cut mid-thought.
• Semantic splitting: Split on sentence or paragraph boundaries using a tokenizer-aware splitter.
• Document-native: Split by headers (Markdown H2/H3), by code block, or by section boundary depending on the source material.

Chunk size is a tuning parameter. Smaller chunks (200-400 tokens) retrieve precisely but may lack context. Larger chunks (600-1000 tokens) carry more context but introduce noise. Most production systems use 300-500 tokens with a 50-token overlap as a starting point.

Step 2: Embedding

An embedding model converts a text chunk into a dense vector—a list of floating-point numbers that encodes semantic meaning. Chunks with similar meaning end up close together in vector space.

ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

async function embedChunk(text: string): Promise<number[]> {
  // Use your preferred embedding provider here.
  // Anthropic's API focuses on generation; pair it with
  // an embedding model from your vector DB provider or
  // a dedicated embeddings API.
  throw new Error("Replace with your embedding provider call");
}

For a deeper look at what embeddings are and how to choose an embedding model, see Embeddings: The Foundation of AI Search.

Step 3: Storing and Retrieving from a Vector Database

Embedded chunks are stored in a vector database (Pinecone, Weaviate, Qdrant, pgvector, or similar). At query time, the user's question is embedded with the same model, and the database returns the top-k most similar chunks by cosine similarity or approximate nearest-neighbor search.

ts
// Pseudocode — adapt to your vector DB client
async function retrieveContext(
  query: string,
  topK = 5
): Promise<string[]> {
  const queryEmbedding = await embedChunk(query);
  const results = await vectorDB.query({
    vector: queryEmbedding,
    topK,
    includeMetadata: true,
  });
  return results.matches.map((m) => m.metadata.text as string);
}

For a full explanation of vector databases, similarity search algorithms, and how to choose a store, see Vector Databases and How Semantic Search Works.

Step 4: Generate with Grounded Context

With the retrieved chunks in hand, construct a prompt that gives the model the context it needs before asking the question.

ts
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

async function ragAnswer(userQuery: string): Promise<string> {
  const chunks = await retrieveContext(userQuery);
  const context = chunks.join("\n\n---\n\n");

  const message = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    system:
      "You are a helpful assistant. Answer the user's question using " +
      "only the provided context. If the context does not contain " +
      "enough information, say so.",
    messages: [
      {
        role: "user",
        content:
          `Context:\n${context}\n\nQuestion: ${userQuery}`,
      },
    ],
  });

  const block = message.content[0];
  if (block.type !== "text") throw new Error("Unexpected content type");
  return block.text;
}

The system prompt instruction—answer only from the provided context—is the key guardrail. It tells the model to treat the retrieved text as its source of truth and admit uncertainty rather than fabricate. This is the primary lever for reducing hallucinations in RAG systems. See Reducing LLM Hallucinations in Production for additional techniques.

Improving Retrieval Quality

Basic top-k semantic search gets you far, but several improvements are worth knowing:

• Hybrid search: Combine vector similarity with keyword (BM25) search. Keyword search catches exact matches that semantic search sometimes misses.
• Re-ranking: After retrieving top-k candidates, run a smaller cross-encoder model to re-rank them for relevance to the specific query before assembly.
• Metadata filtering: Store document source, date, or category as metadata and pre-filter before similarity search so irrelevant corpora are excluded.
• Query rewriting: Ask the model to rewrite or expand the user's query before embedding it, to capture synonyms and related terms.

Handling Long Context

Modern Claude models support long context windows, which raises the question: why not just stuff all documents into the prompt? The answer is cost, latency, and the "lost in the middle" problem—models pay less attention to content in the middle of very long prompts. RAG retrieves the relevant 5-10 chunks rather than sending hundreds of irrelevant ones.

For Building Your First AI Feature with the Claude API, a simple RAG pipeline often provides the biggest quality lift with modest engineering effort.

Putting It Into Production

• Run the ingestion pipeline (chunk, embed, upsert) offline, triggered by document updates, not on every query.
• Cache embeddings for unchanged content.
• Log retrieved chunks alongside the generated answer so you can debug poor responses.
• Monitor retrieval quality independently of generation quality—they fail in different ways.

For a complete end-to-end implementation of semantic search on a blog, see Adding AI-Powered Search to Your Blog with Embeddings. See the interview prep section for common RAG interview questions.

Conclusion

RAG is the practical pattern for grounding LLMs in private, up-to-date, or domain-specific knowledge. The four steps—chunk, embed, retrieve, generate—are each individually simple. The craft is in tuning chunk size, retrieval strategy, and the prompt structure that assembles context into reliable answers.